As far as we know, each Cosmos user in the FIWARE Lab (cosmos.lab.fiware.org) has a maximum of 5GB available in HDFS.
Nevertheless we are getting a DSQuotaExceededException when running our map-reduce Hadoop jobs in spite of the data generated by the job doesn't exceed the 5GB quota.
If we monitor the HDFS usage during the execution of the map-reduce job, we get the following output:
Command: "while true; do date; hadoop fs -count -q . ; sleep 20; done"
Format:
DATE
QUOTA REMAINING_QUOTA SPACE_QUOTA REMAINING_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE FILE_NAME
jue jul 28 18:50:12 CEST 2016
none inf 5368709120 1197734302 19 46 1389627219 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:50:34 CEST 2016
none inf 5368709120 2678747494 16 26 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:50:57 CEST 2016
none inf 5368709120 2678747494 16 26 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:51:20 CEST 2016
none inf 5368709120 2678747494 16 26 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:51:44 CEST 2016
none inf 5368709120 2678747494 16 26 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:52:07 CEST 2016
none inf 5368709120 2678747494 16 26 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:52:28 CEST 2016
none inf 5368709120 1198032544 22 35 1389528792 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:52:50 CEST 2016
none inf 5368709120 1197738517 19 39 1389625814 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:53:11 CEST 2016
none inf 5368709120 2678747494 16 27 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:53:35 CEST 2016
none inf 5368709120 2678747494 16 27 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:53:59 CEST 2016
none inf 5368709120 2678747494 16 27 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:54:22 CEST 2016
none inf 5368709120 2678747494 16 27 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:54:46 CEST 2016
none inf 5368709120 2678747494 16 27 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:55:09 CEST 2016
none inf 5368709120 2477420902 17 28 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:55:31 CEST 2016
none inf 5368709120 1197738514 19 39 1389625815 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:55:55 CEST 2016
none inf 5368709120 1197738514 20 48 1389625815 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:56:17 CEST 2016
none inf 5368709120 2678747506 16 28 895957138 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:56:40 CEST 2016
none inf 5368709120 2678747506 16 28 895957138 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:57:04 CEST 2016
none inf 5368709120 2678747506 16 28 895957138 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:57:28 CEST 2016
none inf 5368709120 2678747506 16 28 895957138 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:57:51 CEST 2016
none inf 5368709120 2678747506 16 28 895957138 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:58:13 CEST 2016
none inf 5368709120 1198032556 16 37 1389528788 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:58:34 CEST 2016
none inf 5368709120 1197738742 19 40 1389625760 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:58:56 CEST 2016
none inf 5368709120 2678747494 16 29 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:59:20 CEST 2016
none inf 5368709120 2678747494 16 29 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:59:43 CEST 2016
none inf 5368709120 2678747494 16 29 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:00:07 CEST 2016
none inf 5368709120 2678747494 16 29 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:00:31 CEST 2016
none inf 5368709120 2678747494 16 29 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:00:54 CEST 2016
none inf 5368709120 1076586601 22 38 1228684181 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:01:18 CEST 2016
none inf 5368709120 1197724648 19 41 1389630437 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:01:41 CEST 2016
none inf 5368709120 1197724648 19 41 1389630437 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:02:05 CEST 2016
none inf 5368709120 1197724648 19 41 1389630437 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:02:29 CEST 2016
none inf 5368709120 1197724648 19 41 1389630437 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:02:53 CEST 2016
none inf 5368709120 1197724648 19 41 1389630437 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:03:14 CEST 2016
none inf 5368709120 364004107 19 46 1667537284 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:03:36 CEST 2016
none inf 5368709120 197959591 20 48 1722885456 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:03:57 CEST 2016
none inf 5368709120 201060881 18 44 1722549413 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:04:19 CEST 2016
none inf 5368709120 201060881 18 44 1722549413 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:04:40 CEST 2016
none inf 5368709120 201060881 18 44 1722549413 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:05:02 CEST 2016
none inf 5368709120 201060881 18 44 1722549413 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:05:23 CEST 2016
none inf 5368709120 201060881 18 44 1722549413 hdfs://cosmosmaster-gi/user/rbarriuso
After a while the execution finishes with this exception:
16/07/28 19:03:11 INFO mapred.JobClient: Task Id : attempt_201604111313_157784_r_000006_0, Status : FAILED
org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/rbarriuso is exceeded: quota=5368709120 diskspace consumed=5.0g
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3778)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3640)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2400(DFSClient.java:2846)
at org.apache.ha...
As you can see at the end of the log above, the maximum HDFS usage corresponds to 1.722.549.413 bytes and 201.060.881 bytes of free quota (according to hadoop fs -count -q), which doesn't sum the 5GB of available user space.
Moreover, the taken space doesn't match the remaining free space.
How is the remaining quota space calculated?
Is there any way to avoid the DSQuotaExceededException?
Thanks in advance.
You have to take into account the replication factor HDFS applies to all the data. By default, this is 3, thus your effective quota is 5GB/3. This quota can be increased by contacting the admin (me :)) via email.
Related
/var/lib/mysql shows files created by systemd-coredump, while running mysql 8.0.31 docker image. Is this an issue, not sure if the database is created properly in the first place?
volumes:
- ./data/keycloak/mysql:/var/lib/mysql
ls -l data/keycloak/mysql/
total 87400
-rw-r----- 1 systemd-coredump systemd-coredump 196608 Oct 19 17:25 '#ib_16384_0.dblwr'
-rw-r----- 1 systemd-coredump systemd-coredump 8585216 Oct 19 15:42 '#ib_16384_1.dblwr'
drwxr-x--- 2 systemd-coredump systemd-coredump 4096 Oct 19 17:23 '#innodb_redo'
drwxr-x--- 2 systemd-coredump systemd-coredump 4096 Oct 19 17:34 '#innodb_temp'
-rw-r----- 1 systemd-coredump systemd-coredump 56 Oct 19 15:42 auto.cnf
-rw-r----- 1 systemd-coredump systemd-coredump 3026699 Oct 19 15:42 binlog.000001
-rw-r----- 1 systemd-coredump systemd-coredump 180 Oct 19 15:48 binlog.000002
-rw-r----- 1 systemd-coredump systemd-coredump 180 Oct 19 15:55 binlog.000003
-rw-r----- 1 systemd-coredump systemd-coredump 180 Oct 19 15:58 binlog.000004
-rw-r----- 1 systemd-coredump systemd-coredump 180 Oct 19 17:22 binlog.000005
-rw-r----- 1 systemd-coredump systemd-coredump 180 Oct 19 17:34 binlog.000006
-rw-r----- 1 systemd-coredump systemd-coredump 96 Oct 19 17:23 binlog.index
-rw------- 1 systemd-coredump systemd-coredump 1676 Oct 19 15:42 ca-key.pem
-rw-r--r-- 1 systemd-coredump systemd-coredump 1112 Oct 19 15:42 ca.pem
-rw-r--r-- 1 systemd-coredump systemd-coredump 1112 Oct 19 15:42 client-cert.pem
-rw------- 1 systemd-coredump systemd-coredump 1680 Oct 19 15:42 client-key.pem
-rw-r----- 1 systemd-coredump systemd-coredump 3540 Oct 19 17:34 ib_buffer_pool
-rw-r----- 1 systemd-coredump systemd-coredump 12582912 Oct 19 17:34 ibdata1
drwxr-x--- 2 systemd-coredump systemd-coredump 4096 Oct 19 15:42 mysql
-rw-r----- 1 systemd-coredump systemd-coredump 31457280 Oct 19 17:23 mysql.ibd
lrwxrwxrwx 1 systemd-coredump systemd-coredump 27 Oct 19 17:23 mysql.sock -> /var/run/mysqld/mysqld.sock
drwxr-x--- 2 systemd-coredump systemd-coredump 4096 Oct 19 15:42 performance_schema
-rw------- 1 systemd-coredump systemd-coredump 1680 Oct 19 15:42 private_key.pem
-rw-r--r-- 1 systemd-coredump systemd-coredump 452 Oct 19 15:42 public_key.pem
-rw-r--r-- 1 systemd-coredump systemd-coredump 1112 Oct 19 15:42 server-cert.pem
-rw------- 1 systemd-coredump systemd-coredump 1676 Oct 19 15:42 server-key.pem
drwxr-x--- 2 systemd-coredump systemd-coredump 4096 Oct 19 15:42 sso#002dkeycloak
drwxr-x--- 2 systemd-coredump systemd-coredump 4096 Oct 19 15:42 sys
-rw-r----- 1 systemd-coredump systemd-coredump 16777216 Oct 19 17:25 undo_001
-rw-r----- 1 systemd-coredump systemd-coredump 16777216 Oct 19 17:25 undo_002
I had some trouble, binlog.* can be deleted and also you need to configure logs size in mysql config, like
expire_logs_days=10
# from MySql 8
# binlog_expire_log_seconds=864000
max_binlog_size=1024M
You can set those parameters in docker-compose file in command parameter like
mysql:
image: mysql:8
container_name: "my-mysql"
restart: always
environment:
MYSQL_ROOT_PASSWORD: ${DB_PASSWORD}
volumes:
- mysql-volume:/var/lib/mysql
command: ['--default-authentication-plugin=mysql_native_password', '--binlog-row-event-max-size=1024M', '--binlog_expire_logs_seconds=864000']
and then re-build container.
function Get_month_column_summary()
{
var month = sheetToPlan.getRange(2, 3).getValue();
Logger.log(month);
var lastCol = summarySheet.getLastColumn();
var lookupRangeValues = summarySheet.getRange(2,1,1,lastCol).getValues();
var concat = [].concat.apply([],lookupRangeValues);
Logger.log(concat);
var index = concat.indexOf(month) + 1;
Logger.log(index);
return index;
}
So basically, I'm trying to find the index of a specific date that I got from sheet 1 and trying to find that same date in sheet 2. But when the index gives me -1 (i added 1 so it shows 0 instead of -1 in logs) for some reason, even though i see that it is in the array, the date that im searching for. It is in the exact same format and wording, 100% the same. but the indexOf still gives me -1. Any idea what is going on here?
[20-05-27 18:21:43:667 HKT] **Fri May 01 00:00:00 GMT+08:00 2020**
[20-05-27 18:21:44:002 HKT] [Sun Jan 01 00:00:00 GMT+08:00 2017,
Wed Feb 01 00:00:00 GMT+08:00 2017,
Wed Mar 01 00:00:00 GMT+08:00 2017,
Sat Apr 01 00:00:00 GMT+08:00 2017,
Mon May 01 00:00:00 GMT+08:00 2017,
Thu Jun 01 00:00:00 GMT+08:00 2017,
Sat Jul 01 00:00:00 GMT+08:00 2017,
Tue Aug 01 00:00:00 GMT+08:00 2017,
Fri Sep 01 00:00:00 GMT+08:00 2017,
Sun Oct 01 00:00:00 GMT+08:00 2017,
Wed Nov 01 00:00:00 GMT+08:00 2017,
Fri Dec 01 00:00:00 GMT+08:00 2017,
Mon Jan 01 00:00:00 GMT+08:00 2018,
Thu Feb 01 00:00:00 GMT+08:00 2018,
Thu Mar 01 00:00:00 GMT+08:00 2018,
Sun Apr 01 00:00:00 GMT+08:00 2018,
Tue May 01 00:00:00 GMT+08:00 2018,
Fri Jun 01 00:00:00 GMT+08:00 2018,
Sun Jul 01 00:00:00 GMT+08:00 2018,
Wed Aug 01 00:00:00 GMT+08:00 2018,
Sat Sep 01 00:00:00 GMT+08:00 2018,
Mon Oct 01 00:00:00 GMT+08:00 2018,
Thu Nov 01 00:00:00 GMT+08:00 2018,
Sat Dec 01 00:00:00 GMT+08:00 2018,
Tue Jan 01 00:00:00 GMT+08:00 2019,
Fri Feb 01 00:00:00 GMT+08:00 2019,
Fri Mar 01 00:00:00 GMT+08:00 2019,
Mon Apr 01 00:00:00 GMT+08:00 2019,
Wed May 01 00:00:00 GMT+08:00 2019,
Sat Jun 01 00:00:00 GMT+08:00 2019,
Mon Jul 01 00:00:00 GMT+08:00 2019,
Thu Aug 01 00:00:00 GMT+08:00 2019,
Sun Sep 01 00:00:00 GMT+08:00 2019,
Tue Oct 01 00:00:00 GMT+08:00 2019,
Fri Nov 01 00:00:00 GMT+08:00 2019,
Sun Dec 01 00:00:00 GMT+08:00 2019,
Wed Jan 01 00:00:00 GMT+08:00 2020,
Sat Feb 01 00:00:00 GMT+08:00 2020,
Sun Mar 01 00:00:00 GMT+08:00 2020,
Wed Apr 01 00:00:00 GMT+08:00 2020,
**Fri May 01 00:00:00 GMT+08:00 2020**,
Mon Jun 01 00:00:00 GMT+08:00 2020,
Wed Jul 01 00:00:00 GMT+08:00 2020,
Sat Aug 01 00:00:00 GMT+08:00 2020,
Tue Sep 01 00:00:00 GMT+08:00 2020,
Thu Oct 01 00:00:00 GMT+08:00 2020,
Sun Nov 01 00:00:00 GMT+08:00 2020,
Tue Dec 01 00:00:00 GMT+08:00 2020,
Fri Jan 01 00:00:00 GMT+08:00 2021,
Mon Feb 01 00:00:00 GMT+08:00 2021,
Mon Mar 01 00:00:00 GMT+08:00 2021,
Thu Apr 01 00:00:00 GMT+08:00 2021,
Sat May 01 00:00:00 GMT+08:00 2021,
Tue Jun 01 00:00:00 GMT+08:00 2021,
Thu Jul 01 00:00:00 GMT+08:00 2021,
Sun Aug 01 00:00:00 GMT+08:00 2021,
Wed Sep 01 00:00:00 GMT+08:00 2021,
Fri Oct 01 00:00:00 GMT+08:00 2021,
Mon Nov 01 00:00:00 GMT+08:00 2021,
Wed Dec 01 00:00:00 GMT+08:00 2021,
Sat Jan 01 00:00:00 GMT+08:00 2022,
Tue Feb 01 00:00:00 GMT+08:00 2022]
[20-05-27 18:21:44:010 HKT] 0.0
This solution may work :
var index = concat.indexOf(month.toString()) + 1;
I you cant to convert all concat data with
var concatstr = []
concat.forEach(e => concatstr.push(e.toString()))
I found questions and answers for this question. But none of them works.
I have a maven project.
I wrote resources/css/default.css.
My final jar looks like this.
97 Mon Nov 07 20:09:24 KST 2016 META-INF/MANIFEST.MF
0 Mon Nov 07 20:09:24 KST 2016 META-INF/
0 Mon Nov 07 20:09:22 KST 2016 resources/
0 Mon Nov 07 20:09:22 KST 2016 resources/css/
0 Mon Nov 07 20:09:22 KST 2016 WEB-INF/
0 Mon Nov 07 20:09:22 KST 2016 WEB-INF/classes/
0 Mon Nov 07 20:09:22 KST 2016 WEB-INF/lib/
1113 Mon Nov 07 19:48:22 KST 2016 index.xhtml
344 Mon Nov 07 20:09:16 KST 2016 resources/css/default.css
339 Mon Nov 07 19:14:30 KST 2016 WEB-INF/faces-config.xml
4007587 Mon Nov 07 19:03:30 KST 2016 WEB-INF/lib/primefaces-6.0.jar
976 Mon Nov 07 19:14:30 KST 2016 WEB-INF/web.xml
Here comes my default.css.
$ unzip -p target/....war resources/css/default.css
/*.ui-widget, .ui-widget .ui-widget {*/
.ui-widget {
font-size: 12px !important;
}
In your JSF page (.xhtml), include:
<h:head>
<h:outputStylesheet name="css/default.css" />
</h:head>
My package.json:
{
"name": "thumbnaillist-gulp",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "",
"license": "ISC",
"dependencies": {
"browserify": "^12.0.1",
"gulp": "^3.9.0",
"gulp-concat": "^2.6.0",
"gulp-react": "^3.1.0",
"gulp-util": "^3.0.7",
"react": "^0.13.3",
"react-script-loader": "0.0.1",
"reactify": "^1.1.1",
"vinyl-source-stream": "^1.1.0",
"watchify": "^3.6.1"
}
}
The weird thing is that after I execute "npm install", besides the modules listed in the dependencies field, other modules are also installed, such as
drwxr-xr-x 14 xxx wheel 476B Nov 18 18:41 acorn
drwxr-xr-x 7 xxx wheel 238B Nov 18 18:41 amdefine
drwxr-xr-x 6 xxx wheel 204B Nov 18 18:41 ansi-green
drwxr-xr-x 6 xxx wheel 204B Nov 18 18:41 ansi-regex
drwxr-xr-x 6 xxx wheel 204B Nov 18 18:41 ansi-styles
drwxr-xr-x 6 xxx wheel 204B Nov 18 18:41 ansi-wrap
drwxr-xr-x 6 xxx wheel 204B Nov 18 18:41 anymatch
drwxr-xr-x 9 xxx wheel 306B Nov 18 18:41 archy
drwxr-xr-x 6 xxx wheel 204B Nov 18 18:41 arr-diff
drwxr-xr-x 6 xxx wheel 204B Nov 18 18:41 arr-flatten
drwxr-xr-x 5 xxx wheel 170B Nov 18 18:41 array-differ
drwxr-xr-x 8 xxx wheel 272B Nov 18 18:41 array-filter
drwxr-xr-x 9 xxx wheel 306B Nov 18 18:41 array-map
drwxr-xr-x 9 xxx wheel 306B Nov 18 18:41 array-reduce
drwxr-xr-x 6 xxx wheel 204B Nov 18 18:41 array-slice
drwxr-xr-x 5 xxx wheel 170B Nov 18 18:41 array-uniq
drwxr-xr-x 6 xxx wheel 204B Nov 18 18:41 array-unique
drwxr-xr-x 6 xxx wheel 204B Nov 18 18:41 arrify
drwxr-xr-x 8 xxx wheel 272B Nov 18 18:41 asn1.js
drwxr-xr-x 10 xxx wheel 340B Nov 18 18:41 assert
drwxr-xr-x 10 xxx wheel 340B Nov 18 18:41 ast-types
drwxr-xr-x 9 xxx wheel 306B Nov 18 18:41 astw
drwxr-xr-x 9 xxx wheel 306B Nov 18 18:41 async-each
drwxr-xr-x 11 xxx wheel 374B Nov 18 18:41 balanced-match
drwxr-xr-x 8 xxx wheel 272B Nov 18 18:41 base62
drwxr-xr-x 9 xxx wheel 306B Nov 18 18:41 base64-js
drwxr-xr-x 6 xxx wheel 204B Nov 18 18:41 beeper
drwxr-xr-x 6 xxx wheel 204B Nov 18 18:41 binary-extensions
drwxr-xr-x 10 xxx wheel 340B Nov 18 18:41 bn.js
drwxr-xr-x 7 xxx wheel 238B Nov 18 18:41 brace-expansion
drwxr-xr-x 7 xxx wheel 238B Nov 18 18:41 braces
drwxr-xr-x 7 xxx wheel 238B Nov 18 18:41 brorand
drwxr-xr-x 13 xxx wheel 442B Nov 18 18:41 browser-pack
drwxr-xr-x 7 xxx wheel 238B Nov 18 18:41 browser-resolve
drwxr-xr-x 12 xxx wheel 408B Nov 18 18:41 browserify
drwxr-xr-x 17 xxx wheel 578B Nov 18 18:41 browserify-aes
drwxr-xr-x 8 xxx wheel 272B Nov 18 18:41 browserify-cipher
drwxr-xr-x 7 xxx wheel 238B Nov 18 18:41 browserify-des
drwxr-xr-x 8 xxx wheel 272B Nov 18 18:41 browserify-rsa
drwxr-xr-x 16 xxx wheel 544B Nov 18 18:41 browserify-sign
drwxr-xr-x 8 xxx wheel 272B Nov 18 18:41 browserify-zlib
drwxr-xr-x 11 xxx wheel 374B Nov 18 18:41 buffer
drwxr-xr-x 11 xxx wheel 374B Nov 18 18:41 buffer-xor
Any hint on this? Is my NPM broken? Or I did not write a package.json in a correct way.
No there's nothing wrong, NPM now installs all dependencies to the root modules folder unless there is a version conflict. All those other modules you are seeing are dependencies of the modules you installed.
From the NPM 3.0.0 release notes:
Your dependencies will now be installed maximally flat. Insofar as is possible, all of your dependencies, and their dependencies, and THEIR dependencies will be installed in your project's node_modules folder with no nesting. You'll only see modules nested underneath one another when two (or more) modules have conflicting dependencies.
I hope Alexander O'Mara answer is helpful for you If you still want or for those who want to use dependencies like older version
you can use
npm install --legacy-bundling
If you always want use
npm set legacy-bundling=true
and use can run as usual
npm install
On my website VPS the MySQL goes down quite often, once a day lately.
I've looked in my syslog and seems like it is an Out Of Memory error.
Is my MySQL not configured the right way or is does my VPS not have enough memory for what I'm running on it?
Here are free and top:
$ free -m
total used free shared buffers cached
Mem: 512 360 151 0 0 42
-/+ buffers/cache: 317 194
Swap: 0 0 0
top - 14:17:47 up 15 days, 9:41, 2 users, load average: 0.13, 0.30, 0.42
Tasks: 58 total, 1 running, 57 sleeping, 0 stopped, 0 zombie
Cpu(s): 19.7%us, 2.7%sy, 0.0%ni, 77.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 524508k total, 415200k used, 109308k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 72404k cached
In my.cnf:
# * Fine Tuning
#
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
I'm using
mysql Ver 14.14 Distrib 5.1.66, for debian-linux-gnu (x86_64) using readline 6.1
on
Debian GNU/Linux 6.0.7 (squeeze)
Here is the block from my syslog:
Jun 23 21:19:01 vps528 kernel: [1182395.730776] mysqld invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
Jun 23 21:19:01 vps528 kernel: [1182395.730789] Pid: 13220, comm: mysqld Not tainted 2.6.24-19-xen #1
Jun 23 21:19:01 vps528 kernel: [1182395.730791]
Jun 23 21:19:01 vps528 kernel: [1182395.730792] Call Trace:
Jun 23 21:19:01 vps528 kernel: [1182395.730802] [<ffffffff80275e26>] oom_kill_process+0xf6/0x110
Jun 23 21:19:01 vps528 kernel: [1182395.730806] [<ffffffff802762ae>] out_of_memory+0x19e/0x1e0
Jun 23 21:19:01 vps528 kernel: [1182395.730810] [<ffffffff80278bd9>] __alloc_pages+0x389/0x3c0
Jun 23 21:19:01 vps528 kernel: [1182395.730815] [<ffffffff8027af54>] __do_page_cache_readahead+0x104/0x260
Jun 23 21:19:01 vps528 kernel: [1182395.730818] [<ffffffff8027520e>] filemap_fault+0x2de/0x3e0
Jun 23 21:19:01 vps528 kernel: [1182395.730822] [<ffffffff80281f3a>] __do_fault+0x6a/0x5d0
Jun 23 21:19:01 vps528 kernel: [1182395.730824] [<ffffffff80287551>] handle_mm_fault+0x1d1/0xd60
Jun 23 21:19:01 vps528 kernel: [1182395.730829] [<ffffffff80229f63>] set_next_entity+0x23/0x50
Jun 23 21:19:01 vps528 kernel: [1182395.730835] [<ffffffff80473553>] do_page_fault+0x1f3/0x11e0
Jun 23 21:19:01 vps528 kernel: [1182395.730840] [<ffffffff8046f7e9>] thread_return+0x3a/0x481
Jun 23 21:19:01 vps528 kernel: [1182395.730845] [<ffffffff80471bf7>] error_exit+0x0/0x79
Jun 23 21:19:01 vps528 kernel: [1182395.730848]
Jun 23 21:19:01 vps528 kernel: [1182395.730850] Mem-info:
Jun 23 21:19:01 vps528 kernel: [1182395.730851] DMA per-cpu:
Jun 23 21:19:01 vps528 kernel: [1182395.730853] CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Jun 23 21:19:01 vps528 kernel: [1182395.730855] DMA32 per-cpu:
Jun 23 21:19:01 vps528 kernel: [1182395.730857] CPU 0: Hot: hi: 186, btch: 31 usd: 172 Cold: hi: 62, btch: 15 usd: 14
Jun 23 21:19:01 vps528 kernel: [1182395.730861] Active:94084 inactive:618 dirty:0 writeback:0 unstable:0
Jun 23 21:19:01 vps528 kernel: [1182395.730861] free:4898 slab:3594 mapped:0 pagetables:3925 bounce:0
Jun 23 21:19:01 vps528 kernel: [1182395.730866] DMA free:11540kB min:28kB low:32kB high:40kB active:0kB inactive:0kB present:16160kB pages_scanned:0 all_unreclaimable? yes
Jun 23 21:19:01 vps528 kernel: [1182395.730869] lowmem_reserve[]: 0 4024 4024 4024
Jun 23 21:19:01 vps528 kernel: [1182395.730875] DMA32 free:8052kB min:8100kB low:10124kB high:12148kB active:376336kB inactive:2472kB present:4120800kB pages_scanned:767723 all_unreclaimable? yes
Jun 23 21:19:01 vps528 kernel: [1182395.730878] lowmem_reserve[]: 0 0 0 0
Jun 23 21:19:01 vps528 kernel: [1182395.730881] DMA: 3*4kB 3*8kB 3*16kB 4*32kB 3*64kB 1*128kB 1*256kB 1*512kB 2*1024kB 0*2048kB 2*4096kB = 11540kB
Jun 23 21:19:01 vps528 kernel: [1182395.730900] DMA32: 62*4kB 4*8kB 3*16kB 4*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 8008kB
Jun 23 21:19:01 vps528 kernel: [1182395.730908] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jun 23 21:19:01 vps528 kernel: [1182395.730910] Free swap = 0kB
Jun 23 21:19:01 vps528 kernel: [1182395.730911] Total swap = 0kB
Jun 23 21:19:01 vps528 kernel: [1182395.730912] Free swap: 0kB
Jun 23 21:19:01 vps528 kernel: [1182395.740886] 1048576 pages of RAM
Jun 23 21:19:01 vps528 kernel: [1182395.740893] 21255 reserved pages
Jun 23 21:19:01 vps528 kernel: [1182395.740894] 39897 pages shared
Jun 23 21:19:01 vps528 kernel: [1182395.740895] 0 pages swap cached
Jun 23 21:19:01 vps528 kernel: [1182395.740898] Out of memory: kill process 25246 (apache2) score 73450 or a child
Jun 23 21:19:01 vps528 kernel: [1182395.740913] Killed process 25246 (apache2)
Looks like the VPS I am running doesn't have enough RAM for the amount of website and databases I am running on it. I'll be upgrading my VPS to one with 1GB of memory.