On my website VPS the MySQL goes down quite often, once a day lately.
I've looked in my syslog and seems like it is an Out Of Memory error.
Is my MySQL not configured the right way or is does my VPS not have enough memory for what I'm running on it?
Here are free and top:
$ free -m
total used free shared buffers cached
Mem: 512 360 151 0 0 42
-/+ buffers/cache: 317 194
Swap: 0 0 0
top - 14:17:47 up 15 days, 9:41, 2 users, load average: 0.13, 0.30, 0.42
Tasks: 58 total, 1 running, 57 sleeping, 0 stopped, 0 zombie
Cpu(s): 19.7%us, 2.7%sy, 0.0%ni, 77.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 524508k total, 415200k used, 109308k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 72404k cached
In my.cnf:
# * Fine Tuning
#
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
I'm using
mysql Ver 14.14 Distrib 5.1.66, for debian-linux-gnu (x86_64) using readline 6.1
on
Debian GNU/Linux 6.0.7 (squeeze)
Here is the block from my syslog:
Jun 23 21:19:01 vps528 kernel: [1182395.730776] mysqld invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
Jun 23 21:19:01 vps528 kernel: [1182395.730789] Pid: 13220, comm: mysqld Not tainted 2.6.24-19-xen #1
Jun 23 21:19:01 vps528 kernel: [1182395.730791]
Jun 23 21:19:01 vps528 kernel: [1182395.730792] Call Trace:
Jun 23 21:19:01 vps528 kernel: [1182395.730802] [<ffffffff80275e26>] oom_kill_process+0xf6/0x110
Jun 23 21:19:01 vps528 kernel: [1182395.730806] [<ffffffff802762ae>] out_of_memory+0x19e/0x1e0
Jun 23 21:19:01 vps528 kernel: [1182395.730810] [<ffffffff80278bd9>] __alloc_pages+0x389/0x3c0
Jun 23 21:19:01 vps528 kernel: [1182395.730815] [<ffffffff8027af54>] __do_page_cache_readahead+0x104/0x260
Jun 23 21:19:01 vps528 kernel: [1182395.730818] [<ffffffff8027520e>] filemap_fault+0x2de/0x3e0
Jun 23 21:19:01 vps528 kernel: [1182395.730822] [<ffffffff80281f3a>] __do_fault+0x6a/0x5d0
Jun 23 21:19:01 vps528 kernel: [1182395.730824] [<ffffffff80287551>] handle_mm_fault+0x1d1/0xd60
Jun 23 21:19:01 vps528 kernel: [1182395.730829] [<ffffffff80229f63>] set_next_entity+0x23/0x50
Jun 23 21:19:01 vps528 kernel: [1182395.730835] [<ffffffff80473553>] do_page_fault+0x1f3/0x11e0
Jun 23 21:19:01 vps528 kernel: [1182395.730840] [<ffffffff8046f7e9>] thread_return+0x3a/0x481
Jun 23 21:19:01 vps528 kernel: [1182395.730845] [<ffffffff80471bf7>] error_exit+0x0/0x79
Jun 23 21:19:01 vps528 kernel: [1182395.730848]
Jun 23 21:19:01 vps528 kernel: [1182395.730850] Mem-info:
Jun 23 21:19:01 vps528 kernel: [1182395.730851] DMA per-cpu:
Jun 23 21:19:01 vps528 kernel: [1182395.730853] CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Jun 23 21:19:01 vps528 kernel: [1182395.730855] DMA32 per-cpu:
Jun 23 21:19:01 vps528 kernel: [1182395.730857] CPU 0: Hot: hi: 186, btch: 31 usd: 172 Cold: hi: 62, btch: 15 usd: 14
Jun 23 21:19:01 vps528 kernel: [1182395.730861] Active:94084 inactive:618 dirty:0 writeback:0 unstable:0
Jun 23 21:19:01 vps528 kernel: [1182395.730861] free:4898 slab:3594 mapped:0 pagetables:3925 bounce:0
Jun 23 21:19:01 vps528 kernel: [1182395.730866] DMA free:11540kB min:28kB low:32kB high:40kB active:0kB inactive:0kB present:16160kB pages_scanned:0 all_unreclaimable? yes
Jun 23 21:19:01 vps528 kernel: [1182395.730869] lowmem_reserve[]: 0 4024 4024 4024
Jun 23 21:19:01 vps528 kernel: [1182395.730875] DMA32 free:8052kB min:8100kB low:10124kB high:12148kB active:376336kB inactive:2472kB present:4120800kB pages_scanned:767723 all_unreclaimable? yes
Jun 23 21:19:01 vps528 kernel: [1182395.730878] lowmem_reserve[]: 0 0 0 0
Jun 23 21:19:01 vps528 kernel: [1182395.730881] DMA: 3*4kB 3*8kB 3*16kB 4*32kB 3*64kB 1*128kB 1*256kB 1*512kB 2*1024kB 0*2048kB 2*4096kB = 11540kB
Jun 23 21:19:01 vps528 kernel: [1182395.730900] DMA32: 62*4kB 4*8kB 3*16kB 4*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 8008kB
Jun 23 21:19:01 vps528 kernel: [1182395.730908] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jun 23 21:19:01 vps528 kernel: [1182395.730910] Free swap = 0kB
Jun 23 21:19:01 vps528 kernel: [1182395.730911] Total swap = 0kB
Jun 23 21:19:01 vps528 kernel: [1182395.730912] Free swap: 0kB
Jun 23 21:19:01 vps528 kernel: [1182395.740886] 1048576 pages of RAM
Jun 23 21:19:01 vps528 kernel: [1182395.740893] 21255 reserved pages
Jun 23 21:19:01 vps528 kernel: [1182395.740894] 39897 pages shared
Jun 23 21:19:01 vps528 kernel: [1182395.740895] 0 pages swap cached
Jun 23 21:19:01 vps528 kernel: [1182395.740898] Out of memory: kill process 25246 (apache2) score 73450 or a child
Jun 23 21:19:01 vps528 kernel: [1182395.740913] Killed process 25246 (apache2)
Looks like the VPS I am running doesn't have enough RAM for the amount of website and databases I am running on it. I'll be upgrading my VPS to one with 1GB of memory.
Related
/var/lib/mysql shows files created by systemd-coredump, while running mysql 8.0.31 docker image. Is this an issue, not sure if the database is created properly in the first place?
volumes:
- ./data/keycloak/mysql:/var/lib/mysql
ls -l data/keycloak/mysql/
total 87400
-rw-r----- 1 systemd-coredump systemd-coredump 196608 Oct 19 17:25 '#ib_16384_0.dblwr'
-rw-r----- 1 systemd-coredump systemd-coredump 8585216 Oct 19 15:42 '#ib_16384_1.dblwr'
drwxr-x--- 2 systemd-coredump systemd-coredump 4096 Oct 19 17:23 '#innodb_redo'
drwxr-x--- 2 systemd-coredump systemd-coredump 4096 Oct 19 17:34 '#innodb_temp'
-rw-r----- 1 systemd-coredump systemd-coredump 56 Oct 19 15:42 auto.cnf
-rw-r----- 1 systemd-coredump systemd-coredump 3026699 Oct 19 15:42 binlog.000001
-rw-r----- 1 systemd-coredump systemd-coredump 180 Oct 19 15:48 binlog.000002
-rw-r----- 1 systemd-coredump systemd-coredump 180 Oct 19 15:55 binlog.000003
-rw-r----- 1 systemd-coredump systemd-coredump 180 Oct 19 15:58 binlog.000004
-rw-r----- 1 systemd-coredump systemd-coredump 180 Oct 19 17:22 binlog.000005
-rw-r----- 1 systemd-coredump systemd-coredump 180 Oct 19 17:34 binlog.000006
-rw-r----- 1 systemd-coredump systemd-coredump 96 Oct 19 17:23 binlog.index
-rw------- 1 systemd-coredump systemd-coredump 1676 Oct 19 15:42 ca-key.pem
-rw-r--r-- 1 systemd-coredump systemd-coredump 1112 Oct 19 15:42 ca.pem
-rw-r--r-- 1 systemd-coredump systemd-coredump 1112 Oct 19 15:42 client-cert.pem
-rw------- 1 systemd-coredump systemd-coredump 1680 Oct 19 15:42 client-key.pem
-rw-r----- 1 systemd-coredump systemd-coredump 3540 Oct 19 17:34 ib_buffer_pool
-rw-r----- 1 systemd-coredump systemd-coredump 12582912 Oct 19 17:34 ibdata1
drwxr-x--- 2 systemd-coredump systemd-coredump 4096 Oct 19 15:42 mysql
-rw-r----- 1 systemd-coredump systemd-coredump 31457280 Oct 19 17:23 mysql.ibd
lrwxrwxrwx 1 systemd-coredump systemd-coredump 27 Oct 19 17:23 mysql.sock -> /var/run/mysqld/mysqld.sock
drwxr-x--- 2 systemd-coredump systemd-coredump 4096 Oct 19 15:42 performance_schema
-rw------- 1 systemd-coredump systemd-coredump 1680 Oct 19 15:42 private_key.pem
-rw-r--r-- 1 systemd-coredump systemd-coredump 452 Oct 19 15:42 public_key.pem
-rw-r--r-- 1 systemd-coredump systemd-coredump 1112 Oct 19 15:42 server-cert.pem
-rw------- 1 systemd-coredump systemd-coredump 1676 Oct 19 15:42 server-key.pem
drwxr-x--- 2 systemd-coredump systemd-coredump 4096 Oct 19 15:42 sso#002dkeycloak
drwxr-x--- 2 systemd-coredump systemd-coredump 4096 Oct 19 15:42 sys
-rw-r----- 1 systemd-coredump systemd-coredump 16777216 Oct 19 17:25 undo_001
-rw-r----- 1 systemd-coredump systemd-coredump 16777216 Oct 19 17:25 undo_002
I had some trouble, binlog.* can be deleted and also you need to configure logs size in mysql config, like
expire_logs_days=10
# from MySql 8
# binlog_expire_log_seconds=864000
max_binlog_size=1024M
You can set those parameters in docker-compose file in command parameter like
mysql:
image: mysql:8
container_name: "my-mysql"
restart: always
environment:
MYSQL_ROOT_PASSWORD: ${DB_PASSWORD}
volumes:
- mysql-volume:/var/lib/mysql
command: ['--default-authentication-plugin=mysql_native_password', '--binlog-row-event-max-size=1024M', '--binlog_expire_logs_seconds=864000']
and then re-build container.
System:Ubuntu 18.04.5 LTS
MYSQL: 5.7.31
Tried following commands all shows the same error
systemctl start mysql
/etc/init.d/mysql start
service mysql start
journalctl -xe shows following
Sep 19 05:38:32 www systemd[1]: Starting MySQL Community Server...
Sep 19 05:38:32 www mysql-systemd-start[2938]: ERROR: Unable to start MySQL server:
**Sep 19 05:38:32 www mysql-systemd-start[2938]: /usr/share/mysql/mysql-systemd-start: line 44: mysqld: command not found**
Sep 19 05:38:32 www mysql-systemd-start[2938]: Please take a look at https://wiki.debian.org/Teams/MySQL/FAQ for tips on fixing common upgrade issues.
Sep 19 05:38:32 www mysql-systemd-start[2938]: Once the problem is resolved, restart the service.
Sep 19 05:38:32 www systemd[1]: mysql.service: Control process exited, code=exited status=1 Sep 19 05:38:32 www systemd[1]: mysql.service: Failed with result 'exit-code'.
Sep 19 05:38:32 www systemd[1]: Failed to start MySQL Community Server. Sep 19 05:38:32 www systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Sep 19 05:38:32 www systemd[1]: mysql.service: Scheduled restart job, restart counter is at 1.
Sep 19 05:38:32 www systemd[1]: Stopped MySQL Community Server.
I've highlighted the line that I think is creating a problem
$PATH is
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
/usrs/bin content related to mysql
-rwxr-xr-x 1 root root 4027496 Jul 20 10:50 myisamchk
-rwxr-xr-x 1 root root 3916200 Jul 20 10:50 myisam_ftdump
-rwxr-xr-x 1 root root 3894296 Jul 20 10:50 myisamlog
-rwxr-xr-x 1 root root 3945448 Jul 20 10:50 myisampack
-rwxr-xr-x 1 root root 3608840 Jul 20 10:50 my_print_defaults
-rwxr-xr-x 1 root root 3908456 Jul 20 10:50 mysql
-rwxr-xr-x 1 root root 3799752 Jul 20 10:50 mysqladmin
lrwxrwxrwx 1 root root 10 Jul 20 10:50 mysqlanalyze -> mysqlcheck
-rwxr-xr-x 1 root root 4068280 Jul 20 10:50 mysqlbinlog
-rwxr-xr-x 1 root root 3825320 Jul 20 10:50 mysqlcheck
-rwxr-xr-x 1 root root 5350 Jul 20 10:50 mysql_config
-rwxr-xr-x 1 root root 3627200 Jul 20 10:50 mysql_config_editor
-rwxr-xr-x 1 root root 26952 Jul 20 10:50 mysqld_multi
-rwxr-xr-x 1 root root 27790 Jul 20 10:50 mysqld_safe
-rwxr-xr-x 1 root root 3875176 Jul 20 10:50 mysqldump
-rwxr-xr-x 1 root root 7865 Jul 20 10:50 mysqldumpslow
-rwxr-xr-x 1 root root 22627736 Jul 20 10:50 mysql_embedded
-rwxr-xr-x 1 root root 3791912 Jul 20 10:50 mysqlimport
-rwxr-xr-x 1 root root 5224224 Jul 20 10:50 mysql_install_db
lrwxrwxrwx 1 root root 10 Jul 20 10:50 mysqloptimize -> mysqlcheck
-rwxr-xr-x 1 root root 3616952 Jul 20 10:50 mysql_plugin
-rwxr-xr-x 1 root root 4286152 Jul 20 10:50 mysqlpump
lrwxrwxrwx 1 root root 10 Jul 20 10:50 mysqlrepair -> mysqlcheck
-rwxr-xr-x 1 root root 39016 Jan 12 2018 mysqlreport
-rwxr-xr-x 1 root root 3788520 Jul 20 10:50 mysql_secure_installation
-rwxr-xr-x 1 root root 3790504 Jul 20 10:50 mysqlshow
-rwxr-xr-x 1 root root 3809512 Jul 20 10:50 mysqlslap
-rwxr-xr-x 1 root root 3653288 Jul 20 10:50 mysql_ssl_rsa_setup
-rwxr-xr-x 1 root root 3569976 Jul 20 10:50 mysql_tzinfo_to_sql
-rwxr-xr-x 1 root root 4446416 Jul 20 10:50 mysql_upgrade
Any idea what's wrong? What to do?
I uninstalled all the mysql packages with apt-get remove one by one and then reinstalled them again. I did not delete /var/lib/mysql directory where all data were there. After re-installation everything worked fine and data are also safe. All tables and sites working. I am not sure but i think last package upgrade caused this.
I've seen similar questions being asked on stack overflow but none of them seem to solve my issue properly. I have recently upgraded my system from Ubuntu 16.04 to Ubuntu 17.10 . Since then, I've been experiencing a problem with mysql where I'm not able to start it up properly and it shows the following error:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket
'/var/run/mysqld/mysqld.sock' (2)
I've searched for many solutions online and as a last effort also tried uninstalling and reinstalling but the problem resurfaces after a few days or few attempts. Upon running "service mysql start", it shows:
Job for mysql.service failed because the control process exited with
error code.
See "systemctl status mysql.service" and "journalctl -xe" for
details.
Upon running the suggested commands it shows the following logs:
systemctl status mysql.service
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor
preset: enabled)
Active: activating (start-post) (Result: exit-code) since Thu 2018-
02-22 09:06:31 IST; 4s ago
Process: 8936 ExecStart=/usr/sbin/mysqld (code=exited,
status=1/FAILURE)
Process: 8930 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre
(code=exited, status=0/SUCCESS)
Main PID: 8936 (code=exited, status=1/FAILURE); Control PID: 8937
(mysql-systemd-s)
Tasks: 2 (limit: 4915)
CGroup: /system.slice/mysql.service
└─control
├─8937 /bin/bash /usr/share/mysql/mysql-systemd-start post
└─8948 sleep 1
Feb 22 09:06:31 vamshi-Lenovo-G50-70 systemd[1]: Starting MySQL
Community Server...
Feb 22 09:06:32 vamshi-Lenovo-G50-70 mysqld[8936]: 2018-02-
22T03:36:31.948822Z 0 [Warning] Changed limits: max_open_files: 1024
(requested 5000)
Feb 22 09:06:32 vamshi-Lenovo-G50-70 mysqld[8936]: 2018-02-
22T03:36:31.948879Z 0 [Warning] Changed limits: table_open_cache: 431
(requested 2000)
Feb 22 09:06:32 vamshi-Lenovo-G50-70 mysqld[8936]: 2018-02-
22T03:36:32.126417Z 0 [Warning] TIMESTAMP with implicit DEFAULT value
is deprecated. Please use --explicit_def
Feb 22 09:06:32 vamshi-Lenovo-G50-70 mysqld[8936]: 2018-02-
22T03:36:32.128149Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.21-
0ubuntu0.17.10.1) starting as process 8936 ...
Feb 22 09:06:32 vamshi-Lenovo-G50-70 mysqld[8936]: 2018-02-
22T03:36:32.130652Z 0 [ERROR] Could not open file
'/var/log/mysql/error.log' for error logging: No such file o
Feb 22 09:06:32 vamshi-Lenovo-G50-70 mysqld[8936]: 2018-02-
22T03:36:32.130675Z 0 [ERROR] Aborting
Feb 22 09:06:32 vamshi-Lenovo-G50-70 mysqld[8936]: 2018-02-
22T03:36:32.130694Z 0 [Note] Binlog end
Feb 22 09:06:32 vamshi-Lenovo-G50-70 mysqld[8936]: 2018-02-
22T03:36:32.130750Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
Feb 22 09:06:32 vamshi-Lenovo-G50-70 systemd[1]: mysql.service: Main
process exited, code=exited, status=1/FAILURE
journalctl -xe
Feb 22 09:11:21 vamshi-Lenovo-G50-70 gnome-shell[2902]:
[AppIndicatorSupport-WARN] Attempting to re-register
:1.69/org/ayatana/NotificationItem/multiload; resetting inst
Feb 22 09:11:21 vamshi-Lenovo-G50-70 gnome-shell[2902]:
[AppIndicatorSupport-WARN] Item
:1.69/org/ayatana/NotificationItem/multiload is already registered
ESCOD
Every time this happens, I uninstall mysql and reinstall(server not client), and then it works alright for the first few attempts, then it goes back to same situation. Thanks in advance!
ls -al
total 32824
drwxrwxr-x 8 root syslog 4096 Feb 22 08:44 .
drwxr-xr-x 14 root root 4096 Feb 16 2017 ..
-rw-r--r-- 1 root root 1527 Feb 17 08:58 alternatives.log
-rw-r----- 1 root adm 0 Feb 22 08:44 apport.log
-rw-r----- 1 root adm 486 Feb 21 07:29 apport.log.1
-rw-r----- 1 root adm 455 Feb 20 20:54 apport.log.2.gz
-rw-r----- 1 root adm 514 Feb 19 21:42 apport.log.3.gz
-rw-r----- 1 root adm 758 Feb 19 18:15 apport.log.4.gz
-rw-r----- 1 root adm 459 Feb 17 23:07 apport.log.5.gz
-rw-r----- 1 root adm 1389 Feb 16 00:33 apport.log.6.gz
-rw-r----- 1 root adm 275 Feb 14 20:16 apport.log.7.gz
drwxr-xr-x 2 root root 4096 Feb 17 08:55 apt
-rw-r----- 1 syslog adm 86558 Feb 22 09:17 auth.log
-rw-r----- 1 syslog adm 97522 Feb 18 18:02 auth.log.1
-rw------- 1 root utmp 0 Feb 14 20:17 btmp
drwxr-x--- 2 root lp 4096 Feb 22 08:44 cups
drwxr-xr-x 2 root root 4096 Feb 5 14:51 dist-upgrade
-rw-r--r-- 1 root root 30036 Feb 17 08:58 dpkg.log
drwx--x--x 2 root gdm 4096 Feb 14 20:18 gdm3
-rw-r--r-- 1 root root 2136 Feb 22 08:39 gpu-manager.log
-rw-r----- 1 syslog adm 3745433 Feb 22 09:28 kern.log
-rw-r----- 1 syslog adm 2302923 Feb 18 18:07 kern.log.1
-rw-rw-r-- 1 root utmp 0 Feb 14 20:17 lastlog
-rw-r----- 1 syslog adm 657 Feb 22 08:40 mail.log
-rw-r----- 1 syslog adm 1971 Feb 18 15:31 mail.log.1
-rw-r----- 1 syslog adm 3706136 Feb 22 09:28 syslog
-rw-r----- 1 syslog adm 18760212 Feb 22 08:44 syslog.1
-rw-r----- 1 syslog adm 989385 Feb 21 07:07 syslog.2.gz
-rw-r----- 1 syslog adm 587416 Feb 20 09:09 syslog.3.gz
-rw-r----- 1 syslog adm 593456 Feb 19 18:21 syslog.4.gz
-rw-r----- 1 syslog adm 672682 Feb 18 18:07 syslog.5.gz
-rw-r----- 1 syslog adm 166575 Feb 17 09:09 syslog.6.gz
-rw-r----- 1 syslog adm 891827 Feb 16 00:44 syslog.7.gz
-rw-r----- 1 syslog adm 572158 Feb 22 09:28 ufw.log
-rw-r----- 1 syslog adm 234885 Feb 18 18:06 ufw.log.1
drwxr-xr-x 2 root root 4096 Feb 15 22:00 unattended-upgrades
drwxr-xr-x 2 root root 4096 Feb 22 08:40 vmware
-rw-rw-r-- 1 root utmp 20736 Feb 22 08:41 wtmp
Create folder /var/log/mysql and make sure mysqld can write to it.
I cannot connect to my compute server VM using SSH. It says "retrying 2/3.. 3/3"
It seems as if the server is actually running and functioning. I have Apache Tomcat with some apps installed and MySql server. I can connect to the apache tomcat manager page and I don't see any problem there. I can connect to my MySQL database using client tool such as MySQL WorkBench. I can insert new rows to tables and delete some. It doesn't look like a ran out of free disk space. Anyways I tried to delete some data from tables and shrink it but apparently free space is not the issue.
Here is a Serial Port log I got. Please, what can cause this? I use to be able to connect under same conditions (same client machine, firewall, etc.) so the issue is from the server side.
[34987645.468284] Free swap = 0kB
[34987645.471547] Total swap = 0kB
[34987645.474838] 157083 pages RAM
[34987645.478109] 0 pages HighMem/MovableOnly
[34987645.482323] 4886 pages reserved
[34987645.485853] 0 pages hwpoisoned
[34987645.489270] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[34987645.497494] [ 311] 0 311 6350 1726 14 0 0 dhclient
[34987645.506101] [ 358] 0 358 6869 60 18 0 0 cron
[34987645.514339] [ 369] 0 369 64668 459 29 0 0 rsyslogd
[34987645.522954] [ 373] 0 373 1064 38 8 0 0 acpid
[34987645.531266] [ 406] 0 406 3604 38 12 0 0 agetty
[34987645.539669] [ 407] 0 407 3604 38 12 0 0 agetty
[34987645.548081] [ 408] 0 408 3604 40 12 0 0 agetty
[34987645.556571] [ 409] 0 409 3604 40 12 0 0 agetty
[34987645.564986] [ 410] 0 410 3604 39 12 0 0 agetty
[34987645.573414] [ 411] 0 411 3604 39 12 0 0 agetty
[34987645.583546] [ 412] 0 412 3559 40 10 0 0 agetty
[34987645.591939] [ 7263] 0 7263 13796 168 29 0 -1000 sshd
[34987645.600464] [11774] 107 11774 10565 124 25 0 -900 dbus-daemon
[34987645.610085] [20473] 104 20473 7292 146 19 0 0 ntpd
[34987645.618585] [27741] 0 27741 13169 2273 30 0 0 google_accounts
[34987645.627869] [27744] 0 27744 13211 2284 29 0 0 google_clock_sk
[34987645.637039] [27747] 0 27747 13211 2244 30 0 0 google_ip_forwa
[34987645.646401] [14807] 0 14807 10176 101 22 0 -1000 systemd-udevd
[34987645.655412] [30481] 0 30481 1085 43 6 0 0 mysqld_safe
[34987645.664327] [30815] 109 30815 221020 12045 81 0 0 mysqld
[34987645.672765] [15984] 0 15984 7218 64 19 0 0 systemd-journal
[34987645.681934] [19949] 108 19949 547956 45017 173 0 0 java
[34987645.690161] [23671] 0 23671 10560 86 26 0 0 cron
[34987645.698380] [23672] 0 23672 1085 22 7 0 0 sh
[34987645.706415] [23673] 0 23673 1057 20 8 0 0 run-parts
[34987645.715058] [23674] 0 23674 1085 37 6 0 0 apt
[34987645.723279] [23793] 0 23793 28771 5257 62 0 0 unattended-upgr
[34987645.732642] [23803] 0 23803 28579 5281 57 0 0 unattended-upgr
[34987645.741809] [24237] 0 24237 4583 418 14 0 0 dpkg
[34987645.750026] [24239] 0 24239 1085 23 7 0 0 google-cloud-sd
[34987645.759214] [24241] 0 24241 62608 51157 126 0 0 python2
[34987645.767782] [24263] 0 24263 12231 155 28 0 0 sshd
[34987645.775989] [24265] 0 24265 10553 79 26 0 0 cron
[34987645.784795] [24266] 0 24266 12203 131 27 0 0 sshd
[34987645.793015] [24267] 0 24267 12203 132 25 0 0 sshd
[34987645.801461] [24268] 0 24268 12203 85 25 0 0 sshd
[34987645.809715] [24270] 0 24270 12203 85 25 0 0 sshd
[34987645.817941] [24271] 0 24271 12203 82 25 0 0 sshd
[34987645.826202] [24272] 0 24272 12203 68 24 0 0 sshd
[34987645.834413] [24273] 0 24273 12203 66 26 0 0 sshd
[34987645.842619] [24274] 0 24274 12203 65 25 0 0 sshd
[34987645.850850] [24275] 0 24275 4527 26 11 0 0 sshd
[34987645.859165] Out of memory: Kill process 24241 (python2) score 326 or sacrifice child
[34987645.867298] Killed process 24241 (python2) total-vm:250432kB, anon-rss:204628kB, file-rss:0kB
Sep 2 08:16:18 instance-1 kernel: [34987645.059900] sshd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=-1000
Sep 2 08:16:18 instance-1 kernel: [34987645.067997] sshd cpuset=/ mems_allowed=0
Sep 2 08:16:18 instance-1 kernel: [34987645.072637] CPU: 0 PID: 7263 Comm: sshd Not tainted 3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-2+deb8u3
Sep 2 08:16:18 instance-1 kernel: [34987645.081960] Hardware name: Google Google/Google, BIOS Google 01/01/2011
Sep 2 08:16:18 instance-1 kernel: [34987645.088955] 0000000000000000 ffffffff8150e08f ffff880025daa9a0 0000000000000000
Sep 2 08:16:18 instance-1 kernel: [34987645.097277] ffffffff8150bc8b 0000000000000000 ffffffff810d51ff 0000000000000000
Sep 2 08:16:18 instance-1 kernel: [34987645.105594] ffffffff815129ae 0000000000000200 ffffffff81067933 ffffffff810c2914
Sep 2 08:16:18 instance-1 kernel: [34987645.113999] Call Trace:
Sep 2 08:16:18 instance-1 kernel: [34987645.116817] [] ? dump_stack+0x5d/0x78
Sep 2 08:16:18 instance-1 kernel: [34987645.122507] [] ? dump_header+0x76/0x1e8
Sep 2 08:16:18 instance-1 kernel: [34987645.128393] [] ? smp_call_function_single+0x5f/0xa0
Sep 2 08:16:18 instance-1 kernel: [34987645.135306] [] ? mutex_lock+0xe/0x2a
Sep 2 08:16:18 instance-1 kernel: [34987645.140921] [] ? put_online_cpus+0x23/0x80
Sep 2 08:16:18 instance-1 kernel: [34987645.147045] [] ? rcu_oom_notify+0xc4/0xe0
Sep 2 08:16:18 instance-1 kernel: [34987645.153083] [] ? do_try_to_free_pages+0x4ac/0x520
Sep 2 08:16:18 instance-1 kernel: [34987645.159842] [] ? oom_kill_process+0x21d/0x370
Sep 2 08:16:18 instance-1 kernel: [34987645.166311] [] ? find_lock_task_mm+0x3d/0x90
Sep 2 08:16:18 instance-1 kernel: [34987645.172614] [] ? out_of_memory+0x473/0x4b0
Sep 2 08:16:18 instance-1 kernel: [34987645.178769] [] ? __alloc_pages_nodemask+0x9cd/0xb30
Sep 2 08:16:18 instance-1 kernel: [34987645.185690] [] ? alloc_pages_current+0x9d/0x150
Sep 2 08:16:18 instance-1 kernel: [34987645.192273] [] ? filemap_fault+0x1a0/0x420
Sep 2 08:16:18 instance-1 kernel: [34987645.198395] [] ? __do_fault+0x3a/0xa0
Sep 2 08:16:18 instance-1 kernel: [34987645.204172] [] ? do_read_fault.isra.54+0x4e/0x300
Sep 2 08:16:18 instance-1 kernel: [34987645.210908] [] ? handle_mm_fault+0x63c/0x11c0
Sep 2 08:16:18 instance-1 kernel: [34987645.217288] [] ? vmacache_find+0x29/0xd0
Sep 2 08:16:18 instance-1 kernel: [34987645.223513] [] ? __do_page_fault+0x177/0x4f0
Sep 2 08:16:18 instance-1 kernel: [34987645.229813] [] ? put_prev_entity+0x57/0x350
Sep 2 08:16:18 instance-1 kernel: [34987645.236015] [] ? set_next_entity+0x56/0x70
Sep 2 08:16:18 instance-1 kernel: [34987645.242146] [] ? pick_next_task_fair+0x6e1/0x820
Sep 2 08:16:18 instance-1 kernel: [34987645.248870] [] ? __switch_to+0x15c/0x5a0
Sep 2 08:16:18 instance-1 kernel: [34987645.254812] [] ? page_fault+0x28/0x30
Sep 2 08:16:18 instance-1 kernel: [34987645.260832] Mem-Info:
Sep 2 08:16:18 instance-1 kernel: [34987645.263586] Node 0 DMA per-cpu:
Sep 2 08:16:18 instance-1 kernel: [34987645.267258] CPU 0: hi: 0, btch: 1 usd: 0
Sep 2 08:16:18 instance-1 kernel: [34987645.272519] Node 0 DMA32 per-cpu:
Sep 2 08:16:18 instance-1 kernel: [34987645.276345] CPU 0: hi: 186, btch: 31 usd: 76
Sep 2 08:16:18 instance-1 kernel: [34987645.281531] active_anon:126526 inactive_anon:3096 isolated_anon:0
Sep 2 08:16:18 instance-1 kernel: [34987645.281531] active_file:813 inactive_file:900 isolated_file:0
Sep 2 08:16:18 instance-1 kernel: [34987645.281531] unevictable:0 dirty:0 writeback:0 unstable:0
Sep 2 08:16:18 instance-1 kernel: [34987645.281531] free:8071 slab_reclaimable:2225 slab_unreclaimable:4588
Sep 2 08:16:18 instance-1 kernel: [34987645.281531] mapped:1028 shmem:3130 pagetables:1213 bounce:0
Sep 2 08:16:18 instance-1 kernel: [34987645.281531] free_cma:0
Sep 2 08:16:18 instance-1 kernel: [34987645.314947] Node 0 DMA free:3088kB min:788kB low:984kB high:1180kB active_anon:11608kB inactive_anon:108kB active_file:12kB inactive_file:152kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:112kB slab_reclaimable:88kB slab_unreclaimable:480kB kernel_stack:16kB pagetables:52kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:522 all_unreclaimable? yes
Sep 2 08:16:18 instance-1 kernel: [34987645.356698] lowmem_reserve[]: 0 576 576 576
Sep 2 08:16:18 instance-1 kernel: [34987645.361878] Node 0 DMA32 free:29196kB min:29300kB low:36624kB high:43948kB active_anon:494496kB inactive_anon:12276kB active_file:3240kB inactive_file:3448kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:612340kB managed:592796kB mlocked:0kB dirty:0kB writeback:0kB mapped:4112kB shmem:12408kB slab_reclaimable:8812kB slab_unreclaimable:17872kB kernel_stack:2320kB pagetables:4800kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:11577 all_unreclaimable? yes
Sep 2 08:16:18 instance-1 kernel: [34987645.406481] lowmem_reserve[]: 0 0 0 0
Sep 2 08:16:18 instance-1 kernel: [34987645.411143] Node 0 DMA: 22*4kB (UE) 19*8kB (UEM) 12*16kB (UEM) 7*32kB (UEM) 4*64kB (UEM) 1*128kB (U) 0*256kB 0*512kB 0*1024kB 1*2048kB (R) 0*4096kB = 3088kB
Sep 2 08:16:18 instance-1 kernel: [34987645.428083] Node 0 DMA32: 383*4kB (E) 310*8kB (UEM) 450*16kB (UEM) 200*32kB (UEM) 109*64kB (EM) 4*128kB (E) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 29196kB
Sep 2 08:16:18 instance-1 kernel: [34987645.445684] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 2 08:16:18 instance-1 kernel: [34987645.454775] 4860 total pagecache pages
Sep 2 08:16:18 instance-1 kernel: [34987645.458908] 0 pages in swap cache
Sep 2 08:16:18 instance-1 kernel: [34987645.462650] Swap cache stats: add 0, delete 0, find 0/0
Sep 2 08:16:18 instance-1 kernel: [34987645.468284] Free swap = 0kB
Sep 2 08:16:18 instance-1 kernel: [34987645.471547] Total swap = 0kB
Sep 2 08:16:18 instance-1 kernel: [34987645.474838] 157083 pages RAM
Sep 2 08:16:18 instance-1 kernel: [34987645.478109] 0 pages HighMem/MovableOnly
Sep 2 08:16:18 instance-1 kernel: [34987645.482323] 4886 pages reserved
Sep 2 08:16:18 instance-1 kernel: [34987645.485853] 0 pages hwpoisoned
Sep 2 08:16:18 instance-1 kernel: [34987645.489270] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Sep 2 08:16:18 instance-1 kernel: [34987645.497494] [ 311] 0 311 6350 1726 14 0 0 dhclient
Sep 2 08:16:18 instance-1 kernel: [34987645.506101] [ 358] 0 358 6869 60 18 0 0 cron
Sep 2 08:16:18 instance-1 kernel: [34987645.514339] [ 369] 0 369 64668 459 29 0 0 rsyslogd
Sep 2 08:16:18 instance-1 kernel: [34987645.522954] [ 373] 0 373 1064 38 8 0 0 acpid
Sep 2 08:16:18 instance-1 kernel: [34987645.531266] [ 406] 0 406 3604 38 12 0 0 agetty
Sep 2 08:16:18 instance-1 kernel: [34987645.539669] [ 407] 0 407 3604 38 12 0 0 agetty
Sep 2 08:16:18 instance-1 kernel: [34987645.548081] [ 408] 0 408 3604 40 12 0 0 agetty
Sep 2 08:16:18 instance-1 kernel: [34987645.556571] [ 409] 0 409 3604 40 12 0 0 agetty
Sep 2 08:16:18 instance-1 kernel: [34987645.564986] [ 410] 0 410 3604 39 12 0 0 agetty
Sep 2 08:16:18 instance-1 kernel: [34987645.573414] [ 411] 0 411 3604 39 12 0 0 agetty
Sep 2 08:16:18 instance-1 kernel: [34987645.583546] [ 412] 0 412 3559 40 10 0 0 agetty
Sep 2 08:16:18 instance-1 kernel: [34987645.591939] [ 7263] 0 7263 13796 168 29 0 -1000 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.600464] [11774] 107 11774 10565 124 25 0 -900 dbus-daemon
Sep 2 08:16:18 instance-1 kernel: [34987645.610085] [20473] 104 20473 7292 146 19 0 0 ntpd
Sep 2 08:16:18 instance-1 kernel: [34987645.618585] [27741] 0 27741 13169 2273 30 0 0 google_accounts
Sep 2 08:16:18 instance-1 kernel: [34987645.627869] [27744] 0 27744 13211 2284 29 0 0 google_clock_sk
Sep 2 08:16:18 instance-1 kernel: [34987645.637039] [27747] 0 27747 13211 2244 30 0 0 google_ip_forwa
Sep 2 08:16:18 instance-1 kernel: [34987645.646401] [14807] 0 14807 10176 101 22 0 -1000 systemd-udevd
Sep 2 08:16:18 instance-1 kernel: [34987645.655412] [30481] 0 30481 1085 43 6 0 0 mysqld_safe
Sep 2 08:16:18 instance-1 kernel: [34987645.664327] [30815] 109 30815 221020 12045 81 0 0 mysqld
Sep 2 08:16:18 instance-1 kernel: [34987645.672765] [15984] 0 15984 7218 64 19 0 0 systemd-journal
Sep 2 08:16:18 instance-1 kernel: [34987645.681934] [19949] 108 19949 547956 45017 173 0 0 java
Sep 2 08:16:18 instance-1 kernel: [34987645.690161] [23671] 0 23671 10560 86 26 0 0 cron
Sep 2 08:16:18 instance-1 kernel: [34987645.698380] [23672] 0 23672 1085 22 7 0 0 sh
Sep 2 08:16:18 instance-1 kernel: [34987645.706415] [23673] 0 23673 1057 20 8 0 0 run-parts
Sep 2 08:16:18 instance-1 kernel: [34987645.715058] [23674] 0 23674 1085 37 6 0 0 apt
Sep 2 08:16:18 instance-1 kernel: [34987645.723279] [23793] 0 23793 28771 5257 62 0 0 unattended-upgr
Sep 2 08:16:18 instance-1 kernel: [34987645.732642] [23803] 0 23803 28579 5281 57 0 0 unattended-upgr
Sep 2 08:16:18 instance-1 kernel: [34987645.741809] [24237] 0 24237 4583 418 14 0 0 dpkg
Sep 2 08:16:18 instance-1 kernel: [34987645.750026] [24239] 0 24239 1085 23 7 0 0 google-cloud-sd
Sep 2 08:16:18 instance-1 kernel: [34987645.759214] [24241] 0 24241 62608 51157 126 0 0 python2
Sep 2 08:16:18 instance-1 kernel: [34987645.767782] [24263] 0 24263 12231 155 28 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.775989] [24265] 0 24265 10553 79 26 0 0 cron
Sep 2 08:16:18 instance-1 kernel: [34987645.784795] [24266] 0 24266 12203 131 27 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.793015] [24267] 0 24267 12203 132 25 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.801461] [24268] 0 24268 12203 85 25 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.809715] [24270] 0 24270 12203 85 25 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.817941] [24271] 0 24271 12203 82 25 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.826202] [24272] 0 24272 12203 68 24 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.834413] [24273] 0 24273 12203 66 26 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.842619] [24274] 0 24274 12203 65 25 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.850850] [24275] 0 24275 4527 26 11 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.859165] Out of memory: Kill process 24241 (python2) score 326 or sacrifice child
Sep 2 08:16:18 instance-1 kernel: [34987645.867298] Killed process 24241 (python2) total-vm:250432kB, anon-rss:204628kB, file-rss:0kB
Sep 2 16:21:27 instance-1 dhclient: DHCPREQUEST on eth0 to 169.254.169.254 port 67
Sep 2 16:21:27 instance-1 dhclient: DHCPACK from 169.254.169.254
Sep 2 16:21:27 instance-1 dhclient: bound to 10.132.0.2 -- renewal in 40860 seconds.
Sep 2 21:14:06 instance-1 systemd[1]: Starting Cleanup of Temporary Directories...
Sep 2 21:14:06 instance-1 systemd[1]: Started Cleanup of Temporary Directories.
Sep 3 03:42:27 instance-1 dhclient: DHCPREQUEST on eth0 to 169.254.169.254 port 67
Sep 3 03:42:27 instance-1 dhclient: DHCPACK from 169.254.169.254
Sep 3 03:42:27 instance-1 dhclient: bound to 10.132.0.2 -- renewal in 34630 seconds.
Sep 3 13:19:37 instance-1 dhclient: DHCPREQUEST on eth0 to 169.254.169.254 port 67
Sep 3 13:19:37 instance-1 dhclient: DHCPACK from 169.254.169.254
Sep 3 13:19:38 instance-1 dhclient: bound to 10.132.0.2 -- renewal in 33626 seconds.
Sep 3 21:14:22 instance-1 systemd[1]: Starting Cleanup of Temporary Directories...
Sep 3 21:14:22 instance-1 systemd[1]: Started Cleanup of Temporary Directories.
Sep 3 22:40:04 instance-1 dhclient: DHCPREQUEST on eth0 to 169.254.169.254 port 67
Sep 3 22:40:04 instance-1 dhclient: DHCPACK from 169.254.169.254
Sep 3 22:40:04 instance-1 dhclient: bound to 10.132.0.2 -- renewal in 38182 seconds.
Sep 4 09:16:26 instance-1 dhclient: DHCPREQUEST on eth0 to 169.254.169.254 port 67
Sep 4 09:16:26 instance-1 dhclient: DHCPACK from 169.254.169.254
Sep 4 09:16:26 instance-1 dhclient: bound to 10.132.0.2 -- renewal in 35099 seconds.
Sep 4 19:01:25 instance-1 dhclient: DHCPREQUEST on eth0 to 169.254.169.254 port 67
Sep 4 19:01:25 instance-1 dhclient: DHCPACK from 169.254.169.254
Sep 4 19:01:25 instance-1 dhclient: bound to 10.132.0.2 -- renewal in 40004 seconds.
Sep 4 21:14:26 instance-1 systemd[1]: Starting Cleanup of Temporary Directories...
Sep 4 21:14:26 instance-1 systemd[1]: Started Cleanup of Temporary Directories.
Okay, I was able to solve the issue.
Apparently, the ssh process in the machine stopped because I ran out of memory. When RAM is full the linux system kills processes.
I couldn't directly see it from the log because it killed something else and indirectly killed ssh.
Here's the solution:
I stopped the machine and then resumed it.
WARNING: my ip address was ephemeral so I changed it to static before stopping the machine. If I wouldn't do that, I would get a different IP address which would force my to update my domain name records.
It can be later switched back to ephemeral.
Google charges for Static IP address.
I can now log in ssh.
I guess a different question is how to prevent SSH from ever being killed.
I hope it helps anyone.
As far as we know, each Cosmos user in the FIWARE Lab (cosmos.lab.fiware.org) has a maximum of 5GB available in HDFS.
Nevertheless we are getting a DSQuotaExceededException when running our map-reduce Hadoop jobs in spite of the data generated by the job doesn't exceed the 5GB quota.
If we monitor the HDFS usage during the execution of the map-reduce job, we get the following output:
Command: "while true; do date; hadoop fs -count -q . ; sleep 20; done"
Format:
DATE
QUOTA REMAINING_QUOTA SPACE_QUOTA REMAINING_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE FILE_NAME
jue jul 28 18:50:12 CEST 2016
none inf 5368709120 1197734302 19 46 1389627219 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:50:34 CEST 2016
none inf 5368709120 2678747494 16 26 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:50:57 CEST 2016
none inf 5368709120 2678747494 16 26 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:51:20 CEST 2016
none inf 5368709120 2678747494 16 26 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:51:44 CEST 2016
none inf 5368709120 2678747494 16 26 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:52:07 CEST 2016
none inf 5368709120 2678747494 16 26 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:52:28 CEST 2016
none inf 5368709120 1198032544 22 35 1389528792 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:52:50 CEST 2016
none inf 5368709120 1197738517 19 39 1389625814 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:53:11 CEST 2016
none inf 5368709120 2678747494 16 27 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:53:35 CEST 2016
none inf 5368709120 2678747494 16 27 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:53:59 CEST 2016
none inf 5368709120 2678747494 16 27 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:54:22 CEST 2016
none inf 5368709120 2678747494 16 27 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:54:46 CEST 2016
none inf 5368709120 2678747494 16 27 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:55:09 CEST 2016
none inf 5368709120 2477420902 17 28 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:55:31 CEST 2016
none inf 5368709120 1197738514 19 39 1389625815 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:55:55 CEST 2016
none inf 5368709120 1197738514 20 48 1389625815 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:56:17 CEST 2016
none inf 5368709120 2678747506 16 28 895957138 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:56:40 CEST 2016
none inf 5368709120 2678747506 16 28 895957138 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:57:04 CEST 2016
none inf 5368709120 2678747506 16 28 895957138 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:57:28 CEST 2016
none inf 5368709120 2678747506 16 28 895957138 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:57:51 CEST 2016
none inf 5368709120 2678747506 16 28 895957138 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:58:13 CEST 2016
none inf 5368709120 1198032556 16 37 1389528788 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:58:34 CEST 2016
none inf 5368709120 1197738742 19 40 1389625760 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:58:56 CEST 2016
none inf 5368709120 2678747494 16 29 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:59:20 CEST 2016
none inf 5368709120 2678747494 16 29 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:59:43 CEST 2016
none inf 5368709120 2678747494 16 29 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:00:07 CEST 2016
none inf 5368709120 2678747494 16 29 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:00:31 CEST 2016
none inf 5368709120 2678747494 16 29 895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:00:54 CEST 2016
none inf 5368709120 1076586601 22 38 1228684181 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:01:18 CEST 2016
none inf 5368709120 1197724648 19 41 1389630437 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:01:41 CEST 2016
none inf 5368709120 1197724648 19 41 1389630437 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:02:05 CEST 2016
none inf 5368709120 1197724648 19 41 1389630437 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:02:29 CEST 2016
none inf 5368709120 1197724648 19 41 1389630437 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:02:53 CEST 2016
none inf 5368709120 1197724648 19 41 1389630437 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:03:14 CEST 2016
none inf 5368709120 364004107 19 46 1667537284 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:03:36 CEST 2016
none inf 5368709120 197959591 20 48 1722885456 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:03:57 CEST 2016
none inf 5368709120 201060881 18 44 1722549413 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:04:19 CEST 2016
none inf 5368709120 201060881 18 44 1722549413 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:04:40 CEST 2016
none inf 5368709120 201060881 18 44 1722549413 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:05:02 CEST 2016
none inf 5368709120 201060881 18 44 1722549413 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:05:23 CEST 2016
none inf 5368709120 201060881 18 44 1722549413 hdfs://cosmosmaster-gi/user/rbarriuso
After a while the execution finishes with this exception:
16/07/28 19:03:11 INFO mapred.JobClient: Task Id : attempt_201604111313_157784_r_000006_0, Status : FAILED
org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/rbarriuso is exceeded: quota=5368709120 diskspace consumed=5.0g
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3778)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3640)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2400(DFSClient.java:2846)
at org.apache.ha...
As you can see at the end of the log above, the maximum HDFS usage corresponds to 1.722.549.413 bytes and 201.060.881 bytes of free quota (according to hadoop fs -count -q), which doesn't sum the 5GB of available user space.
Moreover, the taken space doesn't match the remaining free space.
How is the remaining quota space calculated?
Is there any way to avoid the DSQuotaExceededException?
Thanks in advance.
You have to take into account the replication factor HDFS applies to all the data. By default, this is 3, thus your effective quota is 5GB/3. This quota can be increased by contacting the admin (me :)) via email.