When I read a csv file in my Jupyter notebook I got the following output
54×9 Array{Any,2}:
"\"25 Nov 2015\"" 28637 11314 3804 2536 10675 4808 14122 3955
"\"18 Nov 2015\"" 28634 11314 3804 2536 10672 4808 14122 3955
"\"11 Nov 2015\"" 28635 11314 3805 2536 10672 4808 14122 3955
"\"4 Nov 2015\"" 28607 11314 3810 2536 10672 4808 14089 3955
"\"25 Oct 2015\"" 28539 11298 3806 2535 10672 4808 14061 3955
⋮ ⋮
"\"12 May 2014\"" 260 182 248 171 12 11 "" ""
"\"1 May 2014\"" 239 160 226 149 13 11 "" ""
"\"14 Apr 2014\"" 176 110 168 108 8 2 "" ""
"\"31 Mar 2014\"" 130 82 122 80 8 2 "" ""
"\"22 Mar 2014\"" 49 29 49 29 "" "" "" ""
I want to change the first column content into a day-month-year format in order to calculate the years passed since first observation (which in this case is my last row).
I tried converting the column using string, then using a for loop (Suppose the name of my file is "ebola"
dias = string.(ebola[:,1])
for i = 1:length(dias)
dias[i] = DateTime(dias[i], "d-m-y")
end
After that, I got this error message
ArgumentError: Unable to parse date time. Expected directive DatePart(d) at char 1
I suppose this error arises because the string has a non-conventional format to manipulate it, with four "" symbols and some backslashes:
"\"25 Nov 2015\""
I must add that when i type, for example:
length("\"22 Mar 2014\"")
result is 13 when you can count 17 characters so clearly backslashes are making some "noise".
What can I don to convert the string in a proper format in order to perform the tasks I want to do?
Any help will be much appreciated.
The reason is that "\"22 Mar 2014\"" is a string containing " characters inside. You can check it by printing it:
julia> s = "\"22 Mar 2014\""
"\"22 Mar 2014\""
julia> print(s)
"22 Mar 2014"
In order to parse the date use a proper format string, e.g.:
julia> Date(s,"\"d u y\"")
2014-03-22
Here is how you can perform the conversion from your array:
julia> x = ["\"25 Nov 2015\"" 28637 11314 3804 2536 10675 4808 14122 3955
"\"18 Nov 2015\"" 28634 11314 3804 2536 10672 4808 14122 3955
"\"11 Nov 2015\"" 28635 11314 3805 2536 10672 4808 14122 3955
"\"4 Nov 2015\"" 28607 11314 3810 2536 10672 4808 14089 3955
"\"25 Oct 2015\"" 28539 11298 3806 2535 10672 4808 14061 3955]
5×9 Array{Any,2}:
"\"25 Nov 2015\"" 28637 11314 3804 2536 10675 4808 14122 3955
"\"18 Nov 2015\"" 28634 11314 3804 2536 10672 4808 14122 3955
"\"11 Nov 2015\"" 28635 11314 3805 2536 10672 4808 14122 3955
"\"4 Nov 2015\"" 28607 11314 3810 2536 10672 4808 14089 3955
"\"25 Oct 2015\"" 28539 11298 3806 2535 10672 4808 14061 3955
julia> Date.(x[:,1], "\"d u y\"")
5-element Array{Date,1}:
2015-11-25
2015-11-18
2015-11-11
2015-11-04
2015-10-25
Related
/var/lib/mysql shows files created by systemd-coredump, while running mysql 8.0.31 docker image. Is this an issue, not sure if the database is created properly in the first place?
volumes:
- ./data/keycloak/mysql:/var/lib/mysql
ls -l data/keycloak/mysql/
total 87400
-rw-r----- 1 systemd-coredump systemd-coredump 196608 Oct 19 17:25 '#ib_16384_0.dblwr'
-rw-r----- 1 systemd-coredump systemd-coredump 8585216 Oct 19 15:42 '#ib_16384_1.dblwr'
drwxr-x--- 2 systemd-coredump systemd-coredump 4096 Oct 19 17:23 '#innodb_redo'
drwxr-x--- 2 systemd-coredump systemd-coredump 4096 Oct 19 17:34 '#innodb_temp'
-rw-r----- 1 systemd-coredump systemd-coredump 56 Oct 19 15:42 auto.cnf
-rw-r----- 1 systemd-coredump systemd-coredump 3026699 Oct 19 15:42 binlog.000001
-rw-r----- 1 systemd-coredump systemd-coredump 180 Oct 19 15:48 binlog.000002
-rw-r----- 1 systemd-coredump systemd-coredump 180 Oct 19 15:55 binlog.000003
-rw-r----- 1 systemd-coredump systemd-coredump 180 Oct 19 15:58 binlog.000004
-rw-r----- 1 systemd-coredump systemd-coredump 180 Oct 19 17:22 binlog.000005
-rw-r----- 1 systemd-coredump systemd-coredump 180 Oct 19 17:34 binlog.000006
-rw-r----- 1 systemd-coredump systemd-coredump 96 Oct 19 17:23 binlog.index
-rw------- 1 systemd-coredump systemd-coredump 1676 Oct 19 15:42 ca-key.pem
-rw-r--r-- 1 systemd-coredump systemd-coredump 1112 Oct 19 15:42 ca.pem
-rw-r--r-- 1 systemd-coredump systemd-coredump 1112 Oct 19 15:42 client-cert.pem
-rw------- 1 systemd-coredump systemd-coredump 1680 Oct 19 15:42 client-key.pem
-rw-r----- 1 systemd-coredump systemd-coredump 3540 Oct 19 17:34 ib_buffer_pool
-rw-r----- 1 systemd-coredump systemd-coredump 12582912 Oct 19 17:34 ibdata1
drwxr-x--- 2 systemd-coredump systemd-coredump 4096 Oct 19 15:42 mysql
-rw-r----- 1 systemd-coredump systemd-coredump 31457280 Oct 19 17:23 mysql.ibd
lrwxrwxrwx 1 systemd-coredump systemd-coredump 27 Oct 19 17:23 mysql.sock -> /var/run/mysqld/mysqld.sock
drwxr-x--- 2 systemd-coredump systemd-coredump 4096 Oct 19 15:42 performance_schema
-rw------- 1 systemd-coredump systemd-coredump 1680 Oct 19 15:42 private_key.pem
-rw-r--r-- 1 systemd-coredump systemd-coredump 452 Oct 19 15:42 public_key.pem
-rw-r--r-- 1 systemd-coredump systemd-coredump 1112 Oct 19 15:42 server-cert.pem
-rw------- 1 systemd-coredump systemd-coredump 1676 Oct 19 15:42 server-key.pem
drwxr-x--- 2 systemd-coredump systemd-coredump 4096 Oct 19 15:42 sso#002dkeycloak
drwxr-x--- 2 systemd-coredump systemd-coredump 4096 Oct 19 15:42 sys
-rw-r----- 1 systemd-coredump systemd-coredump 16777216 Oct 19 17:25 undo_001
-rw-r----- 1 systemd-coredump systemd-coredump 16777216 Oct 19 17:25 undo_002
I had some trouble, binlog.* can be deleted and also you need to configure logs size in mysql config, like
expire_logs_days=10
# from MySql 8
# binlog_expire_log_seconds=864000
max_binlog_size=1024M
You can set those parameters in docker-compose file in command parameter like
mysql:
image: mysql:8
container_name: "my-mysql"
restart: always
environment:
MYSQL_ROOT_PASSWORD: ${DB_PASSWORD}
volumes:
- mysql-volume:/var/lib/mysql
command: ['--default-authentication-plugin=mysql_native_password', '--binlog-row-event-max-size=1024M', '--binlog_expire_logs_seconds=864000']
and then re-build container.
I've seen similar questions being asked on stack overflow but none of them seem to solve my issue properly. I have recently upgraded my system from Ubuntu 16.04 to Ubuntu 17.10 . Since then, I've been experiencing a problem with mysql where I'm not able to start it up properly and it shows the following error:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket
'/var/run/mysqld/mysqld.sock' (2)
I've searched for many solutions online and as a last effort also tried uninstalling and reinstalling but the problem resurfaces after a few days or few attempts. Upon running "service mysql start", it shows:
Job for mysql.service failed because the control process exited with
error code.
See "systemctl status mysql.service" and "journalctl -xe" for
details.
Upon running the suggested commands it shows the following logs:
systemctl status mysql.service
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor
preset: enabled)
Active: activating (start-post) (Result: exit-code) since Thu 2018-
02-22 09:06:31 IST; 4s ago
Process: 8936 ExecStart=/usr/sbin/mysqld (code=exited,
status=1/FAILURE)
Process: 8930 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre
(code=exited, status=0/SUCCESS)
Main PID: 8936 (code=exited, status=1/FAILURE); Control PID: 8937
(mysql-systemd-s)
Tasks: 2 (limit: 4915)
CGroup: /system.slice/mysql.service
└─control
├─8937 /bin/bash /usr/share/mysql/mysql-systemd-start post
└─8948 sleep 1
Feb 22 09:06:31 vamshi-Lenovo-G50-70 systemd[1]: Starting MySQL
Community Server...
Feb 22 09:06:32 vamshi-Lenovo-G50-70 mysqld[8936]: 2018-02-
22T03:36:31.948822Z 0 [Warning] Changed limits: max_open_files: 1024
(requested 5000)
Feb 22 09:06:32 vamshi-Lenovo-G50-70 mysqld[8936]: 2018-02-
22T03:36:31.948879Z 0 [Warning] Changed limits: table_open_cache: 431
(requested 2000)
Feb 22 09:06:32 vamshi-Lenovo-G50-70 mysqld[8936]: 2018-02-
22T03:36:32.126417Z 0 [Warning] TIMESTAMP with implicit DEFAULT value
is deprecated. Please use --explicit_def
Feb 22 09:06:32 vamshi-Lenovo-G50-70 mysqld[8936]: 2018-02-
22T03:36:32.128149Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.21-
0ubuntu0.17.10.1) starting as process 8936 ...
Feb 22 09:06:32 vamshi-Lenovo-G50-70 mysqld[8936]: 2018-02-
22T03:36:32.130652Z 0 [ERROR] Could not open file
'/var/log/mysql/error.log' for error logging: No such file o
Feb 22 09:06:32 vamshi-Lenovo-G50-70 mysqld[8936]: 2018-02-
22T03:36:32.130675Z 0 [ERROR] Aborting
Feb 22 09:06:32 vamshi-Lenovo-G50-70 mysqld[8936]: 2018-02-
22T03:36:32.130694Z 0 [Note] Binlog end
Feb 22 09:06:32 vamshi-Lenovo-G50-70 mysqld[8936]: 2018-02-
22T03:36:32.130750Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
Feb 22 09:06:32 vamshi-Lenovo-G50-70 systemd[1]: mysql.service: Main
process exited, code=exited, status=1/FAILURE
journalctl -xe
Feb 22 09:11:21 vamshi-Lenovo-G50-70 gnome-shell[2902]:
[AppIndicatorSupport-WARN] Attempting to re-register
:1.69/org/ayatana/NotificationItem/multiload; resetting inst
Feb 22 09:11:21 vamshi-Lenovo-G50-70 gnome-shell[2902]:
[AppIndicatorSupport-WARN] Item
:1.69/org/ayatana/NotificationItem/multiload is already registered
ESCOD
Every time this happens, I uninstall mysql and reinstall(server not client), and then it works alright for the first few attempts, then it goes back to same situation. Thanks in advance!
ls -al
total 32824
drwxrwxr-x 8 root syslog 4096 Feb 22 08:44 .
drwxr-xr-x 14 root root 4096 Feb 16 2017 ..
-rw-r--r-- 1 root root 1527 Feb 17 08:58 alternatives.log
-rw-r----- 1 root adm 0 Feb 22 08:44 apport.log
-rw-r----- 1 root adm 486 Feb 21 07:29 apport.log.1
-rw-r----- 1 root adm 455 Feb 20 20:54 apport.log.2.gz
-rw-r----- 1 root adm 514 Feb 19 21:42 apport.log.3.gz
-rw-r----- 1 root adm 758 Feb 19 18:15 apport.log.4.gz
-rw-r----- 1 root adm 459 Feb 17 23:07 apport.log.5.gz
-rw-r----- 1 root adm 1389 Feb 16 00:33 apport.log.6.gz
-rw-r----- 1 root adm 275 Feb 14 20:16 apport.log.7.gz
drwxr-xr-x 2 root root 4096 Feb 17 08:55 apt
-rw-r----- 1 syslog adm 86558 Feb 22 09:17 auth.log
-rw-r----- 1 syslog adm 97522 Feb 18 18:02 auth.log.1
-rw------- 1 root utmp 0 Feb 14 20:17 btmp
drwxr-x--- 2 root lp 4096 Feb 22 08:44 cups
drwxr-xr-x 2 root root 4096 Feb 5 14:51 dist-upgrade
-rw-r--r-- 1 root root 30036 Feb 17 08:58 dpkg.log
drwx--x--x 2 root gdm 4096 Feb 14 20:18 gdm3
-rw-r--r-- 1 root root 2136 Feb 22 08:39 gpu-manager.log
-rw-r----- 1 syslog adm 3745433 Feb 22 09:28 kern.log
-rw-r----- 1 syslog adm 2302923 Feb 18 18:07 kern.log.1
-rw-rw-r-- 1 root utmp 0 Feb 14 20:17 lastlog
-rw-r----- 1 syslog adm 657 Feb 22 08:40 mail.log
-rw-r----- 1 syslog adm 1971 Feb 18 15:31 mail.log.1
-rw-r----- 1 syslog adm 3706136 Feb 22 09:28 syslog
-rw-r----- 1 syslog adm 18760212 Feb 22 08:44 syslog.1
-rw-r----- 1 syslog adm 989385 Feb 21 07:07 syslog.2.gz
-rw-r----- 1 syslog adm 587416 Feb 20 09:09 syslog.3.gz
-rw-r----- 1 syslog adm 593456 Feb 19 18:21 syslog.4.gz
-rw-r----- 1 syslog adm 672682 Feb 18 18:07 syslog.5.gz
-rw-r----- 1 syslog adm 166575 Feb 17 09:09 syslog.6.gz
-rw-r----- 1 syslog adm 891827 Feb 16 00:44 syslog.7.gz
-rw-r----- 1 syslog adm 572158 Feb 22 09:28 ufw.log
-rw-r----- 1 syslog adm 234885 Feb 18 18:06 ufw.log.1
drwxr-xr-x 2 root root 4096 Feb 15 22:00 unattended-upgrades
drwxr-xr-x 2 root root 4096 Feb 22 08:40 vmware
-rw-rw-r-- 1 root utmp 20736 Feb 22 08:41 wtmp
Create folder /var/log/mysql and make sure mysqld can write to it.
I cannot connect to my compute server VM using SSH. It says "retrying 2/3.. 3/3"
It seems as if the server is actually running and functioning. I have Apache Tomcat with some apps installed and MySql server. I can connect to the apache tomcat manager page and I don't see any problem there. I can connect to my MySQL database using client tool such as MySQL WorkBench. I can insert new rows to tables and delete some. It doesn't look like a ran out of free disk space. Anyways I tried to delete some data from tables and shrink it but apparently free space is not the issue.
Here is a Serial Port log I got. Please, what can cause this? I use to be able to connect under same conditions (same client machine, firewall, etc.) so the issue is from the server side.
[34987645.468284] Free swap = 0kB
[34987645.471547] Total swap = 0kB
[34987645.474838] 157083 pages RAM
[34987645.478109] 0 pages HighMem/MovableOnly
[34987645.482323] 4886 pages reserved
[34987645.485853] 0 pages hwpoisoned
[34987645.489270] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[34987645.497494] [ 311] 0 311 6350 1726 14 0 0 dhclient
[34987645.506101] [ 358] 0 358 6869 60 18 0 0 cron
[34987645.514339] [ 369] 0 369 64668 459 29 0 0 rsyslogd
[34987645.522954] [ 373] 0 373 1064 38 8 0 0 acpid
[34987645.531266] [ 406] 0 406 3604 38 12 0 0 agetty
[34987645.539669] [ 407] 0 407 3604 38 12 0 0 agetty
[34987645.548081] [ 408] 0 408 3604 40 12 0 0 agetty
[34987645.556571] [ 409] 0 409 3604 40 12 0 0 agetty
[34987645.564986] [ 410] 0 410 3604 39 12 0 0 agetty
[34987645.573414] [ 411] 0 411 3604 39 12 0 0 agetty
[34987645.583546] [ 412] 0 412 3559 40 10 0 0 agetty
[34987645.591939] [ 7263] 0 7263 13796 168 29 0 -1000 sshd
[34987645.600464] [11774] 107 11774 10565 124 25 0 -900 dbus-daemon
[34987645.610085] [20473] 104 20473 7292 146 19 0 0 ntpd
[34987645.618585] [27741] 0 27741 13169 2273 30 0 0 google_accounts
[34987645.627869] [27744] 0 27744 13211 2284 29 0 0 google_clock_sk
[34987645.637039] [27747] 0 27747 13211 2244 30 0 0 google_ip_forwa
[34987645.646401] [14807] 0 14807 10176 101 22 0 -1000 systemd-udevd
[34987645.655412] [30481] 0 30481 1085 43 6 0 0 mysqld_safe
[34987645.664327] [30815] 109 30815 221020 12045 81 0 0 mysqld
[34987645.672765] [15984] 0 15984 7218 64 19 0 0 systemd-journal
[34987645.681934] [19949] 108 19949 547956 45017 173 0 0 java
[34987645.690161] [23671] 0 23671 10560 86 26 0 0 cron
[34987645.698380] [23672] 0 23672 1085 22 7 0 0 sh
[34987645.706415] [23673] 0 23673 1057 20 8 0 0 run-parts
[34987645.715058] [23674] 0 23674 1085 37 6 0 0 apt
[34987645.723279] [23793] 0 23793 28771 5257 62 0 0 unattended-upgr
[34987645.732642] [23803] 0 23803 28579 5281 57 0 0 unattended-upgr
[34987645.741809] [24237] 0 24237 4583 418 14 0 0 dpkg
[34987645.750026] [24239] 0 24239 1085 23 7 0 0 google-cloud-sd
[34987645.759214] [24241] 0 24241 62608 51157 126 0 0 python2
[34987645.767782] [24263] 0 24263 12231 155 28 0 0 sshd
[34987645.775989] [24265] 0 24265 10553 79 26 0 0 cron
[34987645.784795] [24266] 0 24266 12203 131 27 0 0 sshd
[34987645.793015] [24267] 0 24267 12203 132 25 0 0 sshd
[34987645.801461] [24268] 0 24268 12203 85 25 0 0 sshd
[34987645.809715] [24270] 0 24270 12203 85 25 0 0 sshd
[34987645.817941] [24271] 0 24271 12203 82 25 0 0 sshd
[34987645.826202] [24272] 0 24272 12203 68 24 0 0 sshd
[34987645.834413] [24273] 0 24273 12203 66 26 0 0 sshd
[34987645.842619] [24274] 0 24274 12203 65 25 0 0 sshd
[34987645.850850] [24275] 0 24275 4527 26 11 0 0 sshd
[34987645.859165] Out of memory: Kill process 24241 (python2) score 326 or sacrifice child
[34987645.867298] Killed process 24241 (python2) total-vm:250432kB, anon-rss:204628kB, file-rss:0kB
Sep 2 08:16:18 instance-1 kernel: [34987645.059900] sshd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=-1000
Sep 2 08:16:18 instance-1 kernel: [34987645.067997] sshd cpuset=/ mems_allowed=0
Sep 2 08:16:18 instance-1 kernel: [34987645.072637] CPU: 0 PID: 7263 Comm: sshd Not tainted 3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-2+deb8u3
Sep 2 08:16:18 instance-1 kernel: [34987645.081960] Hardware name: Google Google/Google, BIOS Google 01/01/2011
Sep 2 08:16:18 instance-1 kernel: [34987645.088955] 0000000000000000 ffffffff8150e08f ffff880025daa9a0 0000000000000000
Sep 2 08:16:18 instance-1 kernel: [34987645.097277] ffffffff8150bc8b 0000000000000000 ffffffff810d51ff 0000000000000000
Sep 2 08:16:18 instance-1 kernel: [34987645.105594] ffffffff815129ae 0000000000000200 ffffffff81067933 ffffffff810c2914
Sep 2 08:16:18 instance-1 kernel: [34987645.113999] Call Trace:
Sep 2 08:16:18 instance-1 kernel: [34987645.116817] [] ? dump_stack+0x5d/0x78
Sep 2 08:16:18 instance-1 kernel: [34987645.122507] [] ? dump_header+0x76/0x1e8
Sep 2 08:16:18 instance-1 kernel: [34987645.128393] [] ? smp_call_function_single+0x5f/0xa0
Sep 2 08:16:18 instance-1 kernel: [34987645.135306] [] ? mutex_lock+0xe/0x2a
Sep 2 08:16:18 instance-1 kernel: [34987645.140921] [] ? put_online_cpus+0x23/0x80
Sep 2 08:16:18 instance-1 kernel: [34987645.147045] [] ? rcu_oom_notify+0xc4/0xe0
Sep 2 08:16:18 instance-1 kernel: [34987645.153083] [] ? do_try_to_free_pages+0x4ac/0x520
Sep 2 08:16:18 instance-1 kernel: [34987645.159842] [] ? oom_kill_process+0x21d/0x370
Sep 2 08:16:18 instance-1 kernel: [34987645.166311] [] ? find_lock_task_mm+0x3d/0x90
Sep 2 08:16:18 instance-1 kernel: [34987645.172614] [] ? out_of_memory+0x473/0x4b0
Sep 2 08:16:18 instance-1 kernel: [34987645.178769] [] ? __alloc_pages_nodemask+0x9cd/0xb30
Sep 2 08:16:18 instance-1 kernel: [34987645.185690] [] ? alloc_pages_current+0x9d/0x150
Sep 2 08:16:18 instance-1 kernel: [34987645.192273] [] ? filemap_fault+0x1a0/0x420
Sep 2 08:16:18 instance-1 kernel: [34987645.198395] [] ? __do_fault+0x3a/0xa0
Sep 2 08:16:18 instance-1 kernel: [34987645.204172] [] ? do_read_fault.isra.54+0x4e/0x300
Sep 2 08:16:18 instance-1 kernel: [34987645.210908] [] ? handle_mm_fault+0x63c/0x11c0
Sep 2 08:16:18 instance-1 kernel: [34987645.217288] [] ? vmacache_find+0x29/0xd0
Sep 2 08:16:18 instance-1 kernel: [34987645.223513] [] ? __do_page_fault+0x177/0x4f0
Sep 2 08:16:18 instance-1 kernel: [34987645.229813] [] ? put_prev_entity+0x57/0x350
Sep 2 08:16:18 instance-1 kernel: [34987645.236015] [] ? set_next_entity+0x56/0x70
Sep 2 08:16:18 instance-1 kernel: [34987645.242146] [] ? pick_next_task_fair+0x6e1/0x820
Sep 2 08:16:18 instance-1 kernel: [34987645.248870] [] ? __switch_to+0x15c/0x5a0
Sep 2 08:16:18 instance-1 kernel: [34987645.254812] [] ? page_fault+0x28/0x30
Sep 2 08:16:18 instance-1 kernel: [34987645.260832] Mem-Info:
Sep 2 08:16:18 instance-1 kernel: [34987645.263586] Node 0 DMA per-cpu:
Sep 2 08:16:18 instance-1 kernel: [34987645.267258] CPU 0: hi: 0, btch: 1 usd: 0
Sep 2 08:16:18 instance-1 kernel: [34987645.272519] Node 0 DMA32 per-cpu:
Sep 2 08:16:18 instance-1 kernel: [34987645.276345] CPU 0: hi: 186, btch: 31 usd: 76
Sep 2 08:16:18 instance-1 kernel: [34987645.281531] active_anon:126526 inactive_anon:3096 isolated_anon:0
Sep 2 08:16:18 instance-1 kernel: [34987645.281531] active_file:813 inactive_file:900 isolated_file:0
Sep 2 08:16:18 instance-1 kernel: [34987645.281531] unevictable:0 dirty:0 writeback:0 unstable:0
Sep 2 08:16:18 instance-1 kernel: [34987645.281531] free:8071 slab_reclaimable:2225 slab_unreclaimable:4588
Sep 2 08:16:18 instance-1 kernel: [34987645.281531] mapped:1028 shmem:3130 pagetables:1213 bounce:0
Sep 2 08:16:18 instance-1 kernel: [34987645.281531] free_cma:0
Sep 2 08:16:18 instance-1 kernel: [34987645.314947] Node 0 DMA free:3088kB min:788kB low:984kB high:1180kB active_anon:11608kB inactive_anon:108kB active_file:12kB inactive_file:152kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:112kB slab_reclaimable:88kB slab_unreclaimable:480kB kernel_stack:16kB pagetables:52kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:522 all_unreclaimable? yes
Sep 2 08:16:18 instance-1 kernel: [34987645.356698] lowmem_reserve[]: 0 576 576 576
Sep 2 08:16:18 instance-1 kernel: [34987645.361878] Node 0 DMA32 free:29196kB min:29300kB low:36624kB high:43948kB active_anon:494496kB inactive_anon:12276kB active_file:3240kB inactive_file:3448kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:612340kB managed:592796kB mlocked:0kB dirty:0kB writeback:0kB mapped:4112kB shmem:12408kB slab_reclaimable:8812kB slab_unreclaimable:17872kB kernel_stack:2320kB pagetables:4800kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:11577 all_unreclaimable? yes
Sep 2 08:16:18 instance-1 kernel: [34987645.406481] lowmem_reserve[]: 0 0 0 0
Sep 2 08:16:18 instance-1 kernel: [34987645.411143] Node 0 DMA: 22*4kB (UE) 19*8kB (UEM) 12*16kB (UEM) 7*32kB (UEM) 4*64kB (UEM) 1*128kB (U) 0*256kB 0*512kB 0*1024kB 1*2048kB (R) 0*4096kB = 3088kB
Sep 2 08:16:18 instance-1 kernel: [34987645.428083] Node 0 DMA32: 383*4kB (E) 310*8kB (UEM) 450*16kB (UEM) 200*32kB (UEM) 109*64kB (EM) 4*128kB (E) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 29196kB
Sep 2 08:16:18 instance-1 kernel: [34987645.445684] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 2 08:16:18 instance-1 kernel: [34987645.454775] 4860 total pagecache pages
Sep 2 08:16:18 instance-1 kernel: [34987645.458908] 0 pages in swap cache
Sep 2 08:16:18 instance-1 kernel: [34987645.462650] Swap cache stats: add 0, delete 0, find 0/0
Sep 2 08:16:18 instance-1 kernel: [34987645.468284] Free swap = 0kB
Sep 2 08:16:18 instance-1 kernel: [34987645.471547] Total swap = 0kB
Sep 2 08:16:18 instance-1 kernel: [34987645.474838] 157083 pages RAM
Sep 2 08:16:18 instance-1 kernel: [34987645.478109] 0 pages HighMem/MovableOnly
Sep 2 08:16:18 instance-1 kernel: [34987645.482323] 4886 pages reserved
Sep 2 08:16:18 instance-1 kernel: [34987645.485853] 0 pages hwpoisoned
Sep 2 08:16:18 instance-1 kernel: [34987645.489270] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Sep 2 08:16:18 instance-1 kernel: [34987645.497494] [ 311] 0 311 6350 1726 14 0 0 dhclient
Sep 2 08:16:18 instance-1 kernel: [34987645.506101] [ 358] 0 358 6869 60 18 0 0 cron
Sep 2 08:16:18 instance-1 kernel: [34987645.514339] [ 369] 0 369 64668 459 29 0 0 rsyslogd
Sep 2 08:16:18 instance-1 kernel: [34987645.522954] [ 373] 0 373 1064 38 8 0 0 acpid
Sep 2 08:16:18 instance-1 kernel: [34987645.531266] [ 406] 0 406 3604 38 12 0 0 agetty
Sep 2 08:16:18 instance-1 kernel: [34987645.539669] [ 407] 0 407 3604 38 12 0 0 agetty
Sep 2 08:16:18 instance-1 kernel: [34987645.548081] [ 408] 0 408 3604 40 12 0 0 agetty
Sep 2 08:16:18 instance-1 kernel: [34987645.556571] [ 409] 0 409 3604 40 12 0 0 agetty
Sep 2 08:16:18 instance-1 kernel: [34987645.564986] [ 410] 0 410 3604 39 12 0 0 agetty
Sep 2 08:16:18 instance-1 kernel: [34987645.573414] [ 411] 0 411 3604 39 12 0 0 agetty
Sep 2 08:16:18 instance-1 kernel: [34987645.583546] [ 412] 0 412 3559 40 10 0 0 agetty
Sep 2 08:16:18 instance-1 kernel: [34987645.591939] [ 7263] 0 7263 13796 168 29 0 -1000 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.600464] [11774] 107 11774 10565 124 25 0 -900 dbus-daemon
Sep 2 08:16:18 instance-1 kernel: [34987645.610085] [20473] 104 20473 7292 146 19 0 0 ntpd
Sep 2 08:16:18 instance-1 kernel: [34987645.618585] [27741] 0 27741 13169 2273 30 0 0 google_accounts
Sep 2 08:16:18 instance-1 kernel: [34987645.627869] [27744] 0 27744 13211 2284 29 0 0 google_clock_sk
Sep 2 08:16:18 instance-1 kernel: [34987645.637039] [27747] 0 27747 13211 2244 30 0 0 google_ip_forwa
Sep 2 08:16:18 instance-1 kernel: [34987645.646401] [14807] 0 14807 10176 101 22 0 -1000 systemd-udevd
Sep 2 08:16:18 instance-1 kernel: [34987645.655412] [30481] 0 30481 1085 43 6 0 0 mysqld_safe
Sep 2 08:16:18 instance-1 kernel: [34987645.664327] [30815] 109 30815 221020 12045 81 0 0 mysqld
Sep 2 08:16:18 instance-1 kernel: [34987645.672765] [15984] 0 15984 7218 64 19 0 0 systemd-journal
Sep 2 08:16:18 instance-1 kernel: [34987645.681934] [19949] 108 19949 547956 45017 173 0 0 java
Sep 2 08:16:18 instance-1 kernel: [34987645.690161] [23671] 0 23671 10560 86 26 0 0 cron
Sep 2 08:16:18 instance-1 kernel: [34987645.698380] [23672] 0 23672 1085 22 7 0 0 sh
Sep 2 08:16:18 instance-1 kernel: [34987645.706415] [23673] 0 23673 1057 20 8 0 0 run-parts
Sep 2 08:16:18 instance-1 kernel: [34987645.715058] [23674] 0 23674 1085 37 6 0 0 apt
Sep 2 08:16:18 instance-1 kernel: [34987645.723279] [23793] 0 23793 28771 5257 62 0 0 unattended-upgr
Sep 2 08:16:18 instance-1 kernel: [34987645.732642] [23803] 0 23803 28579 5281 57 0 0 unattended-upgr
Sep 2 08:16:18 instance-1 kernel: [34987645.741809] [24237] 0 24237 4583 418 14 0 0 dpkg
Sep 2 08:16:18 instance-1 kernel: [34987645.750026] [24239] 0 24239 1085 23 7 0 0 google-cloud-sd
Sep 2 08:16:18 instance-1 kernel: [34987645.759214] [24241] 0 24241 62608 51157 126 0 0 python2
Sep 2 08:16:18 instance-1 kernel: [34987645.767782] [24263] 0 24263 12231 155 28 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.775989] [24265] 0 24265 10553 79 26 0 0 cron
Sep 2 08:16:18 instance-1 kernel: [34987645.784795] [24266] 0 24266 12203 131 27 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.793015] [24267] 0 24267 12203 132 25 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.801461] [24268] 0 24268 12203 85 25 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.809715] [24270] 0 24270 12203 85 25 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.817941] [24271] 0 24271 12203 82 25 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.826202] [24272] 0 24272 12203 68 24 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.834413] [24273] 0 24273 12203 66 26 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.842619] [24274] 0 24274 12203 65 25 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.850850] [24275] 0 24275 4527 26 11 0 0 sshd
Sep 2 08:16:18 instance-1 kernel: [34987645.859165] Out of memory: Kill process 24241 (python2) score 326 or sacrifice child
Sep 2 08:16:18 instance-1 kernel: [34987645.867298] Killed process 24241 (python2) total-vm:250432kB, anon-rss:204628kB, file-rss:0kB
Sep 2 16:21:27 instance-1 dhclient: DHCPREQUEST on eth0 to 169.254.169.254 port 67
Sep 2 16:21:27 instance-1 dhclient: DHCPACK from 169.254.169.254
Sep 2 16:21:27 instance-1 dhclient: bound to 10.132.0.2 -- renewal in 40860 seconds.
Sep 2 21:14:06 instance-1 systemd[1]: Starting Cleanup of Temporary Directories...
Sep 2 21:14:06 instance-1 systemd[1]: Started Cleanup of Temporary Directories.
Sep 3 03:42:27 instance-1 dhclient: DHCPREQUEST on eth0 to 169.254.169.254 port 67
Sep 3 03:42:27 instance-1 dhclient: DHCPACK from 169.254.169.254
Sep 3 03:42:27 instance-1 dhclient: bound to 10.132.0.2 -- renewal in 34630 seconds.
Sep 3 13:19:37 instance-1 dhclient: DHCPREQUEST on eth0 to 169.254.169.254 port 67
Sep 3 13:19:37 instance-1 dhclient: DHCPACK from 169.254.169.254
Sep 3 13:19:38 instance-1 dhclient: bound to 10.132.0.2 -- renewal in 33626 seconds.
Sep 3 21:14:22 instance-1 systemd[1]: Starting Cleanup of Temporary Directories...
Sep 3 21:14:22 instance-1 systemd[1]: Started Cleanup of Temporary Directories.
Sep 3 22:40:04 instance-1 dhclient: DHCPREQUEST on eth0 to 169.254.169.254 port 67
Sep 3 22:40:04 instance-1 dhclient: DHCPACK from 169.254.169.254
Sep 3 22:40:04 instance-1 dhclient: bound to 10.132.0.2 -- renewal in 38182 seconds.
Sep 4 09:16:26 instance-1 dhclient: DHCPREQUEST on eth0 to 169.254.169.254 port 67
Sep 4 09:16:26 instance-1 dhclient: DHCPACK from 169.254.169.254
Sep 4 09:16:26 instance-1 dhclient: bound to 10.132.0.2 -- renewal in 35099 seconds.
Sep 4 19:01:25 instance-1 dhclient: DHCPREQUEST on eth0 to 169.254.169.254 port 67
Sep 4 19:01:25 instance-1 dhclient: DHCPACK from 169.254.169.254
Sep 4 19:01:25 instance-1 dhclient: bound to 10.132.0.2 -- renewal in 40004 seconds.
Sep 4 21:14:26 instance-1 systemd[1]: Starting Cleanup of Temporary Directories...
Sep 4 21:14:26 instance-1 systemd[1]: Started Cleanup of Temporary Directories.
Okay, I was able to solve the issue.
Apparently, the ssh process in the machine stopped because I ran out of memory. When RAM is full the linux system kills processes.
I couldn't directly see it from the log because it killed something else and indirectly killed ssh.
Here's the solution:
I stopped the machine and then resumed it.
WARNING: my ip address was ephemeral so I changed it to static before stopping the machine. If I wouldn't do that, I would get a different IP address which would force my to update my domain name records.
It can be later switched back to ephemeral.
Google charges for Static IP address.
I can now log in ssh.
I guess a different question is how to prevent SSH from ever being killed.
I hope it helps anyone.
I'm writing a arbitrary JSON parser for golang, the project nearly finished. But I found a confusing problem about the performance:
I want to test the performance about parse a big(100Mb) JSON string, I use the test file itself to init the JSON struct in memory and write the marshaled JSON string to a file, and then read from it, if the file exists already, will not init it in memory, just read from file directly. The performance is totally different: cost about double time to parse when read from file directly. At the same time, I'm testing the performance about parse normal(1Kb) JSON string and deep(2Mb) JSON string, both of these are almost unaffected.
Why? is CPU cache? or GC? or any else?
The code at https://github.com/acrazing/cheapjson, and I opened a issue about this problem at https://github.com/acrazing/cheapjson/issues/1. You can get more detailed information about the benchmark from here.
I am unable to reproduce your issue: https://github.com/acrazing/cheapjson/issues/1. My benchmark results:
$ go version
go version devel +b817359 Sat Jul 22 01:29:58 2017 +0000 linux/amd64
$ rm -rf data
$ ls -la data
ls: cannot access 'data': No such file or directory
$ go test -v -run=! -bench=. -benchmem parser_test.go
2017/07/23 02:42:09 big input size: 117265882, normal input size: 763, deep input size: 2134832
goos: linux
goarch: amd64
BenchmarkUnmarshalBigInput-4 1 1107362634 ns/op 278192256 B/op 8280859 allocs/op
BenchmarkSimpleJsonBigInput-4 1 2680878194 ns/op 569595680 B/op 7575252 allocs/op
BenchmarkUnmarshalNormalInput-4 300000 5685 ns/op 4622 B/op 129 allocs/op
BenchmarkSimpleJsonNormalInput-4 200000 9565 ns/op 5602 B/op 87 allocs/op
BenchmarkUnmarshalDeepInput-4 1000 2085186 ns/op 372922 B/op 5134 allocs/op
BenchmarkSimpleJsonDeepInput-4 100 12311435 ns/op 8911102 B/op 6117 allocs/op
PASS
ok command-line-arguments 15.067s
$ ls -la datatotal 116620
drwxr-xr-x 2 peter peter 4096 Jul 23 02:42 .
drwxr-xr-x 5 peter peter 4096 Jul 23 02:42 ..
-rwxr-xr-x 1 peter peter 117265882 Jul 23 02:42 big.json
-rwxr-xr-x 1 peter peter 2134832 Jul 23 02:42 deep.json
-rwxr-xr-x 1 peter peter 763 Jul 23 02:42 normal.json
$ go test -v -run=! -bench=. -benchmem parser_test.go
2017/07/23 02:42:31 big input size: 117265882, normal input size: 763, deep input size: 2134832
goos: linux
goarch: amd64
BenchmarkUnmarshalBigInput-4 1 1140498937 ns/op 278220704 B/op 8280995 allocs/op
BenchmarkSimpleJsonBigInput-4 1 2685285322 ns/op 569592608 B/op 7575242 allocs/op
BenchmarkUnmarshalNormalInput-4 300000 5685 ns/op 4622 B/op 129 allocs/op
BenchmarkSimpleJsonNormalInput-4 200000 9633 ns/op 5601 B/op 87 allocs/op
BenchmarkUnmarshalDeepInput-4 1000 2086891 ns/op 372927 B/op 5134 allocs/op
BenchmarkSimpleJsonDeepInput-4 100 12387413 ns/op 8911084 B/op 6117 allocs/op
PASS
ok command-line-arguments 11.903s
$ ls -la data
total 116624
drwxr-xr-x 2 peter peter 4096 Jul 23 02:42 .
drwxr-xr-x 5 peter peter 4096 Jul 23 02:42 ..
-rwxr-xr-x 1 peter peter 117265882 Jul 23 02:42 big.json
-rwxr-xr-x 1 peter peter 2134832 Jul 23 02:42 deep.json
-rwxr-xr-x 1 peter peter 763 Jul 23 02:42 normal.json
$
I got similar results on four different machines and two Operating systems: three Linux and one Windows.
This is the old benchmark result, time is at 11:01
This is the final test result currently, the result could not reproduce, it's amazing.
On my website VPS the MySQL goes down quite often, once a day lately.
I've looked in my syslog and seems like it is an Out Of Memory error.
Is my MySQL not configured the right way or is does my VPS not have enough memory for what I'm running on it?
Here are free and top:
$ free -m
total used free shared buffers cached
Mem: 512 360 151 0 0 42
-/+ buffers/cache: 317 194
Swap: 0 0 0
top - 14:17:47 up 15 days, 9:41, 2 users, load average: 0.13, 0.30, 0.42
Tasks: 58 total, 1 running, 57 sleeping, 0 stopped, 0 zombie
Cpu(s): 19.7%us, 2.7%sy, 0.0%ni, 77.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 524508k total, 415200k used, 109308k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 72404k cached
In my.cnf:
# * Fine Tuning
#
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
I'm using
mysql Ver 14.14 Distrib 5.1.66, for debian-linux-gnu (x86_64) using readline 6.1
on
Debian GNU/Linux 6.0.7 (squeeze)
Here is the block from my syslog:
Jun 23 21:19:01 vps528 kernel: [1182395.730776] mysqld invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
Jun 23 21:19:01 vps528 kernel: [1182395.730789] Pid: 13220, comm: mysqld Not tainted 2.6.24-19-xen #1
Jun 23 21:19:01 vps528 kernel: [1182395.730791]
Jun 23 21:19:01 vps528 kernel: [1182395.730792] Call Trace:
Jun 23 21:19:01 vps528 kernel: [1182395.730802] [<ffffffff80275e26>] oom_kill_process+0xf6/0x110
Jun 23 21:19:01 vps528 kernel: [1182395.730806] [<ffffffff802762ae>] out_of_memory+0x19e/0x1e0
Jun 23 21:19:01 vps528 kernel: [1182395.730810] [<ffffffff80278bd9>] __alloc_pages+0x389/0x3c0
Jun 23 21:19:01 vps528 kernel: [1182395.730815] [<ffffffff8027af54>] __do_page_cache_readahead+0x104/0x260
Jun 23 21:19:01 vps528 kernel: [1182395.730818] [<ffffffff8027520e>] filemap_fault+0x2de/0x3e0
Jun 23 21:19:01 vps528 kernel: [1182395.730822] [<ffffffff80281f3a>] __do_fault+0x6a/0x5d0
Jun 23 21:19:01 vps528 kernel: [1182395.730824] [<ffffffff80287551>] handle_mm_fault+0x1d1/0xd60
Jun 23 21:19:01 vps528 kernel: [1182395.730829] [<ffffffff80229f63>] set_next_entity+0x23/0x50
Jun 23 21:19:01 vps528 kernel: [1182395.730835] [<ffffffff80473553>] do_page_fault+0x1f3/0x11e0
Jun 23 21:19:01 vps528 kernel: [1182395.730840] [<ffffffff8046f7e9>] thread_return+0x3a/0x481
Jun 23 21:19:01 vps528 kernel: [1182395.730845] [<ffffffff80471bf7>] error_exit+0x0/0x79
Jun 23 21:19:01 vps528 kernel: [1182395.730848]
Jun 23 21:19:01 vps528 kernel: [1182395.730850] Mem-info:
Jun 23 21:19:01 vps528 kernel: [1182395.730851] DMA per-cpu:
Jun 23 21:19:01 vps528 kernel: [1182395.730853] CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Jun 23 21:19:01 vps528 kernel: [1182395.730855] DMA32 per-cpu:
Jun 23 21:19:01 vps528 kernel: [1182395.730857] CPU 0: Hot: hi: 186, btch: 31 usd: 172 Cold: hi: 62, btch: 15 usd: 14
Jun 23 21:19:01 vps528 kernel: [1182395.730861] Active:94084 inactive:618 dirty:0 writeback:0 unstable:0
Jun 23 21:19:01 vps528 kernel: [1182395.730861] free:4898 slab:3594 mapped:0 pagetables:3925 bounce:0
Jun 23 21:19:01 vps528 kernel: [1182395.730866] DMA free:11540kB min:28kB low:32kB high:40kB active:0kB inactive:0kB present:16160kB pages_scanned:0 all_unreclaimable? yes
Jun 23 21:19:01 vps528 kernel: [1182395.730869] lowmem_reserve[]: 0 4024 4024 4024
Jun 23 21:19:01 vps528 kernel: [1182395.730875] DMA32 free:8052kB min:8100kB low:10124kB high:12148kB active:376336kB inactive:2472kB present:4120800kB pages_scanned:767723 all_unreclaimable? yes
Jun 23 21:19:01 vps528 kernel: [1182395.730878] lowmem_reserve[]: 0 0 0 0
Jun 23 21:19:01 vps528 kernel: [1182395.730881] DMA: 3*4kB 3*8kB 3*16kB 4*32kB 3*64kB 1*128kB 1*256kB 1*512kB 2*1024kB 0*2048kB 2*4096kB = 11540kB
Jun 23 21:19:01 vps528 kernel: [1182395.730900] DMA32: 62*4kB 4*8kB 3*16kB 4*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 8008kB
Jun 23 21:19:01 vps528 kernel: [1182395.730908] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jun 23 21:19:01 vps528 kernel: [1182395.730910] Free swap = 0kB
Jun 23 21:19:01 vps528 kernel: [1182395.730911] Total swap = 0kB
Jun 23 21:19:01 vps528 kernel: [1182395.730912] Free swap: 0kB
Jun 23 21:19:01 vps528 kernel: [1182395.740886] 1048576 pages of RAM
Jun 23 21:19:01 vps528 kernel: [1182395.740893] 21255 reserved pages
Jun 23 21:19:01 vps528 kernel: [1182395.740894] 39897 pages shared
Jun 23 21:19:01 vps528 kernel: [1182395.740895] 0 pages swap cached
Jun 23 21:19:01 vps528 kernel: [1182395.740898] Out of memory: kill process 25246 (apache2) score 73450 or a child
Jun 23 21:19:01 vps528 kernel: [1182395.740913] Killed process 25246 (apache2)
Looks like the VPS I am running doesn't have enough RAM for the amount of website and databases I am running on it. I'll be upgrading my VPS to one with 1GB of memory.