I have a problem with ubuntu killing mySQL server due to out of memory when running Magento. I've ran the mySQLtuner script (which warned that possible memory allocation for mySQL was higher than system memory) and adjusted the mySQL /etc/mysql/my.cnf file accordingly.
According to the script the maximum allowed memory usage for mySQL is 218,9 MB which should not cause a problem. Free memory when the server is just started and everything is working is 277 / 590 MB. After clicking around for a wile in Magento frontend the mySQL server is killed, here is a dump from /var/log/syslog:
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037361] mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037367] mysqld cpuset=/ mems_allowed=0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037370] Pid: 948, comm: mysqld Not tainted 3.2.0-29-virtual #46-Ubuntu
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037372] Call Trace:
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037382] [<ffffffff810bd5fd>] ? cpuset_print_task_mems_allowed+0x9d/0xb0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037386] [<ffffffff81117ce1>] dump_header+0x91/0xe0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037388] [<ffffffff81118065>] oom_kill_process+0x85/0xb0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037391] [<ffffffff8111840a>] out_of_memory+0xfa/0x220
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037394] [<ffffffff8111de3a>] __alloc_pages_nodemask+0x7ea/0x800
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037398] [<ffffffff8165296e>] ? _raw_spin_unlock_irqrestore+0x1e/0x30
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037403] [<ffffffff81154b73>] alloc_pages_current+0xa3/0x110
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037406] [<ffffffff81114a4f>] __page_cache_alloc+0x8f/0xa0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037409] [<ffffffff81114d4e>] ? find_get_page+0x1e/0x90
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037412] [<ffffffff81116bd2>] filemap_fault+0x212/0x3c0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037415] [<ffffffff81136e62>] __do_fault+0x72/0x550
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037419] [<ffffffff8108ca60>] ? lock_hrtimer_base.isra.24+0x30/0x60
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037422] [<ffffffff8113a70a>] handle_pte_fault+0xfa/0x200
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037426] [<ffffffff8100648e>] ? xen_pmd_val+0xe/0x10
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037428] [<ffffffff810052e9>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037431] [<ffffffff8113abc8>] handle_mm_fault+0x1f8/0x350
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037434] [<ffffffff8165625b>] do_page_fault+0x14b/0x520
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037437] [<ffffffff8108d2a4>] ? hrtimer_start_range_ns+0x14/0x20
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037440] [<ffffffff8103cda5>] ? pvclock_clocksource_read+0x55/0xf0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037444] [<ffffffff8100a640>] ? xen_clocksource_read+0x20/0x30
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037447] [<ffffffff8100a729>] ? xen_clocksource_get_cycles+0x9/0x10
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037456] [<ffffffff81093177>] ? getnstimeofday+0x57/0xe0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037459] [<ffffffff81652eb5>] page_fault+0x25/0x30
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037460] Mem-Info:
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037462] Node 0 DMA per-cpu:
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037464] CPU 0: hi: 0, btch: 1 usd: 0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037466] Node 0 DMA32 per-cpu:
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037468] CPU 0: hi: 186, btch: 31 usd: 123
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037471] active_anon:139853 inactive_anon:31 isolated_anon:0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037472] active_file:38 inactive_file:169 isolated_file:0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037473] unevictable:0 dirty:0 writeback:0 unstable:0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037473] free:1375 slab_reclaimable:1934 slab_unreclaimable:2073
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037474] mapped:55 shmem:64 pagetables:2195 bounce:0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037476] Node 0 DMA free:2464kB min:72kB low:88kB high:108kB active_anon:12232kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14524kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:32kB slab_unreclaimable:12kB kernel_stack:0kB pagetables:24kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3 all_unreclaimable? yes
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037484] lowmem_reserve[]: 0 597 597 597
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037487] Node 0 DMA32 free:3036kB min:3088kB low:3860kB high:4632kB active_anon:547180kB inactive_anon:124kB active_file:152kB inactive_file:664kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:611856kB mlocked:0kB dirty:0kB writeback:0kB mapped:220kB shmem:256kB slab_reclaimable:7704kB slab_unreclaimable:8280kB kernel_stack:792kB pagetables:8756kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1943 all_unreclaimable? yes
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037495] lowmem_reserve[]: 0 0 0 0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037498] Node 0 DMA: 8*4kB 6*8kB 5*16kB 2*32kB 3*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 2464kB
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037506] Node 0 DMA32: 247*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3036kB
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037513] 279 total pagecache pages
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037514] 0 pages in swap cache
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037516] Swap cache stats: add 0, delete 0, find 0/0
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037517] Free swap = 0kB
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.037518] Total swap = 0kB
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041176] 159472 pages RAM
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041180] 8376 pages reserved
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041181] 16609 pages shared
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041182] 143523 pages non-shared
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041184] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041194] [ 246] 0 246 4306 49 0 0 0 upstart-udev-br
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041197] [ 250] 0 250 5365 116 0 -17 -1000 udevd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041200] [ 300] 0 300 5364 98 0 -17 -1000 udevd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041203] [ 301] 0 301 5364 97 0 -17 -1000 udevd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041206] [ 393] 0 393 3795 47 0 0 0 upstart-socket-
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041209] [ 444] 0 444 1814 123 0 0 0 dhclient3
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041212] [ 617] 0 617 12487 153 0 -17 -1000 sshd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041215] [ 629] 101 629 63428 103 0 0 0 rsyslogd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041218] [ 636] 102 636 5952 74 0 0 0 dbus-daemon
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041221] [ 692] 0 692 3624 39 0 0 0 getty
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041224] [ 699] 0 699 3624 43 0 0 0 getty
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041227] [ 710] 0 710 3624 41 0 0 0 getty
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041230] [ 711] 0 711 3624 41 0 0 0 getty
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041236] [ 718] 0 718 3624 43 0 0 0 getty
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041239] [ 725] 0 725 1080 36 0 0 0 acpid
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041242] [ 727] 0 727 4776 59 0 0 0 cron
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041245] [ 728] 0 728 4225 40 0 0 0 atd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041248] [ 753] 106 753 203063 19815 0 0 0 mysqld
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041251] [ 808] 103 808 46895 299 0 0 0 whoopsie
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041254] [ 857] 0 857 6274 78 0 0 0 master
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041257] [ 867] 107 867 6831 84 0 0 0 qmgr
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041260] [ 910] 0 910 76016 1435 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041263] [ 953] 33 953 84720 10145 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041266] [ 954] 33 954 90925 15531 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041272] [ 955] 33 955 76062 1474 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041275] [ 957] 33 957 81127 6426 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041278] [ 1065] 0 1065 22444 5138 0 0 0 /usr/share/webm
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041281] [ 1069] 0 1069 3624 41 0 0 0 getty
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041284] [ 1243] 33 1243 90163 15653 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041287] [ 1246] 33 1246 79452 4952 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041290] [ 1247] 33 1247 93679 18462 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041292] [ 1248] 33 1248 93574 19110 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041295] [ 1249] 33 1249 92189 16935 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041298] [ 5116] 107 5116 6790 68 0 0 0 pickup
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041301] [ 5169] 0 5169 18338 200 0 0 0 sshd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041304] [ 5287] 1000 5287 18338 200 0 0 0 sshd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041307] [ 5288] 1000 5288 6248 1457 0 0 0 bash
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041310] [ 5562] 0 5562 22444 5098 0 0 0 miniserv.pl
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041313] [ 5597] 0 5597 18338 201 0 0 0 sshd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041315] [ 5687] 1000 5687 18338 201 0 0 0 sshd
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041318] [ 5688] 1000 5688 6235 1446 0 0 0 bash
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041321] [ 5784] 1000 5784 10474 92 0 0 0 sudo
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041324] [ 5785] 0 5785 3337 171 0 0 0 nano
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041327] [ 5786] 33 5786 84584 10087 0 0 0 apache2
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041330] Out of memory: Kill process 753 (mysqld) score 131 or sacrifice child
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.041396] Killed process 753 (mysqld) total-vm:812252kB, anon-rss:79260kB, file-rss:0kB
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.217049] init: mysql main process (753) killed by KILL signal
Dec 28 11:11:52 ip-10-228-98-15 kernel: [37224.218251] init: mysql main process ended, respawning
Dec 28 11:11:53 ip-10-228-98-15 kernel: [37225.185699] type=1400 audit(1356693113.390:11): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=5794 comm="apparmor_parser"
Dec 28 11:11:54 ip-10-228-98-15 kernel: [37226.152344] init: mysql main process (5798) terminated with status 1
Dec 28 11:11:54 ip-10-228-98-15 kernel: [37226.152382] init: mysql main process ended, respawning
Dec 28 11:11:54 ip-10-228-98-15 kernel: [37226.768092] init: mysql post-start process (5799) terminated with status 1
Dec 28 11:11:54 ip-10-228-98-15 kernel: [37226.784475] type=1400 audit(1356693114.990:12): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=5822 comm="apparmor_parser"
Dec 28 11:11:55 ip-10-228-98-15 kernel: [37226.844954] init: mysql main process (5826) terminated with status 1
Dec 28 11:11:55 ip-10-228-98-15 kernel: [37226.844991] init: mysql respawning too fast, stopped
So basically mySQL is busted for hogging to much memory and then sentenced to death. The question is why? I thought my settings in my.cnf had limited the memory mySQL could use?
Another interesting question is why it won't restart again? It has nothing to do with apparmor, I removed it but same result but without the apparmor reference in the syslog.
my.cnf:
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 1
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
max_connections = 10
#table_cache = 64
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size = 16M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf.
#
# Here you can see queries with especially long duration
#log_slow_queries = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
[isamchk]
key_buffer = 16M
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/
I do realize that these memory settings are way to low for any performance in Magento but I've tried all kinds of value here to the same result. Crasch!
The server is a EC2 micro instance running Ubuntu 12.04 x64 and i don't by any means expect blazing speeds (no production use) but at least I should be able to configure it so it doesn't crasch all the time. I've another EC2 micro instance running on 32bit Ubuntu and the mySQL server has never ever crashed on that one. Could it be related to to 32 vs 64 bit Ubuntu?
As you can see there are many questions but limited knowledge, grateful for any pointers. I'll keep you posted.
In the end I went with the suggestion of Anton S above and simply accepted that the instance did not have enough memory. Launched it as an ec2 small instance instead and now it works like a charm.
Related
I'm trying to restart my mysql/mysqld service on my ubuntu server but I keep getting the same errors.
I'm using mysql version: 8.0.23
Ubuntu version: Ubuntu 20.04.2 LTS
systemctl status mysql.service output:
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2021-03-27 17:15:22 UTC; 10s ago
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
Process: 125121 ExecStartPre=/usr/share/mysql-8.0/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Process: 125175 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
Main PID: 125175 (code=exited, status=1/FAILURE)
Status: "Server startup in progress"
Mar 27 17:13:41 server systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Mar 27 17:13:41 server systemd[1]: mysql.service: Failed with result 'exit-code'.
Mar 27 17:13:41 server systemd[1]: Stopped MySQL Community Server.
Mar 27 17:13:41 server systemd[1]: Starting MySQL Community Server...
Mar 27 17:15:22 server systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Mar 27 17:15:22 server systemd[1]: mysql.service: Failed with result 'exit-code'.
Mar 27 17:15:22 server systemd[1]: Failed to start MySQL Community Server.
journalctl -xe output:
--
-- A start job for unit mysql.service has finished with a failure.
--
-- The job identifier is 26786 and the job result is failed.
Mar 27 17:15:39 server kernel: [UFW BLOCK] IN=eth0 OUT= MAC=MACADDRESS SRC=SRCADDRESS DST=SERVER_IP LEN=52 TOS>
Mar 27 17:15:57 server kernel: [UFW BLOCK] IN=eth0 OUT= MAC=MACADDRESS SRC=SRCADDRESS DST=SERVER_IP LEN=52 TOS>
Mar 27 17:16:19 server kernel: [UFW BLOCK] IN=eth0 OUT= MAC=MACADDRESS SRC=SRCADDRESS DST=SERVER_IP LEN=48 TOS>
Mar 27 17:16:38 server kernel: [UFW BLOCK] IN=eth0 OUT= MAC=MACADDRESS SRC=SRCADDRESS DST=SERVER_IP LEN=52 TOS>
Mar 27 17:16:57 server kernel: [UFW BLOCK] IN=eth0 OUT= MAC=MACADDRESS SRC=SRCADDRESS DST=SERVER_IP LEN=52 TOS>
Mar 27 17:17:01 server CRON[125231]: pam_unix(cron:session): session opened for user root by (uid=0)
Mar 27 17:17:01 server CRON[125232]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Mar 27 17:17:01 server CRON[125231]: pam_unix(cron:session): session closed for user root
Mar 27 17:17:16 server kernel: [UFW BLOCK] IN=eth0 OUT= MAC=MACADDRESS SRC=SRCADDRESS DST=SERVER_IP LEN=48 TOS>
Mar 27 17:17:24 server sshd[125236]: error: kex_exchange_identification: Connection closed by remote host
Mar 27 17:17:37 server kernel: [UFW BLOCK] IN=eth0 OUT= MAC=MACADDRESS SRC=SRCADDRESS DST=SERVER_IP LEN=48 TOS>
Mar 27 17:17:56 server kernel: [UFW BLOCK] IN=eth0 OUT= MAC=MACADDRESS SRC=SRCADDRESS DST=SERVER_IP LEN=52 TOS>
Mar 27 17:18:03 server sshd[125237]: Unable to negotiate with 68.183.4.74 port 49612: no matching key exchange method found. Their offer: diffie-hellm>
Mar 27 17:18:17 server kernel: [UFW BLOCK] IN=eth0 OUT= MAC=MACADDRESS SRC=SRCADDRESS DST=SERVER_IP LEN=40 TOS=>
Mar 27 17:18:30 server systemd[1]: fwupd.service: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit fwupd.service has successfully entered the 'dead' state.
Mar 27 17:18:35 server hv_kvp_daemon[125265]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Mar 27 17:18:35 server hv_kvp_daemon[125267]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found
Mar 27 17:18:35 server hv_kvp_daemon[125275]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Mar 27 17:18:35 server hv_kvp_daemon[125277]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found
Mar 27 17:18:39 server kernel: [UFW BLOCK] IN=eth0 OUT= MAC=MACADDRESS SRC=SRCADDRESS DST=SERVER_IP LEN=48 TOS>
Mar 27 17:18:59 server kernel: [UFW BLOCK] IN=eth0 OUT= MAC=MACADDRESS SRC=SRCADDRESS DST=SERVER_IP LEN=52 TOS>
Mar 27 17:19:17 server kernel: [UFW BLOCK] IN=eth0 OUT= MAC=MACADDRESS SRC=SRCADDRESS DST=SERVER_IP LEN=52 TOS>
Edit 1: new systemctl status mysql.service output
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2021-03-27 18:20:26 UTC; 28s ago
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
Process: 135987 ExecStartPre=/usr/share/mysql-8.0/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Process: 136026 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
Main PID: 136026 (code=exited, status=1/FAILURE)
Status: "Server startup in progress"
Error: 17 (File exists)
Mar 27 18:20:24 emkes-vps systemd[1]: Starting MySQL Community Server...
Mar 27 18:20:26 emkes-vps systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Mar 27 18:20:26 emkes-vps systemd[1]: mysql.service: Failed with result 'exit-code'.
Mar 27 18:20:26 emkes-vps systemd[1]: Failed to start MySQL Community Server.
journalctl -xe output:
Mar 27 18:22:21 emkes-vps hv_kvp_daemon[136366]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Mar 27 18:22:21 emkes-vps hv_kvp_daemon[136368]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found
Mar 27 18:22:21 emkes-vps hv_kvp_daemon[136376]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Mar 27 18:22:21 emkes-vps hv_kvp_daemon[136378]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found
I am unable to start mysql on my EC2 instance. Receiving following errors
Oct 28 05:21:08 ip-172-30-1-66 kernel: [ 841.115678] init: mysql main process (19029) terminated with status 7
Oct 28 05:21:08 ip-172-30-1-66 kernel: [ 841.115687] init: mysql main process ended, respawning
Oct 28 05:21:09 ip-172-30-1-66 kernel: [ 842.011098] init: mysql post-start process (19030) terminated with status 1
Oct 28 05:21:09 ip-172-30-1-66 kernel: [ 842.016704] type=1400 audit(1540704069.482:278): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/mysqld" pid=19084 comm="apparmor_parser"
This is on an Ubuntu system
# uname -a
Linux ip-172-30-1-66 3.13.0-91-generic #138-Ubuntu SMP Fri Jun 24 17:00:34 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
I have xampp installed on my windows 8.1 32-bit. apache and MySQL are running. and i am having kind of weird problem. after a restart of my machine localhost runs but only and if tried reloading it won't load. so i tried some solutions over the internet, include clearing my cache. so i tried clearing my cache, localhost loaded but again only for one time, tried reloading it, but won't load . here is the error log for the day
[Thu Jul 07 01:09:02.134872 2016] [access_compat:error] [pid 8272:tid 1776] [client 100.93.115.237:57084] AH01797: client denied by server configuration: C:/xampp/htdocs/xampp/
[Thu Jul 07 12:27:38.232732 2016] [core:warn] [pid 5548:tid 392] AH00098: pid file C:/xampp/apache/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Thu Jul 07 12:27:38.772405 2016] [ssl:warn] [pid 5548:tid 392] AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]
[Thu Jul 07 12:27:42.577513 2016] [mpm_winnt:notice] [pid 5548:tid 392] AH00455: Apache/2.4.3 (Win32) OpenSSL/1.0.1c PHP/5.4.7 configured -- resuming normal operations
[Thu Jul 07 12:27:42.577513 2016] [mpm_winnt:notice] [pid 5548:tid 392] AH00456: Server built: Aug 18 2012 12:41:37
[Thu Jul 07 12:27:42.577513 2016] [core:notice] [pid 5548:tid 392] AH00094: Command line: 'c:\\xampp\\apache\\bin\\httpd.exe -d C:/xampp/apache'
[Thu Jul 07 12:27:42.624392 2016] [mpm_winnt:notice] [pid 5548:tid 392] AH00418: Parent: Created child process 1128
[Thu Jul 07 12:27:44.139436 2016] [ssl:warn] [pid 1128:tid 356] AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]
[Thu Jul 07 12:27:44.266630 2016] [mpm_winnt:notice] [pid 1128:tid 356] AH00354: Child: Starting 150 worker threads.
[Thu Jul 07 12:30:38.084960 2016] [core:warn] [pid 6960:tid 376] AH00098: pid file C:/xampp/apache/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Thu Jul 07 12:30:38.412969 2016] [ssl:warn] [pid 6960:tid 376] AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]
[Thu Jul 07 12:30:38.472973 2016] [mpm_winnt:notice] [pid 6960:tid 376] AH00455: Apache/2.4.3 (Win32) OpenSSL/1.0.1c PHP/5.4.7 configured -- resuming normal operations
[Thu Jul 07 12:30:38.472973 2016] [mpm_winnt:notice] [pid 6960:tid 376] AH00456: Server built: Aug 18 2012 12:41:37
[Thu Jul 07 12:30:38.472973 2016] [core:notice] [pid 6960:tid 376] AH00094: Command line: 'c:\\xampp\\apache\\bin\\httpd.exe -d C:/xampp/apache'
[Thu Jul 07 12:30:39.464998 2016] [mpm_winnt:notice] [pid 6960:tid 376] AH00418: Parent: Created child process 5448
[Thu Jul 07 12:30:40.893037 2016] [ssl:warn] [pid 5448:tid 348] AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]
[Thu Jul 07 12:30:40.957041 2016] [mpm_winnt:notice] [pid 5448:tid 348] AH00354: Child: Starting 150 worker threads.
[Thu Jul 07 13:16:55.613227 2016] [core:warn] [pid 5772:tid 392] AH00098: pid file C:/xampp/apache/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Thu Jul 07 13:16:55.910117 2016] [ssl:warn] [pid 5772:tid 392] AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]
[Thu Jul 07 13:16:56.925786 2016] [mpm_winnt:notice] [pid 5772:tid 392] AH00455: Apache/2.4.3 (Win32) OpenSSL/1.0.1c PHP/5.4.7 configured -- resuming normal operations
[Thu Jul 07 13:16:56.925786 2016] [mpm_winnt:notice] [pid 5772:tid 392] AH00456: Server built: Aug 18 2012 12:41:37
[Thu Jul 07 13:16:56.925786 2016] [core:notice] [pid 5772:tid 392] AH00094: Command line: 'c:\\xampp\\apache\\bin\\httpd.exe -d C:/xampp/apache'
[Thu Jul 07 13:16:57.082044 2016] [mpm_winnt:notice] [pid 5772:tid 392] AH00418: Parent: Created child process 5560
[Thu Jul 07 13:16:58.566490 2016] [ssl:warn] [pid 5560:tid 396] AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]
[Thu Jul 07 13:16:58.707150 2016] [mpm_winnt:notice] [pid 5560:tid 396] AH00354: Child: Starting 150 worker threads.
I have deployed Orion Context Broker v0.20 on OpenStack - CentOS 6.5 with 4GB RAM | 2 VCPU | 5.0GB Disk.
It works as expected until the multi-threaded HTTP request (for 100 users) test with Apache JMeter 2.13.
The service shuts down with;
Out of memory: Kill process 5640 (contextBroker) score 946 or
sacrifice child Killed process 5640, UID 501, (contextBroker)
total-vm:15377736kB, anon-rss:3822
The problem seems to be the libmicrohttpd.
libmicrohttpd invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
libmicrohttpd cpuset=/ mems_allowed=0
Pid: 2551, comm: libmicrohttpd Not tainted 2.6.32-431.el6.x86_64 #1
Call Trace:
[<ffffffff810d05b1>] ? cpuset_print_task_mems_allowed+0x91/0xb0
[<ffffffff81122960>] ? dump_header+0x90/0x1b0
[<ffffffff8122798c>] ? security_real_capable_noaudit+0x3c/0x70
[<ffffffff81122de2>] ? oom_kill_process+0x82/0x2a0
[<ffffffff81122d21>] ? select_bad_process+0xe1/0x120
[<ffffffff81123220>] ? out_of_memory+0x220/0x3c0
[<ffffffff8112fb3c>] ? __alloc_pages_nodemask+0x8ac/0x8d0
[<ffffffff81167b9a>] ? alloc_pages_vma+0x9a/0x150
[<ffffffff81149c3c>] ? do_wp_page+0x52c/0x920
[<ffffffff8100bc2e>] ? invalidate_interrupt1+0xe/0x20
[<ffffffff8100bc2e>] ? invalidate_interrupt1+0xe/0x20
[<ffffffff8114a82d>] ? handle_pte_fault+0x2cd/0xb00
[<ffffffff8104eeb7>] ? pte_alloc_one+0x37/0x50
[<ffffffff8100bc2e>] ? invalidate_interrupt1+0xe/0x20
[<ffffffff8114b28a>] ? handle_mm_fault+0x22a/0x300
[<ffffffff8104a8d8>] ? __do_page_fault+0x138/0x480
[<ffffffff8105dbef>] ? finish_task_switch+0x4f/0xf0
[<ffffffff81527910>] ? thread_return+0x4e/0x76e
[<ffffffff8152d45e>] ? do_page_fault+0x3e/0xa0
[<ffffffff8152a815>] ? page_fault+0x25/0x30
Mem-Info:
Node 0 DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 30
CPU 1: hi: 186, btch: 31 usd: 30
Node 0 Normal per-cpu:
CPU 0: hi: 186, btch: 31 usd: 48
CPU 1: hi: 186, btch: 31 usd: 51
active_anon:771153 inactive_anon:199950 isolated_anon:352
active_file:17 inactive_file:48 isolated_file:38
unevictable:0 dirty:0 writeback:612 unstable:0
free:21204 slab_reclaimable:2119 slab_unreclaimable:5685
mapped:90 shmem:41 pagetables:3131 bounce:0
Node 0 DMA free:15740kB min:248kB low:308kB high:372kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15352kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 3512 4017 4017
Node 0 DMA32 free:60708kB min:58868kB low:73584kB high:88300kB active_anon:2857740kB inactive_anon:593508kB active_file:72kB inactive_file:48kB unevictable:0kB isolated(anon):640kB isolated(file):0kB present:3596504kB mlocked:0kB dirty:0kB writeback:2444kB mapped:88kB shmem:0kB slab_reclaimable:44kB slab_unreclaimable:400kB kernel_stack:328kB pagetables:868kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1248 all_unreclaimable? no
lowmem_reserve[]: 0 0 505 505
Node 0 Normal free:8368kB min:8464kB low:10580kB high:12696kB active_anon:231444kB inactive_anon:201556kB active_file:0kB inactive_file:144kB unevictable:0kB isolated(anon):896kB isolated(file):24kB present:517120kB mlocked:0kB dirty:0kB writeback:4kB mapped:272kB shmem:164kB slab_reclaimable:8432kB slab_unreclaimable:22340kB kernel_stack:1616kB pagetables:11656kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:312 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 3*4kB 2*8kB 2*16kB 2*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB
Node 0 DMA32: 3*4kB 7*8kB 4*16kB 7*32kB 3*64kB 0*128kB 29*256kB 51*512kB 26*1024kB 0*2048kB 0*4096kB = 60708kB
Node 0 Normal: 346*4kB 188*8kB 101*16kB 47*32kB 15*64kB 5*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8120kB
1824 total pagecache pages
1705 pages in swap cache
Swap cache stats: add 68397, delete 66692, find 1718/1909
Free swap = 0kB
Total swap = 262136kB
1048575 pages RAM
34863 pages reserved
1067 pages shared
988252 pages non-shared
Has anyone faced this issue or have suggestion to workaround it?
The JMeter Test Plan;
<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="2.8" jmeter="2.13 r1665067">
<hashTree>
<HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="HTTP Request" enabled="true">
<elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
<collectionProp name="Arguments.arguments"/>
</elementProp>
<stringProp name="HTTPSampler.domain">ORION_IP_ADDRESS</stringProp>
<stringProp name="HTTPSampler.port">1026</stringProp>
<stringProp name="HTTPSampler.connect_timeout"></stringProp>
<stringProp name="HTTPSampler.response_timeout"></stringProp>
<stringProp name="HTTPSampler.protocol"></stringProp>
<stringProp name="HTTPSampler.contentEncoding"></stringProp>
<stringProp name="HTTPSampler.path">/v1/contextTypes</stringProp>
<stringProp name="HTTPSampler.method">GET</stringProp>
<boolProp name="HTTPSampler.follow_redirects">true</boolProp>
<boolProp name="HTTPSampler.auto_redirects">false</boolProp>
<boolProp name="HTTPSampler.use_keepalive">true</boolProp>
<boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
<boolProp name="HTTPSampler.monitor">false</boolProp>
<stringProp name="HTTPSampler.embedded_url_re"></stringProp>
</HTTPSamplerProxy>
<hashTree/>
</hashTree>
</jmeterTestPlan>
top - 12:24:22 up 1 day, 18:37, 2 users, load average: 1.19, 1.77, 1.59
Tasks: 166 total, 4 running, 162 sleeping, 0 stopped, 0 zombie
Cpu(s): 20.1%us, 5.8%sy, 0.0%ni, 62.4%id, 10.8%wa, 0.0%hi, 1.0%si, 0.0%st
Mem: 987780k total, 979052k used, 8728k free, 17240k buffers
Swap: 2104432k total, 106760k used, 1997672k free, 174100k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4116 mysql 15 0 356m 105m 4176 S 12.0 11.0 139:58.69 mysqld
5722 apache 15 0 160m 17m 4224 S 10.0 1.8 0:00.78 httpd
5741 apache 15 0 161m 17m 4220 S 9.3 1.8 0:00.37 httpd
5840 apache 15 0 161m 17m 4148 S 8.3 1.8 0:00.52 httpd
5846 apache 15 0 161m 17m 4132 S 6.3 1.8 0:00.47 httpd
5744 apache 15 0 162m 18m 4224 S 2.0 1.9 0:00.37 httpd
5725 apache 15 0 161m 17m 4424 S 1.3 1.8 0:00.34 httpd
5755 apache 15 0 105m 14m 4248 R 1.3 1.5 0:00.17 httpd
5564 apache 15 0 163m 19m 4360 S 1.0 2.0 0:00.65 httpd
5322 apache 16 0 162m 19m 4456 S 0.7 2.0 0:02.26 httpd
5586 apache 15 0 161m 18m 4468 S 0.7 1.9 0:01.77 httpd
5852 apache 16 0 99.9m 11m 3424 S 0.7 1.2 0:00.02 httpd
5121 root 18 0 98.3m 10m 4320 S 0.3 1.1 0:00.07 httpd
5723 apache 15 0 161m 17m 4240 S 0.3 1.8 0:00.31 httpd
5833 apeadm 15 0 12740 1128 808 R 0.3 0.1 0:00.03 top
5834 apache 15 0 160m 16m 4172 S 0.3 1.7 0:00.20 httpd
5836 apache 15 0 98.5m 9388 2912 S 0.3 1.0 0:00.01 httpd
1 root 15 0 10348 592 560 S 0.0 0.1 0:00.72 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.01 migration/0
Server Spec:
CENTOS CPU:E5200 RAM:1G
Software: Zencart X 3, Piwik x1
Server always down,How to tuning apache and mysql?
Thanks a lot.
httpd.conf
<IfModule mpm_prefork_module>
StartServers 8
MinSpareServers 8
MaxSpareServers 15
ServerLimit 450
MaxClients 450
MaxRequestsPerChild 10000
</IfModule>
<IfModule mpm_worker_module>
StartServers 2
ServerLimit 450
MaxClients 450
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 10000
</IfModule>
<IfModule mpm_beos_module>
StartThreads 10
MaxClients 50
MaxRequestsPerThread 10000
</IfModule>
<IfModule mpm_netware_module>
ThreadStackSize 65536
StartThreads 250
MinSpareThreads 25
MaxSpareThreads 250
MaxThreads 1000
MaxRequestsPerChild 10000
MaxMemFree 100
</IfModule>
<IfModule mpm_mpmt_os2_module>
StartServers 2
MinSpareThreads 5
MaxSpareThreads 10
MaxRequestsPerChild 10000
</IfModule>
my.cnf
[mysqld]
set-variable = query_cache_limit=1M
set-variable = query_cache_size=16M
set-variable = query_cache_type=1
set-variable = max_connections=400
set-variable = interactive_timeout=100
set-variable = wait_timeout=100
set-variable = connect_timeout=100
set-variable = thread_cache_size=16
#
# Set key_buffer to 5 - 50% of your RAM depending on how much
# you use MyISAM tables, but keep key_buffer_size + InnoDB
# buffer pool size < 80% of your RAM
set-variable = key_buffer=32M
set-variable = join_buffer=1M
set-variable = max_allowed_packet=8M
set-variable = table_cache=1024
set-variable = record_buffer=1M
set-variable = sort_buffer_size=2M
set-variable = read_buffer_size=2M
set-variable = max_connect_errors=10
set-variable = myisam_sort_buffer_size=16M
#Useful for SMP
set-variable = thread_concurrency=8
Server always down,How to tuning apache and mysql?
eh? If its always down how did you get those figures? Tuning isn't going to fix stability issues.
A short answer to how to tune Apache and mysql would fill a large book. Here's some links to books on MysQL: http://forums.mysql.com/read.php?24,92131,92131
And for Apache try searching on Amazon.
Also, you've not provded any information about what's running in between Apache and MySQL - understanding what's going on here is pretty critical too - as is understanding how to improve HTTP and browser performance (caching, keepalives, compression, javascript....).