I'm trying just install mysql/mariadb on fedora 28 but after install it isn't starting server.
When I typed systemctl start mariadb.service got this message :
Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
Analysing journal see these error messages :
*2018-09-03 8:01:20 0 [Note] Recovering after a crash using tc.log
set 03 08:01:20 localhost.localdomain
mysqld[4944]: 2018-09-03 8:01:20 0 [ERROR] Recovery failed! You must enable all engines that were enabled at t>
set 03 08:01:20 localhost.localdomain
mysqld[4944]: 2018-09-03 8:01:20 0 [ERROR] Crash recovery failed. Either correct the problem (if it's, for exa>
set 03 08:01:20 localhost.localdomain
mysqld[4944]: 2018-09-03 8:01:20 0 [ERROR] Can't init tc log
set 03 08:01:20 localhost.localdomain
mysqld[4944]: 2018-09-03 8:01:20 0 [ERROR] Aborting*
Have sombody a light to put over this dark problem ?
It possibly zero byte appear on tc.log . Delete or r and restart the process should works.
sudo mv /var/lib/mysql/tc.log /var/lib/mysql/tc.log.bak
systemctl restart mariadb.service
Ref :
https://fransdejonge.com/2018/02/mariadb-fix-cant-init-tc-log-error/
https://bbs.archlinux.org/viewtopic.php?id=206379
Related
I would like to install MySQL5.7 to my amazon linux in some way.
However I do not know that It is possible to install mysql5.7 in Amazon-linux.
Originally in my EC2 , Maria DB was set as default.
I did these steps.
But it did not start.
$ sudo yum install -y
$ yum list installed | grep mysql
$ sudo yum-config-manager --disable mysql80-community
$ sudo yum-config-manager --enable mysql57-community
$ yum repolist all | grep -E "mysql[0-9]+-community/x86"
$ sudo yum install -y mysql-community-server
Loaded :extras_suggestions, langpacks, priorities, update-motd
amzn2-core | 3.7 kB 00:00:00
385 packages excluded due to repository priority protections
Package mysql-community-server-5.7.36-1.el7.x86_64 is the latest version.
$ sudo systemctl start mysqld
Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
$ systemctl status mysqld.service
● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Fri 2021-12-10 20:52:34 JST; 26s ago
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
Process: 32505 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=1/FAILURE)
Process: 32484 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
12- 10 20:52:34 ip-172-91-96-191.ap-northeast-1.compute.internal systemd[1]: mysqld.service: control process exited, code=exited status=1
12- 10 20:52:34 ip-172-91-96-191.ap-northeast-1.compute.internal systemd[1]: Failed to start MySQL Server.
12- 10 20:52:34 ip-172-91-96-191.ap-northeast-1.compute.internal systemd[1]: Unit mysqld.service entered failed state.
12- 10 20:52:34 ip-172-91-96-191.ap-northeast-1.compute.internal systemd[1]: mysqld.service failed.
12- 10 20:52:34 ip-172-91-96-191.ap-northeast-1.compute.internal systemd[1]: mysqld.service holdoff time over, scheduling restart.
12- 10 20:52:34 ip-172-91-96-191.ap-northeast-1.compute.internal systemd[1]: start request repeated too quickly for mysqld.service
12- 10 20:52:34 ip-172-91-96-191.ap-northeast-1.compute.internal systemd[1]: Failed to start MySQL Server.
12- 10 20:52:34 ip-172-91-96-191.ap-northeast-1.compute.internal systemd[1]: Unit mysqld.service entered failed state.
12- 10 20:52:34 ip-172-91-96-191.ap-northeast-1.compute.internal systemd[1]: mysqld.service failed.
I also did these steps.
I tried look up the way as far I can.
https://aws.amazon.com/amazon-linux-2/
$ /etc/init.d/mysql stop
-bash: /etc/init.d/mysql: No such file or directory
$ service mysql stop
Redirecting to /bin/systemctl stop mysql.service
Failed to stop mysql.service: The name org.freedesktop.PolicyKit1 was not provided by any .service files
See system logs and 'systemctl status mysql.service' for details.
$ killall -KILL mysql mysqld_safe mysqld
mysql: no process found
mysqld_safe: no process found
mysqld: no process found
$ /etc/init.d/mysql start
-bash: /etc/init.d/mysql: No such file or directory
$ sudo systemctl stop mysqld
$ sudo systemctl start mysqld
Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
$ sudo cat /var/log/mysqld.log
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xf525bb]
/usr/sbin/mysqld(handle_fatal_signal+0x486)[0x7e3f06]
/lib64/libpthread.so.0(+0x117e0)[0x7f0dc34517e0]
/lib64/libc.so.6(gsignal+0x110)[0x7f0dc1da0c20]
/lib64/libc.so.6(abort+0x148)[0x7f0dc1da20c8]
/usr/sbin/mysqld[0x7b3eb0]
/usr/sbin/mysqld(_ZN2ib5fatalD1Ev+0xfd)[0x12882cd]
/usr/sbin/mysqld[0x133fce2]
/usr/sbin/mysqld[0x134022b]
/usr/sbin/mysqld(_Z6fil_ioRK9IORequestbRK9page_id_tRK11page_size_tmmPvS8_+0x208)[0x134baf8]
/usr/sbin/mysqld(_Z13buf_read_pageRK9page_id_tRK11page_size_t+0xe4)[0x12f9d74]
/usr/sbin/mysqld(_Z16buf_page_get_genRK9page_id_tRK11page_size_tmP11buf_block_tmPKcmP5mtr_tb+0x42d)[0x12c916d]
/usr/sbin/mysqld(_Z31trx_rseg_get_n_undo_tablespacesPm+0x15d)[0x126742d]
/usr/sbin/mysqld[0x122f258]
/usr/sbin/mysqld(_Z34innobase_start_or_create_for_mysqlv+0x3996)[0x12339a6]
/usr/sbin/mysqld[0x10f8a2f]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x51)[0x836df1]
/usr/sbin/mysqld[0xd35d35]
/usr/sbin/mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x2f0)[0xd3c060]
/usr/sbin/mysqld[0x7dc3e4]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0xa85)[0x7dfc05]
/lib64/libc.so.6(__libc_start_main+0xea)[0x7f0dc1d8e0ba]
/usr/sbin/mysqld[0x7d3864]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2021-12-10T12:20:30.164676Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-12-10T12:20:30.171606Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.36) starting as process 1117 ...
2021-12-10T12:20:30.192632Z 0 [Note] InnoDB: PUNCH HOLE support available
2021-12-10T12:20:30.192667Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-12-10T12:20:30.192671Z 0 [Note] InnoDB: Uses event mutexes
2021-12-10T12:20:30.192675Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2021-12-10T12:20:30.192678Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-12-10T12:20:30.192681Z 0 [Note] InnoDB: Using Linux native AIO
2021-12-10T12:20:30.192972Z 0 [Note] InnoDB: Number of pools: 1
2021-12-10T12:20:30.193097Z 0 [Note] InnoDB: Using CPU crc32 instructions
2021-12-10T12:20:30.203486Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-12-10T12:20:30.218799Z 0 [Note] InnoDB: Completed initialization of buffer pool
2021-12-10T12:20:30.224633Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-12-10T12:20:30.252634Z 0 [ERROR] [FATAL] InnoDB: Table flags are 0 in the data dictionary but the flags in file ./ibdata1 are 0x4000!
2021-12-10 21:20:30 0x7f9a7c57d740 InnoDB: Assertion failure in thread 140301487822656 in file ut0ut.cc line 921
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
12:20:30 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
I am running a AWS Lightsail instance with Ubuntu 18.04 and latest MySQL. Since a few days, the MySQL service stops at random times and then forces me to perform a manual restart after which everything goes back to normal for 24-48 hours until I face another stop.
Terminal echoes this error when running sudo service mysql status:
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2020-04-26 17:54:44 UTC; 10h ago
Process: 30117 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid (code=exited, status=1/FAILURE)
Process: 30095 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Main PID: 14182 (code=killed, signal=KILL)
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: mysql.service: Failed with result 'exit-code'.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: Failed to start MySQL Community Server.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 8.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: Stopped MySQL Community Server.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: mysql.service: Start request repeated too quickly.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: mysql.service: Failed with result 'exit-code'.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: Failed to start MySQL Community Server.
After digging through the /var/log/mysql/error.log file, I found dozens of lines like this:
2020-04-26T10:10:46.710146Z 2 [ERROR] InnoDB: innodb_force_recovery is on. We do not allow database modifications by the user. Shut down mysqld and edit my.cnf to$
Since the error message is truncated at exactly the wrong spot, I did some more digging through other Warning messages and found this one:
2020-04-26T17:54:44.312663Z 0 [ERROR] InnoDB: mmap(137428992 bytes) failed; errno 12
2020-04-26T17:54:44.312671Z 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2020-04-26T17:54:44.312675Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2020-04-26T17:54:44.312680Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2020-04-26T17:54:44.312684Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2020-04-26T17:54:44.312688Z 0 [ERROR] Failed to initialize builtin plugins.
2020-04-26T17:54:44.312691Z 0 [ERROR] Aborting
I am by no means a MySQL expert, but I interpret that mmamp error as the system running out of space, which is a little weird since the instance uses around 10% space and has at most 2 users at the same time (hence little to no CPU usage). I run a single site Wordpress installation (just to divert users) and phpmyadmin plus some minor testing scripts.
Appreciate any help with this, thank you.
The most likely cause is that your configuration is broken in a way that makes mysqld try to allocate more memory that your server has.
my os is Redhat8.
currtenly I install Mysql, version is "mysql Ver 8.0.13 for Linux on x86_64 (Source distribution)"
I use command "sudo yum install mysql, mysql-server"
But i get error when i want to start mysql services. I keep finding soluction, but still I don't know how to solved it.
Can anyone help me to look at it?
#--------------------------------------------------------------------------------------------------------
[yliu#rhlab1 ~]$ sudo systemctl start mysqld
Job for mysqld.service failed because the control process exited with error code.
See "systemctl status mysqld.service" and "journalctl -xe" for details.
# --------------------------------------------------------------------------------------------
[yliu#rhlab1 ~]$ sudo systemctl status mysqld
● mysqld.service - MySQL 8.0 database server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2020-02-19 12:12:15 AST; 15s ago
Process: 13080 ExecStopPost=/usr/libexec/mysql-wait-stop (code=exited, status=0/SUCCESS)
Process: 13052 ExecStart=/usr/libexec/mysqld --basedir=/usr (code=exited, status=1/FAILURE)
Process: 13015 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mysqld.service (code=exited, status=0/SUCCESS)
Process: 12991 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
Main PID: 13052 (code=exited, status=1/FAILURE)
Status: "SERVER_BOOTING"
Error: 2 (No such file or directory)
Feb 19 12:12:12 systemd[1]: Starting MySQL 8.0 database server...
Feb 19 12:12:15 systemd[1]: mysqld.service: Main process exited, code=exited, status=1/FAILURE
Feb 19 12:12:15 systemd[1]: mysqld.service: Failed with result 'exit-code'.
Feb 19 12:12:15 systemd[1]: Failed to start MySQL 8.0 database server.
# ----------------------------------------------------------------------------------------
[yliu#rhlab1 ~]$ sudo cat /var/log/mysql/mysqld.log | nl | tail -n 10
38 2020-02-19T15:48:25.492390Z 0 [System] [MY-010910] [Server] /usr/libexec/mysqld: Shutdown complete (mysqld 8.0.13) Source distribution.
39 2020-02-19T15:48:25.492468Z 0 [ERROR] [MY-010065] [Server] Failed to shutdown components infrastructure.
40 2020-02-19T16:12:13.332724Z 0 [System] [MY-010116] [Server] /usr/libexec/mysqld (mysqld 8.0.13) starting as process 13052
41 2020-02-19T16:12:13.585641Z 1 [ERROR] [MY-010781] [Server] Found ./mysql/index_stats.frm file in mysql schema. DD will create .ibd file with same name. Please rename table and start upgrade process again.
42 2020-02-19T16:12:13.585666Z 1 [ERROR] [MY-010336] [Server] Found .frm file with same name as one of the Dictionary Tables.
43 2020-02-19T16:12:13.585833Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
44 2020-02-19T16:12:13.585871Z 0 [ERROR] [MY-013236] [Server] Newly created data directory /var/lib/mysql/ is unusable. You can safely remove it.
45 2020-02-19T16:12:13.585881Z 0 [ERROR] [MY-010119] [Server] Aborting
46 2020-02-19T16:12:15.000469Z 0 [System] [MY-010910] [Server] /usr/libexec/mysqld: Shutdown complete (mysqld 8.0.13) Source distribution.
47 2020-02-19T16:12:15.000550Z 0 [ERROR] [MY-010065] [Server] Failed to shutdown components infrastructure.
My /etc/my.cnf.d/mysql-server.cnf information is:
[yliu#rhlab1 ~]$ sudo cat /etc/my.cnf.d/mysql-server.cnf
#
# This group are read by MySQL server.
# Use it for options that only the server (but not clients) should see
#
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/en/server-configuration-defaults.html
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mysqld according to the
# instructions in http://fedoraproject.org/wiki/Systemd
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mysql/mysqld.log
pid-file=/run/mysqld/mysqld.pid
I solved the problem, hope it will helpful for everyone who meet the same problem.
Base on the log:
41 2020-02-19T16:12:13.585641Z 1 [ERROR] [MY-010781] [Server] Found ./mysql/index_stats.frm file in mysql schema. DD will create .ibd file with same name. Please rename table and start upgrade process again.
42 2020-02-19T16:12:13.585666Z 1 [ERROR] [MY-010336] [Server] Found .frm file with same name as one of the Dictionary Tables.
43 2020-02-19T16:12:13.585833Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
44 2020-02-19T16:12:13.585871Z 0 [ERROR] [MY-013236] [Server] Newly created data directory /var/lib/mysql/ is unusable. You can safely remove it.
so we can remove all the files under /var/lib/mysql/
and then make sure the mysql file owner and group are mysql.
(Django) [yliu#rhlab1 ops]$ sudo ls -al /var/lib/ | grep mysql
drwxr-xr-x. 7 mysql mysql 4096 Feb 24 10:16 mysql
and then you can start mysqld again
sudo systemctl start mysqld
I have an EC2 instance in which Mysql server is working fine. I created an AMI from this and launched a new instance from this AMI. Now Mysql server is not able to start on this new AMI at all.
Below is the o/p:
ubuntu#ip-172-31-66-160:~$ sudo service mysql status
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: activating (start-post) (Result: exit-code) since Sun 2019-07-21 20:59:25 IST; 28s ago
Process: 1870 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
Process: 1862 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Main PID: 1870 (code=exited, status=1/FAILURE); : 1871 (mysql-systemd-s)
Tasks: 2
Memory: 3.2M
CPU: 247ms
CGroup: /system.slice/mysql.service
└─control
├─1871 /bin/bash /usr/share/mysql/mysql-systemd-start post
└─2529 sleep 1
Jul 21 20:59:25 ip-172-31-66-160 systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Jul 21 20:59:25 ip-172-31-66-160 systemd[1]: Stopped MySQL Community Server.
Jul 21 20:59:25 ip-172-31-66-160 systemd[1]: Starting MySQL Community Server...
Jul 21 20:59:26 ip-172-31-66-160 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
ubuntu#ip-172-31-66-160:~$ sudo service mysql restart
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
I tried everything but not able to get this started.
Closest threads which I came across and followed without any luck:
https://askubuntu.com/questions/916009/mysql-wont-start-because-of-apparmor
https://support.plesk.com/hc/en-us/articles/360004185293-Unable-to-start-MySQL-on-Ubuntu-AVC-apparmor-DENIED-operation-open-
Kindly help, I am just clueless about this.
Finally, I was able to make the server run on my new instance. What I was missing is that mysql also has many config options which gets changed over due course of time and they also depend on the EC2 instance config.
In my case, my new instance has very limited RAM and the mysql.cnf file was trying to allocate more RAM than the machine had hence it was failing. Somehow,
journalctl -xe
is giving a different error which is forcing me to think in wrong direction. I should have looked in to the mysql log file which quickly pointed me to the right issue.
I found below in the error log file which helped me to isolate / fix the issue:
2019-07-21T16:00:55.658985Z 0 [ERROR] InnoDB: mmap(137428992 bytes) failed; errno 12 2019-07-21T16:00:55.658993Z 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2019-07-21T16:00:55.658998Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2019-07-21T16:00:55.659003Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2019-07-21T16:00:55.659620Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2019-07-21T16:00:55.659628Z 0 [ERROR] Failed to initialize builtin plugins.
2019-07-21T16:00:55.659632Z 0 [ERROR] Aborting
So mysql wasn't able to start since requested amount of memory isn't present in the system. I commented all the memory allocation lines in my.cnf file (relying on the default settings) and then it worked.
innodb_buffer_pool_size = 16G
key_buffer_size = 2G
max_allowed_packet = 128M
group_concat_max_len = 50000
query_cache_size = 2147483648
query_cache_limit =67108864
Hope someone would find this information useful.
I rebooted a digital ocean box and now I can't start mysql. When I run start command i get :
Redirecting to /bin/systemctl restart mysql.service
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
Result of systemctl status mysql.service
● mysql.service - Percona XtraDB Cluster
Loaded: loaded (/usr/lib/systemd/system/mysql.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2017-12-11 17:08:18 UTC; 5s ago
Process: 26300 ExecStopPost=/usr/bin/mysql-systemd stop-post (code=exited, status=0/SUCCESS)
Process: 26270 ExecStop=/usr/bin/mysql-systemd stop (code=exited, status=2)
Process: 25674 ExecStartPost=/usr/bin/mysql-systemd start-post $MAINPID (code=exited, status=1/FAILURE)
Process: 25673 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
Process: 25632 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS)
Main PID: 25673 (code=exited, status=0/SUCCESS)
Dec 11 17:08:18 server-name-hidden mysql-systemd[25674]: ERROR! mysqld_safe with PID 25673 has already exited: FAILURE
Dec 11 17:08:18 server-name-hidden systemd[1]: mysql.service: control process exited, code=exited status=1
Dec 11 17:08:18 server-name-hidden mysql-systemd[26270]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Dec 11 17:08:18 server-name-hidden mysql-systemd[26270]: ERROR! mysql already dead
Dec 11 17:08:18 server-name-hidden systemd[1]: mysql.service: control process exited, code=exited status=2
Dec 11 17:08:18 server-name-hidden mysql-systemd[26300]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Dec 11 17:08:18 server-name-hidden mysql-systemd[26300]: WARNING: mysql may be already dead
Dec 11 17:08:18 server-name-hidden systemd[1]: Failed to start Percona XtraDB Cluster.
Dec 11 17:08:18 server-name-hidden systemd[1]: Unit mysql.service entered failed state.
Dec 11 17:08:18 server-name-hidden systemd[1]: mysql.service failed.
/var/run/mysqld/ is owned by mysql user - but it's empty. If i add a mysqld.pid file it gets removed when i run mysql start.
Does anyone know why a reboot would cause this or give me any next steps. I have reviewed the mysqld.log file and can't see anything of use. here are the last 30 lines
017-12-11T15:14:42.473164Z 0 [Note] WSREP: Received shutdown signal. Will sleep for 10 secs before initiating shutdown. pxc_maint_mode switched to SHUTDOWN
2017-12-11T15:14:52.474584Z 0 [Note] WSREP: Stop replication
2017-12-11T15:14:52.474691Z 0 [Note] WSREP: Closing send monitor...
2017-12-11T15:14:52.475160Z 0 [Note] WSREP: Closed send monitor.
2017-12-11T15:14:52.475268Z 0 [Note] WSREP: gcomm: terminating thread
2017-12-11T15:14:52.475305Z 0 [Note] WSREP: gcomm: joining thread
2017-12-11T15:14:52.475886Z 0 [Note] WSREP: gcomm: closing backend
2017-12-11T15:14:52.476079Z 0 [Note] WSREP: Current view of cluster as seen by this node
2017-12-11T15:14:52.476491Z 0 [Note] WSREP: gcomm: closed
2017-12-11T15:14:52.476532Z 0 [Note] WSREP: Received self-leave message.
2017-12-11T15:14:52.476559Z 0 [Note] WSREP: Flow-control interval: [0, 0]
2017-12-11T15:14:52.476595Z 0 [Note] WSREP: Trying to continue unpaused monitor
2017-12-11T15:14:52.476602Z 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2017-12-11T15:14:52.476608Z 0 [Note] WSREP: Shifting SYNCED -> CLOSED (TO: 13842228)
2017-12-11T15:14:52.476632Z 0 [Note] WSREP: RECV thread exiting 0: Success
2017-12-11T15:14:52.477098Z 0 [Note] WSREP: recv_thread() joined.
2017-12-11T15:14:52.477110Z 0 [Note] WSREP: Closing replication queue.
2017-12-11T15:14:52.477116Z 0 [Note] WSREP: Closing slave action queue.
2017-12-11T15:14:52.477123Z 0 [Note] Giving 63 client threads a chance to die gracefully
2017-12-11T15:14:52.478905Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 21 user: ‘hidden’
2017-12-11T15:14:52.478957Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 22 user: 'hidden'
2017-12-11T15:14:52.478994Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 6387 user: 'hidden'
2017-12-11T15:14:52.479034Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 6367 user: 'hidden'
2017-12-11T15:14:52.479084Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 6373 user: 'hidden'
2017-12-11T15:14:52.479130Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 6368 user: 'hidden'
2017-12-11T15:14:54.479289Z 0 [Note] WSREP: Waiting for active wsrep applier to exit
I've also been told to try : systemctl start mysql#bootstrap however this fails with the same error. here is the result from journalctl -xe
Dec 11 17:24:58 dropletname systemd[1]: mysql#bootstrap.service: control process exited, code=exited status=1
Dec 11 17:24:58 dropletname mysql-systemd[28663]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Dec 11 17:24:58 dropletname mysql-systemd[28663]: ERROR! mysql already dead
Dec 11 17:24:58 dropletname systemd[1]: mysql#bootstrap.service: control process exited, code=exited status=2
Dec 11 17:24:58 dropletname mysql-systemd[28694]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Dec 11 17:24:58 dropletname mysql-systemd[28694]: WARNING: mysql may be already dead
Dec 11 17:24:58 dropletname systemd[1]: Failed to start Percona XtraDB Cluster with config /etc/sysconfig/mysql.bootstrap.
Dec 11 17:24:58 dropletname systemd[1]: Unit mysql#bootstrap.service entered failed state.
Dec 11 17:24:58 dropletname systemd[1]: mysql#bootstrap.service failed.
Dec 11 17:24:58 dropletname polkitd[510]: Unregistered Authentication Agent for unix-process:28009:649814 (system bus name :1.159, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_GB.UTF-8) (disconnected from bus)
Dec 11 17:25:02 dropletname sendmail[27970]: unable to qualify my own domain name (dropletname) -- using short name
Dec 11 17:25:02 dropletname sendmail[27970]: vBBHP2cx027970: from=hidden, size=1655, class=-60, nrcpts=1, msgid=<201712111725.vBBHP2cx027970#dropletname>, relay=hidden#localhost
Dec 11 17:25:02 dropletname sendmail[28728]: vBBHP26v028728: from=<hidden#dropletname>, size=1935, class=-60, nrcpts=1, msgid=<201712111725.vBBHP2cx027970#dropletname>, proto=ESMTP, daemon=MTA, relay=dropletname [127.0.0.1]
Dec 11 17:25:02 dropletname sendmail[27970]: vBBHP2cx027970: to=hidden, ctladdr=hidden (1000/1000), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=139655, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (vBBHP26v028728 Message accepted for del
Dec 11 17:25:02 dropletname sendmail[28729]: vBBHP26v028728: to=<hidden#dropletname>, ctladdr=<hidden#dropletname> (1000/1000), delay=00:00:00, xdelay=00:00:00, mailer=local, pri=140148, dsn=2.0.0, stat=Sent
Dec 11 17:25:02 dropletname systemd[1]: Removed slice User Slice of hidden
ps aux|grep mysql
root 11590 0.0 0.0 107924 608 pts/0 T 15:43 0:00 cat /var/log/mysqld.log
root 24666 0.0 0.0 107924 612 pts/0 T 16:52 0:00 cat /var/log/mysqld.log
root 32182 0.0 0.0 112664 972 pts/2 S+ 17:59 0:00 grep --color=auto mysq
I managed to resolve this in the end however i'm not sure what 100% did it. I changed /etc/my.conf to look like this :
!includedir /etc/my.cnf.d/
!includedir /etc/percona-xtradb-cluster.conf.d/
[client]
socket=/var/lib/mysql/mysql.sock
[mysqld]
server-id=1
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
log-bin
log_slave_updates
expire_logs_days=7
innodb_strict_mode=OFF
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
I created a mysqld.pid and mysqld.lock file in /var/run/mysqld/ ensuring the owner and group was mysql:mysql
touch /var/run/mysqld/mysqld.pid;
touch /var/run/mysqld/mysqld.lock;
chown -R mysql:mysql /var/run/mysqld/;
I then ran the following two commands which seems to fail but may have helped
service mysql start --wsrep-cluster-address="gcomm://";
systemctl start mysql --wsrep-new-cluster;
Finally I ran
systemctl start mysql#bootstrap.service
service mysql start
This had previously failed every time I tried. Subsequently I then found that the /var/lib/mysql/mysql.sock had been created which previously did not exist.