Having trouble diagnosing my issue (mysql fails to start). Any help would be greatly appreciated.
Also, this is on Ubuntu 16.04
Step 1 - service mysql start
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
Step 2 - systemctl status mysql.service
mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: activating (start-post) (Result: exit-code) since Mon 2016-10-31 22:40:22 UTC; 11s ago
Process: 9780 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
Process: 9777 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Main PID: 9780 (code=exited, status=1/FAILURE); : 9781 (mysql-systemd-s)
CGroup: /system.slice/mysql.service
└─control
├─9781 /bin/bash /usr/share/mysql/mysql-systemd-start post
└─9806 sleep 1
Step 3 - journalctl -xe
Oct 31 22:42:32 sshd[10033]: Received disconnect from 121.18.238.109 port 34577:11: [preauth]
Oct 31 22:42:32 sshd[10033]: Disconnected from 121.18.238.109 port 34577 [preauth]
Oct 31 22:42:32 sshd[10033]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.109 user=root
Oct 31 22:42:35 sshd[10090]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.23 user=root
Oct 31 22:42:37 sshd[10090]: Failed password for root from 116.31.116.23 port 52791 ssh2
Oct 31 22:42:39 sshd[10090]: Failed password for root from 116.31.116.23 port 52791 ssh2
Oct 31 22:42:41 sshd[10090]: Failed password for root from 116.31.116.23 port 52791 ssh2
Oct 31 22:42:41 sshd[10090]: Received disconnect from 116.31.116.23 port 52791:11: [preauth]
Oct 31 22:42:41 sshd[10090]: Disconnected from 116.31.116.23 port 52791 [preauth]
Oct 31 22:42:41 sshd[10090]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.23 user=root
Oct 31 22:42:43 sshd[10084]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.109 user=root
Oct 31 22:42:45 sshd[10084]: Failed password for root from 121.18.238.109 port 40812 ssh2
Oct 31 22:42:48 sshd[10084]: Failed password for root from 121.18.238.109 port 40812 ssh2
Oct 31 22:42:50 sshd[10084]: Failed password for root from 121.18.238.109 port 40812 ssh2
Oct 31 22:42:50 sshd[10084]: Received disconnect from 121.18.238.109 port 40812:11: [preauth]
Oct 31 22:42:50 sshd[10084]: Disconnected from 121.18.238.109 port 40812 [preauth]
Oct 31 22:42:50 sshd[10084]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.109 user=root
Oct 31 22:42:56 systemd[1]: Failed to start MySQL Community Server.
-- Subject: Unit mysql.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has failed.
--
-- The result is failed.
Oct 31 22:42:56 systemd[1]: mysql.service: Unit entered failed state.
Oct 31 22:42:56 systemd[1]: mysql.service: Failed with result 'exit-code'.
Oct 31 22:42:56 systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Oct 31 22:42:56 systemd[1]: Stopped MySQL Community Server.
-- Subject: Unit mysql.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has finished shutting down.
Oct 31 22:42:56 systemd[1]: Starting MySQL Community Server...
-- Subject: Unit mysql.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has begun starting up.
Oct 31 22:42:56 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Oct 31 22:43:04 sshd[10141]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.109 user=root
Oct 31 22:43:06 sshd[10141]: Failed password for root from 121.18.238.109 port 56169 ssh2
Oct 31 22:43:11 sshd[10141]: Failed password for root from 121.18.238.109 port 56169 ssh2
Oct 31 22:43:13 sshd[10141]: Failed password for root from 121.18.238.109 port 56169 ssh2
Oct 31 22:43:14 sshd[10178]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.23 user=root
Oct 31 22:43:17 sshd[10178]: Failed password for root from 116.31.116.23 port 60094 ssh2
Oct 31 22:43:18 sshd[10178]: Failed password for root from 116.31.116.23 port 60094 ssh2
Oct 31 22:43:20 sshd[10178]: Failed password for root from 116.31.116.23 port 60094 ssh2
Oct 31 22:43:20 sshd[10178]: Received disconnect from 116.31.116.23 port 60094:11: [preauth]
Oct 31 22:43:20 sshd[10178]: Disconnected from 116.31.116.23 port 60094 [preauth]
Oct 31 22:43:20 sshd[10178]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.23 user=root
Oct 31 22:43:21 sshd[10141]: Received disconnect from 121.18.238.109 port 56169:11: [preauth]
Oct 31 22:43:21 sshd[10141]: Disconnected from 121.18.238.109 port 56169 [preauth]
Oct 31 22:43:21 sshd[10141]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.109 user=root
More info that could help. I tired mysql but received ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
Also, here is the Error log
2016-10-31T19:06:58.590888Z 0 [Note] Giving 0 client threads a chance to die gracefully
2016-10-31T19:06:58.591364Z 0 [Note] Shutting down slave threads
2016-10-31T19:06:58.592730Z 0 [Note] Forcefully disconnecting 0 remaining clients
2016-10-31T19:06:58.592792Z 0 [Note] Event Scheduler: Purging the queue. 0 events
2016-10-31T19:06:58.598944Z 0 [Note] Binlog end
2016-10-31T19:06:58.701931Z 0 [Note] Shutting down plugin 'validate_password'
2016-10-31T19:06:58.705709Z 0 [Note] Shutting down plugin 'ngram'
2016-10-31T19:06:58.705751Z 0 [Note] Shutting down plugin 'ARCHIVE'
2016-10-31T19:06:58.705790Z 0 [Note] Shutting down plugin 'partition'
2016-10-31T19:06:58.705812Z 0 [Note] Shutting down plugin 'BLACKHOLE'
2016-10-31T19:06:58.705844Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
2016-10-31T19:06:58.706964Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2016-10-31T19:06:58.707001Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2016-10-31T19:06:58.707020Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2016-10-31T19:06:58.707037Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2016-10-31T19:06:58.707054Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2016-10-31T19:06:58.707208Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2016-10-31T19:06:58.707235Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2016-10-31T19:06:58.707255Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2016-10-31T19:06:58.707271Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2016-10-31T19:06:58.707286Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2016-10-31T19:06:58.707306Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2016-10-31T19:06:58.707322Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2016-10-31T19:06:58.707340Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2016-10-31T19:06:58.707358Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2016-10-31T19:06:58.707411Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2016-10-31T19:06:58.707432Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
2016-10-31T19:06:58.707449Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2016-10-31T19:06:58.707467Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2016-10-31T19:06:58.707484Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2016-10-31T19:06:58.707499Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2016-10-31T19:06:58.707516Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2016-10-31T19:06:58.707534Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2016-10-31T19:06:58.707566Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2016-10-31T19:06:58.707587Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
2016-10-31T19:06:58.707603Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2016-10-31T19:06:58.707619Z 0 [Note] Shutting down plugin 'INNODB_CMP'
2016-10-31T19:06:58.707636Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2016-10-31T19:06:58.707654Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
2016-10-31T19:06:58.707672Z 0 [Note] Shutting down plugin 'INNODB_TRX'
2016-10-31T19:06:58.707698Z 0 [Note] Shutting down plugin 'InnoDB'
2016-10-31T19:06:58.711649Z 0 [Note] InnoDB: FTS optimize thread exiting.
2016-10-31T19:06:58.714623Z 0 [Note] InnoDB: Starting shutdown...
2016-10-31T19:06:58.819989Z 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2016-10-31T19:06:58.820088Z 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2016-10-31T19:06:58.820228Z 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2016-10-31T19:06:58.820416Z 0 [ERROR] InnoDB: Cannot open '/var/lib/mysql/ib_buffer_pool.incomplete' for writing: Permission denied
2016-10-31T19:07:00.692529Z 0 [Note] InnoDB: Shutdown completed; log sequence number 1333673446
2016-10-31T19:07:00.693394Z 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2016-10-31T19:07:00.693505Z 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2016-10-31T19:07:00.693546Z 0 [Note] Shutting down plugin 'MEMORY'
2016-10-31T19:07:00.693586Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2016-10-31T19:07:00.693933Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
2016-10-31T19:07:00.695547Z 0 [Note] Shutting down plugin 'MyISAM'
2016-10-31T19:07:00.696705Z 0 [Note] Shutting down plugin 'CSV'
2016-10-31T19:07:00.696747Z 0 [Note] Shutting down plugin 'sha256_password'
2016-10-31T19:07:00.696758Z 0 [Note] Shutting down plugin 'mysql_native_password'
2016-10-31T19:07:00.696766Z 0 [Note] Shutting down plugin 'keyring_file'
2016-10-31T19:07:00.728590Z 0 [Note] Shutting down plugin 'binlog'
2016-10-31T19:07:00.737815Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
try using sudo service mysql start
to start the mysqld service. the issue seems to be access rights - The error means mysqld does not have the access rights to the directory.
I had a problem with that, in my case the problem was with log file, because this file was deleted. I solved it by recreating this file '/var/log/mysql/error.log', after that my server starting normaly.
Your data directory of mysql is gone for toss in where it holds all the DBs including mysql sys and authentication files.
You need to reinitialize mysql database by making fresh empty mysql base & data directory. You can follow steps from mysqld data directory initialization
You can use following mysqld to reset password easily
$ mysqld --initialize-insecure
to reset password from your Linux Variant root login. Else its little tougher to figure out generated password and that you need to have with init-file for settings I think. Because for me the process doesn't show up any random password.
Both the following option to be used I think. I cannot go over again. But you can try that it is generating and showing random password.
$ mysqld --initialize-insecure --initialize
Article says,
With --initialize-insecure, (either with or without --initialize
because --initialize-insecure implies --initialize), the server does
not generate a password or mark it expired, and writes a warning
message
To reset password,
# mysql -u root --skip-password
....
mysql> ALTER USER 'root'#'localhost' IDENTIFIED BY 'new_password';
Related
Due to a backend application I need MariaDB 5.5.68. I have installed CentOS Web Panel and after with yum remove mariadb, yum remove MariaDB-common and rm -rf /var/lib/mysql. rm /etc/my.cnf I have installed MariaDB 5.5.68.
The problem now is that when I write mysql I get this error:
"ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)".
[root#server720546 ~]# systemctl start mariadb
Failed to start mariadb.service: Unit not found.
UPDATE
[root#server720546 mariadb]# systemctl status mariadb
● mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: failed (Result: exit-code) since Sat 2021-08-07 14:22:22 UTC; 5min ago
Process: 1672 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
Process: 1671 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
Process: 1637 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
Main PID: 1671 (code=exited, status=0/SUCCESS)
Aug 07 14:22:21 server720546.cloudsons.net systemd[1]: Starting MariaDB database server...
Aug 07 14:22:21 server720546.cloudsons.net mariadb-prepare-db-dir[1637]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
Aug 07 14:22:21 server720546.cloudsons.net mysqld_safe[1671]: 210807 14:22:21 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Aug 07 14:22:21 server720546.cloudsons.net mysqld_safe[1671]: 210807 14:22:21 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Aug 07 14:22:22 server720546.cloudsons.net systemd[1]: mariadb.service: control process exited, code=exited status=1
Aug 07 14:22:22 server720546.cloudsons.net systemd[1]: Failed to start MariaDB database server.
Aug 07 14:22:22 server720546.cloudsons.net systemd[1]: Unit mariadb.service entered failed state.
Aug 07 14:22:22 server720546.cloudsons.net systemd[1]: mariadb.service failed.
From mariadb.log:
210807 14:22:21 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
210807 14:22:21 [Note] /usr/libexec/mysqld (mysqld 5.5.68-MariaDB) starting as process 1836 ...
210807 14:22:21 InnoDB: The InnoDB memory heap is disabled
210807 14:22:21 InnoDB: Mutexes and rw_locks use GCC atomic builtins
210807 14:22:21 InnoDB: Compressed tables use zlib 1.2.7
210807 14:22:21 InnoDB: Using Linux native AIO
210807 14:22:21 InnoDB: Initializing buffer pool, size = 128.0M
210807 14:22:21 InnoDB: Completed initialization of buffer pool
InnoDB: Error: checksum mismatch in data file ./ibdata1
210807 14:22:21 InnoDB: Could not open or create data files.
210807 14:22:21 InnoDB: If you tried to add new data files, and it failed here,
210807 14:22:21 InnoDB: you should now edit innodb_data_file_path in my.cnf back
210807 14:22:21 InnoDB: to what it was, and remove the new ibdata files InnoDB created
210807 14:22:21 InnoDB: in this failed attempt. InnoDB only wrote those files full of
210807 14:22:21 InnoDB: zeros, but did not yet use them in any way. But be careful: do not
210807 14:22:21 InnoDB: remove old data files which contain your precious data!
210807 14:22:21 [ERROR] Plugin 'InnoDB' init function returned error.
210807 14:22:21 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
210807 14:22:21 [Note] Plugin 'FEEDBACK' is disabled.
210807 14:22:21 [ERROR] Unknown/unsupported storage engine: InnoDB
210807 14:22:21 [ERROR] Aborting
210807 14:22:21 [Note] /usr/libexec/mysqld: Shutdown complete
210807 14:22:21 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
UPDATE 2
With journalctl -xe I have this:
[root#server720546 ~]# journalctl -xe
--
-- The start-up result is done.
Aug 07 14:52:05 server720546.cloudsons.net systemd[1]: Started Session 6 of user
-- Subject: Unit session-6.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-6.scope has finished starting up.
--
-- The start-up result is done.
Aug 07 14:52:05 server720546.cloudsons.net systemd-logind[482]: New session 6 of
-- Subject: A new session 6 has been created for user root
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
--
-- A new session with the ID 6 has been created for the user root.
--
-- The leading process of the session is 2394.
Aug 07 14:52:05 server720546.cloudsons.net sshd[2394]: pam_unix(sshd:session): s
Aug 07 14:52:06 server720546.cloudsons.net sshd[2396]: Failed password for root
Aug 07 14:52:06 server720546.cloudsons.net sshd[2396]: Received disconnect from
Aug 07 14:52:06 server720546.cloudsons.net sshd[2396]: Disconnected from 198.23.
lines 1206-1228/1228 (END)
--
-- The start-up result is done.
Aug 07 14:52:05 server720546.cloudsons.net systemd[1]: Started Session 6 of user root.
-- Subject: Unit session-6.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-6.scope has finished starting up.
--
-- The start-up result is done.
Aug 07 14:52:05 server720546.cloudsons.net systemd-logind[482]: New session 6 of user root.
-- Subject: A new session 6 has been created for user root
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
--
-- A new session with the ID 6 has been created for the user root.
--
-- The leading process of the session is 2394.
Aug 07 14:52:05 server720546.cloudsons.net sshd[2394]: pam_unix(sshd:session): session opened for user root by (uid=0)
Aug 07 14:52:06 server720546.cloudsons.net sshd[2396]: Failed password for root from 198.23.233.28 port 57126 ssh2
Aug 07 14:52:06 server720546.cloudsons.net sshd[2396]: Received disconnect from 198.23.233.28 port 57126:11: Bye Bye [preauth]
Aug 07 14:52:06 server720546.cloudsons.net sshd[2396]: Disconnected from 198.23.233.28 port 57126 [preauth]
I'm trying to setup a galera cluster, with MariaDB 10.2, and percona-xtrabackup-2.3.10-1.el7.x86_64.
If bootstrap the donor by wsrep_sst_method=xtrabackup or xtrabackup-v2, the joiner will be unable to join in the cluster, and the error message complains about "no valid checkpoint".
However, if first bring up the cluster using wsrep_sst_method=rsync then change it to xtrabackup, then stop all nodes and bootstrap the donor again (by galera_new_cluster), the joiner is able to join in OK.
I suspect there was some data synchronized to the second node when using rsync.
Could you please give me some pointers about why xtrabackup doesn't work the first time?
Is it a common practice to bootstrap first time with rsync?
Or, anything else.
Any hints will be highly appreciated, and just let me know if you need more information.
Thank you for your help.
More details:
When bootstrap the first time using xtrabackup, the joiner is unable to join in, and its log says:
Jan 23 04:09:14 setsv-dr.local.example.com mysqld[6924]: WSREP_SST: [ERROR] xtrabackup_checkpoints missing, failed innobackupex/SST on donor (20210123 04:09:14.948)
Jan 23 04:09:14 setsv-dr.local.example.com mysqld[6924]: WSREP_SST: [ERROR] Cleanup after exit with status:2 (20210123 04:09:14.971)
, and the donor logs say:
Jan 23 04:11:31 setsv mysqld: group UUID = a53cd166-5d68-11eb-aa0f-83c1fd168f1f
Jan 23 04:11:31 setsv mysqld: 2021-01-23 4:11:31 140325406353152 [Note] WSREP: Flow-control interval: [16, 16]
Jan 23 04:11:31 setsv mysqld: 2021-01-23 4:11:31 140325646681856 [Note] WSREP: REPL Protocols: 9 (4, 2)
Jan 23 04:11:31 setsv mysqld: 2021-01-23 4:11:31 140325646681856 [Note] WSREP: New cluster view: global state: a53cd166-5d68-11eb-aa0f-83c1fd168f1f:0, view# 35: Primary, number of nodes: 1, my index: 0, protocol version 3
Jan 23 04:11:31 setsv mysqld: 2021-01-23 4:11:31 140325646681856 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Jan 23 04:11:31 setsv mysqld: 2021-01-23 4:11:31 140325646681856 [Note] WSREP: Assign initial position for certification: 0, protocol version: 4
Jan 23 04:11:31 setsv mysqld: 2021-01-23 4:11:31 140325655074560 [Note] WSREP: Service thread queue flushed.
Jan 23 04:11:32 setsv mysqld: WSREP_SST: [INFO] Streaming the backup to joiner at 192.168.56.71 4444 (20210123 04:11:32.454)
Jan 23 04:11:32 setsv mysqld: WSREP_SST: [INFO] Evaluating innobackupex --no-version-check $tmpopts $INNOEXTRA --galera-info --stream=$sfmt $itmpdir 2>${DATA}/innobackup.backup.log | socat -u stdio TCP:192.168.56.71:4444; RC=( ${PIPESTATUS[#]} ) (20210123 04:11:32.457)
Jan 23 04:11:32 setsv mysqld: 2021/01/23 04:11:32 socat[15384] E connect(6, AF=2 192.168.56.71:4444, 16): Connection refused
Jan 23 04:11:32 setsv mysqld: 2021-01-23 4:11:32 140325134952192 [Warning] Aborted connection 26 to db: 'unconnected' user: 'sst_user' host: 'localhost' (Got an error reading communication packets)
Jan 23 04:11:32 setsv mysqld: WSREP_SST: [ERROR] innobackupex finished with error: 1. Check /var/lib/mysql//innobackup.backup.log (20210123 04:11:32.467)
Jan 23 04:11:32 setsv mysqld: WSREP_SST: [ERROR] Cleanup after exit with status:22 (20210123 04:11:32.469)
Jan 23 04:11:32 setsv mysqld: WSREP_SST: [INFO] Cleaning up temporary directories (20210123 04:11:32.471)
Jan 23 04:11:32 setsv mysqld: 2021-01-23 4:11:32 140324111660800 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.56.71:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --gtid 'a53cd166-5d68-11eb-aa0f-83c1fd168f1f:0' --gtid-domain-id '0' --mysqld-args --basedir=/usr --wsrep-new-cluster --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1
Jan 23 04:11:32 setsv mysqld: 2021-01-23 4:11:32 140324111660800 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.56.71:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --gtid 'a53cd166-5d68-11eb-aa0f-83c1fd168f1f:0' --gtid-domain-id '0' --mysqld-args --basedir=/usr --wsrep-new-cluster --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1: 22 (Invalid argument)
Jan 23 04:11:32 setsv mysqld: 2021-01-23 4:11:32 140324111660800 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.56.71:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --gtid 'a53cd166-5d68-11eb-aa0f-83c1fd168f1f:0' --gtid-domain-id '0' --mysqld-args --basedir=/usr --wsrep-new-cluster --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1
Jan 23 04:11:32 setsv mysqld: 2021-01-23 4:11:32 140325406353152 [Warning] WSREP: Could not find peer: b8dba97a-5d6b-11eb-8d6d-ae8d4ded2a34
Jan 23 04:11:32 setsv mysqld: 2021-01-23 4:11:32 140325406353152 [Warning] WSREP: 0.0 (setsv): State transfer to -1.-1 (left the group) failed: -22 (Invalid argument)
Jan 23 04:11:32 setsv mysqld: 2021-01-23 4:11:32 140325406353152 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 0)
Jan 23 04:11:32 setsv mysqld: 2021-01-23 4:11:32 140325406353152 [Note] WSREP: Member 0.0 (setsv) synced with group.
Jan 23 04:11:32 setsv mysqld: 2021-01-23 4:11:32 140325406353152 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
Jan 23 04:11:32 setsv mysqld: 2021-01-23 4:11:32 140325646681856 [Note] WSREP: Synchronized with group, ready for connections
Jan 23 04:11:32 setsv mysqld: 2021-01-23 4:11:32 140325646681856 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Jan 23 04:11:36 setsv mysqld: 2021-01-23 4:11:36 140325414745856 [Note] WSREP: cleaning up b8dba97a (ssl://192.168.56.71:4567)
, file /var/lib/mysql//innobackup.backup.log says
210123 04:11:32 innobackupex: Starting the backup operation
IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints "completed OK!".
210123 04:11:32 Connecting to MySQL server host: localhost, user: sst_user, password: set, port: not set, socket: /var/lib/mysql/mysql.sock
Using server version 10.2.36-MariaDB
innobackupex version 2.3.10 based on MySQL server 5.6.24 Linux (x86_64) (revision id: bd0d4403f36)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql/
xtrabackup: open files limit requested 0, set to 16384
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 50331648
InnoDB: No valid checkpoint found.
InnoDB: If this error appears when you are creating an InnoDB database,
InnoDB: the problem may be that during an earlier attempt you managed
InnoDB: to create the InnoDB data files, but log file creation failed.
InnoDB: If that is the case, please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/error-creating-innodb.html
So when i try to start mariadb this happens:
Job for mariadb.service failed because a timeout was exceeded.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
systemctl status mariadb.service shows this:
* mariadb.service - MariaDB database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: timeout) since Mon 2019-02-18 10:11:09 UTC; 4min 1s ago
Process: 779 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 692 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]
Process: 690 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 689 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 779 (code=exited, status=0/SUCCESS)
Feb 18 10:09:36 vps-zap348646-1 systemd[1]: Starting MariaDB database server...
Feb 18 10:09:37 vps-zap348646-1 mysqld[779]: 2019-02-18 10:09:37 140439143227328 [Note] /usr/sbin/mysqld (mysqld 10.1.29-MariaDB-6ubuntu2) starting as process
Feb 18 10:11:07 vps-zap348646-1 systemd[1]: mariadb.service: Start operation timed out. Terminating.
Feb 18 10:11:09 vps-zap348646-1 systemd[1]: mariadb.service: Failed with result 'timeout'.
Feb 18 10:11:09 vps-zap348646-1 systemd[1]: Failed to start MariaDB database server.
journalctl -xe shows this:
Feb 18 10:17:09 vps-zap348646-1 systemd[1]: Starting MariaDB database server...
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit mariadb.service has begun starting up.
Feb 18 10:17:10 vps-zap348646-1 mysqld[940]: 2019-02-18 10:17:10 139827003494336 [Note] /usr/sbin/mysqld (mysqld 10.1.29-MariaDB-6ubuntu2)
Feb 18 10:18:39 vps-zap348646-1 systemd[1]: mariadb.service: Start operation timed out. Terminating.
Feb 18 10:18:41 vps-zap348646-1 systemd[1]: mariadb.service: Failed with result 'timeout'.
Feb 18 10:18:41 vps-zap348646-1 systemd[1]: Failed to start MariaDB database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit mariadb.service has failed.
--
-- The result is RESULT.
And finally error.log in mysql:
2019-02-18 10:17:10 139827003494336 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2019-02-18 10:17:10 139827003494336 [Note] InnoDB: The InnoDB memory heap is disabled
2019-02-18 10:17:10 139827003494336 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-02-18 10:17:10 139827003494336 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-02-18 10:17:10 139827003494336 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-02-18 10:17:10 139827003494336 [Note] InnoDB: Using Linux native AIO
2019-02-18 10:17:10 139827003494336 [Note] InnoDB: Using SSE crc32 instructions
2019-02-18 10:17:10 139827003494336 [Note] InnoDB: Initializing buffer pool, size = 1.0G
2019-02-18 10:17:10 139827003494336 [Note] InnoDB: Completed initialization of buffer pool
2019-02-18 10:17:10 139827003494336 [Note] InnoDB: Highest supported file format is Barracuda.
2019-02-18 10:17:10 139827003494336 [Note] InnoDB: 128 rollback segment(s) are active.
2019-02-18 10:17:10 139827003494336 [Note] InnoDB: Waiting for purge to start
2019-02-18 10:17:10 139827003494336 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.36-82.2 started; log sequence number 103241 029
2019-02-18 10:17:10 139827003494336 [Note] Plugin 'FEEDBACK' is disabled.
2019-02-18 10:17:10 139825235617536 [Note] InnoDB: Dumping buffer pool(s) not yet started
2019-02-18 10:17:10 139827003494336 [Note] Server socket created on IP: '127.0.0.1'.
2019-02-18 10:17:10 139827003494336 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.1.29-MariaDB-6ubuntu2' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Ubuntu 18.10
2019-02-18 10:18:39 139827002763008 [Note] /usr/sbin/mysqld: Normal shutdown
2019-02-18 10:18:39 139827002763008 [Note] Event Scheduler: Purging the queue. 0 events
2019-02-18 10:18:39 139825283266304 [Note] InnoDB: FTS optimize thread exiting.
2019-02-18 10:18:39 139827002763008 [Note] InnoDB: Starting shutdown...
2019-02-18 10:18:39 139827002763008 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
2019-02-18 10:18:41 139827002763008 [Note] InnoDB: Shutdown completed; log sequence number 103241039
2019-02-18 10:18:41 139827002763008 [Note] /usr/sbin/mysqld: Shutdown complete
Either I am blind or there just isn't any error and mariadb is just not working.
Maybe one of you guys have an idea. Thanks.
Edit:
I found a "solution" for now, mariadb is shutting down because of a timeout when starting, so for now I set the timeout to 0. For now this solution is enough but there must be a real solution.
Please note that since 10.1.10, MariaDB uses systemd to start the service. The /etc/init.d/mysql script is no longer used, so MYSQLD_STARTUP_TIMEOUT has no effect.
You need to find your mariadb.service file. In our case, it did not contain a timeout so the MariaDB default was being used. Just add /etc/systemd/system/mariadb.service.d/override.conf file, and put in it:
[Service]
TimeoutStartSec = 0
In the [Service] section, and it will never time out. After create the file:
# systemctl daemon-reload
# systemctl restart mysql.service
Today at around 6:23 UTC my Ubuntu Server 16.04 in Amazon EC2 made an automated apt update. From around this time my MySql server refuses to start.
At first he told me the apparmor config was wrong (like here: mysql wont start Ubuntu 16.04.2 x64) which i fixed.
Now when I try to start the mysql service like this:
:~$ sudo /etc/init.d/mysql start
[....] Starting mysql (via systemctl): mysql.serviceJob for mysql.service
failed because the control process exited with error code. See "systemctl
status mysql.service" and "journalctl -xe" for details.
failed!
systemctl status mysql.service :
:~$ sudo systemctl status mysql.service
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: activating (start-post) (Result: exit-code) since Tue 2018-04-24 19:01:31 UTC; 27s ago
Process: 22818 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
Process: 22809 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Main PID: 22818 (code=exited, status=1/FAILURE); : 22819 (mysql-systemd-s)
Tasks: 2
Memory: 324.0K
CPU: 287ms
CGroup: /system.slice/mysql.service
└─control
├─22819 /bin/bash /usr/share/mysql/mysql-systemd-start post
└─22903 sleep 1
Apr 24 19:01:31 ip-172-31-38-155 systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Apr 24 19:01:31 ip-172-31-38-155 systemd[1]: Stopped MySQL Community Server.
Apr 24 19:01:31 ip-172-31-38-155 systemd[1]: Starting MySQL Community Server...
Apr 24 19:01:33 ip-172-31-38-155 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
and the journalctl -ex looks like this:
journalctl -xe
-- The result is failed.
Apr 24 19:05:41 ip-172-31-38-155 systemd[1]: mysql.service: Unit entered failed state.
Apr 24 19:05:41 ip-172-31-38-155 systemd[1]: mysql.service: Failed with result 'exit-code'.
Apr 24 19:05:41 ip-172-31-38-155 systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Apr 24 19:05:41 ip-172-31-38-155 systemd[1]: Stopped MySQL Community Server.
-- Subject: Unit mysql.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has finished shutting down.
Apr 24 19:05:41 ip-172-31-38-155 systemd[1]: Starting MySQL Community Server...
-- Subject: Unit mysql.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has begun starting up.
Apr 24 19:05:43 ip-172-31-38-155 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Apr 24 19:06:12 ip-172-31-38-155 systemd[1]: Failed to start MySQL Community Server.
-- Subject: Unit mysql.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has failed.
--
-- The result is failed.
Apr 24 19:06:12 ip-172-31-38-155 systemd[1]: mysql.service: Unit entered failed state.
Apr 24 19:06:12 ip-172-31-38-155 systemd[1]: mysql.service: Failed with result 'exit-code'.
Apr 24 19:06:12 ip-172-31-38-155 systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Apr 24 19:06:12 ip-172-31-38-155 systemd[1]: Stopped MySQL Community Server.
-- Subject: Unit mysql.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has finished shutting down.
Apr 24 19:06:12 ip-172-31-38-155 systemd[1]: Starting MySQL Community Server...
-- Subject: Unit mysql.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has begun starting up.
Apr 24 19:06:14 ip-172-31-38-155 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
While the the cat /var/log/mysql/error.log | grep ERROR prints loads of these errors:
2018-04-24T19:10:22.941983Z 0 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
2018-04-24T19:10:22.942643Z 0 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.user' doesn't exist
2018-04-24T19:10:22.942683Z 0 [ERROR] Aborting
The one whole cycle of the cat /var/log/error.log would look like this:
2018-04-24T19:14:01.701232Z 0 [Note] Shutting down plugin 'ngram'
2018-04-24T19:14:01.701238Z 0 [Note] Shutting down plugin 'BLACKHOLE'
2018-04-24T19:14:01.701241Z 0 [Note] Shutting down plugin 'partition'
2018-04-24T19:14:01.701243Z 0 [Note] Shutting down plugin 'ARCHIVE'
2018-04-24T19:14:01.701246Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
2018-04-24T19:14:01.701249Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2018-04-24T19:14:01.701251Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2018-04-24T19:14:01.701253Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2018-04-24T19:14:01.701256Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2018-04-24T19:14:01.701258Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2018-04-24T19:14:01.701260Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2018-04-24T19:14:01.701263Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2018-04-24T19:14:01.701265Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2018-04-24T19:14:01.701267Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2018-04-24T19:14:01.701270Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2018-04-24T19:14:01.701272Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2018-04-24T19:14:01.701274Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2018-04-24T19:14:01.701277Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2018-04-24T19:14:01.701279Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2018-04-24T19:14:01.701282Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2018-04-24T19:14:01.701284Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
2018-04-24T19:14:01.701286Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2018-04-24T19:14:01.701289Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2018-04-24T19:14:01.701291Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2018-04-24T19:14:01.701293Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2018-04-24T19:14:01.701296Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2018-04-24T19:14:01.701298Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2018-04-24T19:14:01.701300Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2018-04-24T19:14:01.701303Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
2018-04-24T19:14:01.701305Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2018-04-24T19:14:01.701307Z 0 [Note] Shutting down plugin 'INNODB_CMP'
2018-04-24T19:14:01.701310Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2018-04-24T19:14:01.701312Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
2018-04-24T19:14:01.701314Z 0 [Note] Shutting down plugin 'INNODB_TRX'
2018-04-24T19:14:01.701317Z 0 [Note] Shutting down plugin 'InnoDB'
2018-04-24T19:14:01.701363Z 0 [Note] InnoDB: FTS optimize thread exiting.
2018-04-24T19:14:01.701444Z 0 [Note] InnoDB: Starting shutdown...
2018-04-24T19:14:01.701486Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2018-04-24T19:14:01.701522Z 0 [Note] InnoDB: Buffer pool(s) load completed at 180424 19:14:01 (/var/lib/mysql/ib_buffer_pool was empty)
2018-04-24T19:14:01.701533Z 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2018-04-24T19:14:01.701632Z 0 [Note] InnoDB: Buffer pool(s) dump completed at 180424 19:14:01
2018-04-24T19:14:03.309269Z 0 [Note] InnoDB: Shutdown completed; log sequence number 8715994
2018-04-24T19:14:03.310751Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2018-04-24T19:14:03.310767Z 0 [Note] Shutting down plugin 'MyISAM'
2018-04-24T19:14:03.310781Z 0 [Note] Shutting down plugin 'CSV'
2018-04-24T19:14:03.310794Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2018-04-24T19:14:03.310840Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
2018-04-24T19:14:03.310844Z 0 [Note] Shutting down plugin 'MEMORY'
2018-04-24T19:14:03.310848Z 0 [Note] Shutting down plugin 'sha256_password'
2018-04-24T19:14:03.310850Z 0 [Note] Shutting down plugin 'mysql_native_password'
2018-04-24T19:14:03.310975Z 0 [Note] Shutting down plugin 'binlog'
2018-04-24T19:14:03.311078Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
2018-04-24T19:14:32.669339Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000)
2018-04-24T19:14:32.669378Z 0 [Warning] Changed limits: table_open_cache: 431 (requested 2000)
2018-04-24T19:14:32.820589Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2018-04-24T19:14:32.822172Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.22-0ubuntu0.16.04.1) starting as process 25440 ...
2018-04-24T19:14:32.826095Z 0 [Note] InnoDB: PUNCH HOLE support available
2018-04-24T19:14:32.826121Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-04-24T19:14:32.826125Z 0 [Note] InnoDB: Uses event mutexes
2018-04-24T19:14:32.826129Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-04-24T19:14:32.826133Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-04-24T19:14:32.826136Z 0 [Note] InnoDB: Using Linux native AIO
2018-04-24T19:14:32.826367Z 0 [Note] InnoDB: Number of pools: 1
2018-04-24T19:14:32.826496Z 0 [Note] InnoDB: Using CPU crc32 instructions
2018-04-24T19:14:32.827971Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2018-04-24T19:14:32.835607Z 0 [Note] InnoDB: Completed initialization of buffer pool
2018-04-24T19:14:32.837529Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2018-04-24T19:14:32.849179Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2018-04-24T19:14:32.858128Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2018-04-24T19:14:32.858180Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2018-04-24T19:14:32.892813Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2018-04-24T19:14:32.893685Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2018-04-24T19:14:32.893696Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2018-04-24T19:14:32.893952Z 0 [Note] InnoDB: Waiting for purge to start
2018-04-24T19:14:32.944316Z 0 [Note] InnoDB: 5.7.22 started; log sequence number 8715994
2018-04-24T19:14:32.944678Z 0 [Note] Plugin 'FEDERATED' is disabled.
mysqld: Table 'mysql.plugin' doesn't exist
2018-04-24T19:14:32.944791Z 0 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
2018-04-24T19:14:32.945139Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2018-04-24T19:14:32.945192Z 0 [Warning] Failed to set up SSL because of the following SSL library error: SSL context is not usable without certificate and private key
2018-04-24T19:14:32.945201Z 0 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
2018-04-24T19:14:32.945209Z 0 [Note] - '127.0.0.1' resolves to '127.0.0.1';
2018-04-24T19:14:32.945237Z 0 [Note] Server socket created on IP: '127.0.0.1'.
2018-04-24T19:14:32.945363Z 0 [Warning] Failed to open optimizer cost constant tables
2018-04-24T19:14:32.945455Z 0 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.user' doesn't exist
2018-04-24T19:14:32.945497Z 0 [ERROR] Aborting
2018-04-24T19:14:32.945522Z 0 [Note] Binlog end
On the one hand I have seen that the the error.log tells me to run run mysql_upgrade which i cannot do because the database itself is not running.
The [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.user' doesn't exist should be fixed by an initilize command onto the service which wont work either because the service itself is not running.
I do not have a single clue what to do next. My personal best Idea for now would be to reinstall the MySql server and copy beforehand the /var/lib/mysql/mysql and the /etc/mysql directories to insert them later as a backup.
I hope anyone of you has heard something about the problem or has any kind of idea what to do. Preferably while keeping my Data.
I hope, I provided all information you need. If you need any more, just ask.
Thanks a lot in advance!
I rebooted a digital ocean box and now I can't start mysql. When I run start command i get :
Redirecting to /bin/systemctl restart mysql.service
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
Result of systemctl status mysql.service
● mysql.service - Percona XtraDB Cluster
Loaded: loaded (/usr/lib/systemd/system/mysql.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2017-12-11 17:08:18 UTC; 5s ago
Process: 26300 ExecStopPost=/usr/bin/mysql-systemd stop-post (code=exited, status=0/SUCCESS)
Process: 26270 ExecStop=/usr/bin/mysql-systemd stop (code=exited, status=2)
Process: 25674 ExecStartPost=/usr/bin/mysql-systemd start-post $MAINPID (code=exited, status=1/FAILURE)
Process: 25673 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
Process: 25632 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS)
Main PID: 25673 (code=exited, status=0/SUCCESS)
Dec 11 17:08:18 server-name-hidden mysql-systemd[25674]: ERROR! mysqld_safe with PID 25673 has already exited: FAILURE
Dec 11 17:08:18 server-name-hidden systemd[1]: mysql.service: control process exited, code=exited status=1
Dec 11 17:08:18 server-name-hidden mysql-systemd[26270]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Dec 11 17:08:18 server-name-hidden mysql-systemd[26270]: ERROR! mysql already dead
Dec 11 17:08:18 server-name-hidden systemd[1]: mysql.service: control process exited, code=exited status=2
Dec 11 17:08:18 server-name-hidden mysql-systemd[26300]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Dec 11 17:08:18 server-name-hidden mysql-systemd[26300]: WARNING: mysql may be already dead
Dec 11 17:08:18 server-name-hidden systemd[1]: Failed to start Percona XtraDB Cluster.
Dec 11 17:08:18 server-name-hidden systemd[1]: Unit mysql.service entered failed state.
Dec 11 17:08:18 server-name-hidden systemd[1]: mysql.service failed.
/var/run/mysqld/ is owned by mysql user - but it's empty. If i add a mysqld.pid file it gets removed when i run mysql start.
Does anyone know why a reboot would cause this or give me any next steps. I have reviewed the mysqld.log file and can't see anything of use. here are the last 30 lines
017-12-11T15:14:42.473164Z 0 [Note] WSREP: Received shutdown signal. Will sleep for 10 secs before initiating shutdown. pxc_maint_mode switched to SHUTDOWN
2017-12-11T15:14:52.474584Z 0 [Note] WSREP: Stop replication
2017-12-11T15:14:52.474691Z 0 [Note] WSREP: Closing send monitor...
2017-12-11T15:14:52.475160Z 0 [Note] WSREP: Closed send monitor.
2017-12-11T15:14:52.475268Z 0 [Note] WSREP: gcomm: terminating thread
2017-12-11T15:14:52.475305Z 0 [Note] WSREP: gcomm: joining thread
2017-12-11T15:14:52.475886Z 0 [Note] WSREP: gcomm: closing backend
2017-12-11T15:14:52.476079Z 0 [Note] WSREP: Current view of cluster as seen by this node
2017-12-11T15:14:52.476491Z 0 [Note] WSREP: gcomm: closed
2017-12-11T15:14:52.476532Z 0 [Note] WSREP: Received self-leave message.
2017-12-11T15:14:52.476559Z 0 [Note] WSREP: Flow-control interval: [0, 0]
2017-12-11T15:14:52.476595Z 0 [Note] WSREP: Trying to continue unpaused monitor
2017-12-11T15:14:52.476602Z 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2017-12-11T15:14:52.476608Z 0 [Note] WSREP: Shifting SYNCED -> CLOSED (TO: 13842228)
2017-12-11T15:14:52.476632Z 0 [Note] WSREP: RECV thread exiting 0: Success
2017-12-11T15:14:52.477098Z 0 [Note] WSREP: recv_thread() joined.
2017-12-11T15:14:52.477110Z 0 [Note] WSREP: Closing replication queue.
2017-12-11T15:14:52.477116Z 0 [Note] WSREP: Closing slave action queue.
2017-12-11T15:14:52.477123Z 0 [Note] Giving 63 client threads a chance to die gracefully
2017-12-11T15:14:52.478905Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 21 user: ‘hidden’
2017-12-11T15:14:52.478957Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 22 user: 'hidden'
2017-12-11T15:14:52.478994Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 6387 user: 'hidden'
2017-12-11T15:14:52.479034Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 6367 user: 'hidden'
2017-12-11T15:14:52.479084Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 6373 user: 'hidden'
2017-12-11T15:14:52.479130Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 6368 user: 'hidden'
2017-12-11T15:14:54.479289Z 0 [Note] WSREP: Waiting for active wsrep applier to exit
I've also been told to try : systemctl start mysql#bootstrap however this fails with the same error. here is the result from journalctl -xe
Dec 11 17:24:58 dropletname systemd[1]: mysql#bootstrap.service: control process exited, code=exited status=1
Dec 11 17:24:58 dropletname mysql-systemd[28663]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Dec 11 17:24:58 dropletname mysql-systemd[28663]: ERROR! mysql already dead
Dec 11 17:24:58 dropletname systemd[1]: mysql#bootstrap.service: control process exited, code=exited status=2
Dec 11 17:24:58 dropletname mysql-systemd[28694]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Dec 11 17:24:58 dropletname mysql-systemd[28694]: WARNING: mysql may be already dead
Dec 11 17:24:58 dropletname systemd[1]: Failed to start Percona XtraDB Cluster with config /etc/sysconfig/mysql.bootstrap.
Dec 11 17:24:58 dropletname systemd[1]: Unit mysql#bootstrap.service entered failed state.
Dec 11 17:24:58 dropletname systemd[1]: mysql#bootstrap.service failed.
Dec 11 17:24:58 dropletname polkitd[510]: Unregistered Authentication Agent for unix-process:28009:649814 (system bus name :1.159, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_GB.UTF-8) (disconnected from bus)
Dec 11 17:25:02 dropletname sendmail[27970]: unable to qualify my own domain name (dropletname) -- using short name
Dec 11 17:25:02 dropletname sendmail[27970]: vBBHP2cx027970: from=hidden, size=1655, class=-60, nrcpts=1, msgid=<201712111725.vBBHP2cx027970#dropletname>, relay=hidden#localhost
Dec 11 17:25:02 dropletname sendmail[28728]: vBBHP26v028728: from=<hidden#dropletname>, size=1935, class=-60, nrcpts=1, msgid=<201712111725.vBBHP2cx027970#dropletname>, proto=ESMTP, daemon=MTA, relay=dropletname [127.0.0.1]
Dec 11 17:25:02 dropletname sendmail[27970]: vBBHP2cx027970: to=hidden, ctladdr=hidden (1000/1000), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=139655, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (vBBHP26v028728 Message accepted for del
Dec 11 17:25:02 dropletname sendmail[28729]: vBBHP26v028728: to=<hidden#dropletname>, ctladdr=<hidden#dropletname> (1000/1000), delay=00:00:00, xdelay=00:00:00, mailer=local, pri=140148, dsn=2.0.0, stat=Sent
Dec 11 17:25:02 dropletname systemd[1]: Removed slice User Slice of hidden
ps aux|grep mysql
root 11590 0.0 0.0 107924 608 pts/0 T 15:43 0:00 cat /var/log/mysqld.log
root 24666 0.0 0.0 107924 612 pts/0 T 16:52 0:00 cat /var/log/mysqld.log
root 32182 0.0 0.0 112664 972 pts/2 S+ 17:59 0:00 grep --color=auto mysq
I managed to resolve this in the end however i'm not sure what 100% did it. I changed /etc/my.conf to look like this :
!includedir /etc/my.cnf.d/
!includedir /etc/percona-xtradb-cluster.conf.d/
[client]
socket=/var/lib/mysql/mysql.sock
[mysqld]
server-id=1
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
log-bin
log_slave_updates
expire_logs_days=7
innodb_strict_mode=OFF
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
I created a mysqld.pid and mysqld.lock file in /var/run/mysqld/ ensuring the owner and group was mysql:mysql
touch /var/run/mysqld/mysqld.pid;
touch /var/run/mysqld/mysqld.lock;
chown -R mysql:mysql /var/run/mysqld/;
I then ran the following two commands which seems to fail but may have helped
service mysql start --wsrep-cluster-address="gcomm://";
systemctl start mysql --wsrep-new-cluster;
Finally I ran
systemctl start mysql#bootstrap.service
service mysql start
This had previously failed every time I tried. Subsequently I then found that the /var/lib/mysql/mysql.sock had been created which previously did not exist.