Using AWS Linux 2, on my wp site this error "Error establishing a database connection", I believe I was hacked via the wp blog page and the entire site is disabled. Using MySQL:
$ sudo systemctl start mysqld
Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
At first didn't realize that I needed to run those two commands in the CLI. Here is the output for systemctl status mysqld.service -l:
$ systemctl status mysqld.service -l
● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sun 2022-02-13 21:00:30 UTC; 6h ago
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
Process: 19959 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS (code=exited, status=1/FAILURE)
Process: 19926 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
Main PID: 19959 (code=exited, status=1/FAILURE)
Status: "Server startup in progress"
Error: 13 (Permission denied)
Feb 13 21:00:30 ip-172-31-91-154.ec2.internal mysqld[19959]: 2022-02-13T21:00:30.730728Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.28) starting as process 19959
Feb 13 21:00:30 ip-172-31-91-154.ec2.internal mysqld[19959]: 2022-02-13T21:00:30.734108Z 0 [Warning] [MY-010091] [Server] Can't create test file /var/lib/mysql/mysqld_tmp_file_case_insensitive_test.lower-test
Feb 13 21:00:30 ip-172-31-91-154.ec2.internal mysqld[19959]: 2022-02-13T21:00:30.734121Z 0 [Warning] [MY-010159] [Server] Setting lower_case_table_names=2 because file system for /var/lib/mysql/ is case insensitive
Feb 13 21:00:30 ip-172-31-91-154.ec2.internal mysqld[19959]: 2022-02-13T21:00:30.734720Z 0 [ERROR] [MY-010187] [Server] Could not open file '/var/log/mysqld.log' for error logging: Permission denied
Feb 13 21:00:30 ip-172-31-91-154.ec2.internal mysqld[19959]: 2022-02-13T21:00:30.736749Z 0 [ERROR] [MY-010119] [Server] Aborting
Feb 13 21:00:30 ip-172-31-91-154.ec2.internal mysqld[19959]: 2022-02-13T21:00:30.736912Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.28) MySQL Community Server - GPL.
Feb 13 21:00:30 ip-172-31-91-154.ec2.internal systemd[1]: mysqld.service: main process exited, code=exited, status=1/FAILURE
Feb 13 21:00:30 ip-172-31-91-154.ec2.internal systemd[1]: Failed to start MySQL Server.
Feb 13 21:00:30 ip-172-31-91-154.ec2.internal systemd[1]: Unit mysqld.service entered failed state.
Feb 13 21:00:30 ip-172-31-91-154.ec2.internal systemd[1]: mysqld.service failed.
and running journalctl -xe -l:
Feb 14 03:17:27 ip-172-31-91-154.ec2.internal sshd[29726]: Excess permission or bad ownership on file /var/lo
Feb 14 03:17:27 ip-172-31-91-154.ec2.internal sshd[29726]: Received disconnect from 81.70.242.147 port 43604:
Feb 14 03:17:27 ip-172-31-91-154.ec2.internal sshd[29726]: Disconnected from 81.70.242.147 port 43604 [preaut
Feb 14 03:17:41 ip-172-31-91-154.ec2.internal sshd[29728]: Connection closed by 104.243.26.5 port 46890 [prea
Feb 14 03:18:06 ip-172-31-91-154.ec2.internal sshd[29730]: Invalid user yw from 82.196.5.221 port 47008
Feb 14 03:18:06 ip-172-31-91-154.ec2.internal sshd[29730]: Excess permission or bad ownership on file /var/lo
Feb 14 03:18:06 ip-172-31-91-154.ec2.internal sshd[29730]: input_userauth_request: invalid user yw [preauth]
Feb 14 03:18:06 ip-172-31-91-154.ec2.internal sshd[29730]: pam_unix(sshd:auth): check pass; user unknown
Feb 14 03:18:06 ip-172-31-91-154.ec2.internal sshd[29730]: pam_unix(sshd:auth): authentication failure; logna
Feb 14 03:18:08 ip-172-31-91-154.ec2.internal sshd[29730]: Failed password for invalid user yw from 82.196.5.
Feb 14 03:18:08 ip-172-31-91-154.ec2.internal sshd[29730]: Excess permission or bad ownership on file /var/lo
Feb 14 03:18:08 ip-172-31-91-154.ec2.internal sshd[29730]: Received disconnect from 82.196.5.221 port 47008:1
Feb 14 03:18:08 ip-172-31-91-154.ec2.internal sshd[29730]: Disconnected from 82.196.5.221 port 47008 [preauth
Feb 14 03:18:15 ip-172-31-91-154.ec2.internal sshd[29732]: Invalid user server from 120.53.121.152 port 59070
Feb 14 03:18:15 ip-172-31-91-154.ec2.internal sshd[29732]: Excess permission or bad ownership on file /var/lo
Feb 14 03:18:15 ip-172-31-91-154.ec2.internal sshd[29732]: input_userauth_request: invalid user server [preau
Feb 14 03:18:15 ip-172-31-91-154.ec2.internal sshd[29732]: pam_unix(sshd:auth): check pass; user unknown
Feb 14 03:18:15 ip-172-31-91-154.ec2.internal sshd[29732]: pam_unix(sshd:auth): authentication failure; logna
Feb 14 03:18:18 ip-172-31-91-154.ec2.internal sshd[29732]: Failed password for invalid user server from 120.5
Feb 14 03:18:18 ip-172-31-91-154.ec2.internal sshd[29732]: Excess permission or bad ownership on file /var/lo
Feb 14 03:18:19 ip-172-31-91-154.ec2.internal sshd[29732]: Received disconnect from 120.53.121.152 port 59070
Feb 14 03:18:19 ip-172-31-91-154.ec2.internal sshd[29732]: Disconnected from 120.53.121.152 port 59070 [preau
Feb 14 03:18:27 ip-172-31-91-154.ec2.internal dhclient[2827]: XMT: Solicit on eth0, interval 111530ms.
I tried removing the db and reinstalling but this did not make any difference.
the issue is this line:
[ERROR] [MY-010187] [Server] Could not open file '/var/log/mysqld.log' for error logging: Permission denied
Which is supported by this error message:
access forbidden by rule
So the issue is permissions to the /var/log/mysqld.log file. What is needed is to create permissions for mysql to access this file - running this command with a server reboot solved this issue:
$ sudo chown mysql:mysql /var/log/mysqld.log
Deleting the ec2 instance and starting over was the direction I took. Still no idea of what caused the error however am double checking security holes and patching up. Thanks everyone for responding.
best
Related
Connection to the MySQL db was working fine 2 weeks ago, and without any editing or development activity it dropped out. I am using SQL Workbench to access. I can connect to the AWS EC2 instance via the AWS dashboard as well as SSH tunnel, can view/access files from filezilla connection.
The server is running WordPress application for a single website (https://farrahstribute.com/)
By examining past answers I tried:
(1) editing /etc/my.cnf and added this line
bind-address=0.0.0.0
after adding I rebooted server, however nothing changed, error with db persisting, so I removed the code I added
(2) editing /etc/hosts.allow and added these lines
mysqld: ALL : ALLOW
mysqld-max: ALL : ALLOW
after editing I rebooted server, however nothing changed, error with db persisting, so I removed the code that I added
From the terminal - this command systemctl status mysqld -l gave this result:
● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2022-11-29 01:22:25 UTC; 14h ago
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
Process: 3152 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS (code=exited, status=1/FAILURE)
Process: 2906 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
Main PID: 3152 (code=exited, status=1/FAILURE)
Status: "Server shutdown complete"
Error: 13 (Permission denied)
Nov 29 01:22:20 ip-172-31-28-111.ec2.internal systemd[1]: Starting MySQL Server...
Nov 29 01:22:25 ip-172-31-28-111.ec2.internal mysqld[3152]: 2022-11-29T01:22:25.558132Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.31) starting as process 3152
Nov 29 01:22:25 ip-172-31-28-111.ec2.internal mysqld[3152]: 2022-11-29T01:22:25.615516Z 0 [ERROR] [MY-010187] [Server] Could not open file '/var/log/mysqld.log' for error logging: Permission denied
Nov 29 01:22:25 ip-172-31-28-111.ec2.internal mysqld[3152]: 2022-11-29T01:22:25.615582Z 0 [ERROR] [MY-010119] [Server] Aborting
Nov 29 01:22:25 ip-172-31-28-111.ec2.internal mysqld[3152]: 2022-11-29T01:22:25.640431Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.31) MySQL Community Server - GPL.
Nov 29 01:22:25 ip-172-31-28-111.ec2.internal systemd[1]: mysqld.service: main process exited, code=exited, status=1/FAILURE
Nov 29 01:22:25 ip-172-31-28-111.ec2.internal systemd[1]: Failed to start MySQL Server.
Nov 29 01:22:25 ip-172-31-28-111.ec2.internal systemd[1]: Unit mysqld.service entered failed state.
Nov 29 01:22:25 ip-172-31-28-111.ec2.internal systemd[1]: mysqld.service failed.
Also from the terminal ran: systemctl status mysqld.service with output:
● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2022-11-29 15:42:23 UTC; 2min 7s ago
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
Process: 7247 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS (code=exited, status=1/FAILURE)
Process: 7220 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
Main PID: 7247 (code=exited, status=1/FAILURE)
Status: "Server shutdown complete"
Error: 13 (Permission denied)
Nov 29 15:42:22 ip-172-31-28-111.ec2.internal systemd[1]: Starting MySQL Server...
Nov 29 15:42:23 ip-172-31-28-111.ec2.internal mysqld[7247]: 2022-11-29T15:42:23.292440Z 0 [System] [MY-010116] [Server...7247
Nov 29 15:42:23 ip-172-31-28-111.ec2.internal mysqld[7247]: 2022-11-29T15:42:23.296242Z 0 [ERROR] [MY-010187] [Server]...nied
Nov 29 15:42:23 ip-172-31-28-111.ec2.internal mysqld[7247]: 2022-11-29T15:42:23.297482Z 0 [ERROR] [MY-010119] [Server]...ting
Nov 29 15:42:23 ip-172-31-28-111.ec2.internal mysqld[7247]: 2022-11-29T15:42:23.297889Z 0 [System] [MY-010910] [Server...GPL.
Nov 29 15:42:23 ip-172-31-28-111.ec2.internal systemd[1]: mysqld.service: main process exited, code=exited, status=1/FAILURE
Nov 29 15:42:23 ip-172-31-28-111.ec2.internal systemd[1]: Failed to start MySQL Server.
Nov 29 15:42:23 ip-172-31-28-111.ec2.internal systemd[1]: Unit mysqld.service entered failed state.
Nov 29 15:42:23 ip-172-31-28-111.ec2.internal systemd[1]: mysqld.service failed.
I also examined the error log /var/log/nginx/farrahstribute_error.log and found this error:
2022/11/29 00:55:06 [error] 3111#3111: *39 access forbidden by rule, client: 54.152.255.102, server: farrahstribute.com, request: "GET /.env HTTP/1.1", host: "34.199.116.152"
the issue is this line:
[ERROR] [MY-010187] [Server] Could not open file '/var/log/mysqld.log' for error logging: Permission denied
Which is supported by this error message:
Error: 13 (Permission denied)
So the issue is permissions to the /var/log/mysqld.log file. What is needed is to create permissions for mysql to access this file - running this command with a server reboot solved this issue:
$ sudo chown mysql:mysql /var/log/mysqld.log
I am using MySQL Percona XtraDB Cluster with 3 node and MySQL service in one node did not work because of disk full. after fix the issue we try to start MySQL on this node but the following error occur: (before this issue all node had been working fine)
$ sudo /etc/init.d/mysql start
[....] Starting mysql (via systemctl): mysql.serviceJob for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.
failed!
$ sudo systemctl status mysql.service
● mysql.service - LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon
Loaded: loaded (/etc/init.d/mysql; generated)
Drop-In: /etc/systemd/system/mysql.service.d
└─override.conf
Active: failed (Result: exit-code) since Tue 2020-03-24 11:57:55 +0430; 1h 2min ago
Docs: man:systemd-sysv-generator(8)
Process: 19397 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)
Mar 24 11:57:47 server-3 systemd[1]: Starting LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon...
Mar 24 11:57:47 server-3 mysql[19397]: * Starting MySQL (Percona XtraDB Cluster) database server mysqld
Mar 24 11:57:55 server-3 mysql[19397]: * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
Mar 24 11:57:55 server-3 mysql[19397]: ...fail!
Mar 24 11:57:55 server-3 systemd[1]: mysql.service: Control process exited, code=exited status=1
Mar 24 11:57:55 server-3 systemd[1]: mysql.service: Failed with result 'exit-code'.
Mar 24 11:57:55 server-3 systemd[1]: Failed to start LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon.
Operating System: Ubuntu 18.04.2 LTS
mysql Ver 14.14 Distrib 5.7.25-28, for debian-linux-gnu (x86_64) using 7.0
also please find journalctl log in below:
$ sudo journalctl -xe
Journal file /var/log/journal/7e072ef87bdf452f8a9684d905f87daf/user-1863206337.journal is truncated, ignoring file.
Mar 24 19:05:22 server-3 pmm-agent[30366]: INFO[2020-03-24T19:05:22.235+04:30] time="2020-03-24T19:05:22+04:30" level=error msg="Error pinging mysqld: dial tcp 1
Mar 24 19:05:22 server-3 mysql[6820]: * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
Mar 24 19:05:22 server-3 mysql[6820]: ...fail!
Mar 24 19:05:22 server-3 systemd[1]: mysql.service: Control process exited, code=exited status=1
Mar 24 19:05:22 server-3 systemd[1]: mysql.service: Failed with result 'exit-code'.
Mar 24 19:05:22 server-3 systemd[1]: mysql.service: Failed with result 'exit-code'.
Mar 24 19:05:22 server-3 systemd[1]: mysql.service: Failed with result 'exit-code'.
Mar 24 19:05:22 server-3 systemd[1]: Failed to start LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon.
-- Subject: Unit mysql.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit mysql.service has failed.
--
-- The result is RESULT.
Mar 24 19:05:22 server-3 sudo[6779]: pam_unix(sudo:session): session closed for user root
Mar 24 19:05:22 server-3 kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20170831/exfield-427)
Mar 24 19:05:22 server-3 kernel: No Local Variables are initialized for Method [_PMM]
Mar 24 19:05:22 server-3 kernel: No Arguments are initialized for method [_PMM]
Mar 24 19:05:22 server-3 kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20170831/psparse-550)
Mar 24 19:05:22 server-3 kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20170831/power_meter-338)
Mar 24 19:05:23 server-3 snmpd[1805]: Connection from UDP: [192.168.2.10]:54620->[192.168.8.3]:161
Mar 24 19:05:23 server-3 snmpd[1805]: Connection from UDP: [192.168.2.10]:54647->[192.168.8.3]:161
Mar 24 19:05:25 server-3 pmm-agent[30366]: INFO[2020-03-24T19:05:25.550+04:30] time="2020-03-24T19:05:25+04:30" level=error msg="Error pinging mysqld: dial tcp 1
Mar 24 19:05:26 server-3 snmpd[1805]: Connection from UDP: [192.168.2.10]:55485->[192.168.8.3]:161
Mar 24 19:05:26 server-3 snmpd[1805]: Connection from UDP: [192.168.2.10]:55500->[192.168.8.3]:161
Mar 24 19:05:27 server-3 pmm-agent[30366]: INFO[2020-03-24T19:05:27.235+04:30] time="2020-03-24T19:05:27+04:30" level=error msg="Error pinging mysqld: dial tcp 1
Mar 24 19:05:27 server-3 kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20170831/exfield-427)
Mar 24 19:05:27 server-3 kernel: No Local Variables are initialized for Method [_PMM]
Mar 24 19:05:27 server-3 kernel: No Arguments are initialized for method [_PMM]
Mar 24 19:05:27 server-3 kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20170831/psparse-550)
Mar 24 19:05:27 server-3 kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20170831/power_meter-338)
Mar 24 19:05:30 server-3 pmm-agent[30366]: INFO[2020-03-24T19:05:30.550+04:30] time="2020-03-24T19:05:30+04:30" level=error msg="Error pinging mysqld: dial tcp 1
Mar 24 19:05:31 server-3 pmm-agent[30366]: ERRO[2020-03-24T19:05:31.438+04:30] cannot select ##slow_query_log_file: dial tcp 127.0.0.1:3306: connect: connection
Mar 24 19:05:32 server-3 pmm-agent[30366]: INFO[2020-03-24T19:05:32.234+04:30] time="2020-03-24T19:05:32+04:30" level=error msg="Error pinging mysqld: dial tcp 1
Mar 24 19:05:32 server-3 kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20170831/exfield-427)
Mar 24 19:05:32 server-3 kernel: No Local Variables are initialized for Method [_PMM]
Mar 24 19:05:32 server-3 kernel: No Arguments are initialized for method [_PMM]
Mar 24 19:05:32 server-3 kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20170831/psparse-550)
Mar 24 19:05:32 server-3 kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20170831/power_meter-338)
Mar 24 19:05:35 server-3 pmm-agent[30366]: INFO[2020-03-24T19:05:35.550+04:30] time="2020-03-24T19:05:35+04:30" level=error msg="Error pinging mysqld: dial tcp 1
Mar 24 19:05:36 server-3 sudo[7805]: admin_user : problem with defaults entries ; TTY=pts/0 ; PWD=/etc/systemd/system/mysql.service.d ; USER=root ;
Mar 24 19:05:36 server-3 sudo[7805]: admin_user : TTY=pts/0 ; PWD=/etc/systemd/system/mysql.service.d ; USER=root ; COMMAND=/bin/journalctl -xe
Mar 24 19:05:36 server-3 sudo[7805]: pam_unix(sudo:session): session opened for user root by admin_user(uid=0)
and please find mysqld.log in below:
/var/log$ sudo tail -n50 mysqld.log
2020-03-23T20:44:32.844540Z 36 [ERROR] WSREP: Node consistency compromised, aborting...
2020-03-23T20:44:32.844642Z 13 [ERROR] WSREP: Failed to apply trx: source: 89d92e92-6638-11e9-bdcb-7f2c26717188 version: 4 local: 0 state: APPLYING flags: 1 conn_id: 26113114 trx_id: 5580550846 seqnos (l: 151018291, g: 793430465, s: 793430462, d: 793430355, ts: 33700460567224654)
2020-03-23T20:44:32.844671Z 13 [ERROR] WSREP: Failed to apply trx 793430465 4 times
2020-03-23T20:44:32.844683Z 13 [ERROR] WSREP: Node consistency compromised, aborting...
2020-03-23T20:44:33.255602Z 21 [Note] WSREP: (c0cb4c75, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.8.1:4567 tcp://192.168.8.2:4567
2020-03-23T20:44:37.817120Z 21 [Note] WSREP: declaring node with index 0 suspected, timeout PT5S (evs.suspect_timeout)
2020-03-23T20:44:37.817139Z 21 [Note] WSREP: declaring node with index 1 suspected, timeout PT5S (evs.suspect_timeout)
2020-03-23T20:44:37.817147Z 21 [Note] WSREP: evs::proto(c0cb4c75, LEAVING, view_id(REG,82cab7cb,31)) suspecting node: 82cab7cb
2020-03-23T20:44:37.817152Z 21 [Note] WSREP: evs::proto(c0cb4c75, LEAVING, view_id(REG,82cab7cb,31)) suspected node without join message, declaring inactive
2020-03-23T20:44:37.817158Z 21 [Note] WSREP: evs::proto(c0cb4c75, LEAVING, view_id(REG,82cab7cb,31)) suspecting node: 89d92e92
2020-03-23T20:44:37.817161Z 21 [Note] WSREP: evs::proto(c0cb4c75, LEAVING, view_id(REG,82cab7cb,31)) suspected node without join message, declaring inactive
2020-03-23T20:44:37.817182Z 21 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(NON_PRIM,82cab7cb,31)
memb {
c0cb4c75,0
}
joined {
}
left {
}
partitioned {
82cab7cb,0
89d92e92,0
}
)
2020-03-23T20:44:37.817203Z 21 [Note] WSREP: Current view of cluster as seen by this node
view ((empty))
2020-03-23T20:44:37.817358Z 21 [Note] WSREP: gcomm: closed
2020-03-23T20:44:37.817463Z 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2020-03-23T20:44:37.817504Z 0 [Note] WSREP: Flow-control interval: [10000, 10000]
2020-03-23T20:44:37.817515Z 0 [Note] WSREP: Trying to continue unpaused monitor
2020-03-23T20:44:37.817522Z 0 [Note] WSREP: Received NON-PRIMARY.
2020-03-23T20:44:37.817524Z 13 [ERROR] WSREP: non-standard exception
2020-03-23T20:44:37.817530Z 0 [Note] WSREP: Shifting DONOR/DESYNCED -> OPEN (TO: 793430465)
2020-03-23T20:44:37.817568Z 13 [Note] WSREP: applier thread exiting (code:8)
2020-03-23T20:44:37.817584Z 13 [Note] WSREP: Starting Shutdown
2020-03-23T20:44:37.817587Z 0 [Note] WSREP: Received self-leave message.
2020-03-23T20:44:37.817529Z 36 [ERROR] WSREP: non-standard exception
2020-03-23T20:44:37.817643Z 36 [Note] WSREP: applier thread exiting (code:8)
2020-03-23T20:44:37.817656Z 36 [Note] WSREP: Starting Shutdown
2020-03-23T20:44:37.817759Z 0 [Note] WSREP: Received shutdown signal. Will sleep for 10 secs before initiating shutdown. pxc_maint_mode switched to SHUTDOWN
2020-03-23T20:44:37.817813Z 0 [Note] WSREP: Flow-control interval: [10000, 10000]
2020-03-23T20:44:37.817841Z 0 [Note] WSREP: Trying to continue unpaused monitor
2020-03-23T20:44:37.817852Z 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2020-03-23T20:44:37.817860Z 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 793430465)
2020-03-23T20:44:37.817870Z 0 [Note] WSREP: RECV thread exiting 0: Success
2020-03-23T20:44:37.818185Z 21 [Note] WSREP: recv_thread() joined.
2020-03-23T20:44:37.818192Z 21 [Note] WSREP: Closing replication queue.
2020-03-23T20:44:37.818195Z 21 [Note] WSREP: Closing slave action queue.
2020-03-23T20:44:37.818247Z 21 [Note] WSREP: /usr/sbin/mysqld: Terminated.
How to start MySQL in this node?
thanks in advance.
As your myslqd.log shows, the disk full condition has left your node in a state, where it can find no way to achieve consistency with the cluster.
This implies, that you have to remove the node from the cluster, empty it out and then re-add it like a new node.
I have a relatively new install of MySQL 5.7 server on Fedora Core 24 that will not start up.
I had successfully moved the datadir to /data02/mysql/ from the default and started up the database just fine a couple of time, even loaded a 7GB database into it from a mysqldump.
Now I'm getting the below error "Job for mysqld.service failed because the control process exited with error code."
The MySQL Error log seems to suggest this is the issue
2016-08-27T04:57:06.855134Z 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2016-08-27T04:57:06.855195Z 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2016-08-27T04:57:06.855201Z 0 [ERROR] InnoDB: os_file_get_status() failed on './ibdata1'. Can't determine file permissions
2016-08-27T04:57:06.855220Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
I have verified that:
partition /data02 is read/write
chown is set to mysql:mysql recursively on all files in /data02/mysql
chmod is set to 755 on /data02/mysql recursively.
NOTE: I did try 777 and that did not work either
pid file and mysql.sock are in locations that should be writeable.
Here's the directory listing showing 777 permissions
[root#localhost data02]# ls -al /data02/mysql
total 910768
drwxrwxrwx. 6 mysql mysql 4096 Aug 10 10:35 .
drwxrwxrwx. 6 root root 4096 Aug 14 03:19 ..
-rwxrwxrwx. 1 mysql mysql 56 Aug 4 03:47 auto.cnf
-rwxrwxrwx. 1 mysql mysql 1680 Aug 4 03:47 ca-key.pem
-rwxrwxrwx. 1 mysql mysql 1075 Aug 4 03:47 ca.pem
-rwxrwxrwx. 1 mysql mysql 1079 Aug 4 03:47 client-cert.pem
-rwxrwxrwx. 1 mysql mysql 1680 Aug 4 03:47 client-key.pem
-rwxrwxrwx. 1 mysql mysql 485738 Aug 10 10:35 ib_buffer_pool
-rwxrwxrwx. 1 mysql mysql 817889280 Aug 10 10:40 ibdata1
-rwxrwxrwx. 1 mysql mysql 50331648 Aug 10 10:38 ib_logfile0
-rwxrwxrwx. 1 mysql mysql 50331648 Aug 10 10:22 ib_logfile1
-rwxrwxrwx. 1 mysql mysql 12582912 Aug 4 22:38 ibtmp1
drwxrwxrwx. 2 mysql mysql 4096 Aug 4 03:47 mysql
-rwxrwxrwx. 1 mysql mysql 7 Aug 4 03:47 mysql_upgrade_info
drwxrwxrwx. 2 mysql mysql 4096 Aug 4 03:47 performance_schema
-rwxrwxrwx. 1 mysql mysql 1680 Aug 4 03:47 private_key.pem
-rwxrwxrwx. 1 mysql mysql 452 Aug 4 03:47 public_key.pem
-rwxrwxrwx. 1 mysql mysql 1079 Aug 4 03:47 server-cert.pem
-rwxrwxrwx. 1 mysql mysql 1676 Aug 4 03:47 server-key.pem
drwxrwxrwx. 2 mysql mysql 12288 Aug 4 03:47 sys
drwxrwxrwx. 2 mysql mysql 4096 Aug 4 22:55 test_tmp
Here's the server config
[root#localhost data02]# cat /etc/my.cnf.d/community-mysql-server.cnf
[server]
[mysqld]
datadir=/data02/mysql
socket=/data02/mysql/mysql.sock
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
innodb_buffer_pool_size = 5G
I'm launching the process as 'root' but it starts up as 'mysql' as it should.
It last worked on Aug 10th. But was shutdown since then, An OS Patching had been run since then, but that's all to my knowledge.
Here's the startup cmds executed.
[root#localhost ~]# systemctl start mysqld
Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
[root#localhost ~]# systemctl status mysqld.service
● mysqld.service - MySQL 5.7 database server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: inactive (dead) (Result: exit-code) since Sat 2016-08-27 04:26:52 UTC; 40s ago
Process: 5100 ExecStart=/usr/libexec/mysqld --daemonize --basedir=/usr --pid-file=/var/run/mysqld/mysqld.pid (code=exited, status=1/FAILURE)
Process: 5066 ExecStartPre=/usr/libexec/mysql-prepare-db-dir %n (code=exited, status=0/SUCCESS)
Process: 5043 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
Aug 27 04:26:52 localhost.localdomain systemd[1]: Failed to start MySQL 5.7 database server.
Aug 27 04:26:52 localhost.localdomain systemd[1]: mysqld.service: Unit entered failed state.
Aug 27 04:26:52 localhost.localdomain systemd[1]: mysqld.service: Failed with result 'exit-code'.
Aug 27 04:26:52 localhost.localdomain systemd[1]: mysqld.service: Service hold-off time over, scheduling restart.
Aug 27 04:26:52 localhost.localdomain systemd[1]: Stopped MySQL 5.7 database server.
Aug 27 04:26:52 localhost.localdomain systemd[1]: mysqld.service: Start request repeated too quickly.
Aug 27 04:26:52 localhost.localdomain systemd[1]: Failed to start MySQL 5.7 database server.
I was trying to install the mysqlgovernor, and in the end I got this mess, and now all my websites are down.
root#panel [~]# systemctl status mariadb.service
● mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: failed (Result: exit-code) since Sun 2016-03-06 06:35:31 CET; 5s ago
Process: 6244 ExecStartPost=/usr/sbin/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
Process: 6243 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
Process: 6205 ExecStartPre=/usr/sbin/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
Process: 6179 ExecStartPre=/usr/sbin/mariadb-check-socket (code=exited, status=0/SUCCESS)
Main PID: 6243 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/mariadb.service
Mar 06 06:35:28 panel.uhlhosting.ch mariadb-prepare-db-dir[6205]: touch: cannot touch 'panel.uhlhosting.ch.err': Permission denied
Mar 06 06:35:28 panel.uhlhosting.ch mariadb-prepare-db-dir[6205]: chown: cannot access 'panel.uhlhosting.ch.err': No such file or directory
Mar 06 06:35:28 panel.uhlhosting.ch mariadb-prepare-db-dir[6205]: chmod: cannot access 'panel.uhlhosting.ch.err': No such file or directory
Mar 06 06:35:29 panel.uhlhosting.ch mysqld_safe[6243]: 160306 06:35:29 mysqld_safe Logging to '/var/lib/mysql/panel.uhlhosting.ch.err'.
Mar 06 06:35:29 panel.uhlhosting.ch mysqld_safe[6243]: 160306 06:35:29 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Mar 06 06:35:30 panel.uhlhosting.ch mysqld_safe[6243]: 160306 06:35:30 mysqld_safe mysqld from pid file /var/lib/mysql/panel.uhlhosting.ch.pid ended
Mar 06 06:35:31 panel.uhlhosting.ch systemd[1]: mariadb.service: control process exited, code=exited status=1
Mar 06 06:35:31 panel.uhlhosting.ch systemd[1]: Failed to start MariaDB database server.
Mar 06 06:35:31 panel.uhlhosting.ch systemd[1]: Unit mariadb.service entered failed state.
Mar 06 06:35:31 panel.uhlhosting.ch systemd[1]: mariadb.service failed.
root#panel [~]#
Here is my Error list:
InnoDB: Assertion failure in thread 140010623854336 in file buf0buf.cc line 2850
Here is the working MariaDB after reinstall, and disabling the InnoDB db's yet we do not want to be without them.
root#panel [~]# systemctl status mariadb.service
● mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: active (running) since Sun 2016-03-06 10:16:12 CET; 29min ago
Main PID: 16551 (mysqld)
Status: "Taking your SQL requests now..."
CGroup: /system.slice/mariadb.service
└─16551 /usr/sbin/mysqld
Mar 06 10:16:12 panel.uhlhosting.ch mysqld[16551]: 2016-03-06 10:16:12 140128996632704 [Note] /usr/sbin/mysqld (mysqld 10.1.12-MariaDB) starting as process 16551 ...
Mar 06 10:16:12 panel.uhlhosting.ch mysqld[16551]: 2016-03-06 10:16:12 140128996632704 [Note] Plugin 'InnoDB' is disabled.
Mar 06 10:16:12 panel.uhlhosting.ch mysqld[16551]: 2016-03-06 10:16:12 140128996632704 [Note] Plugin 'FEEDBACK' is disabled.
Mar 06 10:16:12 panel.uhlhosting.ch mysqld[16551]: 2016-03-06 10:16:12 140128996632704 [Note] Server socket created on IP: '::'.
Mar 06 10:16:12 panel.uhlhosting.ch mysqld[16551]: 2016-03-06 10:16:12 140128995883776 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1286: Unknown storage engine 'InnoDB'
Mar 06 10:16:12 panel.uhlhosting.ch mysqld[16551]: 2016-03-06 10:16:12 140128996632704 [Note] /usr/sbin/mysqld: ready for connections.
Mar 06 10:16:12 panel.uhlhosting.ch mysqld[16551]: Version: '10.1.12-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
Mar 06 10:16:12 panel.uhlhosting.ch systemd[1]: Started MariaDB database server.
Mar 06 10:16:54 panel.uhlhosting.ch systemd[1]: Started MariaDB database server.
Mar 06 10:17:00 panel.uhlhosting.ch systemd[1]: Started MariaDB database server.
Server.err
my.cnf file
Please look at the first three lines of the error-log:
MariaDB has no permission to write on the directory where panel.uhlhosting.ch.err is located
You can fix this by changing the owner/group of this directory to mysql:mysql by chown -R mysql:mysql /path/to/directory/ if it's on linux.
Here is the error i am getting
mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled)
Active: failed (Result: exit-code) since Mon 2014-10-13 14:31:02 CEST; 10s ago
Process: 23609 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
Process: 23608 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
Process: 23506 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
Main PID: 23608 (code=exited, status=0/SUCCESS)
Oct 13 14:30:59 localhost.localdomain mariadb-prepare-db-dir[23506]: Support MariaDB development by buying support/new features from
Oct 13 14:30:59 localhost.localdomain mariadb-prepare-db-dir[23506]: SkySQL Ab. You can contact us about this at sales#skysql.com.
Oct 13 14:30:59 localhost.localdomain mariadb-prepare-db-dir[23506]: Alternatively consider joining our community based development effort:
Oct 13 14:30:59 localhost.localdomain mariadb-prepare-db-dir[23506]: http://mariadb.com/kb/en/contributing-to-the-mariadb-project/
Oct 13 14:30:59 localhost.localdomain mysqld_safe[23608]: 141013 14:30:59 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Oct 13 14:30:59 localhost.localdomain mysqld_safe[23608]: 141013 14:30:59 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Oct 13 14:31:02 localhost.localdomain mysqld_safe[23608]: 141013 14:31:02 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
Oct 13 14:31:02 localhost.localdomain systemd[1]: mariadb.service: control process exited, code=exited status=1
Oct 13 14:31:02 localhost.localdomain systemd[1]: Failed to start MariaDB database server.
Oct 13 14:31:02 localhost.localdomain systemd[1]: Unit mariadb.service entered failed state.
THis is the mariadb.log file:
Im not really understanding this error, because the sock file has not been created and my server hasn't succesfully started...
141013 14:37:28 [ERROR] Can't start server : Bind on unix socket: Permission denied
141013 14:37:28 [ERROR] Do you already have another mysqld server running on socket: /var/run/mysql/mysql.sock ?
141013 14:37:28 [ERROR] Aborting
141013 14:37:28 InnoDB: Starting shutdown...
141013 14:37:29 InnoDB: Shutdown completed; log sequence number 1597945
141013 14:37:29 [Note] /usr/libexec/mysqld: Shutdown complete
I installed it fresh several times and I can't seem to figure out what the error is in the logs. I am running everything as root. Thanks.