MySQL server crash on TRUNCATE table with foreign key constraints - mysql

I was running a web UI named pterodactyl panel : https://pterodactyl.io/ and after days of working fine, it crashed with the error message "Out of storage".
Obviously I did check and found that the disk was full, upon further investigation, a table named activity_logs in the panel database was using up almost 120 GB of space. I decided to clear the logs as I did not need them. I went ahead with a vacate command TRUNCATE. This was followed by an error:
Database changed
MariaDB [panel]> TRUNCATE TABLE activity_logs;
ERROR 1701 (42000): Cannot truncate a table referenced in a foreign key constraint (`panel`.`activity_log_subjects`, CONSTRAINT `activity_log_subjects_activity_log_id_foreign` FOREIGN KEY (`activity_log_id`) REFERENCES `panel`.`activity_logs` (`id`))
I went ahead and disabled key constrains and ran TRUNCATE in the hopes that I would enable forgeign key checks once the table is empty.
Expected:
SET FOREIGN_KEY_CHECKS = 0;
TRUNCATE TABLE activity_logs;
SET FOREIGN_KEY_CHECKS = 1;
exit;
What I got instead was:
MariaDB [panel]> SET FOREIGN_KEY_CHECKS = 0;
Query OK, 0 rows affected (0.000 sec)
MariaDB [panel]> TRUNCATE TABLE activity_logs;
ERROR 2013 (HY000): Lost connection to MySQL server during query
MariaDB [panel]> TRUNCATE TABLE activity_logs;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
ERROR: Can't connect to the server
unknown [panel]> exit
Bye
Not sure what to do in order to fix the issue.
Here is the log that i ran after:
root#Panel:/var/lib/mysql/panel# systemctl status mysql
● mariadb.service - MariaDB 10.4.27 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: failed (Result: exit-code) since Tue 2023-01-24 09:20:30 CET; 2s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 4680 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
Process: 4677 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 6101 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 5952 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] & Process: 5943 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 5931 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 6101 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"
Jan 24 09:20:29 Panel mysqld[6101]: 2023-01-24 9:20:29 0 [Note] InnoDB: Starting shutdown...
Jan 24 09:20:30 Panel mysqld[6101]: 2023-01-24 9:20:30 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
Jan 24 09:20:30 Panel mysqld[6101]: 2023-01-24 9:20:30 0 [ERROR] Plugin 'InnoDB' init function returned error.
Jan 24 09:20:30 Panel mysqld[6101]: 2023-01-24 9:20:30 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Jan 24 09:20:30 Panel mysqld[6101]: 2023-01-24 9:20:30 0 [Note] Plugin 'FEEDBACK' is disabled.
Jan 24 09:20:30 Panel mysqld[6101]: 2023-01-24 9:20:30 0 [ERROR] Unknown/unsupported storage engine: InnoDB
Jan 24 09:20:30 Panel mysqld[6101]: 2023-01-24 9:20:30 0 [ERROR] Aborting
Jan 24 09:20:30 Panel systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Jan 24 09:20:30 Panel systemd[1]: mariadb.service: Failed with result 'exit-code'.
Jan 24 09:20:30 Panel systemd[1]: Failed to start MariaDB 10.4.27 database server.```

Related

MySQL Workbench error: Lost connection to MySQL server; Terminal error: Error: 13 (Permission denied)

Connection to the MySQL db was working fine 2 weeks ago, and without any editing or development activity it dropped out. I am using SQL Workbench to access. I can connect to the AWS EC2 instance via the AWS dashboard as well as SSH tunnel, can view/access files from filezilla connection.
The server is running WordPress application for a single website (https://farrahstribute.com/)
By examining past answers I tried:
(1) editing /etc/my.cnf and added this line
bind-address=0.0.0.0
after adding I rebooted server, however nothing changed, error with db persisting, so I removed the code I added
(2) editing /etc/hosts.allow and added these lines
mysqld: ALL : ALLOW
mysqld-max: ALL : ALLOW
after editing I rebooted server, however nothing changed, error with db persisting, so I removed the code that I added
From the terminal - this command systemctl status mysqld -l gave this result:
● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2022-11-29 01:22:25 UTC; 14h ago
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
Process: 3152 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS (code=exited, status=1/FAILURE)
Process: 2906 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
Main PID: 3152 (code=exited, status=1/FAILURE)
Status: "Server shutdown complete"
Error: 13 (Permission denied)
Nov 29 01:22:20 ip-172-31-28-111.ec2.internal systemd[1]: Starting MySQL Server...
Nov 29 01:22:25 ip-172-31-28-111.ec2.internal mysqld[3152]: 2022-11-29T01:22:25.558132Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.31) starting as process 3152
Nov 29 01:22:25 ip-172-31-28-111.ec2.internal mysqld[3152]: 2022-11-29T01:22:25.615516Z 0 [ERROR] [MY-010187] [Server] Could not open file '/var/log/mysqld.log' for error logging: Permission denied
Nov 29 01:22:25 ip-172-31-28-111.ec2.internal mysqld[3152]: 2022-11-29T01:22:25.615582Z 0 [ERROR] [MY-010119] [Server] Aborting
Nov 29 01:22:25 ip-172-31-28-111.ec2.internal mysqld[3152]: 2022-11-29T01:22:25.640431Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.31) MySQL Community Server - GPL.
Nov 29 01:22:25 ip-172-31-28-111.ec2.internal systemd[1]: mysqld.service: main process exited, code=exited, status=1/FAILURE
Nov 29 01:22:25 ip-172-31-28-111.ec2.internal systemd[1]: Failed to start MySQL Server.
Nov 29 01:22:25 ip-172-31-28-111.ec2.internal systemd[1]: Unit mysqld.service entered failed state.
Nov 29 01:22:25 ip-172-31-28-111.ec2.internal systemd[1]: mysqld.service failed.
Also from the terminal ran: systemctl status mysqld.service with output:
● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2022-11-29 15:42:23 UTC; 2min 7s ago
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
Process: 7247 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS (code=exited, status=1/FAILURE)
Process: 7220 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
Main PID: 7247 (code=exited, status=1/FAILURE)
Status: "Server shutdown complete"
Error: 13 (Permission denied)
Nov 29 15:42:22 ip-172-31-28-111.ec2.internal systemd[1]: Starting MySQL Server...
Nov 29 15:42:23 ip-172-31-28-111.ec2.internal mysqld[7247]: 2022-11-29T15:42:23.292440Z 0 [System] [MY-010116] [Server...7247
Nov 29 15:42:23 ip-172-31-28-111.ec2.internal mysqld[7247]: 2022-11-29T15:42:23.296242Z 0 [ERROR] [MY-010187] [Server]...nied
Nov 29 15:42:23 ip-172-31-28-111.ec2.internal mysqld[7247]: 2022-11-29T15:42:23.297482Z 0 [ERROR] [MY-010119] [Server]...ting
Nov 29 15:42:23 ip-172-31-28-111.ec2.internal mysqld[7247]: 2022-11-29T15:42:23.297889Z 0 [System] [MY-010910] [Server...GPL.
Nov 29 15:42:23 ip-172-31-28-111.ec2.internal systemd[1]: mysqld.service: main process exited, code=exited, status=1/FAILURE
Nov 29 15:42:23 ip-172-31-28-111.ec2.internal systemd[1]: Failed to start MySQL Server.
Nov 29 15:42:23 ip-172-31-28-111.ec2.internal systemd[1]: Unit mysqld.service entered failed state.
Nov 29 15:42:23 ip-172-31-28-111.ec2.internal systemd[1]: mysqld.service failed.
I also examined the error log /var/log/nginx/farrahstribute_error.log and found this error:
2022/11/29 00:55:06 [error] 3111#3111: *39 access forbidden by rule, client: 54.152.255.102, server: farrahstribute.com, request: "GET /.env HTTP/1.1", host: "34.199.116.152"
the issue is this line:
[ERROR] [MY-010187] [Server] Could not open file '/var/log/mysqld.log' for error logging: Permission denied
Which is supported by this error message:
Error: 13 (Permission denied)
So the issue is permissions to the /var/log/mysqld.log file. What is needed is to create permissions for mysql to access this file - running this command with a server reboot solved this issue:
$ sudo chown mysql:mysql /var/log/mysqld.log

Mariadb not starting

systemctl status mariadb:
root#server:/var/www/pterodactyl# systemctl status mariadb
● mariadb.service - MariaDB 10.7.3 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: failed (Result: exit-code) since Fri 2022-02-18 11:25:25 UTC; 8min ago
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
Process: 21815 ExecStartPost=/etc/mysql/debian-start (code=exited, status=203/EXEC)
Process: 21814 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 21802 ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 21784 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status
Process: 21779 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 21771 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 21802 (code=exited, status=0/SUCCESS)
Status: "MariaDB server is down"
Feb 18 11:25:24 server.hostname.com mariadbd[21802]: 2022-02-18 11:25:24 0 [Note] InnoDB: FTS optimize thread exiting.
Feb 18 11:25:24 server.hostname.com mariadbd[21802]: 2022-02-18 11:25:24 0 [Note] InnoDB: Buffer pool(s) load aborted due to user instigated abort at 220218 11:25:24
Feb 18 11:25:24 server.hostname.com mariadbd[21802]: 2022-02-18 11:25:24 0 [Note] InnoDB: Dumping of buffer pool not started as load was incomplete
Feb 18 11:25:24 server.hostname.com mariadbd[21802]: 2022-02-18 11:25:24 0 [Note] InnoDB: Starting shutdown...
Feb 18 11:25:24 server.hostname.com mariadbd[21802]: 2022-02-18 11:25:24 0 [Note] InnoDB: Dumping of buffer pool not started as load was incomplete
Feb 18 11:25:25 server.hostname.com mariadbd[21802]: 2022-02-18 11:25:25 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
Feb 18 11:25:25 server.hostname.com mariadbd[21802]: 2022-02-18 11:25:25 0 [Note] InnoDB: Shutdown completed; log sequence number 8689222; transaction id 15019
Feb 18 11:25:25 server.hostname.com mariadbd[21802]: 2022-02-18 11:25:25 0 [Note] /usr/sbin/mariadbd: Shutdown complete
Feb 18 11:25:25 server.hostname.com systemd[1]: mariadb.service: Failed with result 'exit-code'.
Feb 18 11:25:25 server.hostname.com systemd[1]: Failed to start MariaDB 10.7.3 database server.
lines 1-26/26 (END)
Mariadb: 10.7.3
It stopped working when I installed and uninstalled DirectAdmin, reinstalling won't do anything

Mysql server not running on EC2 instance created from AMI

I have an EC2 instance in which Mysql server is working fine. I created an AMI from this and launched a new instance from this AMI. Now Mysql server is not able to start on this new AMI at all.
Below is the o/p:
ubuntu#ip-172-31-66-160:~$ sudo service mysql status
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: activating (start-post) (Result: exit-code) since Sun 2019-07-21 20:59:25 IST; 28s ago
Process: 1870 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
Process: 1862 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Main PID: 1870 (code=exited, status=1/FAILURE); : 1871 (mysql-systemd-s)
Tasks: 2
Memory: 3.2M
CPU: 247ms
CGroup: /system.slice/mysql.service
└─control
├─1871 /bin/bash /usr/share/mysql/mysql-systemd-start post
└─2529 sleep 1
Jul 21 20:59:25 ip-172-31-66-160 systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Jul 21 20:59:25 ip-172-31-66-160 systemd[1]: Stopped MySQL Community Server.
Jul 21 20:59:25 ip-172-31-66-160 systemd[1]: Starting MySQL Community Server...
Jul 21 20:59:26 ip-172-31-66-160 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
ubuntu#ip-172-31-66-160:~$ sudo service mysql restart
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
I tried everything but not able to get this started.
Closest threads which I came across and followed without any luck:
https://askubuntu.com/questions/916009/mysql-wont-start-because-of-apparmor
https://support.plesk.com/hc/en-us/articles/360004185293-Unable-to-start-MySQL-on-Ubuntu-AVC-apparmor-DENIED-operation-open-
Kindly help, I am just clueless about this.
Finally, I was able to make the server run on my new instance. What I was missing is that mysql also has many config options which gets changed over due course of time and they also depend on the EC2 instance config.
In my case, my new instance has very limited RAM and the mysql.cnf file was trying to allocate more RAM than the machine had hence it was failing. Somehow,
journalctl -xe
is giving a different error which is forcing me to think in wrong direction. I should have looked in to the mysql log file which quickly pointed me to the right issue.
I found below in the error log file which helped me to isolate / fix the issue:
2019-07-21T16:00:55.658985Z 0 [ERROR] InnoDB: mmap(137428992 bytes) failed; errno 12 2019-07-21T16:00:55.658993Z 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2019-07-21T16:00:55.658998Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2019-07-21T16:00:55.659003Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2019-07-21T16:00:55.659620Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2019-07-21T16:00:55.659628Z 0 [ERROR] Failed to initialize builtin plugins.
2019-07-21T16:00:55.659632Z 0 [ERROR] Aborting
So mysql wasn't able to start since requested amount of memory isn't present in the system. I commented all the memory allocation lines in my.cnf file (relying on the default settings) and then it worked.
innodb_buffer_pool_size = 16G
key_buffer_size = 2G
max_allowed_packet = 128M
group_concat_max_len = 50000
query_cache_size = 2147483648
query_cache_limit =67108864
Hope someone would find this information useful.

Unable to start MariaDB - Ubuntu 18.04

I installed mariadb in my system. But I forgot the root password... I tried to reset root password by following "https://tecadmin.net/steps-to-reset-mariadb-root-password-in-linux/" and mysql official documentation.But no success.
But when I install fresh mariadb it is working. As per the official documention, while we installing mariadb it will ask root password. However in my case, it is never asking. However the server working on fresh install.
When i stop and restart server, it is keep on failing..
The error message I am getting is
mariadb.service - MariaDB 10.4.6 database server Loaded: loaded
(/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf Active: failed (Result: timeout) since Mon 2019-07-15 14:52:16 IST; 1min 46s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/ Process: 10988 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER
$_WSREP_START_POSITION (code=exited, status=1/FAILURE) Process:
10829 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] &&
VAR= || VAR=/usr/bin/galera_recovery; [ $? -eq 0 ] && syst
Process: 10827 ExecStartPre=/bin/sh -c systemctl unset-environment
_WSREP_START_POSITION (code=exited, status=0/SUCCESS) Process: 10826 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d
/var/run/mysqld (code=exited, status=0/SUCCESS) Main PID: 10988
(code=exited, status=1/FAILURE) Status: "MariaDB server is down"
Jul 15 14:52:16 sonu mysqld[10988]: 2019-07-15 14:52:16 0 [Note]
InnoDB: Starting shutdown... Jul 15 14:52:16 sonu mysqld[10988]:
2019-07-15 14:52:16 0 [ERROR] Plugin 'InnoDB' init function returned
error. Jul 15 14:52:16 sonu mysqld[10988]: 2019-07-15 14:52:16 0
[ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. Jul
15 14:52:16 sonu mysqld[10988]: 2019-07-15 14:52:16 0 [Note] Plugin
'FEEDBACK' is disabled. Jul 15 14:52:16 sonu mysqld[10988]: 2019-07-15
14:52:16 0 [ERROR] Could not open mysql.plugin table. Some plugins may
be not loaded Jul 15 14:52:16 sonu mysqld[10988]: 2019-07-15 14:52:16
0 [ERROR] Failed to initialize plugins. Jul 15 14:52:16 sonu
mysqld[10988]: 2019-07-15 14:52:16 0 [ERROR] Aborting Jul 15 14:52:16
sonu systemd[1]: mariadb.service: Main process exited, code=exited,
status=1/FAILURE Jul 15 14:52:16 sonu systemd[1]: mariadb.service:
Failed with result 'timeout'. Jul 15 14:52:16 sonu systemd[1]: Failed
to start MariaDB 10.4.6 database server.
Please guide me towards resolution. Thanks for your time and waiting for your response.
Nagarajan S

Error: Trying to do i/o to a tablespace which exists without .ibd data file

I can't restart my mariadb server because I'm getting this error message and I ran systemctl status mariadb.service -l to see the full error message but it doesn't tell me what table and where the table can be found so I'm completely lost.
mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─limits.conf, migrated-from-my.cnf-settings.conf
Active: activating (auto-restart) (Result: signal) since Thu 2017-04-06 10:55:41 EDT; 4s ago
Process: 2324 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=killed, signal=ABRT)
Process: 2202 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
Process: 2199 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Main PID: 2324 (code=killed, signal=ABRT)
Status: "InnoDB: Error: Trying to do i/o to a tablespace which exists without .ibd data file. i/o type 10, space id 840720, page no 0, i/o length 16384 bytes"
CGroup: /system.slice/mariadb.service
Apr 06 10:55:41 www.ooples.com systemd[1]: Failed to start MariaDB database server.
Apr 06 10:55:41 www.ooples.com systemd[1]: Unit mariadb.service entered failed state.
Apr 06 10:55:41 www.ooples.com systemd[1]: mariadb.service failed.
This error was caused by an orphaned idb file in the /var/lib/mysql/database_name folder and I deleted the ibd file and I was able to start the mysql server