I get this error when I try to start mysql with service mysql start under a Debian machine (a Docker container actually):
[FAIL] Starting MySQL database server: mysqld . . . . . failed!
I think the problem is in the Debian itself, but I'll explain the context just in case. I have the following context:
A Debian Docker container based in this image running over an Ubuntu 18.04 host.
This container was runninng properly in previous Ubuntu 16.04 and actually is still running in another computer with Ubuntu 16.04. Of course, the container has been rebuilt in new OS.
If I try sh -x /etc/init.d/mysql start to get some feedback info all I get is messages like:
+ echo -n Starting MySQL database server: mysqld
Starting MySQL database server: mysqld+ log_daemon_msg_post Starting MySQL database server mysqld
+ :
+ mysqld_status check_alive nowarn
+ /usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping
+ ping_output=/usr/bin/mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
I'm quite stucked in this, I've been several days searching and moving in circles to no avail.
EDIT: I'll add some tries and fails I did in order to get some output or logging:
$/usr/bin/mysqld_safe -v
181010 15:30:51 mysqld_safe Can't log to error log and syslog at the same time. Remove all --log-error configuration options for --syslog to take effect.
181010 15:30:51 mysqld_safe Logging to '/var/log/mysql/error.log'.
181010 15:30:51 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
181010 15:30:51 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
$mysqld --print-defaults
(empty output)
$mysqld --v
(empty output)
$tail -f /var/log/mysql/error.log
(last log entry is 1 month old)
180921 10:59:40 InnoDB: highest supported file format is Barracuda.
180921 10:59:40 InnoDB: Waiting for the background threads to start
180921 10:59:41 InnoDB: 5.5.60 started; log sequence number 1595675
ERROR: 1050 Table 'plugin' already exists
180921 10:59:41 [ERROR] Aborting
180921 10:59:41 InnoDB: Starting shutdown...
180921 10:59:42 InnoDB: Shutdown completed; log sequence number 1595675
180921 10:59:42 [Note] /usr/sbin/mysqld: Shutdown complete
Related
I cannot use MySQL anymore in my Docker container:
root#mysql-container:/# mysql -uroot -proot
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
mysqld is running:
root#mysql-spirit-ssl:/etc/mysql/conf.d# /etc/init.d/mysql start
[info] A MySQL Server is already started.
Trying to stop mysqld timed out:
root#mysql-container:/# /etc/init.d/mysql stop
............................................................[info] Attempt to shutdown MySQL Community Server 5.7.17 timed out.
So I tried to start using the mysqladmin way:
root#mysql-container:/# /usr/bin/mysqladmin --port=8889 -u root shutdown
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)'
So I checked that MySQL daemon is running:
root#mysql-container:/# ps -eax
PID TTY STAT TIME COMMAND
1 ? Ssl 0:01 mysqld
And that socket exists:
root#mysql-container:/# ls -l /var/run/mysqld/mysqld.sock
-rwxrwxrwx. 1 mysql mysql 0 Jan 4 10:12 /var/run/mysqld/mysqld.sock
I already tried to:
restart my Docker container
comment bind address in my.cnf and restart my Docker container
kill mysqld process => does not work, process is still listed by ps -eax
recreate my Docker container
restart Docker
restart the server
delete pid and sock files, and /etc/init.d/mysql start
Result of cat /var/log/mysql/error.log:
2018-02-27T15:27:35.966028Z 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
2018-02-27T15:27:35.966061Z 0 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
However I cannot kill that mysqld process, either with pkill mysqld, kill -9 1 or initctl --system stop mysql.
Could this be related to Docker?
Remark: The MySQL daemon could not be killed because it was owned by Docker user systemd+ and it was the entry point of the container. Indeed mysqld was process with PID 1. This means that MySQL daemon could be restarted by simply restarting the Docker container, and that MySQL configuration could be modified in between.
I noticed in MySQL logs tail -f /var/log/mysql/error.log that a data recovery was triggered on daemon start due to an anomaly detected during internal log scan: the database was not closed properly. However the recovery could not repair the data and an intentional crash was performed. As a consequence, the container was restarted and so on. This infinite loop prevented mysqld to start and the socket to be used by the client mysql.
1) This configuration of /etc/mysql/conf.d/my.cnf enabled to skip the recovery:
[mysqld]
innodb_force_recovery=4
and to use mysql client with socket to dump important schemas and/or delete corrupted schemas.
Do not forget to remove this line from my.cnf after you're done!
2) Perfoming a mysql upgrade and repair could also have been beneficial:
docker exec -it mysql-container mysql_upgrade -u root -p --force
mysqlcheck -u root -p --auto-repair --check --all-databases
Restarting the Docker container is necessary after this step.
3) Also, deleting MySQL internal logs (that were scanned and triggered the recovery) was necessary:
cd /var/lib/mysql/mysql/
rm ibdata1 ib_logfile0 ib_logfile1
Now I can use MySQL again, from inside and outside the container.
I am on Mac 10.11.3 (15D21)
About Docker I have:
$ docker version
Client:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 20f81dd
Built: Thu Mar 10 21:49:11 2016
OS/Arch: darwin/amd64
Server:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 20f81dd
Built: Thu Mar 10 21:49:11 2016
OS/Arch: linux/amd64
For Mysql
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
mysql latest 7a5866c2edbf 12 days ago 361.3 MB
...
I did the following:
mkdir -p /Users/me/docker/mysql/data
chmod -R 777 /Users/me/docker/mysql/data
(Even with sudo for each command)
It from Pro Docker and from: Permission denied when mounting Docker volume in OSX
Even doing:
docker -v /Users/me/docker/mysql/data:/var/lib/mysql mysql chown -R mysql /var/lib/mysql
It from: Database Fails to Start - Host Directory as a Data Volume
When I execute: (showing in multiple lines for better visualisation)
docker run -v /Users/me/docker/mysql/data:/var/lib/mysql --name manolitomysql
-e MYSQL_DATABASE='mysqldb' -e MYSQL_USER='mysql' -e MYSQL_PASSWORD='mysql'
-e MYSQL_ALLOW_EMPTY_PASSWORD='yes' -e MYSQL_ROOT_PASSWORD='' -d mysql
I always get:
$ docker logs manolitomysql
Initializing database
2016-03-27T16:04:33.520170Z 0 [Warning] Setting lower_case_table_names=2 because file system for /var/lib/mysql/ is case insensitive
2016-03-27T16:04:35.089907Z 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2016-03-27T16:04:35.090006Z 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2016-03-27 16:04:35 0x7f061a050740 InnoDB: Assertion failure in thread 139664183068480 in file fil0fil.cc line 868
InnoDB: Failing assertion: success
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
16:04:35 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68188 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
mysqld(my_print_stacktrace+0x2c)[0xe57c7c]
mysqld(handle_fatal_signal+0x459)[0x7896b9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7f0619c2d8d0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f061841b067]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f061841c448]
mysqld[0x75fc29]
mysqld[0x1180f68]
mysqld(_Z40fil_open_log_and_system_tablespace_filesv+0xe4)[0x1185a74]
mysqld[0x107afe0]
mysqld(_Z34innobase_start_or_create_for_mysqlv+0x3692)[0x107eb12]
mysqld[0xf4a05d]
mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x51)[0x7d3981]
mysqld[0xc53716]
mysqld(_Z11plugin_initPiPPci+0x5c8)[0xc5a5c8]
mysqld[0x782164]
mysqld(_Z11mysqld_mainiPPc+0x7b2)[0x7835d2]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f0618407b45]
mysqld[0x77a274]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
What extra configuration is need it?
For the audience, the following works:
1 Create a Dockerfile with
FROM mysql:latest
RUN deluser mysql
RUN useradd mysql
RUN mkdir -p /Users/me/docker/mysql/data
RUN chmod -R 777 /Users/me/docker/mysql/data
2 Create a new image using the previous Dockerfile
3 Run your new container using the previous new image
You may have better control using a Dockerfile with the following instructions:
RUN useradd mysql
RUN mkdir /var/lib/mysql && chown -R mysql:mysql /var/lib/mysql
start mysql server as root failed.
1.service mysqld start
Starting mysqld: [FAILED]
2.mysqld
error log:
2016-02-23T01:19:16.961857Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-02-23T01:19:16.963350Z 0 [Note] mysqld (mysqld 5.7.10) starting as process 20924 ...
2016-02-23T01:19:16.965669Z 0 [ERROR] Fatal error: Please read "Security" section of the manual to find out how to run mysqld as root!
2016-02-23T01:19:16.965715Z 0 [ERROR] Aborting
2016-02-23T01:19:16.965737Z 0 [Note] Binlog end
2016-02-23T01:19:16.965818Z 0 [Note] mysqld: Shutdown complete
but can start mysql service as mysql user, or use mysqld_safe as root can start too. what's wrong?
/etc/init.d/msyqld start
MySQL Daemon failed to start.
Starting mysqld: [FAILED]
if ! /bin/kill -0 $safe_pid 2>/dev/null; then
echo "MySQL Daemon failed to start."
ret=1
break
fi
... ...
if [ $ret -eq 0 ]; then
action $"Starting $prog: " /bin/true
touch $lockfile
else
action $"Starting $prog: " /bin/false
fi
I found this Code Snippets in the script /etc/init.d/mysqld. but don't know why?
Add this line to /etc/my.cnf in [mysqld] section
explicit_defaults_for_timestamp = 1
Check this link
Since you can start as the mysql user, it is likely you have it currently configured to run as root.
Mysql's reference manual says:
On Linux, for installations performed using a MySQL repository, RPM packages, or Debian packages, the MySQL server mysqld should be started by the local mysql operating system user. Starting by another operating system user is not supported by the init scripts that are included as part of the installation.
/etc/selinux/config
#SELINUX=enforcing
SELINUX=disabled
I have a little Problem with a MySQL Server running on a Debian Server managed by Plesk. If I try to start the MySQL Server the following message is logged at MySQL_error.log:
150518 02:54:42 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150518 02:55:15 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150518 2:55:15 [Warning] Using unique option prefix w instead of wait_timeout is deprecated and will be removed in a future release. Please use the full name instead.
150518 2:55:15 [ERROR] /usr/sbin/mysqld: option '--wait_timeout' requires an argument
150518 2:55:15 [ERROR] Aborting
150518 02:55:15 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
But I never modified my config file and I did never set a --wait_timeout argument.
Can someone help me out?
Pay attention for this string:
"Using unique option prefix w instead of wait_timeout is deprecated"
Check that there is no "w" command argument or option in my.cnf.
I'm having issues starting MySQL after it randomly stopped working a few minutes ago. I'm getting this error while trying to connect:
Connect failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
So I tried to restart MySQL (this had worked for me before) and I got this:
Stopping mysqld: [ OK ]
MySQL Daemon failed to start.
Starting mysqld: [FAILED]
Here's my error log:
130414 20:03:45 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130414 20:03:45 [Note] Plugin 'FEDERATED' is disabled.
130414 20:03:45 InnoDB: The InnoDB memory heap is disabled
130414 20:03:45 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130414 20:03:45 InnoDB: Compressed tables use zlib 1.2.5
130414 20:03:45 InnoDB: Using Linux native AIO
130414 20:03:45 InnoDB: Initializing buffer pool, size = 128.0M
130414 20:03:45 InnoDB: Completed initialization of buffer pool
InnoDB: The first specified data file ./ibdata1 did not exist:
InnoDB: a new database to be created!
130414 20:03:45 InnoDB: Setting file ./ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
130414 20:03:46 InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 5 MB
InnoDB: Database physically writes the file full: wait...
130414 20:03:46 InnoDB: Log file ./ib_logfile1 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile1 size to 5 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: 127 rollback segment(s) active.
InnoDB: Creating foreign key constraint system tables
Can anyone offer some tips? I'm pretty noobish at this server stuff :P
Many thanks!
I fixed this by restarting my EC2 instance.
Restarting the instance helps, but it doesn't resolve it permanently.
This is what I used to resolve it:
http://www.prowebdev.us/2012/05/amazon-ec2-linux-micro-swap-space.html
Amazon EC2 Micro Instance Swap Space - Linux
I have a Amazon EC2 Linux Micro instance. Since Micro instances have only 613MB of memory, MySQL crashed every now and then. After a long search about MySQL, Micro Instance and Memory Managment I found out there is no default SWAP space for Micro instance. So if you want to avoid the crash you may need to setup a swap space for your micro instance. Actually performance wise is better to enable swap.
Steps below show how to make a swap space for your Micro instance. I assume you have AWS Account with a Micro instance running.
1.) Run dd if=/dev/zero of=/swapfile bs=1M count=1024
2.) Run mkswap /swapfile
3.) Run swapon /swapfile
4.) Add this line /swapfile swap swap defaults 0 0 to /etc/fstab
Step 4 is needed if you would like to automatically enable swap file after each reboot.
Some useful command related to SWAP space:
$ swapon -s
$ free -k
$ swapoff -a
$ swapon -a
References:
http://www.thegeekstuff.com/2010/08/how-to-add-swap-space/
http://cloudstory.in/2012/02/getting-the-best-out-of-amazon-ec2-micro-instances/
http://cloudstory.in/2012/02/adding-swap-space-to-amazon-ec2-linux-micro-instance-to-increase-the-performance/
http://aws.amazon.com/ec2/instance-types/
In My case i solved it by clearing some space in server.
I was taking backup of database on daily basis which results in consumption of lots of space.
I deleted all those backup and started mysql.
I also had faced the same challenge this is what I explored, the reason for restarting can be one of the below:
There have been updates applied at your EC2 instance, causing the MySql service to stop and not able to restart again.
There may be other processes running along which cause a crunch in memory, thus not allowing the MySql to restart.
To Tackle this you can either use one of the below:
If your MySql service is going down because of memory issue -- Upgrade your instance.
If your mysqld restart command fails, try to restart the httpd service first and then your mysql service. Here are the commands:
sudo service httpd restart
sudo service mysqld restart
If none of the above work restart your EC2 Instance. Not a permanent fix, but help if you want your services to be up and running, and later want to do an RCA of the issue
If you want you can create a script shown below, and execute it after via a cron-job every 5-10 mins depending on your requirement :
#!/bin/bash
dateFormat=`date "+%Y-%m-%d %T"`
log_file_path="/home/ec2-user/mysql_restart_log.dat"
sudo service mysqld status | grep "is running"
if [ $? -ne 0 ]; then
echo "HTTPD restart attempted ${dateFormat}" >> ${log_file_path}
sudo service httpd restart
if [ $? -ne 0 ]; then
echo "HTTPD restart failed... ${dateFormat}" >> ${log_file_path}
else
echo "HTTPD restart success... ${dateFormat}" >> ${log_file_path}
fi
echo "MYSQL restart attempted ${dateFormat}" >> ${log_file_path}
sudo service mysqld restart
if [ $? -ne 0 ]; then
echo "MYSQL restart failed... ${dateFormat}" >> ${log_file_path}
else
echo "MYSQL restart success... ${dateFormat}" >> ${log_file_path}
fi
fi
Also found this link helpful: mysql on amazon linux - MySQL Daemon failed to start
I had this issue too on an EC2 micro instance I was running WordPress on. Turned out that Apache was being a memory hog, which was not allowing mysqld to start/re-start. Once I edited httpd.conf to tame apache's memory use, httpd started with no issues.
I have seen this issue multiple times on multiple AWS servers, main root cause in my cases was lack of space. I deleted some unused files from the server which were taking space and problem was resolved.
This issue of lack of space will also stop your apache from running.