I restarted MySQL server in one of the nodes in Percona Cluster.
Since the restart took much time, I interrupted the process. I tried restarting the MySQL server again. I got the following error:
Stale sst_in_progress file in datadir
I followed this link, https://www.percona.com/forums/questions-discussions/percona-xtradb-cluster/46846-sql-cluster-issue-need-help-please, and deleted the sst_in_progress file as mentioned in it.
Now, when I try restarting the MySQL server, I am getting this:
● mysql.service - LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon
Loaded: loaded (/etc/init.d/mysql; bad; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2018-03-14 11:04:07 IST; 16min ago
Docs: man:systemd-sysv-generator(8)
Process: 23568 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)
Mar 14 11:04:00 systemd[1]: Starting LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon...
Mar 14 11:04:00 mysql[23568]: * Starting MySQL (Percona XtraDB Cluster) database server mysqld
Mar 14 11:04:00 /etc/init.d/mysql[23614]: MySQL PID not found, pid_file detected/guessed: /var/run/mysqld/mysqld.pid
Mar 14 11:04:07 mysql[23568]: * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
Mar 14 11:04:07 mysql[23568]: ...fail!
Mar 14 11:04:07 systemd[1]: mysql.service: Control process exited, code=exited status=1
Mar 14 11:04:07 systemd[1]: Failed to start LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon.
Mar 14 11:04:07 systemd[1]: mysql.service: Unit entered failed state.
Mar 14 11:04:07 systemd[1]: mysql.service: Failed with result 'exit-code'.
One more thing is that no log is getting written to mysql-error.log file during restart because of which I am not able to continue debugging.
The best solution here, without being able to see more information, is to simply rm -rf $datadir and start the node back up. It will indeed SST which, depending on the dataset, will take a while. Estimate 1 hour for every 100GB of data over gigE.
If joiner node taking too much time so you can increase gcache.size upto 1 GB then restart joiner node, so if data is already there on new node so it will choose IST instead of SST.
set in my.cnf
wsrep_provider_options="gcache.size=1G"
Related
I wanted to import a sql dump
mysql -u username mydb < file.sql
And of course i get the classic ERROR 2006 (HY000) at line 1701: MySQL server has gone away message. No big deal i know that i need to increase the packet size. So i edit the /etc/mysql/mysql.cnf file and add the line max_allowed_packet=64M. But when i try to start the mysql server i get an error.
mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2019-01-21 16:27:47 CET; 15s ago
Process: 9151 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid (code=exited, status=1/FAILURE)
Process: 9112 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Main PID: 27620 (code=exited, status=0/SUCCESS)
Jan 21 16:27:47 aleksandar-pc systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Jan 21 16:27:47 aleksandar-pc systemd[1]: mysql.service: Scheduled restart job, restart counter is at 5.
Jan 21 16:27:47 aleksandar-pc systemd[1]: Stopped MySQL Community Server.
Jan 21 16:27:47 aleksandar-pc systemd[1]: mysql.service: Start request repeated too quickly.
Jan 21 16:27:47 aleksandar-pc systemd[1]: mysql.service: Failed with result 'exit-code'.
If i remove the code added for packets, the server can be started normally.
Can anyone help me with this, i really need to import this large db and don't know how to fix this mysql error.
Mysql verison: Ver 14.14 Distrib 5.7.25, for Linux (x86_64)
Linux: Ubuntu 18.04 LTS
Ok i forgot this [mysqld] above:
[mysqld]
max_allowed_packet=128M
I'm using Ubuntu 16.04 and installed MySQL version 5.7 via apt-get in AWS.
After that, I've managed to upgrade the MySQL to Galera Cluster by following the Documentation from the official website using apt-get install galera-3 galera-arbitrator-3 mysql-wsrep-5.6
After that I've configured the file /etc/mysql/conf.d/galera.cnf to be:
[mysqld]
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0
# Galera Provider Configuration
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
# Galera Cluster Configuration
wsrep_cluster_name="test_cluster"
wsrep_cluster_address="gcomm://this_node_ip,second_ip,third_ip"
# Galera Synchronization Configuration
wsrep_sst_method=rsync
# Galera Node Configuration
wsrep_node_address="this_node_ip"
wsrep_node_name="this_node_name"
The step above is repeated for the other 2 nodes (with it's own IP address). And stop the mysql with the following command sudo service mysql stop for each of the node.
And in the first node, I'm trying to start with the following command:
sudo service mysql start --wsrep-new-cluster
But it returns: Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
In the systemctl status mysql.service, it says:
● mysql.service - LSB: start and stop MySQL
Loaded: loaded (/etc/init.d/mysql; bad; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2018-02-09 15:53:16 +08; 33s ago
Docs: man:systemd-sysv-generator(8)
Process: 6264 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)
Feb 09 15:53:15 systemd[1]: Starting LSB: start and stop MySQL...
Feb 09 15:53:15 mysql[6264]: Starting MySQL
Feb 09 15:53:15 mysql[6264]: .180209 15:53:15 mysqld_safe Can't log to error log and syslog at the same time. Remove all --log-error configuration options for --syslog to take effect.
Feb 09 15:53:15 mysql[6264]: 180209 15:53:15 mysqld_safe Directory '/var/run/mysqld' for UNIX socket file don't exists.
Feb 09 15:53:16 mysql[6264]: * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
Feb 09 15:53:16 systemd[1]: mysql.service: Control process exited, code=exited status=1
Feb 09 15:53:16 systemd[1]: Failed to start LSB: start and stop MySQL.
Feb 09 15:53:16 systemd[1]: mysql.service: Unit entered failed state.
Feb 09 15:53:16 systemd[1]: mysql.service: Failed with result 'exit-code'.
Hope these information helps.
I have Percona MySQL server 5.7 running under CentOS 7 and although mysql is running without any noticeable errors, it is filling my /var/log/messages with the following every ten seconds:
Nov 15 10:07:27 server systemd: mysqld.service holdoff time over, scheduling restart.
Nov 15 10:07:27 server systemd: Starting MySQL Percona Server...
Nov 15 10:07:27 server mysqld_safe: 171115 10:07:27 mysqld_safe Adding '/usr/lib64/libjemalloc.so.1' to LD_PRELOAD for mysqld
Nov 15 10:07:27 server mysqld_safe: 171115 10:07:27 mysqld_safe Logging to '/var/lib/mysql/server.local.err'.
Nov 15 10:07:27 server mysqld_safe: 171115 10:07:27 mysqld_safe A mysqld process already exists
Nov 15 10:07:27 server systemd: mysqld.service: main process exited, code=exited, status=1/FAILURE
Nov 15 10:07:28 server systemd: Failed to start MySQL Percona Server.
Nov 15 10:07:28 server systemd: Unit mysqld.service entered failed state.
Nov 15 10:07:28 server systemd: Triggering OnFailure= dependencies of mysqld.service.
Nov 15 10:07:28 server systemd: mysqld.service failed.
Nov 15 10:07:28 server systemd: Started Service Status Monitor.
Nov 15 10:07:28 server systemd: Starting Service Status Monitor...
Even though it's stating in there that it failed to start the Percona server, it appears to be working as my website is still doing mysql queries. I know very little about mysql admin and was hoping a mysql guru could shed some light on what is happening.
The clue is here: "A mysqld process already exists". It can't start mysqld because another mysqld process is already running, and using the same port. You need to kill that process before the one you tried to start can start.
Re your comment:
Since this is CentOS 7, I assume mysql.service is being called by systemd.
In my experience, if you start mysqld "ad hoc" without using systemd, then systemd has no idea that it's running, and tries to start mysqld on its own. Systemd also cannot shut down an instance of mysqld unless it started that instance.
the mysqld process is active,ps -ef |grep mysqld ,kill -9 {}
For a particular project I wanted to have Mint (Sonja) appliance with all MariaDB and only MariaDB doing MySQL's work, without a hint of MySQL so I know I'm working with MariaDB.
The appliance is one where I've installed multiple open source projects that power their own website (Alfresco, Request Tracker, SuiteCRM, etc.), and all of them seem to have worked either with MariaDB, or just used Postgres without interesting difficulty, and up until I tried to create a new MariaDB database to add a clone of a specific Wordpress site, MariaDB worked predictably well with no headaches on the same system. Until now, where I can't seem to find a pulse.
The MariaDB troubleshooting page confirmed what I'd found in my investigations (in particular, a ps wwaux | grep mysql only turned up the grep process). The basic problem as I'd encountered it was when I tried to log in to create a database:
monk#toolchest ~ $ mysql -uroot -p
Enter password:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket
'/var/run/mysqld/mysqld.sock' (2 "No such file or directory")
I've never gotten involved with MySQL's or MariaDB's /etc configuration files, but I looked briefly. They appeared sane to my uneducated eye.
Neither an aptitude reinstall mariadb-server nor a service mysql start produce any changes so far as I could tell.
For service mysql start, I got:
# service mysql start
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
The details for systemctl status mysql.service were:
● mysql.service - LSB: Start and stop the mysql database server daemon
Loaded: loaded (/etc/init.d/mysql; bad; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2017-07-14 18:16:38 EEST; 1min 11s ago
Docs: man:systemd-sysv-generator(8)
Process: 5011 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)
Jul 14 18:16:38 toolchest /etc/init.d/mysql[5479]: 0 processes alive and '/usr/bin/mysql
Jul 14 18:16:38 toolchest /etc/init.d/mysql[5479]: [61B blob data]
Jul 14 18:16:38 toolchest /etc/init.d/mysql[5479]: error: 'Can't connect to local MySQL
Jul 14 18:16:38 toolchest /etc/init.d/mysql[5479]: Check that mysqld is running and that
Jul 14 18:16:38 toolchest /etc/init.d/mysql[5479]:
Jul 14 18:16:38 toolchest mysql[5011]: ...fail!
Jul 14 18:16:38 toolchest systemd[1]: mysql.service: Control process exited, code=exited
Jul 14 18:16:38 toolchest systemd[1]: Failed to start LSB: Start and stop the mysql data
Jul 14 18:16:38 toolchest systemd[1]: mysql.service: Unit entered failed state.
Jul 14 18:16:38 toolchest systemd[1]: mysql.service: Failed with result 'exit-code'.
The recent (i.e. non-cronned) content of journalctl -xe ran:
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has failed.
--
-- The result is failed.
Jul 14 18:16:38 toolchest systemd[1]: mysql.service: Unit entered failed state.
Jul 14 18:16:38 toolchest systemd[1]: mysql.service: Failed with result 'exit-code'.
/etc/mysql/mariadb.cnf and /etc/mysql/my.cnf both read (comments stripped):
[client-server]
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mariadb.conf.d/
/etc/mysql/my.cnf.fallback dropps the latter !includedir. Commenting out the latter /etc/mysql/my.cnf to match the fallback file produced identical results.
I can edit my question to include more of /etc/mysql/*, but I wanted to ask. Error messages similar to the ERROR 2002 are to be found in questions like Can't connect to local MySQL server through socket '/var/mysql/mysql.sock' (38), but these appear to be MySQL rather than MariaDB, and I suspect this may be a MariaDB-specific fluke.
So I've been working on a rails server with MySQL and suddenly can't access my database. When I try to log in with mysql -u root -p I get ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2). I've read most of the forums I can find and looked through all the logs trying to reset mysqld.sock. I've tried reinstalling and repackaging unsuccessfully. The weird thing is that var/run/mysqld/ doesn't even exist after reinstalling.
So I decided to just back up my databases and overhaul everything. I cleaned out both apache and mysql with apt-get remove --purge and reinstalled. All went fine, apache launched fine, and then tried to relaunch mysql with systemctl start mysql which gave a prompt to check the log which says:
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: inactive (dead) (Result: exit-code) since Fri 2016-12-16 23:56:19 UTC; 17s ago
Process: 15690 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=1/FAILURE)
Dec 16 23:56:19 ip-172-31-0-55 systemd[1]: mysql.service: Control process exited, code=exited status=1
Dec 16 23:56:19 ip-172-31-0-55 systemd[1]: Failed to start MySQL Community Server.
Dec 16 23:56:19 ip-172-31-0-55 systemd[1]: mysql.service: Unit entered failed state.
Dec 16 23:56:19 ip-172-31-0-55 systemd[1]: mysql.service: Failed with result 'exit-code'.
Dec 16 23:56:19 ip-172-31-0-55 systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Dec 16 23:56:19 ip-172-31-0-55 systemd[1]: Stopped MySQL Community Server.
Dec 16 23:56:19 ip-172-31-0-55 systemd[1]: mysql.service: Start request repeated too quickly.
Dec 16 23:56:19 ip-172-31-0-55 systemd[1]: Failed to start MySQL Community Server.
Update - I created the folder mysqld, which seems to allow sudo mysqld --initialize to run. Unfortunately this yielded the error
2016-12-17T00:16:36.298825Z 0 [ERROR] Can't change data directory owner to mysql
2016-12-17T00:16:36.299212Z 0 [ERROR] Aborting
So no party yet. Any thoughts would be hugely appreciated.
edit /etc/mysql/conf.d/mysql.cnf :
sudo nano /etc/mysql/conf.d/mysql.cnf
add this line :
socket=/var/run/mysqld/mysqld.sock
then restart mysql service :
sudo service mysql restart