I'm having trouble deleting a table in MySQL v8.0 (on Windows 10) either from MySQL Workbench or via Python script (using mysql-connector-python). In both cases, the drop table command times out with "Error Code: 2013. Lost connection to MySQL server during query"
I previously set DBMS connection read timeout interval to 500 sec to try and work around this, but no luck.
The table in question has several hundred rows of data, and the entire .ibd file is 176kb. I suppose deleting the .ibd file directly isn't the greatest database practice?
I can create a new table and delete it, no problem. I'm running MySQL server locally.
Any suggestions on what to try next?
#obe's suggestion to restart the server resolved the issue. So it seems like that particular table got locked due to access from both Workbench and python. Database itself was not locked, since I could create/drop other tables.
FINAL UPDATE: We fixed this problem by finding a way to accomplish our goals without forking. But forking was the cause of the problem.
---Original Post---
I'm running a ruby on rails stack, our mysql server is separate, but housed at the same site as our app servers. (we've tried swapping it out for a different mysql server with double the specs, but no improvement was seen.
during business hours we get a handful of these from no particular query.
ActiveRecord::StatementInvalid: Mysql2::Error: Lost connection to MySQL server during query
most of the queries that fail are really simple, and there seems to be no pattern between one query and another. This all started when I upgraded from Rails 4.1 to 4.2.
I'm at a loss as to what to try. Our database server is less than 5% CPU throughout the day. I do get bug reports from users who have random interactions fail due to this, so it's not queries that have been running for hours or anything like that, of course when they retry the exact same thing it works.
Our servers are configured by cloud66.
So in short: our mysql server is going away for some reason, but it's not because of lack of resources, it's also a brand new server as we migrated from another server when this problem started.
this also happens to me on localhost while developing features sometimes, so I don't believe it's a load issue.
We're running the following:
ruby 2.2.5
rails 4.2.6
mysql2 0.4.8
UPDATE: per the first answer below I increased our max_connections variable to 500 last night, and confirmed the increase via
show global variables like 'max_connections';
I'm still getting dropped connection, the first one today was dropped only a few minutes ago....
ActiveRecord::StatementInvalid: Mysql2::Error: Lost connection to MySQL server during query
I ran select * from information_schema.processlist; and I got 36 rows back. Does this mean my app servers were running 36 connections at that moment? or can a process be multiple connections?
UPDATE: I just set net_read_timeout = 60 (it was 30 before) I'll see if that helps
UPDATE: It didn't help, I'm still looking for a solution...
Heres my Database.yml with credentials removed.
production:
adapter: mysql2
encoding: utf8
host: localhost
database:
username:
password:
port: 3306
reconnect: true
The connection to MySQL can be disrupted by a number of means, but I would recommend revisiting Mario Carrion's answer since it's a very wise answer.
It seems likely that connection is disrupted because it's being shared with the other processes, causing communication protocol errors...
...this could easily happen if the connection pool is process bound, which I believe it is, in ActiveRecord, meaning that the same connection could be "checked-out" a number of times simultaneously in different processes.
The solution is that database connections must be established only AFTER the fork statement in the application server.
I'm not sure which server you're using, but if you're using a warmup feature - don't.
If you're running any database calls before the first network request - don't.
Either of these actions could potentially initialize the connection pool before forking occurs, causing the MySQL connection pool to be shared between processes while the locking system isn't.
I'm not saying this is the only possible reason for the issue, as stated by #sloth-jr, there are other options... but most of them seem less likely according to your description.
Sidenote:
I ran select * from information_schema.processlist; and I got 36 rows back. Does this mean my app servers were running 36 connections at that moment? or can a process be multiple connections?
Each process could hold a number of connections. In your case, you might have up to 500X36 connections. (see edit)
In general, the number of connections in the pool can often be the same as the number of threads in each process (it shouldn't be less than the number of thread, or contention will slow you down). Sometimes it's good to add a few more depending on your application.
EDIT:
I apologize for ignoring the fact that the process count was referencing the MySQL data and not the application data.
The process count you showed is the MySQL server data, which seems to use a thread per connection IO scheme. The "Process" data actually counts active connections and not actual processes or threads (although it should translate to the number of threads as well).
This means that out of possible 500 connections per application processes (i.e., if you're using 8 processes for your application, that would be 8X500=4,000 allowed connections) your application only opened 36 connections so far.
This indicates a timeout error. It's usually a general resource or connection error.
I would check your MySQL config for max connections on MySQL console:
show global variables like 'max_connections';
And ensure the number of pooled connections used by Rails database.yml is less than that:
pool: 10
Note that database.yml reflects number of connections that will be pooled by a single Rails process. If you have multiple processes or other servers like Sidekiq, you'll need to add them together.
Increase max_connections if necessary in your MySQL server config (my.cnf), assuming your kit can handle it.
[mysqld]
max_connections = 100
Note other things might be blocking too, e.g. open files, but looking at connections is a good starting point.
You can also monitor active queries:
select * from information_schema.processlist;
as well as monitoring the MySQL slow log.
One issue may be a long-running update command. If you have a slow-running command that affects a lot of records (e.g. a whole table), it might be blocking even the simplest queries. This means you could see random queries timeout, but if you check MySQL status, the real cause is another long-running query.
Things you did not mention but you should take a look:
Are you using unicorn? If so, are your reconnecting and disconnecting in your after_fork and before_fork?
Is reconnect: true set in your database.yml configuration?
Well,at first glance this sounds like your webserver is keeping the mysql sessions open and sometimes a user runs into a timeout. Try disabling the keep mysql sessions alive.
It will be a hog but you only use 5% ...
other tipps:
Enable the mysql "Slow Query Log" and take a look.
write a short script which pulls and logs the mysql processlist every minute and cross check the log with timeouts
look at the pool size in your db connection or set one!
http://guides.rubyonrails.org/configuring.html#database-pooling
should be equal to the max-connections mysql likes to have!
Good luck!
Find out if your database is limited in terms of multiple connections. Because normally a SQL database is supposed to have more than one active connection.
(Contact your network provider)
Would you mind posting some of your queries? The MySQL documentation has this to say about it:
https://dev.mysql.com/doc/refman/5.7/en/error-lost-connection.html
TL;DR:
Network problems; are any of your boxes renewing leases
periodically, or experiencing other network connection errors
(netstat / ss), firewall timeouts, etc. Not sure how managed your
hosts are by cloud66....
Query timed out. This can happen if you've got commands backed up
behind blocking statements (eg, alters/locking backups on MyISAM
tables). How simple are your queries? No cartesian products in-play?
EXPLAIN query could help.
Exceeding MAX_PACKET_SIZE. Are you storing pictures, video content, etc.?
There are lots of possibilities here, and without more information, will be difficult to pinpoint this.
Would look first at mysql_error.log, then work your way from the DB server back to your application.
UPDATE: this didn't work.
Heres the solution, special thanks to #Myst for pointing out that forking can cause issues, I had no idea to look at this particular code. As the errors seemed random because we forked in this fashion in several places.
It turns out that when I was forking processes, rails was using the same database connection for all forked processes, This created a situation where when one of the processes (the parent process?) terminated the database connection, the remaining process would have its connection interrupted.
The solution was to change this code:
def recalculate_completion
Process.fork do
if self.course
self.course.user_groups.includes(user:[:events]).each do |ug|
ug.recalculate_completion
end
end
end
end
into this code:
def recalculate_completion
ActiveRecord::Base.remove_connection
Process.fork do
ActiveRecord::Base.establish_connection
if self.course
self.course.user_groups.includes(user:[:events]).each do |ug|
ug.recalculate_completion
end
end
ActiveRecord::Base.remove_connection
end
ActiveRecord::Base.establish_connection
end
Making this change stopped the errors from our servers and everything appears to be working well now. If anyone has any more info as to why this worked I would be happy to hear it, as I would like to have a deeper understanding of this.
Edit: it turns out this didn't work either.... we still got dropped connections but not as often.
If you have query cache enabled, please reset it and it should work.
RESET QUERY CACHE;
Firstly this isn't a duplicate of this question because I'm not behind a load balancer at the moment. Also randomly restarting the instance isn't a satisfactory solution either.
I moved our store from using a local mysql db on the EC2 to RDS recently and we're getting some errors on the backend when they're moving categories around / adding products. Mostly they look like SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction but I've seen other variations of PDO errors.
How would one go about eliminating these issues?
I've read suggestions to just 'keep retrying' but this seems stupid.
I've tried tinkering with innodb_lock_wait_timeout and others but doesn't seem to resolve it.
I have finally disabled magento logging to the DB - will see if this helps going forward. Unfortunately I have to wait a while to let the staff tinker on the backend and then see if any fix has an effect.
Thanks
UPDATE
Now also seeing a bunch of these
User Error: DDL statements are not allowed in transactions
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Lost connection to MySQL server during query
I am importing some data from a large csv to a mysql table. I am losing the connection to the server during the process of importing the file to the table.
What is going wrong?
The error code is 2013: Lost connection to the mySql server during the query.
I am running these queries from a ubuntu machine remotely on a windows server.
Try the following 2 things...
1) Add this to your my.cnf / my.ini in the [mysqld] section
max_allowed_packet=32M
(you might have to set this value higher based on your existing database).
2) If the import still does not work, try it like this as well...
mysql -u <user> --password=<password> <database name> <file_to_import
Usually that happens when you exhaust one resource for the db session, such as memory, and mysql closes the connection.
Can you break the CSV file into smaller ones and process them? or do commit every 100 rows? The idea is that the transaction you're running shouldn't try to insert a large amount of data.
I forgot to add, this error is related to the configuration property max_allowed_packet, but I can't remember the details of what to change.
The easiest solution I found to this problem was to downgrade the MySql from MySQL Workbench to MySQL Version 1.2.17. I had browsed some MySQL Forums, where it was said that the timeout time in MySQL Workbech has been hard coded to 600 and some suggested methods to change it didn't work for me. If someone is facing the same problem with workbench you could try downgrading too.
1) you may have to increase the timeout on your connection.
2)You can get more information about the lost connections by starting mysqld with the --log-warnings=2 option.
This logs some of the disconnected errors in the hostname.err file
You can use that for further investigation
3) if you are trying to send the data to BLOB columns, check server's max_allowed_packet variable, which has a default value of 1MB. You may also need to increase the maximum packet size on the client end. More information on setting the packet size is given in following link, “Packet too large”.
4) you can check the following url link
5) you should check your available disk space is bigger than the table you're trying to update link
You might like to read this - http://dev.mysql.com/doc/refman/5.0/en/gone-away.html - that very well explains the reasons and fixes for "lost connection during query" scenarios.
In your case, it might be because of the max allowed packet size as pointed by Augusto. Or if you've verified it isn't the case, then it might be the connection wait timeout setting due to which the client is losing connection. However, I do not think latter is true here because it's a CSV file and not containing queries.
I think you can use mysql_ping() function.
This function checks for connection to the server alive or not. if it fails then you can reconnect and proceed with your query.
I'm using SQLyog to sync a production database to a dev db. On 4 tables, I'm getting:
Error No. 1205 Lock wait timeout exceeded; try restarting transaction
Researching the web seems to indicate that a transaction has begun, locked tables, but has not committed. One post said to SHOW PROCESSLIST; but the only processes appear to be my own, via SQLyog.
I have also tried a Restart of MySQL, but that didn't help either.
As a relative novice in MySQL, I'm stuck: I can't determine what transaction or process is locking the tables, nor how to clear this situation.
Any suggestions would be gratefully accepted!
MTIA
Having the same problem on MySQL-cluster, I've solved (at least it looks being solved now - no fail have occured during last two days) it by performing commit/rollback after SELECTs too.
Export and re-import your database; this can often fix a lot of mysterious problems. You can do this through phpMyAdmin or from the command line.
This page at MediaTemple has a good set of instructions:
http://kb.mediatemple.net/questions/129/Export+and+import+MySQL+databases#gs
(Well, it worked for me!)