I'm having trouble diagnosing a problem I'm having on my ubuntu scalr/ec2 production environment.
The trouble is apparently randomly, database queries and/or memcache queries will take MUCH longer than they should. I've seen a simple select statement take 130ms or a Memcache fetch take 65ms! It can happen a handful of times per request, causing some requests to take twice as long as they should.
To diagnose the problem, I wrote a very simple script which will just connect to the MySql server and run a query.
require 'mysql'
mysql = Mysql.init
mysql.real_connect('', '', '', '')
max = 0
100.times do
start = Time.now
mysql.query('select * from navigables limit 1')
stop = Time.now
total = stop - start
max = total if total > max
end
puts "Max Time: #{max * 1000}"
mysql.close
This script consistently returned a really high max time, so I eliminated any Rails as the source of the problem. I also wrote the same thing in Python to eliminate Ruby. And indeed the Python one took inordinate amounts of time as well!
Both MySql and Memcache are on their own boxes, so I considered network latency, but watching pings and tracerouteing look normal.
Also running the queries/fetches on the respective machines returns expected times, and I'm running the same version gems on my staging machine without this issue.
I'm really stumped on this one... any thoughts on something I could try to diagnose this?
Thanks
My only thought is that it might be disk?
Mysql uses query cache to store SELECT together with its result. That could explain the constant speed you are getting in continous selecting. Try EXPLAIN-inig the query to see if you are using indexes.
I don't see why memcache would be a problem (unles it's crashing and restarting?). Check your server logs for suspicious service failures.
Related
I have a MySQL database that I am running very simple queries against as part of a webapp. I have received reports from users starting today that they got an error saying that their account doesn't exist, and when they log in again, it does (this happened to only a few people, and only once to each, so clearly it is rare). Based on my backend code, this error can only occur if the same query returns 0 row the first time, and 1 row the second. My query is basically SELECT * FROM users WHERE username="...". How is this possible? My suspicion is that the hard disk is having I/O failures, but I am unsure because I would not expect MySQL to fail silently in this case. That said, I don't know what else it could be.
This could be a bug with your mysql client (Though I'm unsure as to how the structure of your code is, it could just be bad query). However let's assume that your query has been working fine up until now with no prior issues, so we'll rule out bad code.
With that in mind, I'm assuming it's either a bug in your mysql client or your max connection count is reached (Had this issue with my previous host - Hostinger).
Let's say your issue is a bug in your mysql client, set your sessions to per session basis by running this
SET SESSION optimizer_switch="index_merge_intersection=off";
or in your my.cnf you can set it globally
[mysqld] optimizer_switch=index_merge_intersection=off
As for max connection you can either increase your max_connection value (depending if your host allows it), or you'll have to make a logic to close the mysql connection after a query execution.
$mysqli->close();
I'm trying to import a large (~30M row) MySQL database to ElasticSearch. Cool cool, there's a logstash tool that looks like it's built for this sort of thing; its JDBC plugin will let me connect right to the database and slurp up the rows real fast.
However! When I try it, it bombs with java.lang.OutOfMemoryError. Okay. It's probably trying to batch up too many rows or something. So I add jdbc_fetch_size => 1000 to my configuration. No dice, still out of memory. Okay, maybe that option doesn't work, or doesn't do what I think?
So I try adding jdbc_paging_enabled => true and jdbc_page_size => 10000 to my config. Success! It starts adding rows in batches of 10k to my index.
But it slows down. At first I'm running 100k rows/minute; by the time I'm at 2M rows, however, I'm at maybe a tenth of that. And no surprise; I'm pretty sure this is using LIMIT and OFFSET, and using huge OFFSETs in queries is real slow, so I'm dealing with an O(n^2) kind of thing here.
I'd really like to just run the whole big query and let the cursor iterate over the resultset, but it looks like that isn't working for some reason. If I had more control over the query, I could change the LIMIT/OFFSET thing out for an WHERE id BETWEEN val1 AND val2 kind of thing, but I can't see where I could get in to do that.
Any suggestions on how I can not crash, but still run at a reasonable speed?
Okay! After searching the issues for the logstash-input-jdbc github page for "Memory" I found this revelation:
It seems that an additional parameter ?useCursorFetch=true needs to be added to the connection string of mysql 5.x.
It turns out that the MySQL JDBC client does not use a cursor for fetching rows by default because of some reason, and the logstash client doesn't warn you that it's not able to use a cursor to iterate over the resultset even though you've set a jdbc_fetch_size because of some other reason.
The obvious way to know about this, of course, would have been to have carefully read the MySQL Connector/J documentation which does mention that cursors are off by default, though not why.
Anyhow, I added useCursorFetch=true to the connection string, kicked jdbc_query_paging to the curb, and imported 26M rows into my index in 2.5 hours, on an aging Macbook Pro with 8G memory.
Thanks to github user axhiao for the helpful comment!
I i'm doing select from 3 joined tables on MySql server 5.6 running on azure instance with inno_db set to 2GB. I used to have 14GB ram and 2core server and I just doubled ram and cores hoping this will result positive on my select but it didn't happen.
My 3 tables I'm doing select from are 90mb,15mb and 3mb.
I believe I don't do anything crazy in my request where I select few booleans however i'm seeing this select is hangind the server pretty bad and I can't get my data. I do see traffic increasing to like 500MB/s via Mysql workbench but can't figure out what to do with this.
Is there anything I can do to get my sql queries working? I don't mind to wait for 5 minutes to get that data, but i need to figure out how to get it.
==================== UPDATE ===============================
I was able to get it done via cloning the table that is 90 mb and forfilling it with filtered original table. It ended up to be ~15mb, then I just did select all 3 tables joining then via ids. So now request completes in 1/10 of a second.
What did I do wrong in the first place? I feel like there is a way to increase some sizes of some packets to get such queries to work? Any suggestions on what shall I google?
Just FYI, my select query looked like this
SELECT
text_field1,
text_field2,
text_field3 ,..
text_field12
FROM
db.major_links,db.businesses, db.emails
where bool1=1
and bool2=1
and text_field is not null or text_field!=''
and db.businesses.major_id=major_links.id
and db.businesses.id=emails.biz_id;
So bool1,2 and textfield i'm filtering are the filds from that 90mb table
I know this might be a bit late, but I have some suggestions.
First take a look the max_allowed_packet in your my.ini file. This is usually found here in Windows:
C:\ProgramData\MySQL\MySQL Server 5.6
This controls the packet size, and usually causes errors in large queries if it isn't set correctly. I have mine set to 100M
Here is some documentation for you:
Official documentation
In addition I've slow queries when there are a lot of items in the where statement and here you have several. Make sure you have indexes and compound indexes on the values in your where clause especially related to the joins.
I have a mysql SELECT query which is fast (<0.1 sec) but only the first time I run it. It joins 3 tables together (using indices) and has a relatively simple WHERE statement. When I run it by hand in the phpmyadmin (always changing numbers in the WHERE so that it isn't cached) it is always fast but when I have php run several copies of it in a row, the first one is fast and the others hang for ~400 sec. My only guess is that somehow mysql is running out of memory for the connection and then has to do expensive paging.
My general question is how can I fix this behavior, but my specific questions are without actually closing and restarting the connection how can I make these queries coming from php be seen as separate just like the queries coming from phpmyadmin, how can I tell mysql to flush any memory when the request is done, and does this sound like a memory issue to you?
Well I found the answer at least in my case and I'm putting it here for anyone in the future who runs into a similar issue. The query I was running had a lot of results returned and MYSQL's query cache was causing a lot of overhead. When you run a query MYSQL will save it and its output so that it can quickly answer future identical requests quickly. All I had to do was put SQL_NO_CACHE and the speed was back to normal. Just look out if your incoming query is large or the results are very large because it can take considerable resources for MYSQL to decide when to kick things out.
What can be done to identify the reason for DB slowness?
When i ran the query in the morning it ran quickly & i got the output.
When i run the same query after 1 hr, it took more than 2mins.
What can be checked to identify this slowness?
All the tables are properly indexed.
If it's just a single query which is running slowly, EXPLAIN SELECT... as mentioned by arex1337 may help you see the reason.
It would also be worth looking at the output of e.g. vmstat on the box whilst running the query to see what it's doing - you should be able to get a feel for whether the machine is swapping, IO-bound, CPU-bound etc.
Check also with top to look for any rogue processes hogging CPU time.
Finally, if the machine is using RAID, it's possible that, if a drive has failed, the RAID array could be in a degraded state, which could make disc access slower (this is only applicable in certain RAID configurations, but worth considering and ruling out).
You can use EXPLAIN <your query> to get information about how MySQL executes your query. Maybe you get some hints about why it's slow.
EXPLAIN SELECT ... FROM ... WHERE ...;
Also, maybe you just have a slow query, and it was fast the second time because the result was cached?