I'm really new to server-side and I've been administrating a dedicated server for my website.
At peak hours the website is very slow, in order to know why I've been monitoring it using htop.
Some really long-lasting (more than few hours) mysql processes use up to 95% of the CPU !
The thing is I don't know what queries might be the cause of it nor how to monitor it.
I do have a cron every quarter of hour that sometimes takes a long time to run but the time of the slow down is not always matching cron's.
I've heard of a solution using a cron job killing too long mysql processes but wouldn't it cause discrepancies in the db ?
Configure MySQL server to log slow queries, then look at them. You need to understand why these queries are so slow.
Most likely, these queries can be sped up by adding proper indexes, but this is not fast and hard rule - you need to understand what is really happening.
You can kill long-running queries, but if you have MyISAM engine it can corrupt your tables. In that case, seriously consider switching to Innodb engine. With transactional engine like Innodb, currently running transactions will likely be rolled back, but data should not be corrupted.
Related
We are planning to rewrite legacy system that is using MySQL InnoDB database and trying to analyse main bottlenecks that should be avoided in next version.
System has many services/jobs that runs over night that generates data - inserts/updates, that mainly should be optimized. Jobs runs avg. 2-3 hours now.
We already gathered long running queries that must be optimized.
But I am wondering if it is possible to gather information and statistics about long running transactions.
Very helpful will be information which tables is locked by transaction the most - average locking time, lock type, periods.
Could somebody advice any tool or script that can gather such information?
Or maybe someone can share own experience in database analyse and optimization?
MySQL has built in capability for capturing "slow" query statistics (but to get an accurate picture you need to set the slow threshold as 0). You can turn the log into useful information with mysqldumpslow (bundled with mysql). I like the percona toolkit, but there are lots of other tools available.
We have a query which most of the time takes about 150ms to complete. Thousands go through the system each day without any issue. But every few days, something happens, and one of these queries suddenly takes roughly 30 minutes to complete. Any subsequent queries also run slow. The only way we have been able to recover is to kill ever single one of these queries. Soon as we do that, any subsequent queries again run at usual 150ms speed. For security reasons, I am not allowed to post the query itself. But it is nothing special.
The dB is MySQL 5.5.40 and using the innodb engine. During this period, all the usual system resources look fine - memory, cpu, disk space, disk i/o, network i/o.
Can someone give me some ideas about how I can troubleshoot the nature of this issue? I do not believe it is the query, since it seems to work just great 99% of the time. So I am thinking there is some kind of MySQL bug or a weird race condition going on.
This appears to have been some kind of issue with the planner in MySQL 5.5. Since our upgrade to 5.6, the problem has gone away. Also the explain of the query is completely different.
All my websites are running super slow because I tried to optimize the Magento DB in phpmyadmin in an effort to get the sites to speed UP. They're so slow they might as well be down. I want the pain to stop. Can I kill the optimize and how would I do that? Or is it better just to wait until it's finished.
For the record, I have a backup of the database.
Usually you can KILL any operation you don't want to finish, but be aware that the time to undo the operation may in fact be longer than it takes to simply complete.
Unless you're running a massive installation it sounds like your database needs to be tuned better. MySQL comes with a default my.cnf configuration that is terrible, barely any memory allocated to it, and runs slowly.
Secondly, you might be running a lot of queries that are slow because they're missing indexes. How much data is in this table you're trying to optimize? SHOW TABLE STATUS will give you an idea.
I'm thinking about moving our production env from a self hosted solution to amazon aws. I took a look at the different services and thought about using RDS as replacement for our mysql instances. The hardware we're using for our master seems to be better than the best hardware we can get when using rds (Quadruple Extra Large DB Instance). Since I can't simply move our production env to aws and see if the performance is still good enough I'd love to make some tests in advance.
I thought about creating a full query log from our current master, configure the rds instance and start to replay the full query log against it. Actually I don't even know if this kind of testing is a good idea but I guess you'll tell me if there are better ways to make sure the performance of mysql won't drop dramatically when making the move to rds.
Is there a preferred tool to replay the full query log?
at what metrics should I take a look while running the test
cpu usage?
memory usage?
disk usage?
query time?
anything else?
Thanks in advance
I'd recommend against replaying the query log - it's almost certainly not going to give you the information you want, and will take a significant amount of effort.
Firstly, you'd need to prepare your database so that replaying the query log won't break constraints when inserting, updating or deleting data, and that subsequent "select" queries will find the records they should find. This is distinctly non-trivial on anything other than a toy database - just taking a back-up and replaying the log doesn't necessarily guarantee the ordering of DML statements will match what happened on production. This may well give you a false sense of comfort - all your select statements return in a few milliseconds, because the data they're looking for doesn't exist!
Secondly, load and performance testing rarely works by replaying what happened on production - that doesn't (usually) reflect the peak conditions that will bring your system to its knees. For instance, most production systems run happily most of the time at <50% capacity, but go through spikes during the day, when they might reach 80% or more of capacity - that's what you care about, can your new environment handle the peaks.
My recommendation would be to use a tool like JMeter to write performance scripts (either directly to the database using the JDBC driver, or through the front end if you've got a web appilcation). Your performance scripts should reflect the behaviour you see from users, and be parameterized so they're not dependent on the order in which records are created.
Set yourself some performance targets (ideally based on current production levels, with a multiplier to cover you against spikes), e.g. "100 concurrent users, with no query taking more than 1 second"), and use JMeter to simulate that load. If you reach it first time, congratulations - go home! If not, look at the performance counters to see where the bottleneck is; see if you can alleviate that bottleneck (or tune your queries, your awesome on-premise hardware may be hiding some performance issues). Typical bottlenecks are CPU, RAM, and disk I/O.
Experiment with different test scenarios - "lots of writes", "lots of reads", "lots of reporting queries", and mix them up.
The idea is to understand the bottlenecks on the system, and see how far you are from those bottleneck, and understand what you can do to alleviate them. Once you know that, your decision to migrate will be far more robust.
This is the most puzzling MySQL problem that I've encountered in my career as an administrator. Can anyone with MySQL mastery help me a bit on this?:
Right now, I run an application that queries my MySQL/InnoDB tables many times a second. These queries are simple and optimized -- either single row inserts or selects with an index.
Usually, the queries are super fast, running under 10 ms. However, once every hour or so, all the queries slow down. For example, at 5:04:39 today, a bunch of simple queries all took more than 1-3 seconds to run, as shown in my slow query log.
Why is this the case, and what do you think the solution is?
I have some ideas of my own: maybe the hard drive is busy during that time? I do run a cloud server (rackspace) But I have flush_log_at_trx_commit set to 0 and tons of buffer memory (10x the table size on disk). So the inserts and selects should be done from memory right?
Has anyone else experience something like this before? I've searched all over this forum and others, and it really seems like no other MySQL problem I've seen before.
There are many reasons for sudden stalls. For example - even if you are using flush_log_at_trx_commit=0, InnoDB will need to pause briefly as it extends the size of data files.
My experience with the smaller instance types on Rackspace is that IO is completely awful. I've seen random writes (which should take 10ms) take 500ms.
There is nothing in built-in MySQL that will help you identify the problem easier. What you might want to do is take a look at Percona Server's slow query log enhancements. There's a specific feature called "profiling_server" which can break down time:
http://www.percona.com/docs/wiki/percona-server:features:slow_extended#changes_to_the_log_format