Mysql query taking too much time - mysql

I have problem related to mysql database. i am linux webserver admin and i am facing a problem with a mysql query. The database is very small. I tried to track in logs and found that a query is taking minimum 5 sec to respond . The first page of site is coming from the database. Client are using cms. when the server gets some number of hits database server starts to give response very slowly and wait time increases from 5 sec to several seconds.
I checked slow query logs
{
Query_time: 11.480138 Lock_time: 0.003837 Rows_sent: 921 Rows_examined: 3333
SET timestamp=1346656767;
SELECT `Tender`.`id`,
`Tender`.`department_id`,
`Tender`.`title_english`,
`Tender`.`content_english`,
`Tender`.`title_hindi`,
`Tender`.`content_hindi`,
`Tender`.`file_name`,
`Tender`.`start_publish`,
`Tender`.`end_publish`,
`Tender`.`publish`,
`Tender`.`status`,
`Tender`.`createdBy`,
`Tender`.`created`,
`Tender`.`modifyBy`,
`Tender`.`modified`
FROM `mcms_tenders` AS `Tender`
WHERE `Tender`.`department_id` IN ( 31, 33, 32, 30 );
}
Every line in the log is same only there is diff in Query time.
Is there any way tweak the performance?
Update: Here is the EXPLAIN result:
+----+-------------+--------+------+---------------+------+---------+------+-‌-----+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------+------+---------------+------+---------+------+----‌​--+-------------+
| 1 | SIMPLE | Tender | ALL | NULL | NULL | NULL | NULL | 3542 | Using where |
+----+-------------+--------+------+---------------+------+---------+------+----‌​--+-------------+
1 row in set, 1 warning (0.00 sec)
client is saying they are using Index so i run the command to check the indexing.
I got following output. Does It means they are using Indexing.
+--------------+------------+----------+--------------+-------------+-----------+------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+--------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| mcms_tenders | 0 | PRIMARY | 1 | id | A | 4264 | NULL | NULL | | BTREE | |
+--------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+

The normal way to tweak the performance of a query like this is to create an index on department_id.
However, this assumes that Tenders is actually a table and not a view. You should confirm this, since the problem may be in a view.
Also, from what you describe the issue may be the connection from the server to the end users. I would try running the query locally on the server (or checking the execute time strictly on the server) to see if the query is really taking that long.

"when the server gets some number of hits"
Define 'some number'. It makes sense that reading the database is slower when it is more heavily used. Also, MySQL has a query cache that is fully invalidated when changes are made to the data. So every time someone inserts, deletes or modifies a record in this table, the next queries will be slower because the table date is still uncached.
But 11 seconds for a query like this is very slow, so either the load is way too high, the hardware insufficient or broken or your database lacks indexes (I always forget to mention that at first, because I assume adding indexes to be a second nature for anyone working with databases).

Related

MariaDB - Should I add index to my table?

Recently I was checking my system logs and I noticed some of my queries are very slow.
I have a table that stores user activites. The table structure is id (int), user (int), type (int), object (varchar), extra (mediumtext) and date (timestamp).
Also I only have index for id (BTREE, unique).
I have performance issues for following query;
SELECT DISTINCT object as usrobj
from ".MV15_PREFIX."useractivities
WHERE user='".$user_id."'
and type = '3'
limit 0,1000000"
Question is, should I also index user same as id? What should be the best practise I should follow?
This table is actively used and has over 500k+ rows in it. And there are 2k~ concurrent users online average on site.
The reason I am asking this question is I am not really good at managing DB, and also I have slow query issue on another table which has proper indexes.
Thanks in advance for suggestions.
Side note:
Result of mysqltuner
General recommendations:
Reduce or eliminate persistent connections to reduce connection usage
Adjust your join queries to always utilize indexes
Temporary table size is already large - reduce result set size
Reduce your SELECT DISTINCT queries without LIMIT clauses
Consider installing Sys schema from https://github.com/mysql/mysql-sys
Variables to adjust:
max_connections (> 768)
wait_timeout (< 28800)
interactive_timeout (< 28800)
join_buffer_size (> 64.0M, or always use indexes with joins)
(I will set max_connections > 768, not really sure about timeouts and as far I read topics/suggestions in Stackoverflow I think I shouldn't increase the size of join_buffer_size but I'd really appreciate getting feedback about these variables too.)
EDIT - SHOW INDEX result;
+--------------------+------------+-----------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------------------+------------+-----------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| ***_useractivities | 0 | PRIMARY | 1 | id | A | 434006 | NULL | NULL | | BTREE | | |
| ***_useractivities | 1 | user_index | 1 | user | A | 13151 | NULL | NULL | | BTREE | | |
| ***_useractivities | 1 | user_type_index | 1 | user | A | 10585 | NULL | NULL | | BTREE | | |
| ***_useractivities | 1 | user_type_index | 2 | type | A | 13562 | NULL | NULL | | BTREE | | |
+--------------------+------------+-----------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
Most of these rules of thumb for PostgreSQL indexing apply to most SQL database management systems.
https://dba.stackexchange.com/a/31517/1064
So, yes, you will probably benefit from an index on user and an index on type. You might benefit more from an index on the pair user, type.
You'll benefit from learning how to read an execution plan, too.
For that query, either of these is optimal:
INDEX(user, type)
INDEX(type, user)
Separate indexes (INDEX(user), INDEX(type)) is likely to be not nearly as good.
MySQL's InnoDB has only BTree, not Hash. Anyway, BTree is essentially as good as Hash for 'point queries', and immensely better for 'range' queries.
Indexing tips.
Indexes help SELECTs and UPDATEs, sometimes a lot. Use them. The side effects are minor -- such as extra disk space used.

Slow query after upgrade mysql from 5.5 to 5.6

We're upgrading mysql from 5.5 to 5.6 and some queries are deadly slow now.
Queries that took 0.005 seconds before are now taking 49 seconds.
Queries on 5.6 are skipping the indexes, it seems:
+----+-------------+-------+-------+----------------------------------------------------+---------+---------+------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+----------------------------------------------------+---------+---------+------+--------+-------------+
| 1 | SIMPLE | pens | index | index_contents_on_slug,index_contents_on_slug_hash | PRIMARY | 4 | NULL | 471440 | Using where |
+----+-------------+-------+-------+----------------------------------------------------+---------+---------+------+--------+-------------+
1 row in set (0.00 sec)
But are not being skipped on 5.5:
+----+-------------+-------+-------------+----------------------------------------------------+----------------------------------------------------+---------+------+------+----------------------------------------------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------------+----------------------------------------------------+----------------------------------------------------+---------+------+------+----------------------------------------------------------------------------------------------+
| 1 | SIMPLE | pens | index_merge | index_contents_on_slug,index_contents_on_slug_hash | index_contents_on_slug_hash,index_contents_on_slug | 768,768 | NULL | 2 | Using union(index_contents_on_slug_hash,index_contents_on_slug); Using where; Using filesort |
+----+-------------+-------+-------------+----------------------------------------------------+----------------------------------------------------+---------+------+------+----------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
Both DBs were created from the same mysql dump.
Are these indexes not being constructed when I do the import on 5.6? How do I force the index creation?
The query:
SELECT `pens`.* FROM `pens` WHERE (slug_hash = 'style' OR slug = 'style') ORDER BY `pens`.`id` DESC LIMIT 1
Edit: Removed the schema
Ultimately the accepted answer above is correct.
The help from #RandomSeed got me thinking in the right direction. Basically the optimization plans created in 5.6 are significantly different from those in 5.5, so you'll probably have to rework your query, much like I did.
I did not end up using the FORCE INDEX, but instead removed portions of the query until I determined what was causing 5.6 to miss the index. Then I reworked the application logic to deal with that.
The slow query in v5.6 is caused by the engine being unable to, or deciding not to, merge the two relevant indexes (index_contents_on_slug_hash, index_contents_on_slug) in order to process your query. Remember that a query may only use one index per table. In order to be able to take advantage of several indexes on the same table, it needs to pre-merge on-the-fly these indexes into one (in memory). This is the meaning of the index_merge and Using union(...) notices in your execution plan. This consumes time and memory, obviously.
Quick fix (and probably preferred solution anyways): add a two-colums index on slug and slug_hash.
ALTER TABLE pens ADD INDEX index_contents_on_slug_and_slug_hash (
slug, slug_hash
);
Now your new server is probably unable to merge these indexes because it results in an index too large to fit in the buffer. Your new server probably has a much smaller value for key_buffer_size (if the table is MyISAM) or for innodb_buffer_pool_size (if InnoDB) than there used to be in your older installation.

can mysqldump on a large database be causing my long queries to hang?

I have a large database (approx 50GB). It is on a server I have little control over, but I know they are using mysqldump to do backups nightly.
I have a query that takes hours to finish. I set it to run, but it never actually finishes.
I've noticed that after the backup time, all the tables have a lock request (SHOW OPEN TABLES WHERE in_use > 0; lists all tables).
The tables from my query have in_use = 2, all other tables have in_use = 1.
So... what is happening here?
a) my query is running normally, blocking the dump from happening. I should just wait?
b) the dump is causing the server to hang (maybe lack of memory/disk space?)
c) something else?
EDIT: using MyISAM tables
There is a server admin who is not very competent, but if I ask him specific things he does them. What should I get him to check?
EDIT: adding query
SELECT citing.article_id as citing, citing.year, r.id_when_cited, cited_issue.country
FROM isi_lac_authored_articles as citing # 1M records
JOIN isi_citation_references r ON (citing.article_id = r.article_id) # 400M records
JOIN isi_articles cited ON (cited.id_when_cited = r.id_when_cited) # 25M records
JOIN isi_issues cited_issue ON (cited.issue_id = cited_issue.issue_id) # 1M records
This is what EXPLAIN has to say:
+----+-------------+-------------+------+--------------------------------------------------------------------------+---------------------------------------+---------+-------------------------------+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------+------+--------------------------------------------------------------------------+---------------------------------------+---------+-------------------------------+---------+-------------+
| 1 | SIMPLE | cited_issue | ALL | NULL | NULL | NULL | NULL | 1156856 | |
| 1 | SIMPLE | cited | ref | isi_articles_id_when_cited,isi_articles_issue_id | isi_articles_issue_id | 49 | func | 19 | Using where |
| 1 | SIMPLE | r | ref | isi_citation_references_article_id,isi_citation_references_id_when_cited | isi_citation_references_id_when_cited | 17 | mimir_dev.cited.id_when_cited | 4 | Using where |
| 1 | SIMPLE | citing | ref | isi_lac_authored_articles_article_id | isi_lac_authored_articles_article_id | 16 | mimir_dev.r.article_id | 1 | |
+----+-------------+-------------+------+--------------------------------------------------------------------------+---------------------------------------+---------+-------------------------------+---------+-------------+
I actually don't understand why it needs to look at all the records in isi_issues table. Shouldn't it just be matching up by the isi_articles (cited) on issue_id? Both fields are indexed.
For a MySQL database of that size, you may want to consider setting up replication to a slave node, and then have your nightly database backups performed on the slave.
Yes -- some options to mysqldump will have the effect of locking all MyISAM tables while the backup is in progress, so that the backup is a consistent "snapshot" of a point in time.
InnoDB supports transactions, which make this unnecessary. It's also generally faster than MyISAM. You should use it. :)

how can I optimize a query with multiple joins (already have indexes)?

SELECT citing.article_id as citing, lac_a.year, r.id_when_cited, cited_issue.country, citing.num_citations
FROM isi_lac_authored_articles as lac_a
JOIN isi_articles citing ON (lac_a.article_id = citing.article_id)
JOIN isi_citation_references r ON (citing.article_id = r.article_id)
JOIN isi_articles cited ON (cited.id_when_cited = r.id_when_cited)
JOIN isi_issues cited_issue ON (cited.issue_id = cited_issue.issue_id);
I have indexes on all the fields being JOINED on.
Is there anything I can do? My tables are large (some 1 Million records, the references tables has 500 million records, the articles table has 25 Million).
This is what EXPLAIN has to say:
+----+-------------+-------------+--------+--------------------------------------------------------------------------+---------------------------------------+---------+-------------------------------+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------+--------+--------------------------------------------------------------------------+---------------------------------------+---------+-------------------------------+---------+-------------+
| 1 | SIMPLE | cited_issue | ALL | NULL | NULL | NULL | NULL | 1156856 | |
| 1 | SIMPLE | cited | ref | isi_articles_id_when_cited,isi_articles_issue_id | isi_articles_issue_id | 49 | func | 19 | Using where |
| 1 | SIMPLE | r | ref | isi_citation_references_article_id,isi_citation_references_id_when_cited | isi_citation_references_id_when_cited | 17 | mimir_dev.cited.id_when_cited | 4 | Using where |
| 1 | SIMPLE | lac_a | eq_ref | PRIMARY | PRIMARY | 16 | mimir_dev.r.article_id | 1 | |
| 1 | SIMPLE | citing | eq_ref | PRIMARY | PRIMARY | 16 | mimir_dev.r.article_id | 1 | |
+----+-------------+-------------+--------+--------------------------------------------------------------------------+---------------------------------------+---------+-------------------------------+---------+-------------+
5 rows in set (0.07 sec)
If you realy need all the returned data, I would suggest two things:
You, probably, know the data better than MySQL and you can try to make advantage of it if MySQL is not correct in its assumptions. Currently, MySQL thinks that it is easier to full scan the whole isi_issues table at the beginning, and if the result is really going to include all issues, than the assumption is correct. But if there are many issues that should not be in the result, you may want to force another order of the joins that you consider more correct. It is you, who knows which table applies the strongest restrictions and which are the smallest to full scan (you will anyway need to full scan something, since there is no WHERE clause).
You can make profit from covering indexes (that is indexes that contain enough data in itself and not needing to touch the row data). For example, having an index (article_id, num_citations) on isi_articles and (article_id, year) on isi_lac_authored_articles and even (country) on isi_issues will significantly speed up that query as long as the indexes fit in memory, but, from the other side, will make you indexes larger and slightly slow dow inserts into the table.
i think it's the best you can do. i mean at least it's not using nested/multiple queries. you should do a little benchmark on the sql. you could at least limit your results at the least as possible. 15-30 rows for a return set is pretty fine per page (this depends on the app, but 15-30 for me is the tolerance range)
i believe in mySQL (phpMyAdmin, console, GUI whatever) they return some sort of "execution time" which is the time that it took to the query to process. compare that with a benchmark of the query using your server-side code. then compare that with the query run using the server-side code and outputting it with your app interface included after that.
by this, you can see where your bottle-neck is - that is where you optimize.
Unless the result of your query is input to some other query or system, it is useless to return that much(3M) rows. That would be clever to return just an acceptable amount of rows per query(like 1000) that is for visualizing.
Looking at your SQL - the lack of a WHERE clause means it is pulling all rows from:
JOIN isi_issues cited_issue ON (cited.issue_id = cited_issue.issue_id)
You could look at partitioning the large isi_issues table, this would allow MySQL to perform a bit quicker (smaller files are easier to handle)
Or alternatively you can loop the statement and use a LIMIT clause.
LIMIT 0,100000
then
LIMIT 100001, 200000
This will let the statements run quicker and you can deal with the data in batches.

MySQL query not going away after being killed

I have a MySQL query that is copying data from one table to another for processing. For some reason, this query that normally takes a few seconds locked up overnight and ran for several hours. When I logged in this morning, I tried to kill the query, but it is still listed in the process list.
| Id | User | Host | db | Command | Time | State | Info |
+---------+----------+-----------+------+---------+-------+--------------+--------------------------------------------------------------------------------------+
| 1061763 | tb_admin | localhost | dw | Killed | 45299 | Sending data | INSERT INTO email_data_inno_stage SELECT * FROM email_data_test LIMIT 4480000, 10000 |
| 1062614 | tb_admin | localhost | dw | Killed | 863 | Sending data | INSERT INTO email_data_inno_stage SELECT * FROM email_data_test LIMIT 4480000, 10000 |
What could have caused this, and how can I kill this process so I can get on with my work?
If the table email_data_test is MyISAM and it was locked, that would have held up the the INSERT.
If the table email_data_test is InnoDB, then a lot of MVCC data was being written in ib_logfiles, which may not have occurred yet.
In both cases, you had the LIMIT clause scroll through 4,480,000 rows just to get to 10,000 rows you actually needed to INSERT.
Killing the query only causes the InnoDB table email_data_inno_stage to execute a rollback.