mysql query to required to be faster

mysql query to required to be faster - mysql

Hi Is it possible to speed the following query up? I have tried using different indexes, Would I be right in thinking that If its constantly taking more than 0.20 seconds that the query cache is not being used?
Also is forcing an index a good idea in this case?
FYI all three columns used after the where clause are indexed.
mysql> SELECT count( item_t0.PK ) FROM table item_t0 WHERE ( item_t0.PK <>87984555070 AND item_t0.p_internalurl ='replicated273784712' AND ( item_t0.p_datapk =8987719073822 OR ( item_t0.p_datapk IS NULL AND item_t0.PK =8987719073822))) AND (item_t0.TypePkString IN (N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N) );
+---------------------+
| count( item_t0.PK ) |
+---------------------+
| 0 |
+---------------------+
1 row in set (0.32 sec)
mysql> explain extended SELECT count( item_t0.PK ) FROM medias item_t0 WHERE ( item_t0.PK <>8798340055070 AND item_t0.p_internalurl ='replicated273654712' AND ( item_t0.p_datapk =8987719073822 OR ( item_t0.p_datapk IS NULL AND item_t0.PK =8987717719073822))) AND (item_t0.TypePkString IN (N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N,N) ); 7961003
+----+-------------+---------+------+--------------------------------------------------------+-----------------+---------+-------+-------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------+------+--------------------------------------------------------+-----------------+---------+-------+-------+----------+-------------+
| 1 | SIMPLE | item_t0 | ref | PRIMARY,dataPK_idx_30,internalurl_idx,typepkstring_idx | internalurl_idx | 768 | const | 74102 | 100.00 | Using where |
+----+-------------+---------+------+--------------------------------------------------------+-----------------+---------+-------+-------+----------+-------------+
1 row in set, 1 warning (0.00 sec)
I have looked at whether the query_cache needs to be looked into and saw the following values, Can anyone suggest if the following results show anything...
mysql> show global status like "%Qcache_hits%";
+---------------+--------+
| Variable_name | Value |
+---------------+--------+
| Qcache_hits | 352251 |
+---------------+--------+
mysql> show global status like "%com_select%";
+---------------+----------+
| Variable_name | Value |
+---------------+----------+
| Com_select | 33525493 |
+---------------+----------+
1 row in set (0.00 sec)
mysql> show global status like "%prunes%";
+----------------------+----------+
| Variable_name | Value |
+----------------------+----------+
| Qcache_lowmem_prunes | 15729370 |
+----------------------+----------+
1 row in set (0.00 sec)

Related

Select count takes two minutes on a two column table

I have a MariaDB table with just under 100000 rows, and selecting the count takes a very long time (almost 2 minutes).
Selecting anything by id from the table though takes only 4 milliseconds.
The text field here contains on average 5000 characters.
How can I speed this up?
MariaDB [companies]> describe company_details;
+---------+------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------+------------------+------+-----+---------+-------+
| id | int(10) unsigned | NO | PRI | NULL | |
| details | text | YES | | NULL | |
+---------+------------------+------+-----+---------+-------+
MariaDB [companies]> explain select count(id) from company_details;
+------+-------------+-----------------+-------+---------------+---------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-----------------+-------+---------------+---------+---------+------+-------+-------------+
| 1 | SIMPLE | company_details | index | NULL | PRIMARY | 4 | NULL | 71267 | Using index |
+------+-------------+-----------------+-------+---------------+---------+---------+------+-------+-------------+
MariaDB [companies]> analyze table company_details;
+---------------------------+---------+----------+----------+
| Table | Op | Msg_type | Msg_text |
+---------------------------+---------+----------+----------+
| companies.company_details | analyze | status | OK |
+---------------------------+---------+----------+----------+
1 row in set (0.098 sec)
MariaDB [companies]> select count(id) from company_details;
+-----------+
| count(id) |
+-----------+
| 96544 |
+-----------+
1 row in set (1 min 43.199 sec)
This becomes an even bigger problem when I try to join the table.
For example, to find the number of companies which do not have associated details:
MariaDB [companies]> SELECT COUNT(*) FROM company c LEFT JOIN company_details cd ON c.id = cd.id WHERE cd.id IS NULL;
+----------+
| count(*) |
+----------+
| 42178 |
+----------+
1 row in set (10 min 28.846 sec)
Edit:
After running OPTIMIZE on the table, the select count has improved speed from 1min 43sec to just 5 sec, and the join has improved speed from 10 minutes to 25 seconds.
MariaDB [companies]> optimize table company_details;
+---------------------------+----------+----------+-------------------------------------------------------------------+
| Table | Op | Msg_type | Msg_text |
+---------------------------+----------+----------+-------------------------------------------------------------------+
| companies.company_details | optimize | note | Table does not support optimize, doing recreate + analyze instead |
| companies.company_details | optimize | status | OK |
+---------------------------+----------+----------+-------------------------------------------------------------------+
2 rows in set (11 min 21.195 sec)

I think there's OPTIMIZE -command for rebuilding indexes.
OPTIMIZE company_details;
This usually takes some time to complete. More details: https://mariadb.com/kb/en/optimize-table/

Mysql Why is the count(*) performance very fast after locking a table

If I don't lock the table, the count(*) performance was bad like this:
mysql> select count(*) from titles;
+----------+
| count(*) |
+----------+
| 443308 |
+----------+
1 row in set (8.79 sec)
mysql> explain select count(*) from titles;
+----+-------------+--------+------------+-------+---------------+---------+---------+------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------+------------+-------+---------------+---------+---------+------+--------+----------+-------------+
| 1 | SIMPLE | titles | NULL | index | NULL | PRIMARY | 209 | NULL | 442843 | 100.00 | Using index |
+----+-------------+--------+------------+-------+---------------+---------+---------+------+--------+----------+-------------+
1 row in set, 1 warning (0.00 sec)
It looks so normal. But when I lock the table, the count(*) performance was so fast.
mysql> lock tables titles read;
Query OK, 0 rows affected (0.00 sec)
mysql> select count(*) from titles;
+----------+
| count(*) |
+----------+
| 443308 |
+----------+
1 row in set (0.13 sec)
What happens to the operation of count after locking the table?
update at 2021-5-29:
mysql> show index from titles;
+--------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | Visible | Expression |
+--------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+
| titles | 0 | PRIMARY | 1 | emp_no | A | 301411 | NULL | NULL | | BTREE | | | YES | NULL |
| titles | 0 | PRIMARY | 2 | title | A | 441772 | NULL | NULL | | BTREE | | | YES | NULL |
| titles | 0 | PRIMARY | 3 | from_date | A | 442843 | NULL | NULL | | BTREE | | | YES | NULL |
| titles | 1 | idx_emp_no | 1 | emp_no | A | 300876 | NULL | NULL | | BTREE | | | YES | NULL |
+--------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+
4 rows in set (0.04 sec)
mysql> explain select count(*) from titles;
+----+-------------+--------+------------+-------+---------------+------------+---------+------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------+------------+-------+---------------+------------+---------+------+--------+----------+-------------+
| 1 | SIMPLE | titles | NULL | index | NULL | idx_emp_no | 4 | NULL | 442835 | 100.00 | Using index |
+----+-------------+--------+------------+-------+---------------+------------+---------+------+--------+----------+-------------+
1 row in set, 1 warning (0.00 sec)
mysql> select count(*) from titles;
+----------+
| count(*) |
+----------+
| 443304 |
+----------+
1 row in set (8.79 sec)
I create an smaller index. But it doesn't have any change.

I've found no relation between table locks and Query time, but looking into the Mysql Manual I've found some clues:
The COUNT(*) is calculated by scanning the smallest secondary index if any, otherwise it uses the clustered index (primary key or auto-generated)
Locking the table could lead to an update of all records into the buffer pool, so it does not need to perform additional scans and then returns quicky
Your primary index key length is 209 which makes the index scan slow, and any SELECT statement could be subject to different optimizations depending on the current table and buffer pool status. (try using a small numeric primary key with unique row id and additional index with all your fields in the current primary key)
In the following link you can find more information
https://dev.mysql.com/doc/refman/5.6/en/aggregate-functions.html#function_count
COUNT(expr) [over_clause]
As of MySQL 8.0.13, SELECT COUNT(*) FROM tbl_name query performance
for InnoDB tables is optimized for single-threaded workloads if there
are no extra clauses such as WHERE or GROUP BY.
InnoDB processes SELECT COUNT(*) statements by traversing the smallest
available secondary index unless an index or optimizer hint directs
the optimizer to use a different index. If a secondary index is not
present, InnoDB processes SELECT COUNT(*) statements by scanning the
clustered index.
Processing SELECT COUNT(*) statements takes some time if index records
are not entirely in the buffer pool.
If an approximate row count is sufficient, use SHOW TABLE STATUS.
InnoDB handles SELECT COUNT(*) and SELECT COUNT(1) operations in the
same way. There is no performance difference.
For MyISAM tables, COUNT(*) is optimized to return very quickly if the
SELECT retrieves from one table, no other columns are retrieved, and
there is no WHERE clause.
This optimization only applies to
MyISAM tables, because an exact row count is stored for this storage
engine and can be accessed very quickly. COUNT(1) is only subject to
the same optimization if the first column is defined as NOT NULL.

Strange. I get the opposite (6x slower with the LOCK):
mysql> SELECT COUNT(*) FROM allcountries;
+----------+
| COUNT(*) |
+----------+
| 11082175 |
+----------+
1 row in set (0.26 sec)
mysql> LOCK TABLES allcountries READ;
mysql> SELECT COUNT(*) FROM allcountries;
+----------+
| COUNT(*) |
+----------+
| 11082175 |
+----------+
1 row in set (1.59 sec)
mysql> UNLOCK TABLES;
mysql> LOCK TABLES allcountries READ;
mysql> SELECT COUNT(*) FROM allcountries;
+----------+
| COUNT(*) |
+----------+
| 11082175 |
+----------+
1 row in set (1.50 sec)
mysql> UNLOCK TABLES;
mysql> SELECT COUNT(*) FROM allcountries;
+----------+
| COUNT(*) |
+----------+
| 11082175 |
+----------+
1 row in set (0.28 sec)
mysql> SELECT ##version;
+-------------------------+
| ##version |
+-------------------------+
| 8.0.25-0ubuntu0.20.10.1 |
+-------------------------+
1 row in set (0.00 sec)
The table is not PARTITIONed. innodb_parallel_read_threads = 4.
Parallel execution
8.0 has a few cases where multiple threads are used for queries. Here are some entries from the Changelogs. (Still, it does not explain why LOCK TABLES is relevant.)
8.0.14 Parallel scanning of by PRIMARY KEY (cf innodb_parallel_read_threads) COUNT(*) w/o WHERE
8.0.17 Parallel scanning of partitions (cf innodb_parallel_read_threads)
8.0.20 Changes to parallel read threads functionality introduced in MySQL 8.0.17 caused a degradation in SELECT COUNT(*) performance. Pages were read from disk unnecessarily. (Bug #30766089)
What was the value of innodb_parallel_read_threads? (SHOW VARIABLES LIKE 'innodb_parallel_read_threads';)

How to improve Limit clause in MySQL

I have the posts table with 10k rows and I want to create pagination by that. So I have the next query for that purpose:
SELECT post_id
FROM posts
LIMIT 0, 10;
When I Explain that query I get the next result:
So I don't understand why MySql need to iterate thru 9976 rows for finding the 10 first rows? I will be very thankful if somebody help me to optimize this query.
Also I know about that topic MySQL ORDER BY / LIMIT performance: late row lookups, but the problem still exist even if I modify the query to the next one:
SELECT t.post_id
FROM (
SELECT post_id
FROM posts
ORDER BY
post_id
LIMIT 0, 10
) q
JOIN posts t
ON q.post_id = t.post_id
Update
#pala_'s solution works great for above simple case but now while I am testing a more complex query with inner join. My purpose is to join comment table with post table and unfortunately when I Explain new query is still iterate through 9976 rows.
Select comm.comment_id
from comments as comm
inner join (
SELECT post_id
FROM posts
ORDER BY post_id
LIMIT 0, 10
) as paged_post on comm.post_id = paged_post.post_id;
Do you have some idea what is the reason of such MySQL behavior ?

Try this:
SELECT post_id
FROM posts
ORDER BY post_id DESC
LIMIT 0, 10;
Pagination via LIMIT doesn't make much sense without ordering anyway, and it should fix your problem.
mysql> explain select * from foo;
+----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
| 1 | SIMPLE | foo | index | NULL | PRIMARY | 4 | NULL | 20 | Using index |
+----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
1 row in set (0.00 sec)
mysql> explain select * from foo limit 0, 10;
+----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
| 1 | SIMPLE | foo | index | NULL | PRIMARY | 4 | NULL | 20 | Using index |
+----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
1 row in set (0.00 sec)
mysql> explain select * from foo order by id desc limit 0, 10;
+----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
| 1 | SIMPLE | foo | index | NULL | PRIMARY | 4 | NULL | 10 | Using index |
+----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
1 row in set (0.00 sec)
Regarding your last comments about the comment join. Do you have an index on comment(post_id)? with my test data I'm getting the following results:
mysql> alter table comments add index pi (post_id);
Query OK, 0 rows affected (0.15 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> explain select c.id from comments c inner join (select id from posts o order by id limit 0, 10) p on c.post_id = p.id;
+----+-------------+------------+-------+---------------+---------+---------+------+------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+-------+---------------+---------+---------+------+------+--------------------------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 10 | |
| 1 | PRIMARY | c | ref | pi | pi | 5 | p.id | 4 | Using where; Using index |
| 2 | DERIVED | o | index | NULL | PRIMARY | 4 | NULL | 10 | Using index |
+----+-------------+------------+-------+---------------+---------+---------+------+------+--------------------------+
and for table size reference:
mysql> select count(*) from posts;
+----------+
| count(*) |
+----------+
| 15021 |
+----------+
1 row in set (0.01 sec)
mysql> select count(*) from comments;
+----------+
| count(*) |
+----------+
| 1000 |
+----------+
1 row in set (0.00 sec)

MySql 5.6: escaping consecutive percent signs in like query

Consider 2 queries:
SELECT * FROM T1 WHERE NAME LIKE '%\%%';
SELECT * FROM T1 WHERE NAME LIKE '%\%\%%';
Assume that T1 has records where NAME is %, %%, or %%%.
I would expect the second query to return fewer results but it is including the record where T1.NAME = '%'! Is there a way to filter out that record using like query? Something like SELECT * FROM T1 WHERE NAME LIKE '%\%\%%' AND NAME <> '%'; is not what I am looking for.
mysql> explain table1;
+-------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+------------+------+-----+---------+-------+
| id | int(11) | YES | | NULL | |
| text | varchar(4) | YES | | NULL | |
+-------+------------+------+-----+---------+-------+
2 rows in set (0.00 sec)
mysql> explain select * from table1
-> where text like '%\%%';
+----+-------------+--------+------+---------------+------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------+------+---------------+------+---------+------+------+-------------+
| 1 | SIMPLE | table1 | ALL | NULL | NULL | NULL | NULL | 4 | Using where |
+----+-------------+--------+------+---------------+------+---------+------+------+-------------+
1 row in set (0.00 sec)
mysql> explain select * from table1
-> where text like '%\%\%%';
+----+-------------+--------+------+---------------+------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------+------+---------------+------+---------+------+------+-------------+
| 1 | SIMPLE | table1 | ALL | NULL | NULL | NULL | NULL | 4 | Using where |
+----+-------------+--------+------+---------------+------+---------+------+------+-------------+
1 row in set (0.00 sec)
Here is my config if it helps:
[mysqld]
innodb_buffer_pool_size=402653184
innodb_log_file_size=262144000
innodb_log_buffer_size=8388608
max_allowed_packet=5241856
innodb_additional_mem_pool_size=20971520
I need to investigate further but it appears that the problem has something to do with the way that the database is being created:
create database testdb character set utf8 collate utf8_unicode_ci;
I was able to get the correct results when I create the database in a less specific way:
create database testdb;
Any idea why?

As far as I can tell, this appears to be a bug in mysql... it is sometimes impossible to search for consecutive percent signs using LIKE. (I was experiencing the same issue you described)
Still not sure what conditions are required to trigger the bug.
One workaround, however, would be to use REGEXP:
SELECT * FROM T1 WHERE NAME REGEXP "%%";

str_to_date returns null in query but fine on its own

I'm trying to sort some results by time. I've gathered str_to_date is the way to go, but I appear to be using it wrong, and I can't tell for sure, but I think it's converting to NULL and then not sorting in a meaningful way:
mysql> SELECT member_id, result_result, str_to_date('result_result','%i:%s.%f') FROM results WHERE workout_id = '2' ORDER BY str_to_date('result_result','%i:%s.%f') LIMIT 5;
+-----------+---------------+-----------------------------------------+
| member_id | result_result | str_to_date('result_result','%i:%s.%f') |
+-----------+---------------+-----------------------------------------+
| 0 | 1:35.0 | NULL |
| 1 | 1:35.0 | NULL |
| 3 | 1:40 | NULL |
| 4 | 1:37.8 | NULL |
| 7 | 1:27.3 | NULL |
+-----------+---------------+-----------------------------------------+
5 rows in set, 5 warnings (0.00 sec)
but the two result types seem to convert fine if I do it manually:
mysql> select str_to_date('1:40','%i:%s.%f');
+--------------------------------+
| str_to_date('1:40','%i:%s.%f') |
+--------------------------------+
| 00:01:40 |
+--------------------------------+
1 row in set (0.00 sec)
mysql> select str_to_date('1:35.0','%i:%s.%f');
+----------------------------------+
| str_to_date('1:35.0','%i:%s.%f') |
+----------------------------------+
| 00:01:35 |
+----------------------------------+
1 row in set (0.00 sec)
Any ideas what's happening / how to fix it?
Thanks!

You don't need the quotes inside the function. Try
str_to_date( result_result, '%i:%s.%f' )

We Keep Coding

html mysql json google-apps-script actionscript-3 ms-access google-chrome google-maps reporting-services sql-server-2008

mysql query to required to be faster - mysql

Related

Select count takes two minutes on a two column table

Mysql Why is the count(*) performance very fast after locking a table

How to improve Limit clause in MySQL

MySql 5.6: escaping consecutive percent signs in like query

str_to_date returns null in query but fine on its own

Categories

Resources