mysql query taking more than 30 seconds - mysql

Below is the output of "show full processlist" on my-sql command line. this query calculates the users total upload and download bandwidth for specified period of time. (the database is from freeradius). How can i make the query faster. [ Memory:150G , processr:8 centos 6.8]
15260415|radiusremote|panel.example.com:57526|radius|Query|35|Copying to tmp table
SELECT sum(acctinputoctets) as upload,sum(acctoutputoctets) as download
FROM radacct a
INNER JOIN
( SELECT acctuniqueid, MIN( radacctid ) radacctid
FROM radacct
WHERE username='nyjohan'
and acctstarttime between '2016-01-15 13:50:05'
AND '2016-08-07 13:16:36'
GROUP BY acctuniqueid
) b ON a.acctuniqueid = b.acctuniqueid
AND a.radacctid = b.radacctid
There are indexes created on the table , below is the output of indexes on table
mysql> show index from radacct;
+---------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+---------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+
| radacct | 0 | PRIMARY | 1 | radacctid | A | 161791738 | NULL | NULL | | BTREE | |
| radacct | 1 | username | 1 | username | A | 15 | NULL | NULL | | BTREE | |
| radacct | 1 | framedipaddress | 1 | framedipaddress | A | 458333 | NULL | NULL | | BTREE | |
| radacct | 1 | acctsessionid | 1 | acctsessionid | A | 161791738 | NULL | NULL | | BTREE | |
| radacct | 1 | acctsessiontime | 1 | acctsessiontime | A | 46332 | NULL | NULL | YES | BTREE | |
| radacct | 1 | acctuniqueid | 1 | acctuniqueid | A | 161791738 | NULL | NULL | | BTREE | |
| radacct | 1 | acctstarttime | 1 | acctstarttime | A | 40447934 | NULL | NULL | YES | BTREE | |
| radacct | 1 | acctstoptime | 1 | acctstoptime | A | 80895869 | NULL | NULL | YES | BTREE | |
| radacct | 1 | nasipaddress | 1 | nasipaddress | A | 15 | NULL | NULL | | BTREE | |
+---------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+
Schema of table
mysql> describe radacct;
+----------------------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------------+-------------+------+-----+---------+----------------+
| radacctid | bigint(21) | NO | PRI | NULL | auto_increment |
| acctsessionid | varchar(32) | NO | MUL | | |
| acctuniqueid | varchar(32) | NO | MUL | | |
| username | varchar(64) | NO | MUL | | |
| groupname | varchar(64) | NO | | | |
| realm | varchar(64) | YES | | | |
| nasipaddress | varchar(15) | NO | MUL | | |
| nasportid | varchar(15) | YES | | NULL | |
| nasporttype | varchar(32) | YES | | NULL | |
| acctstarttime | datetime | YES | MUL | NULL | |
| acctstoptime | datetime | YES | MUL | NULL | |
| acctsessiontime | int(12) | YES | MUL | NULL | |
| acctauthentic | varchar(32) | YES | | NULL | |
| connectinfo_start | varchar(50) | YES | | NULL | |
| connectinfo_stop | varchar(50) | YES | | NULL | |
| acctinputoctets | bigint(20) | YES | | NULL | |
| acctoutputoctets | bigint(20) | YES | | NULL | |
| calledstationid | varchar(50) | NO | | | |
| callingstationid | varchar(50) | NO | | | |
| acctterminatecause | varchar(32) | NO | | | |
| servicetype | varchar(32) | YES | | NULL | |
| framedprotocol | varchar(32) | YES | | NULL | |
| framedipaddress | varchar(15) | NO | MUL | | |
| acctstartdelay | int(12) | YES | | NULL | |
| acctstopdelay | int(12) | YES | | NULL | |
| xascendsessionsvrkey | varchar(10) | YES | | NULL | |
+----------------------+-------------+------+-----+---------+----------------+
Explain output:-
explain SELECT sum(acctinputoctets) as upload,sum(acctoutputoctets) as download
FROM radacct a
INNER JOIN (
SELECT acctuniqueid, MIN( radacctid ) radacctid
FROM radacct
WHERE username='dave137' and acctstarttime between '2016-08-03 00:00:00' and '2016-08-07 14:47:54' GROUP BY acctuniqueid
)b ON a.acctuniqueid = b.acctuniqueid
AND a.radacctid = b.radacctid ;
+----+-------------+------------+--------+------------------------+----------+---------+-------------+-------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+--------+------------------------+----------+---------+-------------+-------+----------------------------------------------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 10 | |
| 1 | PRIMARY | a | eq_ref | PRIMARY,acctuniqueid | PRIMARY | 8 | b.radacctid | 1 | Using where |
| 2 | DERIVED | radacct | ref | username,acctstarttime | username | 66 | | 10164 | Using where; Using temporary; Using filesort |
+----+-------------+------------+--------+------------------------+----------+---------+-------------+-------+----------------------------------------------+
3 rows in set (9.91 sec)

For this query:
SELECT sum(acctinputoctets) as upload,sum(acctoutputoctets) as download
FROM radacct a INNER JOIN
(SELECT acctuniqueid, MIN( radacctid ) as radacctid
FROM radacct
WHERE username = 'dave137' and acctstarttime between '2016-08-03 00:00:00' and '2016-08-07 14:47:54'
GROUP BY acctuniqueid
) b
ON a.acctuniqueid = b.acctuniqueid AND a.radacctid = b.radacctid ;
I would recommend two indexes, one is already present: radacct(radacctid) and radacct(username, acctstarttime, acctuniqueid).
In addition, I would simplify the ON clause to:
ON a.radacctid = b.radacctid ;
radacctid is unique so there is no need for another condition.

Using a JOIN on a sub-query will slow down your query regardless of indexes. From what I can see this should do the same thing without needing to do the sub-join:
SELECT acctinputoctets as upload, acctoutputoctets as download
FROM radacct a
WHERE username = 'dave137'
and acctstarttime between '2016-08-03 00:00:00' and '2016-08-07 14:47:54'
ORDER BY radacctid
LIMIT 1

Related

mysql join query not using index

im stuck with a query which take more than 13 secs to return output
i did create the indexes but the execution plan doesnt seem to use them. Below are some information regarding
Query
SELECT userinfo.username,userinfo.firstname,userinfo.lastname,userinfo.email,
radcheck.attribute,radcheck.value,radusergroup.groupname, userinfo.id,
userinfo.workphone,userinfo.homephone,
userinfo.mobilephone,userinfo.address, userinfo.zone,
userinfo.account_state,userinfo.device_owner,userinfo.link_type,userinfo.account_type
FROM userinfo
INNER JOIN radcheck ON userinfo.username = radcheck.username
INNER JOIN radusergroup ON userinfo.username = radusergroup.username
WHERE radcheck.attribute='Expiration' and
STR_TO_DATE(radcheck.value, '%d %M %Y') < CURDATE()
and radusergroup.groupname='Customer30M';
Explain output
+----+-------------+--------------+------+--------------------+-----------+---------+--------------------------------+------+-----------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------------+------+--------------------+-----------+---------+--------------------------------+------+-----------------------------------+
| 1 | SIMPLE | radusergroup | ref | username,groupname | groupname | 66 | const | 5 | Using where with pushed condition |
| 1 | SIMPLE | radcheck | ref | username,attribute | attribute | 34 | const | 9 | Using where |
| 1 | SIMPLE | userinfo | ref | username | username | 131 | ctradius.radusergroup.username | 10 | Using where |
+----+-------------+--------------+------+--------------------+-----------+---------+--------------------------------+------+-----------------------------------+
3 rows in set (0.00 sec)
Table Info
Table: radusergroup
+-----------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| username | varchar(64) | NO | MUL | | |
| groupname | varchar(64) | NO | MUL | | |
| priority | int(11) | NO | | 1 | |
+-----------+-------------+------+-----+---------+----------------+
radusergroup table Index
+--------------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| radusergroup | 0 | PRIMARY | 1 | id | A | 11292 | NULL | NULL | | BTREE | | |
| radusergroup | 1 | username | 1 | username | A | NULL | NULL | NULL | | BTREE | | |
| radusergroup | 1 | groupname | 1 | groupname | A | NULL | NULL | NULL | | BTREE | | |
| radusergroup | 1 | indexradusergroup | 1 | username | A | NULL | NULL | NULL | | BTREE | | |
| radusergroup | 1 | indexradusergroup | 2 | groupname | A | NULL | NULL | NULL | | BTREE | | |
Table : userinfo
---------------------+------------------+------+-----+---------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------------+------------------+------+-----+---------------------+----------------+
| id | int(11) unsigned | NO | PRI | NULL | auto_increment |
| username | varchar(128) | YES | MUL | NULL | |
| firstname | varchar(200) | YES | | NULL | |
| lastname | varchar(200) | YES | | NULL | |
| email | varchar(200) | YES | | NULL | |
| workphone | varchar(200) | YES | | NULL | |
| homephone | varchar(200) | YES | | NULL | |
| mobilephone | varchar(200) | YES | | NULL | |
| address | varchar(200) | YES | | NULL | |
| zone | varchar(20) | YES | | NULL | |
| account_state | varchar(10) | YES | | active | |
| change_commit | tinyint(1) | YES | | NULL | |
| link_type | varchar(8) | YES | | NULL | |
| account_type | varchar(255) | YES | | NULL | |
| device_owner | varchar(255) | YES | | NULL | |
| account_name | varchar(255) | YES | | NULL | |
| account_email | varchar(255) | YES | | NULL | |
| account_mobile | varchar(255) | YES | | NULL | |
| groupname | varchar(255) | YES | | NULL | |
+---------------------+------------------+------+-----+---------------------+----------------+
Indexes on userinfo
----------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+----------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| userinfo | 0 | PRIMARY | 1 | id | A | 11282 | NULL | NULL | | BTREE | | |
| userinfo | 1 | username | 1 | username | A | NULL | NULL | NULL | YES | BTREE | | |
+----------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
Table: Radcheck
+-----------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------------+------+-----+---------+----------------+
| id | int(11) unsigned | NO | PRI | NULL | auto_increment |
| username | varchar(64) | NO | MUL | | |
| attribute | varchar(32) | NO | MUL | | |
| op | char(2) | NO | | == | |
| value | varchar(253) | NO | MUL | | |
+-----------+------------------+------+-----+---------+----------------+
indexes on radcheck
+----------+------------+---------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+----------+------------+---------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| radcheck | 0 | PRIMARY | 1 | id | A | 33425 | NULL | NULL | | BTREE | | |
| radcheck | 1 | username | 1 | username | A | NULL | NULL | NULL | | BTREE | | |
| radcheck | 1 | value | 1 | value | A | NULL | NULL | NULL | | BTREE | | |
| radcheck | 1 | attribute | 1 | attribute | A | NULL | NULL | NULL | | BTREE | | |
The function invalidates the use of the index. Despite your cardinality showing 11k rows, the table scan ensues and it takes 13 seconds.
Though you have a join between Radcheck and userinfo, it is the STR_TO_DATE function in the where clause that kills the index use.

Optimize sql statement with OR in where clause

running this query on 5Gb sized messages table.
The problem is that the execution takes > 3 minutes.
SELECT m.id FROM messages m
LEFT JOIN dialog d on m.id=d.mid
WHERE (SELECT count(*)
FROM dialog
WHERE (m.from_id=uid1 and m.user_id=uid2)
OR (m.from_id=uid2 and m.user_id=uid1))=0 && read_state=0
LIMIT 100
I understand, that this is a bad practice to search by NESTED SELECT IN WHERE CLAUSE, but yet didn`t find another way to select such rows. Tried to split OR to 2 UNION statements, but it was long either.
messages table structure:
+------------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+-------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| from_id | int(11) | NO | MUL | NULL | |
| user_id | int(11) | NO | MUL | NULL | |
| group_id | int(11) | NO | | NULL | |
| to_number | varchar(30) | NO | MUL | NULL | |
| msg | text | NO | | NULL | |
| image | varchar(20) | NO | | NULL | |
| date | bigint(20) | NO | | NULL | |
| read_state | tinyint(1) | NO | | 0 | |
| removed | tinyint(1) | NO | | NULL | |
+------------+-------------+------+-----+---------+----------------+
dialog table structure
+-----------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| uid1 | int(11) | NO | MUL | NULL | |
| uid2 | int(11) | NO | MUL | NULL | |
| mid | int(11) | NO | | NULL | |
| anonym_id | int(10) unsigned | NO | | NULL | |
+-----------+------------------+------+-----+---------+----------------+
mysql> show index from messages;
+----------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+----------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| messages | 0 | PRIMARY | 1 | id | A | 12560908 | NULL | NULL | | BTREE | | |
| messages | 1 | to_number | 1 | to_number | A | 161037 | NULL | NULL | | BTREE | | |
| messages | 1 | from_id | 1 | from_id | A | 157011 | NULL | NULL | | BTREE | | |
| messages | 1 | from_id | 2 | to_number | A | 169742 | NULL | NULL | | BTREE | | |
| messages | 1 | user_id_2 | 1 | user_id | A | 314022 | NULL | NULL | | BTREE | | |
| messages | 1 | user_id_2 | 2 | read_state | A | 380633 | NULL | NULL | | BTREE | | |
| messages | 1 | user_id_2 | 3 | removed | A | 392528 | NULL | NULL | | BTREE | | |
+----------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
mysql> show index from dialog;
+--------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| dialog | 0 | PRIMARY | 1 | id | A | 3125615 | NULL | NULL | | BTREE | | |
| dialog | 1 | uid1 | 1 | uid1 | A | 520935 | NULL | NULL | | BTREE | | |
| dialog | 1 | uid1 | 2 | uid2 | A | 3125615 | NULL | NULL | | BTREE | | |
| dialog | 1 | uid2 | 1 | uid2 | A | 1562807 | NULL | NULL | | BTREE | | |
+--------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
EXPLAIN EXTENDED
+----+--------------------+--------+-------+---------------+------+---------+------+----------+----------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+--------------------+--------+-------+---------------+------+---------+------+----------+----------+--------------------------+
| 1 | PRIMARY | m | ALL | NULL | NULL | NULL | NULL | 22190398 | 100.00 | Using where |
| 1 | PRIMARY | d | ALL | NULL | NULL | NULL | NULL | 3125621 | 100.00 | |
| 2 | DEPENDENT SUBQUERY | dialog | index | uid1,uid2 | uid1 | 8 | NULL | 3125621 | 100.00 | Using where; Using index |
+----+--------------------+--------+-------+---------------+------+---------+------+----------+----------+--------------------------+
The first thought is to change the subquery query to use not exists instead of count(*). The second is to split this into two separate subqueries. The third is to add indexes:
create index idx_messages_read_state_4 on messages_read_state(user_id, from_id, user_id, id);
create index idx_dialog_2 on dialog(uid1, uid2)
And the fourth is to remove the left join to dialog in the outer query. No fields from dialog are being used and the left join means it is not being used for filtering.
The query is then:
select m.id
from messages m
where m.read_state = 0 and
not exists (select 1
from dialog d
where m.from_id = d.uid1 and m.user_id = d.uid2
) and
not exists (select 1
from dialog d
where m.from_id = d.uid2 and m.user_id = d.uid1
)
limit 100;

Why I can use index on "select count(*) from table_name" but cannot work on "select * from table_name" ? Is there any difference?

I have a table like this:
+--------------+--------------+------+-----+--------------------------------------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------+--------------+------+-----+--------------------------------------+-------+
| id | varchar(36) | NO | PRI | NULL | |
| provider_id | varchar(36) | YES | MUL | 00000000-0000-0000-0000-000000000000 | |
| to_provider | int(11) | NO | | NULL | |
| to_customer | int(11) | NO | | NULL | |
| published_at | datetime | NO | | NULL | |
| expired_at | datetime | NO | | NULL | |
| title | varchar(512) | NO | | NULL | |
| content | text | YES | | NULL | |
| created_at | datetime | NO | | NULL | |
| created_by | varchar(255) | YES | | NULL | |
| updated_at | datetime | YES | | NULL | |
| updated_by | varchar(255) | YES | | NULL | |
| deleted_at | datetime | YES | | NULL | |
| deleted_by | varchar(255) | YES | | NULL | |
+--------------+--------------+------+-----+--------------------------------------+-------+
And I create an index like this :
+---------------+------------+-------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------------+------------+-------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| announcements | 0 | PRIMARY | 1 | id | A | 76184 | NULL | NULL | | BTREE | | |
| announcements | 1 | idx_announcements | 1 | provider_id | A | 7 | NULL | NULL | YES | BTREE | | |
| announcements | 1 | idx_announcements | 2 | deleted_at | A | 7 | NULL | NULL | YES | BTREE | | |
| announcements | 1 | idx_announcements | 3 | published_at | A | 7 | NULL | NULL | | BTREE | | |
| announcements | 1 | idx_announcements | 4 | to_provider | A | 7 | NULL | NULL | | BTREE | | |
| announcements | 1 | idx_announcements | 5 | to_customer | A | 7 | NULL | NULL | | BTREE | | |
| announcements | 1 | idx_announcements | 6 | expired_at | A | 7 | NULL | NULL | | BTREE | | |
| announcements | 1 | idx_announcements | 7 | updated_at | A | 7 | NULL | NULL | YES | BTREE | | |
+---------------+------------+-------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
When I run
MariaDB [table_name]> EXPLAIN SELECT COUNT(*) FROM `announcements` `t` WHERE (((`t`.provider_id = "3c5e63df-cb9b-f5a8-4eaf-7ed0061b797d") OR (provider_id="00000000-0000-0000-0000-000000000000")) AND (`t`.deleted_at IS NULL))AND(((published_at <= "2015-07-17 14:54:36") AND ( to_provider IN (1))) AND (to_customer IN (0, 1)));
+------+-------------+-------+-------+-------------------+-------------------+---------+------+-------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------+-------+-------------------+-------------------+---------+------+-------+--------------------------+
| 1 | SIMPLE | t | range | idx_announcements | idx_announcements | 64 | NULL | 38093 | Using where; Using index |
+------+-------------+-------+-------+-------------------+-------------------+---------+------+-------+--------------------------+
See, I can use the index, but when I run
MariaDB [table_name]> EXPLAIN SELECT * FROM `announcements` `t` WHERE (((`t`.provider_id = "3c5e63df-cb9b-f5a8-4eaf-7ed0061b797d") OR (provider_id="00000000-0000-0000-0000-000000000000")) AND (`t`.deleted_at IS NULL))AND(((published_at <= "2015-07-17 14:54:36") AND ( to_provider IN (1))) AND (to_customer IN (0, 1)));
+------+-------------+-------+------+-------------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------+------+-------------------+------+---------+------+-------+-------------+
| 1 | SIMPLE | t | ALL | idx_announcements | NULL | NULL | NULL | 76184 | Using where |
+------+-------------+-------+------+-------------------+------+---------+------+-------+-------------+
See, the key is null! I really don't get it. use the same where, but "select *" can not use the index.
Any one could tell me ?
Oh, and I got a SQL like this
SELECT * FROM `announcements` `t`
WHERE
(((`t`.provider_id = :provider_id) OR (provider_id=:ycp0)) AND (`t`.deleted_at IS NULL))
AND (published_at<="2015-07-17 16:58:57" AND (to_provider IN(1) AND to_customer IN (0,1))
AND expired_at>="2015-07-17 16:58:57"
AND (updated_at > "2015-07-10 16:58:57" || published_at > "2015-07-10 16:58:57" ))
ORDER BY updated_at DESC LIMIT 10
How to create index for this query ? I got a problem on "OR" thing ~~~
One way round the issue of MySQL only using a single index on any instance of a table in a select is to split the select into 2 and union them together.
For example with your select, if both updated_at and published_at would be useful indexes then you could allow MySQL to use them both as follows:-
SELECT *
FROM announcements t
WHERE t.provider_id IN ( :provider_id, :ycp0)
AND t.deleted_at IS NULL
AND published_at<="2015-07-17 16:58:57"
AND to_provider IN(1)
AND to_customer IN (0,1)
AND expired_at>="2015-07-17 16:58:57"
AND updated_at > "2015-07-10 16:58:57"
UNION
SELECT *
FROM announcements t
WHERE t.provider_id IN ( :provider_id, :ycp0)
AND t.deleted_at IS NULL
AND published_at<="2015-07-17 16:58:57"
AND to_provider IN(1)
AND to_customer IN (0,1)
AND expired_at>="2015-07-17 16:58:57"
AND published_at > "2015-07-10 16:58:57"
ORDER BY updated_at DESC LIMIT 10

MySQL Queries with a range on the Join (speed issue)

I'm trying to run the following query:
SELECT formatted_journeys.*, MAX(speed)
FROM formatted_journeys
JOIN tracker.g_log
ON imeiid = vehicle
AND g_logid BETWEEN start_g_log AND end_g_log
GROUP BY id
however, it seems to be extremely slow. Here's the explain for that query.
+----+-------------+--------------------+------+-------------------------------+-------+---------+-----------------------------------+------+----------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------------------+------+-------------------------------+-------+---------+-----------------------------------+------+----------+---------------------------------+
| 1 | SIMPLE | formatted_journeys | ALL | vehicle,start_g_log,end_g_log | NULL | NULL | NULL | 824 | 100.00 | Using temporary; Using filesort |
| 1 | SIMPLE | g_log | ref | PRIMARY,Dupes | Dupes | 4 | motrak.formatted_journeys.vehicle | 1985 | 100.00 | Using where |
+----+-------------+--------------------+------+-------------------------------+-------+---------+-----------------------------------+------+----------+---------------------------------+
2 rows in set, 1 warning (0.02 sec)
The tables are as follows:
formatted_journeys:
+-----------------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+------------------+------+-----+---------+----------------+
| id | int(11) unsigned | NO | PRI | NULL | auto_increment |
| start_g_log | int(11) | YES | MUL | NULL | |
| end_g_log | int(11) | YES | MUL | NULL | |
| start_latitude | decimal(18,12) | YES | | NULL | |
| start_longitude | decimal(18,12) | YES | | NULL | |
| end_latitude | decimal(18,12) | YES | | NULL | |
| end_longitude | decimal(18,12) | YES | | NULL | |
| start_location | text | YES | | NULL | |
| end_location | text | YES | | NULL | |
| distance | decimal(10,5) | YES | | NULL | |
| start_date | datetime | YES | | NULL | |
| end_date | datetime | YES | | NULL | |
| vehicle | int(11) | YES | MUL | NULL | |
| private | bit(1) | NO | | b'0' | |
+-----------------+------------------+------+-----+---------+----------------+
Indices:
+--------------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+--------------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| formatted_journeys | 0 | PRIMARY | 1 | id | A | 830 | NULL | NULL | | BTREE | |
| formatted_journeys | 1 | vehicle | 1 | vehicle | A | 4 | NULL | NULL | YES | BTREE | |
| formatted_journeys | 1 | start_g_log | 1 | start_g_log | A | 830 | NULL | NULL | YES | BTREE | |
| formatted_journeys | 1 | end_g_log | 1 | end_g_log | A | 830 | NULL | NULL | YES | BTREE | |
+--------------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
tracker.g_log:
+-----------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------------+------+-----+---------+----------------+
| g_logid | int(10) unsigned | NO | PRI | NULL | auto_increment |
| imeiid | int(10) unsigned | NO | MUL | NULL | |
| latitude | decimal(18,12) | YES | | NULL | |
| longitude | decimal(18,12) | YES | | NULL | |
| speed | int(4) | YES | MUL | NULL | |
| bearing | int(4) | YES | | NULL | |
| distance | decimal(10,5) | YES | | NULL | |
| eventcode | int(10) | YES | | NULL | |
| status | int(10) | YES | | NULL | |
| date | datetime | YES | | NULL | |
+-----------+------------------+------+-----+---------+----------------+
Indices:
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| g_log | 0 | PRIMARY | 1 | g_logid | A | 31760 | NULL | NULL | | BTREE | |
| g_log | 0 | Dupes | 1 | imeiid | A | 16 | NULL | NULL | | BTREE | |
| g_log | 0 | Dupes | 2 | date | A | 31760 | NULL | NULL | YES | BTREE | |
| g_log | 0 | Dupes | 3 | eventcode | A | 31760 | NULL | NULL | YES | BTREE | |
| g_log | 1 | speed | 1 | speed | A | 423 | NULL | NULL | YES | BTREE | |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
Now I know that the filesort isn't a good thing but how would I get rid of it?
You will hardly get rid of Using temporary; Using filesort, but the query might be faster.
First, try to rewrite the query as:
SELECT *
FROM (
SELECT id, MAX(speed) as max_speed
FROM formatted_journeys
JOIN tracker.g_log
ON imeiid = vehicle
AND g_logid BETWEEN start_g_log AND end_g_log
GROUP BY id) as maxspeeds
JOIN formatted_journeys USING (id);
Then, you can try to force the query to use a coverage index, though that is not easy for the query.
First try: add a composite index (vehicle, start_g_log, end_g_log) and look in explain whether it is used (you should see "Using index" there)

Why doesn't mysql use my index?

I have two tables:
mysql> desc myps3t_gameusertrophyinfo;
+-----------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| user_id | int(11) | NO | MUL | NULL | |
| trophy_id | int(11) | NO | MUL | NULL | |
| date | datetime | NO | MUL | NULL | |
| date_read | varchar(100) | NO | | NULL | |
+-----------+--------------+------+-----+---------+----------------+
5 rows in set (0.19 sec)
mysql> show index from myps3t_gameusertrophyinfo;
+---------------------------+------------+------------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+---------------------------+------------+------------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| myps3t_gameusertrophyinfo | 0 | PRIMARY | 1 | id | A | 4004589 | NULL | NULL | | BTREE | |
| myps3t_gameusertrophyinfo | 0 | user_id | 1 | user_id | A | 7686 | NULL | NULL | | BTREE | |
| myps3t_gameusertrophyinfo | 0 | user_id | 2 | trophy_id | A | 4004589 | NULL | NULL | | BTREE | |
| myps3t_gameusertrophyinfo | 1 | myps3t_gameusertrophyinfo_403f60f | 1 | user_id | A | 7686 | NULL | NULL | | BTREE | |
| myps3t_gameusertrophyinfo | 1 | myps3t_gameusertrophyinfo_61a683d8 | 1 | trophy_id | A | 22624 | NULL | NULL | | BTREE | |
| myps3t_gameusertrophyinfo | 1 | idx_gameusertrophyinfo_date | 1 | date | A | 4004589 | NULL | NULL | | BTREE | |
+---------------------------+------------+------------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
7 rows in set (0.06 sec)
the other table:
mysql> desc myps3t_gametrophyinfo ;
+-----------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| game_id | int(11) | NO | MUL | NULL | |
| name | varchar(500) | NO | | NULL | |
| desc | varchar(500) | NO | | NULL | |
| type | varchar(20) | NO | | NULL | |
| pic_url | varchar(200) | NO | | NULL | |
| desc_pt | varchar(500) | NO | | NULL | |
| name_pt | varchar(500) | NO | | NULL | |
| hidden_id | int(11) | NO | | NULL | |
| total_id | int(11) | NO | | NULL | |
| trophy_id | int(11) | NO | | NULL | |
| addon_id | int(11) | YES | | NULL | |
| points | double | NO | | 0 | |
| sony_id | int(11) | YES | | NULL | |
+-----------+--------------+------+-----+---------+----------------+
14 rows in set (0.00 sec)
mysql> show index from myps3t_gametrophyinfo;
+-----------------------+------------+-------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-----------------------+------------+-------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| myps3t_gametrophyinfo | 0 | PRIMARY | 1 | id | A | 25976 | NULL | NULL | | BTREE | |
| myps3t_gametrophyinfo | 1 | myps3t_gametrophyinfo_game_id | 1 | game_id | A | 764 | NULL | NULL | | BTREE | |
+-----------------------+------------+-------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
when i do this query:
mysql> explain select * from myps3t_gameusertrophyinfo a, myps3t_gametrophyinfo b where a.trophy_id = b.id and b.addon_id = 58; +----+-------------+-------+--------+--------------------------------------------------------------+---------+---------+-----------------------------+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+--------------------------------------------------------------+---------+---------+-----------------------------+---------+-------------+
| 1 | SIMPLE | a | ALL | myps3t_gameusertrophyinfo_61a683d8,myps3t_gameusertrophyinfo | NULL | NULL | NULL | 4004592 | |
| 1 | SIMPLE | b | eq_ref | PRIMARY | PRIMARY | 4 | fabriciols_ps3t.a.trophy_id | 1 | Using where |
+----+-------------+-------+--------+--------------------------------------------------------------+---------+---------+-----------------------------+---------+-------------+
2 rows in set (0.00 sec)
why mysql do not use my keys ?
this query take more than 30 seconds, the first table has 4milion records ...
-- edit --
for quasnoi
mysql> SELECT COUNT(*), COUNT(DISTINCT addon_id), SUM(addon_id = 58) FROM myps3t_gametrophyinfo;
+----------+--------------------------+--------------------+
| COUNT(*) | COUNT(DISTINCT addon_id) | SUM(addon_id = 58) |
+----------+--------------------------+--------------------+
| 25976 | 160 | 6 |
+----------+--------------------------+--------------------+
1 row in set (0.00 sec)
MySQL chooses a as a leading table and b as a driven table. It does use a PRIMARY KEY on b for the joins.
Create an index on myps3t_gametrophyinfo (addon_id), this way b will be more probably chosen as a leading table.
You can try
select * from
myps3t_gametrophyinfo b
STRAIGHT_JOIN myps3t_gameusertrophyinfo a ON (a.trophy_id = b.id)
WHERE b.addon_id = 58;
I would probably rewrite the query to try to get a more sane execution path. I think something like below is more likely to get you the performance you want and is more clear in what you are doing to a human reader
SELECT * FROM myps3t_gametrophyinfo a LEFT JOIN myps3t_gameusertrophyinfo b ON a.id = b.trophy_id WHERE a.addon_id=58;