I have a myisam table with a primary key spanning 5 columns. I do a select using a WHERE on every of those 5 columns ANDed. Using the primary key (multicolumn index) it takes 25s, using a single index in one of the columns it takes 1 sec. I did a profiling and most of the 25s is taken in “Sending data” stage. The primary key has cardinality of about 7M and the single column about 80. Am i missing somehting?
CREATE TABLE `mytable` (
`a` int(11) unsigned NOT NULL,
`b` varchar(2) NOT NULL,
`c` int(11) unsigned NOT NULL,
`d` varchar(560) NOT NULL,
`e` varchar(45) NOT NULL,
PRIMARY KEY (`a`,`e`,`d`,`b`,`c`),
KEY `d` (`d`),
KEY `e` (`e`),
KEY `b` (`b`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
EXPLAIN SELECT * FROM mytable USE INDEX (PRIMARY)
WHERE a=12 AND e=1319677200 AND d='69.171.242.53' AND b='*' AND c=0;
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE i ref PRIMARY PRIMARY 4 const 5912231 Using where
EXPLAIN SELECT * FROM mytable
WHERE a=12 AND e=1319677200 AND d='69.171.242.53' AND b='*' AND c=0;
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE i ref PRIMARY,d,e,b d 562 const 158951 Using where
The problem is caused by casting,
try quote every varchar column b,d,e
SELECT * FROM mytable USE INDEX (PRIMARY)
WHERE a=12 AND e='1319677200' AND d='69.171.242.53' AND b='*' AND c=0;
Related
I have these tables.
CREATE TABLE `movements` (
`movementId` mediumint(8) UNSIGNED NOT NULL,
`movementType` tinyint(3) UNSIGNED NOT NULL,
`deleted` tinyint(1) UNSIGNED NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
ALTER TABLE `movements`
ADD PRIMARY KEY (`movementId`),
ADD KEY `movementType` (`movementType`) USING BTREE,
ADD KEY `deleted` (`deleted`),
ADD KEY `movementId` (`movementId`,`deleted`);
CREATE TABLE `movements_items` (
`movementId` mediumint(8) UNSIGNED NOT NULL,
`itemId` mediumint(8) UNSIGNED NOT NULL,
`qty` decimal(10,3) UNSIGNED NOT NULL,
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
ALTER TABLE `movements_items`
ADD KEY `movementId` (`movementId`),
ADD KEY `itemId` (`itemId`),
ADD KEY `movementId_2` (`movementId`,`itemId`);
and this view called "movements_items_view".
SELECT
movements_items.itemId, movements_items.qty,
movements.movementId, movements.movementType
FROM movements_items
JOIN movements ON (movements.movementId=movements_items.movementId
AND movements.deleted=0)
The first table has 5913 rows, the second one has 144992.
The view is very fast, it loads 20 result in PhpMyAdmin in 0.0011s but as soon as I ask for a GROUP BY on it (I need it to do statistics with SUM()) es:
SELECT * FROM movements_items_view GROUP BY itemId LIMIT 0,20
time jumps to 0.2s or more and it causes "Using where; Using temporary; Using filesort" on movements join.
Any help appreciated, thanks.
EDIT:
I also run via phpMyAdmin this query to try to not use the view:
SELECT movements.movementId, movements.movementType, movements_items.qty
FROM movements_items
JOIN movements ON movements.movementId=movements_items.movementId
GROUP BY itemId LIMIT 0,20
And the performance is the same.
Edit. Here is the EXPLAIN
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE movements index PRIMARY,movementid movement_type 1 NULL 5913 Using index; Using temporary; Using filesort
1 SIMPLE movements_items ref movementId,itemId,movementId_2 movementId_2 3 movements.movementId 12 Using index
Turn it inside out. See if this works:
SELECT movementId, m.movementType, mi.qty
FROM
( SELECT movementId, qty
FROM movements_items
GROUP BY itemId
ORDER BY itemId
LIMIT 20
) AS mi
JOIN movements AS m USING(movementId)
The trick is to do the LIMIT sooner. The original way had all the data being hauled around, not just 20 rows.
In movements_items, is no column or combination of columns 'unique'? If so, make it the PRIMARY KEY.
In movement, KEY movementId (movementId,deleted) is redundant and should be dropped.
In movement_items, KEY movementId (movementId) is redundant and should be dropped.
I have the following MySQL query which takes about 40 seconds on a linux VM:
SELECT
* FROM `clients_event_log`
WHERE
`ex_long` = 1475461 AND
`type` in (2, 1) AND NOT
(
(category=1 AND error=-2147212542) OR
(category=7 AND error=67)
)
ORDER BY `ev_time` DESC LIMIT 100
The table has around 7 million rows, aprox. 800 MB in size and it has indexes on all the fields used in the WHERE and ORDER BY clauses.
Now if I change the query in such a way that the ordering is done in an outer SELECT, everything works much faster (around 100ms):
SELECT res.* FROM
(
SELECT * FROM `clients_event_log`
WHERE
`ex_long` = 1475461 AND
`type` in (2, 1) AND NOT
(
(category=1 AND error=-2147212542) OR
(category=7 AND error=67)
)
) AS res
ORDER BY res.ev_time DESC LIMIT 0, 100
Do you have any idea why the first query takes such a long time? Thank you.
Later Update:
1st Query EXPLAIN:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE clients_event_log index category,ex_long,type,error,categ_error ev_time 4 NULL 5636 Using where
2nd Query EXPLAIN:
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY <derived2> system NULL NULL NULL NULL 1
2 DERIVED clients_event_log ref category,ex_long,type,error,categ_error ex_long 5 131264 Using where
Table definition:
CREATE TABLE `clients_event_log` (
`ev_id` int(11) NOT NULL,
`type` int(6) NOT NULL,
`ev_time` int(11) NOT NULL,
`category` smallint(6) NOT NULL,
`error` int(11) NOT NULL,
`ev_text` varchar(1024) DEFAULT NULL,
`userid` varchar(20) DEFAULT NULL,
`ex_long` int(11) DEFAULT NULL,
`client_ex_long` int(11) DEFAULT NULL,
`ex_text` varchar(1024) DEFAULT NULL,
PRIMARY KEY (`ev_id`),
KEY `category` (`category`),
KEY `ex_long` (`ex_long`),
KEY `type` (`type`),
KEY `ev_time` (`ev_time`),
KEY `error` (`error`),
KEY `categ_error` (`category`,`error`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
I ended up using the second query (inner SELECT) because the MySQL optimiser decided to always use the ev_time index even if I tried multiple versions of a composite index containing the columns in the WHERE and ORDER BY clauses.
Using force index (ex_long) also worked.
The MySQL version was 5.5.38
Thank you.
Add these
INDEX(ev_long, ev_time),
INDEX(ev_long, type)
and use the first format of the query and let the optimizer decide which is better based on the statistics.
I have this query:
SELECT `variationID`, count(DISTINCT(`userID`))
FROM data WHERE `testID` = XXXX AND `visit` = 1 GROUP BY `variationID`
;
that takes a lot of time to query.How I can speed up the query.
select_type table type possible_keys key key_len ref rows
filtered Extra SIMPLE data
ref dc3_testIDPage,dc3_testIDvarIDPage,user_test_varID_url
dc3_testIDvarIDPage 8 const 33106102 100.00 Using where
This is the output of the create table:
CREATE TABLE `data` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`userID` bigint(17) NOT NULL,
`testID` bigint(20) NOT NULL,
`variationID` bigint(20) NOT NULL,
`url` bigint(20) NOT NULL,
`time` bigint(20) NOT NULL,
`visit` bigint(20) NOT NULL DEFAULT '1',
`isTestPage` tinyint(1) NOT NULL,
PRIMARY KEY (`id`,`testID`),
KEY `url` (`url`),
KEY `dc3_testIDPage` (`testID`,`url`),
KEY `dc3_testIDvarIDPage` (`testID`,`variationID`,`url`),
KEY `user_test_url` (`userID`,`testID`,`url`),
KEY `user_test_varID_url` (`userID`,`testID`,`variationID`,`url`)
) ENGINE=InnoDB AUTO_INCREMENT=67153224 DEFAULT CHARSET=latin1
The easiest thing you can do to speed up your query is to make sure you are not doing full table scans. All the columns in your where clause should appear in indexes. So in your case testID and visit should have indexes and even better you can create a single index with both testID and visit. If visit is a true/false boolean that won't help narrow the index search much but testID certainly will.
Create index documentation is here: http://dev.mysql.com/doc/refman/5.7/en/create-index.html
Based on the create table you have id and testID are in a single primary key. Add a new key or index that only has testID in it. That should help quite a bit. Since it looks like visit is not a boolean adding an index with both visit and testID will give you the best performance boost.
I have a pretty simple query that gives me headaches because of a very long execution time which I cannot explain. The query:
explain select A.* from NAV_ADRESSEN A left outer join MITGL_KENNZEICHEN K on (K.MNR=A.MNR)
where ((A.MNR='19012546') or (IMPORTID='19012546') or (K.KENNZEICHEN='19012546')) and
(not UNGUELTIG) order by ZUNAME, VORNAME limit 0, 25;
The query (the real one, not the explain) takes about 17 seconds, whether or not any matches are found.
The explain result:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE A index MNR,IMPORTID ZUNAME 164 NULL 25 Using where
1 SIMPLE K ref MNR MNR 23 gsco-test.A.MNR 1 Using where
This looks pretty normal to me. All relevant columns have keys (A.MNR, K.MNR, A.IMPORTID, K.KENNZEICHEN); the tables contain ~600 000 rows (NAV_ADRESSEN) and 180 rows (MITGL_KENNZEICHEN).
What could be the problem?
Edited to add:
The explain looks slightly different when leaving out the limit clause (but the execution time doubles):
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE A ALL MNR,IMPORTID NULL NULL NULL 544587 Using where; Using filesort
1 SIMPLE K ref MNR MNR 23 gsco-test.A.MNR 1 Using where
The table definition:
CREATE TABLE `MITGL_KENNZEICHEN` (
`PK` int(11) unsigned NOT NULL AUTO_INCREMENT,
`MNR` varchar(20) DEFAULT NULL,
`KENNZEICHEN` varchar(80) DEFAULT NULL,
`DESCRIPTION` varchar(80) DEFAULT NULL,
PRIMARY KEY (`PK`),
KEY `MNR` (`MNR`),
KEY `KENNZEICHEN` (`KENNZEICHEN`)
) ENGINE=InnoDB AUTO_INCREMENT=247 DEFAULT CHARSET=latin1;
... and ...
CREATE TABLE `NAV_ADRESSEN` (
`PK` int(11) NOT NULL AUTO_INCREMENT,
`MNR` varchar(20) NOT NULL,
-- ... 50 fields omitted for brevity ...
`UNGUELTIG` tinyint(1) NOT NULL,
`IMPORTID` varchar(20) NOT NULL,
`MATCHCODE` varchar(255) DEFAULT NULL,
PRIMARY KEY (`PK`),
UNIQUE KEY `MNR` (`MNR`),
KEY `ZUNAME` (`ZUNAME`,`VORNAME`),
KEY `IMPORTID` (`IMPORTID`),
KEY `MATCHCODE` (`MATCHCODE`),
KEY `ANGELEGTDAT` (`ANGELEGTDAT`)
) ENGINE=InnoDB AUTO_INCREMENT=1076829 DEFAULT CHARSET=latin1;
Just FYI, that query can (presumably) be rewritten as follows:
SELECT a.*
FROM nav_adressen a
JOIN mitgl_kennzeichen k
ON k.mnr = a.mnr
AND 19012546 IN (a.mnr,importid,k.kennzeichen)
AND NOT ungueltid
ORDER
BY zuname
, vorname
LIMIT 0,25;
I am doing a simple Select from a single table.
CREATE TABLE `book` (
`Book_Id` int(10) NOT NULL AUTO_INCREMENT,
`Book_Name` varchar(100) COLLATE utf8_turkish_ci DEFAULT NULL ,
`Book_Active` bit(1) NOT NULL DEFAULT b'1' ,
`Author_Id` int(11) NOT NULL,
PRIMARY KEY (`Book_Id`),
KEY `FK_Author` (`Author_Id`),
CONSTRAINT `FK_Author` FOREIGN KEY (`Author_Id`) REFERENCES `author` (`Author_Id`) ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=5947698 DEFAULT CHARSET=utf8 COLLATE=utf8_turkish_ci ROW_FORMAT=COMPACT
table : book
columns :
Book_Id (INTEGER 10) | Book_Name (VARCHAR 100) | Author_Id (INTEGER 10) | Book_Active (Boolean)
I have Indexes on three columns : Book_Id (PRIMARY key) , Author_Id (FK) , Book_Active .
first query :
SELECT * FROM book WHERE Author_Id = 1 AND Book_Active = 1
EXPLAIN :
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE book ref FK_Author,index_Book_Active FK_Author 4 const 4488510 Using where
second query :
SELECT b.* FROM book b
WHERE Book_Active=1
AND Book_Id IN (SELECT Book_Id FROM book WHERE Author_Id=1)
EXPLAIN :
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY book ref index_Book_Active index_Book_Active 1 const 9369399 Using where
2 DEPENDENT SUBQUERY book unique_subquery PRIMARY,FK_Author PRIMARY 4 func 1 Using where
The data statistics is like this :
16.8 million books
10.5 million Book_Active=true
6.3 million Book_Active = false
And For Author_Id=1
2.4 million Book_Active=false
5000 Book_Active=true
The first query takes 6.7 seconds . The second query takes 0.0002 seconds
What is the cause of this enormous difference ? is it the right thing to use the nested select query ?
edit: added "sql explain"
In the first case: MySQL uses FK_Author index (this gives us ~4.5M rows), then it has to match every row against Book_Active = 1 condition — index cannot be used here.
The second case: InnoDB implicitly adds primary key to every index. Thus, when MySQL executes this part: SELECT book.* FROM book WHERE Book_Active=1 it has Book_Id from the index. Then for the subquery it has to match Book_Id with Author_Id; Author_Id is a constant and is a prefix of the index; implicitly included primary key can be matched against implicit primary key from Book_Active index. In your case it is faster to intersect two indices than to use index to retrieve 4.5M rows and scan them sequentially.