Running a simple query on an indexed column, yet it's taking over 500ms. These queries run often so it's affecting performance in a big way.
Just under 1M rows in the table, a very simple table. Using MyISAM. I don't understand why it's examining all rows, seems like it's ignoring the index! I tried adding a second index on the field, a normal one instead of a unique index, didn't make any difference. Thanks for looking.
# Time: 130730 22:00:07
# User#Host: engine[engine] # engine [10.0.0.6]
# Query_time: 0.511209 Lock_time: 0.000050 Rows_sent: 1 Rows_examined: 932048
SET timestamp=1375236007;
SELECT * FROM `marketplacesales`.`sales_order` WHERE `marketplace_orderid` = 823123693003;
+---------------------+-----------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------------+-----------------------+------+-----+---------+----------------+
| id | mediumint(8) unsigned | NO | PRI | NULL | auto_increment |
| marketplace_id | smallint(5) unsigned | NO | MUL | NULL | |
| marketplace_orderid | varchar(255) | NO | UNI | NULL | |
| datetime | datetime | NO | | NULL | |
| added | datetime | NO | MUL | NULL | |
| phone | varchar(255) | YES | | NULL | |
| email | varchar(255) | YES | | NULL | |
| company | varchar(255) | NO | | NULL | |
| name | varchar(255) | NO | | NULL | |
| address1 | varchar(255) | NO | | NULL | |
| address2 | varchar(255) | NO | | NULL | |
| city | varchar(255) | NO | | NULL | |
| state | varchar(255) | NO | | NULL | |
| zip | varchar(255) | NO | | NULL | |
| country | varchar(255) | NO | | NULL | |
+---------------------+-----------------------+------+-----+---------+----------------+
EXPLAIN SELECT * FROM `marketplacesales`.`sales_order` WHERE `marketplace_orderid` = 823123693003;
+----+-------------+-------------+------+-------------------------------------------+------+---------+------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------+------+-------------------------------------------+------+---------+------+--------+-------------+
| 1 | SIMPLE | sales_order | ALL | marketplace_orderid,marketplace_orderid_2 | NULL | NULL | NULL | 932053 | Using where |
+----+-------------+-------------+------+-------------------------------------------+------+---------+------+--------+-------------+
1 row in set (0.00 sec)
You need to use explain to see what is happening. My guess is that the index is not being used.
The where clause is:
WHERE `marketplace_orderid` = 823123693003;
As explained here, the conversion will take place as floating point numbers. This requires a conversion on marketplace_orderid.
Either fix the field in the table so it is numeric. Or, put the value in quotes in the where clause:
WHERE `marketplace_orderid` = '823123693003';
The problem with quotes is that the actual value might have leading zeros, which would cause the match to fail.
Try using:
SELECT * FROM `marketplacesales`.`sales_order` WHERE `marketplace_orderid LIKE '823123693003%'
That should force the use of the index.
Also, if possible, I would consider changing the field to bigint intead of varchar
Related
Query is taking more than one hour.
Anyone can help with this? thanks in advance.
My query is
UPDATE coupons SET expiration_date='2020-06-06'
My table structure
+-----------------------+--------------------------------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------------+--------------------------------------------+------+-----+---------+----------------+
| id | int(11) unsigned | NO | PRI | NULL | auto_increment |
| vendor_id | int(11) | NO | MUL | NULL | |
| type | enum('food','shopping','home','lifestyle') | NO | | NULL | |
| title | varchar(255) | NO | MUL | NULL | |
| slug | varchar(255) | YES | | NULL | |
| thumbnail_image | varchar(600) | YES | | NULL | |
| coupon_name | varchar(255) | NO | | NULL | |
| expiration_date | date | YES | MUL | NULL | |
| discount_type | enum('d','p') | YES | | NULL | |
| discount_rate | double(7,2) | YES | | NULL | |
| code_type | enum('i','c','u','b') | NO | | NULL | |
| code | text | YES | | NULL |
| created_at | datetime | YES | | NULL | |
| updated_at | datetime | YES | | NULL |
+-----------------------+--------------------------------------------+------+-----+---------+----------------+
Others have pointed out that your UPDATE statement updates every row in the table. That, inherently, takes a long time. And, because of transaction / rollback in the database it will take a lot of RAM and disk space.
You didn't tell us how you defined your index or indexes on expiration_date, so this answer is a is a bit of guesswork. Specifically, unless expiration_date is in its own index or is the first column in a compound index this answer won't perform very well.
Try using an UPDATE query like this, to do the update 1000 rows at a time.
UPDATE coupons
SET expiration_date='2020-06-06'
WHERE expiration_date <> '2020-06-06'
LIMIT 1000
Repeat the query until it updates no more rows. Completing the update will still take a while, but it won't monopolize the table, nor will it generate vast transactions.
Caution, don't try to run more than one of these queries at a time.
I am performing a simple select query to extract username from table logs(containing 54864 rows).
It took about 7.836s to retrieve data.
How can I speed up the performace???
SELECT username FROM `logs`
WHERE
logs.branch=1
and
logs.added_on > '2016-11-27 00:00:00'
On describing table,
+-------------------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------------+-------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| username | char(255) | YES | MUL | NULL | |
| fullname | char(255) | YES | | NULL | |
| package | char(255) | YES | | NULL | |
| prev_expiry | date | YES | | NULL | |
| recharged_upto | date | YES | | NULL | |
| payment_option | int(11) | YES | MUL | NULL | |
| amount | float(14,2) | YES | | NULL | |
| branch | int(11) | YES | MUL | NULL | |
| added_by | int(11) | YES | | NULL | |
| added_on | datetime | YES | MUL | NULL | |
| remark | text | YES | | NULL | |
| payment_mode | char(255) | YES | | NULL | |
| recharge_duration | char(255) | YES | | NULL | |
| invoice_number | char(255) | YES | | NULL | |
| cheque_no | char(255) | YES | | NULL | |
| bank_name | char(255) | YES | | NULL | |
| verify_by_ac | int(11) | YES | | 0 | |
| adjusted_days | int(11) | YES | | NULL | |
| adjustment_note | text | YES | | NULL | |
+-------------------+-------------+------+-----+---------+----------------+
20 rows in set
On explaining query,
+----+-------------+--------------------------+------+-----------------------------+--------------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------------------------+------+-----------------------------+--------------+---------+-------+------+-------------+
| 1 | SIMPLE | logs | ref | branch_index,added_on_index | branch_index | 5 | const | 37 | Using where |
+----+-------------+--------------------------+------+-----------------------------+--------------+---------+-------+------+-------------+
1 row in set
updated:: explaing query after adding composite index(branch_added_index )
+----+-------------+--------------------------+------+------------------------------------------------+--------------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------------------------+------+------------------------------------------------+--------------+---------+-------+------+-------------+
| 1 | SIMPLE | logs | ref | branch_index,added_on_index,branch_added_index | branch_index | 5 | const | 37 | Using where |
+----+-------------+--------------------------+------+------------------------------------------------+--------------+---------+-------+------+-------------+
1 row in set
Add a composite key on branch,added_on so you cover all the WHERE conditions since you use AND.
ALTER TABLE logs ADD KEY(branch,added_on)
This should be much faster,also you can drop the branch_index key since the above index can replace it.You only return 37 rows from 54000 so the cardinality is OK.
ALTER TABLE logs DROP INDEX `branch_index`;
Or you can use index hints
SELECT username FROM `logs` USE INDEX (branch_added_index) WHERE
logs.branch=1
and
logs.added_on > '2016-11-27 00:00:00'
if your table's existing index is already used for other queries try adding new composite index like below
create index <indexname> on logs(branch,added_on)
create a composite index on 2 fields (branch, added_on) like:
ALTER TABLE `logs` ADD KEY idx_branch_added (branch, added_on);
This will be even faster, because it is "covering":
INDEX(branch, added_on, username) -- in exactly that order.
(And drop any indexes that are prefixes of this.)
Index Cookbook
"Cardinality" is rarely of importance. And EXPLAIN often gets the value wrong.
The EXPLAIN shows 5 for the size of branch -- does it really need to be NULLable? Will you have 2 billion branches? Consider using something smaller, such as the 1-byte TINYINT UNSIGNED NOT NULL (values 0..255).
Also, shrink the 255 to something reasonable.
When describing a table, please use SHOW CREATE TABLE; it is more descriptive. It might help to know the Engine, Charset, etc.
I have two tables: gpnxuser and key_value
mysql> describe gpnxuser;
+--------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+----------------+
| id | bigint(20) | NO | PRI | NULL | auto_increment |
| version | bigint(20) | NO | | NULL | |
| email | varchar(255) | YES | | NULL | |
| uuid | varchar(255) | NO | MUL | NULL | |
| partner_id | bigint(20) | NO | MUL | NULL | |
| password | varchar(255) | YES | | NULL | |
| date_created | datetime | YES | | NULL | |
| last_updated | datetime | YES | | NULL | |
+--------------+--------------+------+-----+---------+----------------+
and
mysql> describe key_value;
+----------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+----------------+
| id | bigint(20) | NO | PRI | NULL | auto_increment |
| version | bigint(20) | NO | | NULL | |
| date_created | datetime | YES | | NULL | |
| last_updated | datetime | YES | | NULL | |
| upkey | varchar(255) | NO | MUL | NULL | |
| user_id | bigint(20) | YES | MUL | NULL | |
| security_level | int(11) | NO | | NULL | |
+----------------+--------------+------+-----+---------+----------------+
key_value.user_id is FK that references gpnxuser.id. I also have an index in gpnxuser.partner_id which is a FK that references a table called "partner" (which, I think, does not matter much to this question).
For partner_id = 64, I have 500K rows in gpnxuser which have relationship with approximatelly 6M rows in key_value.
I wanted to have a query that returned all distinct 'key_value.upkey' for user´s belonging to a given partner. I did something like this:
select upkey from gpnxuser join key_value on gpnxuser.id=key_value.user_id where partner_id=64 group by upkey;
which takes forever to run. The explain for the query looks like:
mysql> explain select upkey from gpnxuser join key_value on gpnxuser.id=key_value.user_id where partner_id=64 group by upkey;
+----+-------------+-----------+------+----------------------------+--------------------+---------+-----------------------------+--------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------+------+----------------------------+--------------------+---------+-----------------------------+--------+----------------------------------------------+
| 1 | SIMPLE | gpnxuser | ref | PRIMARY,FKB2D9FEBE725C505E | FKB2D9FEBE725C505E | 8 | const | 259640 | Using index; Using temporary; Using filesort |
| 1 | SIMPLE | key_value | ref | FK9E0C0F912D11F5A9 | FK9E0C0F912D11F5A9 | 9 | gpnx_finance_db.gpnxuser.id | 14 | Using where |
+----+-------------+-----------+------+----------------------------+--------------------+---------+-----------------------------+--------+----------------------------------------------+
My question is: is there a query that can run fast and obtain the result that I want?
what you need to do is utilize EXISTS statement: This will cause only partial table scan until a match found and not more.
select upkey from (select distinct upkey from key_value) upk
where EXISTS
(select 1 from gpnxuser u, key_value kv
where u.id=kv.user_id and partner_id=1 and kv.upkey = upk.upkey)
NB. In the original query, group by is misused: distinct looks better there.
select DISTINCT upkey from gpnxuser join key_value on
gpnxuser.id=key_value.user_id where partner_id=1
I would look into partitioning your key_value table on user_id, if you typically run queries based on this column.
http://dev.mysql.com/doc/refman/5.1/en/partitioning.html
I have the following simple MySQL query:
SELECT SQL_NO_CACHE mainID
FROM tableName
WHERE otherID3=19
AND dateStartCol >= '2012-08-01'
AND dateStartCol <= '2012-08-31';
When I run this it takes 0.29 seconds to bring back 36074 results. When I increase my date period to bring back more results (65703) it runs in 0.56. When I run other similar SQL queries on the same server but on different tables (some tables are larger) the results come back in approximately 0.01 seconds.
Although 0.29 isn't slow - this is a basic part for a complex query and this timing means that it is not scalable.
See below for the table definition and indexes.
I know it's not server load as I have the same issue on a development server which has very little usage.
+---------------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------------------+--------------+------+-----+---------+----------------+
| mainID | int(11) | NO | PRI | NULL | auto_increment |
| otherID1 | int(11) | NO | MUL | NULL | |
| otherID2 | int(11) | NO | MUL | NULL | |
| otherID3 | int(11) | NO | MUL | NULL | |
| keyword | varchar(200) | NO | MUL | NULL | |
| dateStartCol | date | NO | MUL | NULL | |
| timeStartCol | time | NO | MUL | NULL | |
| dateEndCol | date | NO | MUL | NULL | |
| timeEndCol | time | NO | MUL | NULL | |
| statusCode | int(1) | NO | MUL | NULL | |
| uRL | text | NO | | NULL | |
| hostname | varchar(200) | YES | MUL | NULL | |
| IPAddress | varchar(25) | YES | | NULL | |
| cookieVal | varchar(100) | NO | | NULL | |
| keywordVal | varchar(60) | NO | | NULL | |
| dateTimeCol | datetime | NO | MUL | NULL | |
+---------------------------+--------------+------+-----+---------+----------------+
+--------------------+------------+-------------------------------+--------------+---------------------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+--------------------+------------+-------------------------------+--------------+---------------------------+-----------+-------------+----------+--------+------+------------+---------+
| tableName | 0 | PRIMARY | 1 | mainID | A | 661990 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_otherID1 | 1 | otherID1 | A | 330995 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_otherID2 | 1 | otherID2 | A | 25 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_otherID3 | 1 | otherID3 | A | 48 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_dateStartCol | 1 | dateStartCol | A | 187 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_timeStartCol | 1 | timeStartCol | A | 73554 | NULL | NULL | | BTREE | |
|tableName | 1 | idx_dateEndCol | 1 | dateEndCol | A | 188 | NULL | NULL | | BTREE | |
|tableName | 1 | idx_timeEndCol | 1 | timeEndCol | A | 73554 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_keyword | 1 | keyword | A | 82748 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_hostname | 1 | hostname | A | 2955 | NULL | NULL | YES | BTREE | |
| tableName | 1 | idx_dateTimeCol | 1 | dateTimeCol | A | 220663 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_statusCode | 1 | statusCode | A | 2 | NULL | NULL | | BTREE | |
+--------------------+------------+-------------------------------+--------------+---------------------------+-----------+-------------+----------+--------+------+------------+---------+
Explain Output:
+----+-------------+-----------+-------+----------------------------------+-------------------+---------+------+-------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-----------+-------+----------------------------------+-------------------+---------+------+-------+----------+-------------+
| 1 | SIMPLE | tableName | range | idx_otherID3,idx_dateStartCol | idx_dateStartCol | 3 | NULL | 66875 | 75.00 | Using where |
+----+-------------+-----------+-------+----------------------------------+-------------------+---------+------+-------+----------+-------------+
If that is really your query (and not a simplified version of same), then this ought to achieve best results:
CREATE INDEX table_ndx on tableName( otherID3, dateStartCol, mainID);
The first index entry means that the first match in the WHERE is very fast; the same also applies with dateStartCol. The third field is very small and does not slow the index appreciably, but allows for the datum you require to be found immediately in the index with no table access at all.
It is important that the keys are in the same index. In the EXPLAIN you posted, each key is in an index of its own, so even if MySQL chooses the best index, the performances will not be optimal. I'd try and use less indexes, for they also have a cost (shameless plug: Can Indices actually decrease SELECT performance? ).
First try to add the right key. It seems like dateStartCol is more selective than otherID3
ALTER TABLE tableName ADD KEY idx_dates(dateStartCol, dateStartCol)
Second - please make sure you select only rows you need by adding LIMIT clause to the SELECT. This will should up the query. Try like this:
SELECT SQL_NO_CACHE mainID
FROM tableName
WHERE otherID3=19
AND dateStartCol >= '2012-08-01'
AND dateStartCol <= '2012-08-31'
LIMIT 10;
Please also make sure that your MySQL tuned up properly. You may want to check key_buffer_size and innodb_buffer_pool_size as described in http://astellar.com/2011/12/why-is-stock-mysql-slow/
If this is a recurrent or important query then create a multiple column index:
CREATE INDEX index_name ON tableName (otherID3, dateStartCol)
Delete the non used indexes as they make table changes more expensive.
BTW you don't need two separate columns for date and time. You can combine then in a datetime or timestamp type. One less column and one less index.
The explain output shows it chose the dateStartCol index so you could try the opposite I suggested above:
CREATE INDEX index_name ON tableName (dateStartCol, otherID3)
Notice that the query's dateStartCol condition will still get 75% of the rows so not much improvement, if any, in using that single index.
How unique is otherID3? If there are not many repeated otherID3 you can hint the engine to use it.
I have a query that is taking several seconds to complete for certain WHERE conditions. To the best of my knowledge, everything is indexed correctly, but performance is still poor. All tables are InnoDB, blog_users and blog_users_profiles contain about 5 million rows and forum_contacts has about 10K rows.
The query
SELECT blog.*
FROM blog_users AS blog
INNER JOIN blog_user_profiles AS profile ON blog.user_id = profile.user_id
INNER JOIN forum_contacts AS contact ON profile.forum_id = contact.user_id
WHERE blog.comments > :comment_cutoff
AND blog.last_active > :time_cutoff
AND contact.type = :contact_type
AND contact.area = :location
LIMIT 0, 100
Explain output
+----+-------------+---------+--------+------------------------------+-----------+---------+------------------------------+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------+--------+------------------------------+-----------+---------+------------------------------+---------+-------------+
| 1 | SIMPLE | blog | range | PRIMARY,last_active,comments | comments | 3 | NULL | 1313813 | Using where |
| 1 | SIMPLE | profile | eq_ref | PRIMARY,forum_id | PRIMARY | 3 | xc_db.blog.user_id | 1 | |
| 1 | SIMPLE | contact | eq_ref | PRIMARY,type,area,user_id | PRIMARY | 80 | xc_db.profile.forum_id,const | 1 | Using where |
+----+-------------+---------+--------+------------------------------+-----------+---------+------------------------------+---------+-------------+
Table structure (irrelevant rows snipped)
blog_users
+-------------+-----------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+-----------------------+------+-----+---------+----------------+
| user_id | mediumint(8) unsigned | NO | PRI | NULL | auto_increment |
| username | varchar(16) | NO | UNI | NULL | |
| comments | mediumint(7) unsigned | NO | MUL | NULL | |
| last_active | int(10) unsigned | NO | MUL | NULL | |
+-------------+-----------------------+------+-----+---------+----------------+
blog_user_profiles
+----------+-----------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+-----------------------+------+-----+---------+-------+
| user_id | mediumint(8) unsigned | NO | PRI | NULL | |
| forum_id | mediumint(8) unsigned | NO | MUL | 0 | |
+----------+-----------------------+------+-----+---------+-------+
forum_contacts
+---------+-----------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------+-----------------------+------+-----+---------+-------+
| user_id | mediumint(8) unsigned | NO | PRI | NULL | |
| type | varchar(25) | NO | PRI | NULL | |
| area | varchar(255) | NO | MUL | NULL | |
+---------+-----------------------+------+-----+---------+-------+
The strange thing is that the execution time seems to be inversely related to the total number of rows returned. For example, a query like contact.area = 'United States' which returns a few thousand rows executes in less than 0.01 second. However, queries with fewer result rows such as contact.area = 'Egypt' which returns 20 rows takes in excess of 5 seconds to complete. Does my query need to be rewritten or is there a problem with the indexes?