This question already has answers here:
MySQL varchar index length
(2 answers)
Closed 1 year ago.
Note:
First I want to mention that I read all other related questions and answers.
Question:
I had created 2 indexes for the table I have but whenever I use my queries it's not using the index. even when forced to use the index it's not using it.
The table has 1.5M rows and it will be increased, and the query is taking 35+ seconds.
Query 1
explain analyze
SELECT sum(cid_user_usd_earned)
FROM `bbtv_adv_records`
WHERE (user_id =2
and `cid_assign_month` = '2020-08-01'
And `content_type` = 'UGC'
);
explain analyze
SELECT sum(cid_user_usd_earned)
FROM `bbtv_adv_records`
USE INDEX (bbtv_adv_records_user_id_cid_assign_month_content_type_index)
WHERE (user_id =2
and `cid_assign_month` = '2020-08-01'
And `content_type` = 'UGC'
);
Query 2
explain analyze
SELECT *
FROM `bbtv_adv_records`
WHERE (user_id =2
and `cid_assign_month` = '2020-08-01'
);
Query 3
explain analyze
SELECT sum(cid_user_usd_earned),channel_id
FROM `bbtv_adv_records`
WHERE (user_id =2
and `cid_assign_month` = '2020-08-01'
Group by `channel_id`
);
Table
CREATE TABLE `bbtv_adv_records` (
`id` bigint UNSIGNED NOT NULL,
`content_type` varchar(400) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
....
`channel_id` varchar(400) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
...
`cid_assign_month` date NOT NULL,
`cid_process_state` enum('process','done','fail') CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
`user_id` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
--
-- Indexes for table `bbtv_adv_records`
--
ALTER TABLE `bbtv_adv_records`
ADD PRIMARY KEY (`id`),
ADD KEY `bbtv_adv_records_user_id_cid_assign_month_index
` (`user_id`,`cid_assign_month`),
ADD KEY `bbtv_adv_records_user_id_cid_assign_month_content_type_index`
(`user_id`,`cid_assign_month`,`content_type`);
--
-- AUTO_INCREMENT for table `bbtv_adv_records`
--
ALTER TABLE `bbtv_adv_records`
MODIFY `id` bigint UNSIGNED NOT NULL AUTO_INCREMENT, AUTO_INCREMENT=1543191;
COMMIT;
mysql> show index from bbtv_adv_records
+------------------+------------+--------------------------------------------------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | Visible | Expression |
+------------------+------------+--------------------------------------------------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+
| bbtv_adv_records | 0 | PRIMARY | 1 | id | A | 1456578 | NULL | NULL | | BTREE | | | YES | NULL |
| bbtv_adv_records | 1 | bbtv_adv_records_user_id_cid_assign_month_index | 1 | user_id | A | 8 | NULL | NULL | YES | BTREE | | | YES | NULL |
| bbtv_adv_records | 1 | bbtv_adv_records_user_id_cid_assign_month_index | 2 | cid_assign_month | A | 47 | NULL | NULL | | BTREE | | | YES | NULL |
| bbtv_adv_records | 1 | bbtv_adv_records_user_id_cid_assign_month_content_type_index | 1 | user_id | A | 8 | NULL | NULL | YES | BTREE | | | YES | NULL |
| bbtv_adv_records | 1 | bbtv_adv_records_user_id_cid_assign_month_content_type_index | 2 | cid_assign_month | A | 49 | NULL | NULL | | BTREE | | | YES | NULL |
| bbtv_adv_records | 1 | bbtv_adv_records_user_id_cid_assign_month_content_type_index | 3 | content_type | A | 58 | NULL | NULL | YES | BTREE | | | YES | NULL |
+------------------+------------+--------------------------------------------------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+
6 rows in set (0.01 sec)
note the best result I have 10 seconds when creating only the cid_assign_month index alone, but all my queries have the cid_assign_month with the user_id always.
could you show the result of "show index from bbtv_adv_records ".
The mysql query optimizer is a cost based optimiser.
It try to find out the best execution plan.
If the cost of ALL( full table scan) is less then REF_OR_NULL (using secondary index fetch data), It will use full table scan where time complexity is O(n) .
for example, there are the commonest cost matrix
the IO cost of Reading 1 block page is 1.
the CPU cost of comparing 1 record is 0.2.
the query 1:
SELECT sum(cid_user_usd_earned) FROM bbtv_adv_records WHERE (user_id =2 and cid_assign_month = '2020-08-01' And content_type = 'UGC');
because the column cid_user_usd_earned is not in secondary index bbtv_adv_records_user_id_cid_assign_month_content_type_index, mysql will return to the primary index to get cid_user_usd_earned, after using binary search in secondary index.
Solved
It looks that MySQL indexing will not work with varchar(400) or any large length.
change from
`content_type` varchar(400) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`channel_id` varchar(400) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`user_id` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
to
`content_type` varchar(30) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`channel_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`user_id` bigint UNSIGNED DEFAULT NULL,
now the Query show indexing and faster from 34s to 8s
mysql> explain analyze SELECT sum(cid_user_usd_earned) FROM `bbtv_adv_records` WHERE (user_id =2 and `cid_assign_month` = '2020-08-01' And `content_type` = 'UGC');
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| EXPLAIN |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| -> Aggregate: sum(bbtv_adv_records.cid_user_usd_earned) (actual time=8247.975..8247.981 rows=1 loops=1)
-> Index lookup on bbtv_adv_records using bbtv_adv_records_user_id_cid_assign_month_content_type_index (user_id=2, cid_assign_month=DATE'2020-08-01', content_type='UGC') (cost=116370.56 rows=496622) (actual time=0.373..6175.144 rows=259373 loops=1)
|
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (8.25 sec)
I have a table:
CREATE TABLE `test` (
`t` varchar(255) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8
The data in the table is:
mysql> select * from test;
+----------------------+
| t |
+----------------------+
| 12511023900355495873 |
| 12511023900355495872 |
| 12511023900355495874 |
+----------------------+
When I select from the table like this:
select * from test where t = 12511023900355495873;
The result is:
+----------------------+
| t |
+----------------------+
| 12511023900355495873 |
| 12511023900355495872 |
| 12511023900355495874 |
+----------------------+
What implicit conversion happens when I use number to select a varchar column?
The manual page https://dev.mysql.com/doc/refman/5.7/en/type-conversion.html In all other cases, the arguments are compared as floating-point (real) numbers.
so
drop table if exists t;
CREATE TABLE t (
t varchar(255) DEFAULT NULL,
t1 float
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
insert into t values
( 12511023900355495873,12511023900355495873 ),
( 12511023900355495872,12511023900355495872 ),
( 12511023900355495874,12511023900355495874 );
select t.*,#t2 from t where t = 12511023900355495873;
results in
+----------------------+-----------+------+
| t | t1 | #t2 |
+----------------------+-----------+------+
| 12511023900355495873 | 1.2511e19 | NULL |
| 12511023900355495872 | 1.2511e19 | NULL |
| 12511023900355495874 | 1.2511e19 | NULL |
+----------------------+-----------+------+
3 rows in set (0.00 sec)
Note that all float values in the column t1 are the same.
It goes without saying that you should avoid type conversion.
I'm using MariaDb server (Ver 15.1 Distrib 10.2.7-MariaDB).
When I execute
CREATE TABLE `my_table` (
`id` INT NOT NULL,
`name` NVARCHAR(64) NULL,
PRIMARY KEY (`id`)
);
Describe output:
MariaDB [db]> describe my_table;
+-------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+-------------+------+-----+---------+-------+
| id | int(11) | NO | PRI | NULL | |
| name | varchar(64) | YES | | NULL | |
+-------+-------------+------+-----+---------+-------+
2 rows in set (0.00 sec)
Why there is no error, and "name" column datatype is varchar (not nvarchar)?
db schema details:
Default collation: utf8_general_ci
Default characterset: utf8
NVARCHAR is a synonym for VARCHAR in MySQL/MariaDB. But you need to add the CHARACTER SET utf8mb4 to be sure that you get full UTF-8 support.
What you show as the default for that database is only the subset, called 'utf8'. It will not handle Emoji or some of Chinese.
I do not understand the difference (line 2) of those two EXPLAINs. Maybe someone has a hint for me why mysql acts so different on those, which heavily affects query speed.
The slow query lasts 12 seconds (which equals querying all rows with that query) and uses a join on integer columns while the joined table has just 3 records:
SELECT `inv_assets`.`id` AS `id`, `site`.`description` AS `sitename`,
(SELECT COALESCE(DATE_FORMAT(CONVERT_TZ(MIN(inspdate),'UTC','Europe/Vienna'),'%Y-%m-%d'),'')
FROM `mobuto_inv_inspections` AS `nextinsp`
WHERE ((`nextinsp`.`objectlink` = `inv_assets`.`id`
AND `nextinsp`.`inspdate` >= NOW()))
) AS `nextinsp`
FROM `mobuto_inv_assets` AS `inv_assets`
LEFT JOIN `mobuto_inv_sites` AS `site`
ON (`site`.`siteid` = `inv_assets`.`site`
AND `site`.`_state` IN (2,0))
ORDER BY `inv_assets`.`type` ASC LIMIT 0, 20;
+----+--------------------+------------+--------+----------------+---------+---------+------------------------------+-------+----------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+------------+--------+----------------+---------+---------+------------------------------+-------+----------------------------------------------------+
| 1 | PRIMARY | inv_assets | ALL | NULL | NULL | NULL | NULL | 24857 | Using temporary; Using filesort |
| 1 | PRIMARY | site | ALL | PRIMARY,_state | NULL | NULL | NULL | 3 | Using where; Using join buffer (Block Nested Loop) |
| 2 | DEPENDENT SUBQUERY | nextinsp | ALL | inspdate | NULL | NULL | NULL | 915 | Using where |
+----+--------------------+------------+--------+----------------+---------+---------+------------------------------+-------+----------------------------------------------------+
The fast query consumes just a few fractions of a second, uses a join on varchar(32) columns and the joined table has 1352 records:
SELECT `inv_assets`.`id` AS `id`, `guarantor`.`lastname` AS `guarantoruname`,
(SELECT COALESCE(DATE_FORMAT(CONVERT_TZ(MIN(inspdate),'UTC','Europe/Vienna'),'%Y-%m-%d'),'')
FROM `mobuto_inv_inspections` AS `nextinsp`
LEFT JOIN `users` AS `saveuser`
ON (`saveuser`.`uid` = `nextinsp`.`saveuser`
AND `saveuser`.`_state` = '0')
WHERE ((`nextinsp`.`objectlink` = `inv_assets`.`id`
AND `nextinsp`.`inspdate` >= NOW()))
) AS `nextinsp`
FROM `mobuto_inv_assets` AS `inv_assets`
LEFT JOIN `users` AS `guarantor`
ON (`guarantor`.`uid` = `inv_assets`.`guarantor`
AND `guarantor`.`_state` = '0')
ORDER BY `inv_assets`.`type` ASC LIMIT 0, 20;
+----+--------------------+------------+--------+----------------+---------+---------+---------------------------------+-------+----------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+------------+--------+----------------+---------+---------+---------------------------------+-------+----------------+
| 1 | PRIMARY | inv_assets | ALL | NULL | NULL | NULL | NULL | 24857 | Using filesort |
| 1 | PRIMARY | guarantor | eq_ref | PRIMARY,_state | PRIMARY | 98 | mobuto_dev.inv_assets.guarantor | 1 | Using where |
| 2 | DEPENDENT SUBQUERY | nextinsp | ALL | inspdate | NULL | NULL | NULL | 915 | Using where |
| 2 | DEPENDENT SUBQUERY | saveuser | eq_ref | PRIMARY,_state | PRIMARY | 98 | mobuto_dev.nextinsp.saveuser | 1 | Using where |
+----+--------------------+------------+--------+----------------+---------+---------+---------------------------------+-------+----------------+
The strange thing to me is, when I remove the column (description) of the joined table in the 'column-select-part' (while the join still persists and IMHO mysql does not optimize it away when not used), the speed is back (because mysql does not use a temporary table any longer and the explain looks same as the fast one, having type=eq_ref).
But why does this work for the first sample only when no column selected, whereas I can select one in the second one!?
CREATE TABLE `mobuto_inv_assets` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`invnum` varchar(10) NOT NULL,
`oebglcat` varchar(4) NOT NULL,
`mark` varchar(100) NOT NULL,
`type` varchar(100) NOT NULL,
`serialnum` varchar(100) NOT NULL,
`desc` varchar(100) NOT NULL,
`site` int(11) NOT NULL DEFAULT '0',
`licnum` varchar(20) NOT NULL DEFAULT '',
`inquirer` varchar(100) NOT NULL DEFAULT '',
`inqdate` date NOT NULL DEFAULT '0000-00-00',
`supplier` varchar(100) NOT NULL DEFAULT '',
`suppldate` date NOT NULL DEFAULT '0000-00-00',
`supplnumber` varchar(30) NOT NULL DEFAULT '',
`invoicedate` date NOT NULL DEFAULT '0000-00-00',
`invoicenumber` varchar(30) NOT NULL DEFAULT '',
`purchaseprice` decimal(11,2) NOT NULL DEFAULT '0.00',
`leased` varchar(1) NOT NULL DEFAULT 'N',
`leasingcompany` varchar(100) NOT NULL DEFAULT '',
`leasingnumber` varchar(30) NOT NULL DEFAULT '',
`notes` text NOT NULL,
`inspnotes` text NOT NULL,
`inactive` varchar(1) NOT NULL DEFAULT 'N',
`maintain` varchar(1) NOT NULL DEFAULT 'Y',
`asset` varchar(1) NOT NULL DEFAULT 'Y',
`inspection` varchar(1) NOT NULL DEFAULT '',
`inspperson` varchar(100) NOT NULL DEFAULT '',
`guarantor` varchar(32) NOT NULL DEFAULT '',
`saveuser` varchar(32) NOT NULL,
`savetime` int(11) NOT NULL,
`recordid` varchar(32) NOT NULL,
`_state` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`id`),
UNIQUE KEY `invnum` (`invnum`),
KEY `_state` (`_state`),
KEY `site` (`site`)
) ENGINE=InnoDB AUTO_INCREMENT=30707 DEFAULT CHARSET=utf8;
CREATE TABLE `mobuto_inv_sites` (
`siteid` int(11) NOT NULL AUTO_INCREMENT,
`description` varchar(100) NOT NULL,
`saveuser` varchar(32) NOT NULL,
`savetime` int(11) NOT NULL,
`recordid` varchar(32) NOT NULL,
`_state` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`siteid`),
KEY `_state` (`_state`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=utf8;
mysql> SHOW INDEX FROM mobuto_inv_assets;
+-------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| mobuto_inv_assets | 0 | PRIMARY | 1 | id | A | 24857 | NULL | NULL | | BTREE | | |
| mobuto_inv_assets | 0 | invnum | 1 | invnum | A | 24857 | NULL | NULL | | BTREE | | |
| mobuto_inv_assets | 1 | _state | 1 | _state | A | 4 | NULL | NULL | | BTREE | | |
+-------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
Changes as requested by #Wilson Hauck:
Added index to column site in mobuto_inv_assets (reduced execution speed by almost half a second)
Seems that column nextinsp was missing in first query. Maybe lost while formatting the query. Of course there should be the same as in the fast one
Removed the saveuser join as it is not used there (saved another 2 seconds) and updated its EXPLAIN (last line removed)
SHOW INDEX FROM mobuto_inv_sites added
+------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| mobuto_inv_sites | 0 | PRIMARY | 1 | siteid | A | 3 | NULL | NULL | | BTREE | | |
| mobuto_inv_sites | 1 | _state | 1 | _state | A | 3 | NULL | NULL | | BTREE | | |
+------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
Your first query is making less use of keys than the second. The possible_keys column in the explain plan shows where keys are available to be used, however, the key column shows where they are actually being used.
I would advise, short of seeing the structure of your DB, to make more use of these keys in your JOIN and WHERE clauses to speed it up.
I'd make sure that the query isn't being cached when you say you're modifying the select columns and the speed is varying.
12 seconds first query likely caused by ROWS column clues of simply 24857*3*915*1 = 68,232,465 total rows considered. Less than 1 second for second query ROWS column clues of simply 24857*1*915*1 = 22,744,155 total rows considered. The first query's use of Block Nested Loop processing is a major contributor to delaying the response.
Please post results of SHOW CREATE TABLE mobuto_inv_assets and mobuto_inv_sites. Please also post results of SHOW INDEX FROM mobuto_inv_assets and mobuto_inv_sites. With this additional information someone may be able to suggest improvements to SELECT .... queries that will avoid Block Nested Loop processing that is very time CPU intense with RBAR (Row By Agonizing Row processing). Additional indexing may be required.
Thanks for posting your two SHOW CREATE TABLE's, immensely helpful.
Please consider adding index with
ALTER TABLE mobuto_inv_sites ADD INDEX site --
if space permits on your system.
Also, the EXPLAIN showing for query1 is mismatched to the query.
The query does not refer to nextinsp or saveused that I can see in EXPLAIN.
Please replace the EXPLAIN for query1 after creating the index when you have an opportunity to test again and indicate any reduction in execution time required.
It would also be nice if you could post results of
SHOW INDEX FROM mobuto_inv_sites so we can see the scope of your data and cardinality.
If the inv_assets rows are populated with ACCURATE _state data
consider changing query1 to something like the following:
SELECT inv_assets.id AS id, site.description AS sitename,
(SELECT COALESCE(DATE_FORMAT(CONVERT_TZ(MIN(inspdate),'UTC','Europe/Vienna'),'%Y-%m-%d'),'')
FROM mobuto_inv_inspections AS nextinsp
WHERE ((nextinsp.objectlink = inv_assets.id
AND nextinsp.inspdate >= NOW()))
) AS nextinsp
FROM mobuto_inv_assets AS inv_assets
WHERE inv_assets._state = 2 OR inv_assets._state = 0
LEFT JOIN mobuto_inv_sites AS site
ON (site.siteid = inv_assets.site
AND site._state IN (2,0))
ORDER BY inv_assets.type ASC LIMIT 0, 20;
EXPLAIN should avoid table scan and subsequent Block Nested Loop processing.
If _state data in inv_assets is not ACCURATE on every row, this will not work.
2017-08-10 update 09:42 CT please post QUERY, EXPLAIN result, SHOW CREATE TABLE tblname for tables involved and SHOW INDEX FROM tblname for tables involved.
I use following query to create table news:
CREATE TABLE IF NOT EXISTS `news` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`news_title` varchar(500) NOT NULL,
`news_detail` varchar(5000) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=1 ;
mysql> desc news;
+-------------+---------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+---------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| news_title | varchar(500) | NO | | | |
| news_detail | varchar(5000) | NO | | | |
+-------------+---------------+------+-----+---------+----------------+
mysql> insert into news (news_title, news_detail) values ('test','demod demo');
mysql> select * from news;
+----+--------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------+
| id | news_title | news_detail |
+----+--------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------+
| 3 | Advani wants to shift from Gujarat, BJP trying to convince him otherwise | testt |
| 5 | test | demod demo |
+----+--------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------+
as you see in the select query the id is increment like 1,3,5,7.... means it increment by 2. So what is the problem here?
actually in my local, it is increment by 1 and working perfectly. but in my server it creates the problem.
Thanks in advance.
Why ?
The auto_increment value can be change with the variable auto_increment_increment.Normally, it’s always 1, but for some weird reason it was set to 2 in my case. I think MySQL Workbench may be involed.
You can change it be doing one of those :
SET ##auto_increment_increment=1
SET GLOBAL auto_increment_increment=1;
More information
You can find some information here and here.
Check system variable ##set_auto_increment_increment.
it should be
SET ##auto_increment_increment=1;