I have table banners with following column
column
type
index
id
BIGINT
primary
start_date
DATETIME
key
end_date
DATETIME NULLABLE
key
Executing this query gives me result
EXPLAIN SELECT * FROM banners WHERE end_date IS NULL;
id
select_type
table
partitions
type
possible_keys
key
key_len
ref
rows
filtered
Extra
1
SIMPLE
banners
(NULL)
ref
banners_end_date_index
banners_end_date_index
6
const
2
100.00
Using index condition
However, this query gives me result
EXPLAIN SELECT * FROM banners WHERE end_date > '2022-04-26';
id
select_type
table
partitions
type
possible_keys
key
key_len
ref
rows
filtered
Extra
1
SIMPLE
banners
(NULL)
ALL
banners_end_date_index
(NULL)
(NULL)
(NULL)
9,980
50.00
Using where
So does that mean the 2nd query is ignoring the index and doing full table scan?
If the index is being ignored, what query should I use instead to use the index?
Also, I read that OR statement is also ignoring index. Is it always true, or are there exception for index and OR to work together?
Edit:
As pointed out by Bill Karwin in the comment, if most of the data match the condition, it will skip the index.
So, I tried again with end_date so far in the future, and here's the EXPLAIN result. Column key and key_len are not NULL, which mean the index is indeed being used.
EXPLAIN SELECT * FROM banners WHERE end_date > DATE('2022-07-20');
id
select_type
table
partitions
type
possible_keys
key
key_len
ref
rows
filtered
Extra
1
SIMPLE
banners
(NULL)
ALL
banners_end_date_index
banners_end_date_index
6
(NULL)
74
100.00
Using index condition
Related
I have a datehelper table with every YYYY-MM-DD as DATE between the years 2000 and 2100. To this I'm joining a subquery for all unit transactions. unit.end is a DATETIME so my subquery simplifies it to DATE and uses that to join to the datehelper table.
In 5.6 this query takes a couple seconds to run a massive amount of transactions, and it derives a table that is auto keyed based on the DATE(unit.end) in the subquery and uses that to join everything else fairly quickly.
In 5.7, it takes 600+ seconds and I can't get it to derive a table or follow the much better execution plan that 5.6 used. Is there a flag I need to set or some way to prefer the old execution plan?
Here's the query:
EXPLAIN SELECT datehelper.id AS date, MONTH(datehelper.id)-1 AS month, DATE_FORMAT(datehelper.id,'%d')-1 AS day,
IFNULL(SUM(a.total),0) AS total, IFNULL(SUM(a.tax),0) AS tax, IFNULL(SUM(a.notax),0) AS notax
FROM datehelper
LEFT JOIN
(SELECT
DATE(unit.end) AS endDate,
getFinalPrice(unit.id) AS total, tax, getFinalPrice(unit.id)-tax AS notax
FROM unit
INNER JOIN products ON products.id=unit.productID
INNER JOIN prodtypes FORCE INDEX(primary) ON prodtypes.id=products.prodtypeID
WHERE franchiseID='1' AND void=0 AND checkout=1
AND end BETWEEN '2020-01-01' AND DATE_ADD('2020-01-01', INTERVAL 1 YEAR)
AND products.prodtypeID NOT IN (1,10)
) AS a ON a.endDate=datehelper.id
WHERE datehelper.id BETWEEN '2020-01-01' AND '2020-12-31'
GROUP BY datehelper.id ORDER BY datehelper.id;
5.6 result (much faster):
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY datehelper range PRIMARY PRIMARY 3 NULL 365 Using where; Using index
1 PRIMARY <derived2> ref <auto_key0> <auto_key0> 4 datehelper.id 10 NULL
2 DERIVED prodtypes index PRIMARY PRIMARY 4 NULL 10 Using where; Using index
2 DERIVED products ref PRIMARY,prodtypeID prodtypeID 4 prodtypes.id
9 Using index
2 DERIVED unit ref productID,end,void,franchiseID productID 9 products.id 2622 Using where
5.7 result (much slower, no auto key found):
id select_type table partitions type possible_keys key key_len ref rows filtered Extra
1 SIMPLE datehelper NULL range PRIMARY PRIMARY 3 NULL 366 100.00 Using where; Using index
1 SIMPLE unit NULL ref productID,end,void,franchiseID franchiseID 4 const 181727 100.00 Using where
1 SIMPLE products NULL eq_ref PRIMARY,prodtypeID PRIMARY 8 barkops3.unit.productID 1 100.00 Using where
1 SIMPLE prodtypes NULL eq_ref PRIMARY PRIMARY 4 barkops3.products.prodtypeID 1 100.00 Using index
I found the problem. It was the optimizer_switch 'derived_merge' flag which is new to 5.7.
https://dev.mysql.com/doc/refman/5.7/en/derived-table-optimization.html
This flag overrides materialization of derived tables if the optimizer thinks the outer WHERE can be pushed down into a subquery. In this case, that optimization was enormously more costly than joining a materialized table on an auto_key.
I have the following Query
SELECT
product.AID,
product.ART_ID,
product.EAN,
productdetails.DESCRIPTION_SHORT,
MAX(
(100000 * (MATCH(productdetails.DESCRIPTION_SHORT) AGAINST ('"psen in1p"' IN BOOLEAN MODE)))+
(100000 * (MATCH(product.ART_ID) AGAINST ('"psen in1p"' IN BOOLEAN MODE)))+
(100000 * (MATCH(product.EAN) AGAINST ('"psen in1p"' IN BOOLEAN MODE)))+
(100000 * (MATCH(product.SUPPLIER_ALT_PID) AGAINST ('"psen in1p"' IN BOOLEAN MODE)))+
(10 * (MATCH(productdetails.DESCRIPTION_LONG) AGAINST ('*psen in1p*' IN BOOLEAN MODE)))+
(2 * (MATCH(productdetails.KEYWORD) AGAINST ('+psen +in1p' IN BOOLEAN MODE)))
) AS relevance
FROM
tbl_product as product
INNER JOIN
`tbl_product_details` as productdetails ON product.AID = productdetails.AID
WHERE MATCH
(product.ART_ID,
product.EAN,
product.SUPPLIER_ALT_PID,
product.ERP_GROUP_SUPPLIER) AGAINST ('*psen* *in1p*' IN BOOLEAN MODE)
OR MATCH
(productdetails.DESCRIPTION_SHORT,
productdetails.DESCRIPTION_LONG,
productdetails.MANUFACTURER_TYPE_DESC,
productdetails.KEYWORD) AGAINST ('*psen* *in1p*' IN BOOLEAN MODE)
GROUP BY
product.AID
ORDER BY
relevance DESC
My Problem is, that the Query takes about ~3 Second which is ways to much. If i run the Statement without ORDER BY it takes about 0,0096 Seconds which is perfect. I dont know why it takes so long. I already tried to Subselect and Order the Subselect with the Same Result (About 3 Seconds to Finish. Same goes for a Subselect without ORDER BY.
The Database have about 600k Records and over 1 Million Records in tbl_product_details.
I'm thankfull for any Help on this Problem.
Explain for the Query with Order By (3 Seconds)
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE product index PRIMARY,tbl_product_catalog_id_foreign,tbl_product_supplier_id_foreign,tbl_product_art_id_index,tbl_product_ean_index,SUPPLIER_ALT_PID,ART_ID_2,ft_artid,ft_ean,ft_sapid PRIMARY 4 NULL 569643 Using temporary; Using filesort
1 SIMPLE productdetails ref tbl_product_details_aid_foreign tbl_product_details_aid_foreign 5 shop_meyle1.product.AID 1 Using where
Explain for the Query without Order By (0,01 Seconds)
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE product index PRIMARY,tbl_product_catalog_id_foreign,tbl_product... PRIMARY 4 NULL 569643 NULL
1 SIMPLE productdetails ref tbl_product_details_aid_foreign tbl_product_details_aid_foreign 5 shop_meyle1.product.AID 1 Using where
Explain for the Query without Order By and with Subselect (3 Seconds)
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY <derived2> ALL NULL NULL NULL NULL 569643 NULL
2 DERIVED product index PRIMARY,tbl_product_catalog_id_foreign,tbl_product... PRIMARY 4 NULL 569643 NULL
2 DERIVED productdetails ref tbl_product_details_aid_foreign tbl_product_details_aid_foreign 5 shop_meyle1.product.AID 1 Using where
I have a problem with the count() of paginate still in cakephp version 3.3:
My table has 6,000,000 records.
The fields involved here are name and cityf. Both have index in MySQL
I'm showing 10 and 10 and despite the query is very fast, the count() of paginate is taking more than 50 seconds.
How to solve this in version 3.3 of cakephp. Follows the two SQL statements and times below:
Select query main:
SELECT
Rr.id AS `Rr__id`,
Rr.idn AS `Rr__idn`,
Rr.aniver AS `Rr__aniver`,
Rr.pessoa AS `Rr__pessoa`,
Rr.name AS `Rr__name`,
Rr.phoner AS `Rr__phoner`,
Rr.tipolf AS `Rr__tipolf`,
Rr.addressf AS `Rr__addressf`,
Rr.num_endf AS `Rr__num_endf`,
Rr.complem AS `Rr__complem`,
Rr.bairrof AS `Rr__bairrof`,
Rr.cityf AS `Rr__cityf`,
Rr.statef AS `Rr__statef`,
Rr.cepf AS `Rr__cepf`,
Rr.n1 AS `Rr__n1`,
Rr.n2 AS `Rr__n2`,
Rr.smerc AS `Rr__smerc`,
Rr.n3 AS `Rr__n3`,
Rr.n4 AS `Rr__n4`,
Rr.fone AS `Rr__fone`,
Rr.numero AS `Rr__numero`
FROM
`MG` Rr
WHERE
(
Rr.name like 'MARCOS%'
AND Rr.cityf like 'BELO HORIZONTE%'
)
ORDER BY
name asc
LIMIT
10 OFFSET 0
= 10 ms
Explain:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE Rr range NAME,CITYF,cityfbairrof,cityfaddressf,cityfbairrofaddressf,namen1n2n3n4 NAME 63 NULL 21345 Using index condition; Using where
-
Select query count:
SELECT
(
COUNT(*)
) AS `count`
FROM
`MG` Rr
WHERE
(
Rr.name like 'MARCOS%'
AND Rr.cityf like 'BELO HORIZONTE%'
)
= 51.247 ms
Explain:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE Rr range NAME,CITYF,cityfbairrof,cityfaddressf,cityfbairrofaddressf,namen1n2n3n4 NAME 63 NULL 21345 Using index condition; Using where
It's happening in several other cases: Always count of query is very slow.
I appreciate any help.
Marcos
I have a question about this query and indexing
my query is :
EXPLAIN SELECT * FROM s1_training where amt > 0 LIMIT 500
and i indexed ( amt ) But its not effected if i have this wheres :
WHERE amt != 0
WHERE amt > 0
WHERE amt < 0
the result is :
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE s1_training ALL amt NULL NULL NULL 64 Using where
The only way index will be used is to set
WHERE amt = number ( like amt = 2 )
and the result will be
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE s1_training ref amt amt 4 const 1
thanks in advance for any help
best regards.
It depends on the number of rows and the cardinality of values.
For example if most values are >0 then mysql optimizer will decide to to a table scan since is faster.
For the second condition WHERE amt=2 this will probably return far fewer values so the index is used because is faster.
I have been going through my slow queries and doing what I can to property optimize each one. I ran across this one, that I have been stuck on.
EXPLAIN SELECT pID FROM ds_products WHERE pLevel >0
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE ds_products ALL pLevel NULL NULL NULL 45939 Using where
I have indexed pLevel [tinyint(1)], but the query is not using it and doing a full table scan.
Here is the row count of this table for each value of pLevel:
pLevel count
0 34040
1 3078
2 7143
3 865
4 478
5 279
6 56
if I do the query for a specific value of pLevel, it does use the index:
EXPLAIN SELECT pID FROM ds_products WHERE pLevel =6
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE ds_products ref pLevel pLevel 1 const 1265
I've tried pLevel>=1 and pLevel<=6... but it still does a full scan
I've tried (pLevel=1 or pLevel=2 or pLevel=3 or pLevel=4 or pLevel=5 or pLevel=6) .... but it still does a full table scan.
Try using MySQL GROUP BY.
SELECT pLevel, COUNT(*)
FROM ds_products
GROUP BY pLevel
Edit:
This MySQL documentation article may be useful to you. How to Avoid Table Scans