I have a SQL query like this, and it takes high amount of loading...
Not sure what is happening here...if anyone could help me with this.
SELECT `posts`.* FROM `posts`
WHERE `posts`.`type`
IN ('MySubDomainSitePost')
AND `posts`.`aasm_state` = 'published'
AND (published_at <= '2015-05-12 01:01:01')
AND `posts`.`on_frontpage` = 1
AND `posts`.`is_pinned` = 0
ORDER BY published_at DESC LIMIT 16
You need to check the query performance using explain select. Now for large data-set the query will perform very poorly if the columns are not indexed.
From the given query you may need to add the following indexes
alter table posts
add index p_search_idx(type,aasm_state,published_at,on_frontpage,is_pinned);
This will boost up the speed of the query.
Make sure to take a backup of the table before applying the index.
And no need to use IN in the query it could be as
SELECT `posts`.* FROM `posts`
WHERE `posts`.`type` = 'MySubDomainSitePost'
AND `posts`.`aasm_state` = 'published'
AND (published_at <= '2015-05-12 01:01:01')
AND `posts`.`on_frontpage` = 1
AND `posts`.`is_pinned` = 0
ORDER BY published_at DESC LIMIT 16
Related
I have the following mysql query that I think should be faster. The database table has 1 million records and the query table 3.5 seconds
set #numberofdayssinceexpiration = 1;
set #today = DATE(now());
set #start_position = (#pagenumber-1)* #pagesize;
SELECT *
FROM (SELECT ad.id,
title,
description,
startson,
expireson,
ad.appuserid UserId,
user.email UserName,
ExpiredCount.totalcount
FROM advertisement ad
LEFT JOIN (SELECT servicetypeid,
Count(*) AS TotalCount
FROM advertisement
WHERE Datediff(#today,expireson) =
#numberofdayssinceexpiration
AND sendreminderafterexpiration = 1
GROUP BY servicetypeid) AS ExpiredCount
ON ExpiredCount.servicetypeid = ad.servicetypeid
LEFT JOIN aspnetusers user
ON user.id = ad.appuserid
WHERE Datediff(#today,expireson) = #numberofdayssinceexpiration
AND sendreminderafterexpiration = 1
ORDER BY ad.id) AS expiredAds
LIMIT 20 offset 1;
Here's the execution plan:
Here are the indexes defined on the table:
I wonder what I am doing wrong.
Thanks for any help
First, I would like to point out some problems. Then I will get into your Question.
LIMIT 20 OFFSET 1 gives you 20 rows starting with the second row.
The lack of an ORDER BY in the outer query may lead to an unpredictable ordering. In particular, the Limit and Offset can pick whatever they want. New versions will actually throw away the ORDER BY in the subquery.
DATEDIFF, being a function, makes that part of the WHERE not 'sargeable'. That is it can't use an INDEX. The usual way (which is sargeable) to compare dates is (assuming expireson is of datatype DATE):
WHERE expireson >= CURDATE() - INTERVAL 1 DAY
Please qualify each column name. With that, I may be able to advise on optimal indexes.
Please provide SHOW CREATE TABLE so that we can see what column(s) are in each index.
I want to improve my current query. So I have this table called Incomes. Where I have a sourceId varchar field. I have a single SELECT for the fields I need, but I needed to add an extra field called isFirstTime to represent if it was the first time on the row on what that sourceId was used. This is my current query:
SELECT DISTINCT
`income`.*,
CASE WHEN (
SELECT
`income2`.id
FROM
`income` as `income2`
WHERE
`income2`."sourceId" = `income`."sourceId"
ORDER BY
`income2`.created asc
LIMIT 1
) = `income`.id THEN true ELSE false END
as isFirstIncome
FROM
`income` as `income`
WHERE `income`.incomeType IN ('passive', 'active') AND `income`.status = 'paid'
ORDER BY `income`.created desc
LIMIT 50
The query works but slows down if I keep increasing the LIMIT or OFFSET. Any suggestions?
UPDATE 1:
Added WHERE statements used on the original query
UPDATE 2:
MYSQL version 5.7.22
You can achieve it using Ordered Analytical Function.
You can use ROW_NUMBER or RANK to get the desired result.
Below query will give the desired output.
SELECT *,
CASE
WHEN Row_number()
OVER(
PARTITION BY sourceid
ORDER BY created ASC) = 1 THEN true
ELSE false
END AS isFirstIncome
FROM income
WHERE incomeType IN ('passive', 'active') AND status = 'paid'
ORDER BY created desc
DB Fiddle: See the result here
My first thought is that isFirstIncome should be an extra column in the table. It should be populated as the data is inserted.
If you don't like that, let's try to optimize the query...
Let's avoid doing the subquery more than 50 times. This requires turning the query inside-out. (It's like "explode-implode", where the query gathers lots of stuff, then sorts it and throws most of the rows away.)
To summarize:
do the least amount of effort to just identify the 5 rows.
JOIN to whatever tables are needed (including itself if appropriate); this is to get any other columns desired (including isFirstIncome).
SELECT i3.*,
( ... using i3 ... ) as isFirstIncome
FROM (
SELECT i1.id, i1.sourceId
FROM `income` AS i1
WHERE i1.incomeType IN ('passive', 'active')
AND i1.status = 'paid'
ORDER BY i1.created DESC
LIMIT 50
) AS i2
JOIN income AS i3 USING(id)
ORDER BY i2.created DESC -- yes, repeated
(I left out the computation of isFirstIncome; it is discussed in other Answers. But note that it will be executed at most 50 times.)
(The aliases -- i1, i2, i3 -- are numbered in the order they will be "used"; this is to assist in following the SQL.)
To assist in performance, add
INDEX(status, incomeType, created, id, sourceId)
It should help with my formulation, but probably not for the other versions. Your version would benefit from
INDEX(sourceId, created, id)
SELECT `f`.*
FROM `files_table` `f`
WHERE f.`application_id` IN(6)
AND `f`.`project_id` IN(130418)
AND `f`.`is_last_version` = 1
AND `f`.`temporary` = 0
AND f.deleted_by is null
ORDER BY `f`.`date` DESC
LIMIT 5
When I remove the ORDER BY, query executes in 0.1 seconds. With the ORDER BY it takes 3 seconds.
There is an index on every WHERE column and there is also an index on ORDER BY field (date).
What can I do to make this query faster? Why is ORDER BY slowing it down so much? Table has 3M rows.
instead of an index on each column in where be sure you have a composite index that cover all the columns in where
eg
create index idx1 on files_table (application_id, project_id,is_last_version,temporary,deleted_by)
avoid IN clause for single value use = for these
SELECT `f`.*
FROM `files_table` `f`
WHERE f.`application_id` = 6
AND `f`.`project_id` = 130418
AND `f`.`is_last_version` = 1
AND `f`.`temporary` = 0
AND f.deleted_by is null
ORDER BY `f`.`date` DESC
LIMIT 5
the date or others column in select could be useful retrive all info using the index and avoiding the access to the table data .. but for select all (select *)
you probably need severl columns an then the access to the table data is done however .. but you can try an eval the performance ..
be careful to place the data non involved in where at the right of all the column involved in where
create index idx1 on files_table (application_id, project_id,is_last_version,temporary,deleted_by, date)
i try to optimize my query because it takes 3.5 seconds and its too long.
this is my query:
SELECT
`products`.*,
IFNULL(SUM(`products_uses`.`quantity`),0) as `usesQuantity`,
IF((`products`.`productQuantity` - IFNULL(SUM(`products_uses`.`quantity`),0)) > 0, (`products`.`productQuantity` - IFNULL(SUM(`products_uses`.`quantity`),0)), 0) AS `totalUses`
FROM `products`
LEFT JOIN `products_uses` ON `products`.`id` = `products_uses`.`productId`
WHERE `products`.`nurseForm` = 1
GROUP BY `products`.`id`
ORDER BY `products`.`fav` DESC, `products`.`productName` ASC
i tried to optimize with variables but nothing changed:
SELECT
`products`.*,
#usesQuantity := IFNULL(SUM(`products_uses`.`quantity`),0) as `usesQuantity`,
IF((`products`.`productQuantity` - #usesQuantity) > 0, (`products`.`productQuantity` - #usesQuantity), 0) AS `totalUses`
FROM `products`
LEFT JOIN `products_uses` ON `products`.`id` = `products_uses`.`productId`
WHERE `products`.`nurseForm` = 1
GROUP BY `products`.`id`
ORDER BY `products`.`fav` DESC, `products`.`productName` ASC
this query
sum how much quantity used each product - IFNULL(SUM(products_uses.quantity),0)
,
how much uses each product -
IF((`products`.`productQuantity` - IFNULL(SUM(`products_uses`.`quantity`),0)) > 0, (`products`.`productQuantity` - IFNULL(SUM(`products_uses`.`quantity`),0)), 0)
i tried to changed structure of the tables to myISAM and InnoDB nothing changed.
what can i do to optimize this query?
tnx
I'm not sure that introducing user variables will change much. But stategically adding indices on your two tables might help. Try this:
ALTER TABLE products ADD INDEX nurse_index (nurseForm);
ALTER TABLE products_uses ADD INDEX product_index (productId);
The first index, on the products.nurseForm column, might help the WHERE clause. In particular, this index would be a big help if only a few records match.
The second index, on products_uses.productId, might help the join go faster. Again, this would depend on how large your tables are.
You may also run EXPLAIN to see if any other bottlenecks stand out.
Providing SHOW CREATE TABLE would help. Meanwhile, I will guess.
This may (or may not) help. It attempts to do the summing without hauling all the columns of products around. It avoids the GROUP BY.
SELECT p2.*,
IFNULL(x.usesQuantity, 0) as `usesQuantity`,
GREATEST(p2.`productQuantity` - IFNULL(x.usesQuantity), 0) AS `totalUses`
FROM `products` AS p2
LEFT JOIN
( SELECT p.id AS xid,
SUM(pu.quantity) as `usesQuantity`
FROM products_uses AS pu
JOIN products AS p ON p.id = pu.productId
WHERE p.nurseForm = 1
) AS x ON x.xid = p2.id
ORDER BY p2.`fav` DESC,
p2.`productName` ASC
These indexes should help:
products: INDEX(nurseForm, id)
products: PRIMARY KEY(id) -- I am assuming this??
products_uses: INDEX(productId, quantity)
If LEFT were unnecessary there would be other optimizations.
MySQL 5.6 would help with the subquery.
MySQL 8.0 could help with the ORDER BY, meanwhile, a sort is required due to the mixture of DESC and ASC.
I have the following query:
SELECT *
FROM products
INNER JOIN product_meta
ON products.id = product_meta.product_id
JOIN sales_rights
ON product_meta.product_id = sales_rights.product_id
WHERE ( products.categories REGEXP '[[:<:]]5[[:>:]]' )
AND ( active = '1' )
AND ( products.show_browse = 1 )
AND ( product_meta.software_platform_mac IS NOT NULL )
AND ( sales_rights.country_id = '240'
OR sales_rights.country_id = '223' )
GROUP BY products.id
ORDER BY products.avg_rating DESC
LIMIT 0, 18;
Running the query with the omission of the ORDER BY section and the query runs in ~90ms, with the ORDER BY section and the query takes ~8s.
I've browsed around SO and have found the reason for this could be that the sort is being executed before all the data is returned, and instead we should be running ORDER BY on the result set instead? (See this post: Slow query when using ORDER BY)
But I can't quite figure out the definitive way on how I do this?
I've browsed around SO and have found the reason for this could be
that the sort is being executed before all the data is returned, and
instead we should be running ORDER BY on the result set instead?
I find that hard to believe, but if that's indeed the issue, I think you'll need to do something like this. (Note where I put the parens.)
select * from
(
SELECT products.id, products.avg_rating
FROM products
INNER JOIN product_meta
ON products.id = product_meta.product_id
JOIN sales_rights
ON product_meta.product_id = sales_rights.product_id
WHERE ( products.categories REGEXP '[[:<:]]5[[:>:]]' )
AND ( active = '1' )
AND ( products.show_browse = 1 )
AND ( product_meta.software_platform_mac IS NOT NULL )
AND ( sales_rights.country_id = '240'
OR sales_rights.country_id = '223' )
GROUP BY products.id
) as X
ORDER BY avg_rating DESC
LIMIT 0, 18;
Also, edit your question and include a link to that advice. I think many of us would benefit from reading it.
Additional, possibly unrelated issues
Every column used in a WHERE clause should probably be indexed somehow. Multi-column indexes might perform better for this particular query.
The column products.categories seems to be storing multiple values that you filter with regular expressions. Storing multiple values in a single column is usually a bad idea.
MySQL's GROUP BY is indeterminate. A standard SQL statement using a GROUP BY might return fewer rows, and it might return them faster.
If you can, you may want to index your ID columns so that the query will run quicker. This is a DBA-level solution, rather than a SQL solution - tuning the database will help overall performance.
The issue in the instance of this query, was that by using GROUP BY and ORDER BY in a query, MySQL is unable to use the index if the GROUP BY and ORDER BY expressions are different.
Related Reading:
http://dev.mysql.com/doc/refman/5.0/en/order-by-optimization.html
http://mysqldba.blogspot.co.uk/2008/06/how-to-pick-indexes-for-order-by-and.html