Mysql optimization based on explain - mysql

I have the following query that takes a really long time to execute. I need to speed it up, but I'm at a lost as to what technique to use. This is the query:
SELECT
`User`.`id`,
`User`.`username`,
`User`.`password`,
`User`.`role`,
`User`.`created`,
`User`.`modified`,
`User`.`email`,
`User`.`other_user_id`,
`User`.`first_name`,
`User`.`last_name`,
`User`.`place_id`,
`Resume`.`id`,
`Resume`.`user_id`,
`Resume`.`other_resume_id`,
`Resume`.`other_user_id`,
`Resume`.`file_extension`,
`Resume`.`created`,
`Resume`.`modified`,
`Resume`.`is_deleted`,
`Resume`.`has_file`,
`Resume`.`is_stamped`,
`Resume`.`is_active`
FROM
`streetofwalls`.`users` AS `User`
LEFT JOIN `my_database`.`attempts` AS `Attempt`
ON (`Attempt`.`user_id` = `User`.`id` AND `Attempt`.`test_id` != 5)
LEFT JOIN `my_database`.`reports` AS `Resume`
ON (`Resume`.`user_id` = `User`.`id`)
WHERE
`Attempt`.`test_id` = 8
AND `Attempt`.`score` > 60
AND `User`.`id` IN (
SELECT
`User1`.`id`
FROM
`my_database`.`users` AS User1
LEFT JOIN `my_database`.`tags_users` AS TagUser
ON (`User1`.`id`= `TagUser`.`user_id`)
LEFT JOIN `my_database`.`tags` AS Tag
ON (`TagUser`.`tag_id`= `Tag`.`id`)
WHERE `Tag`.`id` = (8) )
AND `User`.`id` NOT IN (
SELECT
`User1`.`id`
FROM
`my_database`.`users` AS User1
LEFT JOIN `my_database`.`tags_users` AS TagUser
ON (`User1`.`id`= `TagUser`.`user_id`)
LEFT JOIN `my_database`.`tags` AS Tag
ON (`TagUser`.`tag_id`= `Tag`.`id`)
WHERE `Tag`.`id` = (3) )
AND `Resume`.`has_file` = 1
GROUP BY `User`.`id`
ORDER BY `Attempt`.`score` DESC;
This query generates the following explain:
As you can see, I have several indexes on this query. At the moment only the resume table is not able to be indexed. Is is possible to index this table in the context of this query? Is there some other way to speed this query up that I have not thought of? Its prohibitively slow for its intended function and I'm out of ideas. Thank you to anyone who can help. Please let me know if any other information is needed.

try inner join instead of sub-query
it is default to guide query without running on data,but may be following the query will help you.
SELECT User.id, User.username, User.password, User.role, User.created, User.modified, User.email, User.other_user_id, User.first_name, User.last_name, User.place_id, Resume.id, Resume.user_id, Resume.other_resume_id, Resume.other_user_id, Resume.file_extension, Resume.created, Resume.modified, Resume.is_deleted, Resume.has_file, Resume.is_stamped, Resume.is_active
FROM
streetofwalls.users AS User
LEFT JOIN my_database.attempts AS Attempt ON (Attempt.user_id = User.id AND Attempt.test_id != 5)
LEFT JOIN my_database.reports AS Resume ON (Resume.user_id = User.id)
, my_database.users AS User1
LEFT JOIN my_database.tags_users AS TagUser on (User1.id= TagUser.user_id)
LEFT JOIN my_database.tags AS Tag ON (TagUser.tag_id= Tag.id)
WHERE
User.id = User1.id
AND Attempt.test_id = 8
AND Attempt.score > 60
AND Resume.has_file = 1
AND Tag.id = '8' AND Tag.id != '3'
GROUP BY User.id
ORDER BY Attempt.score DESC;

Related

Optimising the query which has joins and unions in mysql

I am using the laravel to build my web app. In that for listing all orders I have an query but it takes upto 40s in my localhost whereas in goddady shared hosting it takes upto 95s. Although I implemented the server side datatable yet the query takes bit long to execute. I need someone to suggest or help to make the query to excute more efficient. There are majorly two sets and I make that union.
select * from ((
select `subscription`.`id`,
`subscription`.`order_id`,
`users`.`name` as `user_name`,
`subscription`.`new_dated`,
`subscription`.`rescheduling_delivery_date` as expected_date,
COUNT(`subscription_items`.`product_id`) as items,
`subscription`.`od_dis_total` as `total`,
`subscription`.`order_discount`,
`area`.`area_name`,
`subscribe_orders`.`action` as ordertype,
driver.name as driver_name,
'NA' as payment_mode,
`payment_status`.`status_name` as payment_status,
`subscription_status`.`status_name` as order_status,
`subscription`.`od_payment_method`,
`subscription_items`.`product_id`,
subscription.delivered_dated
from `subscription`
left join `subscription_items` on `subscription`.`id` = `subscription_items`.`subscription_id`
left join `users` on `subscription`.`user_id` = `users`.`id`
left join `users` as `driver` on `subscription`.`driver_id` = `driver`.`id`
left join `ycias_address` on `subscription`.`add_id` = `ycias_address`.`id`
left join `area` on `ycias_address`.`area_id` = `area`.`id`
left join `subscribe_order_items` on `subscription`.`subscribe_order_id` = `subscribe_order_items`.`subscribe_order_id`
left join `subscribe_orders` on `subscription`.`subscribe_order_id` = `subscribe_orders`.`id`
left join `payment_status` on `subscription`.`payment_status` = `payment_status`.`id`
left join `subscription_status` on `subscription`.`od_status` = `subscription_status`.`id`
group by `subscription_items`.`subscription_id`)
union (
select `orders`.`id`,
`orders`.`order_id`,
`users`.`name` as `user_name`,
`orders`.`new_dated`,
`orders`.`expected_delivery_date` as expected_date,
COUNT(order_items.product_id) as items,
`orders`.`od_dis_total` as `total`,
`orders`.`order_discount`,
`area`.`area_name`,
'Instant Order' as ordertype,
driver.name as driver_name,
orders.payment_mode,
`payment_status`.`status_name` as payment_status,
`order_status`.`status_name` as order_status,
`orders`.`od_payment_method`,
`order_items`.`product_id`,
`orders`.`delivered_dated`
from `orders`
left join `order_items` on `orders`.`id` = `order_items`.`order_id`
left join `users` on `users`.`id` = `orders`.`user_id`
left join `users` as `driver` on `driver`.`id` = `orders`.`driver_id`
left join `ycias_address` on `orders`.`add_id` = `ycias_address`.`id`
left join `area` on `ycias_address`.`area_id` = `area`.`id`
left join `payment_status` on `orders`.`payment_status` = `payment_status`.`id`
left join `order_status` on `orders`.`od_status` = `order_status`.`id`
group by `order_items`.`order_id`)) as all_orders limit 10 offset 0
As per the suggestion given by Andy Lester I have added the index for the necessary columns. So the execution time is reduced to half that is 18 secs in localhost. I further investigated that the index columns are of datatype VARCHAR. So I changed that to INT and then executed, got results within 0.9 secs. Thanks to Andy Lester.

How to optimize performance of count query in symfony doctrine (mysql)?

I used to below query to count rows (~ 1M record and left join many table) :
SELECT COUNT(DISTINCT u0_.id) AS sclr_0
FROM user u0_
LEFT JOIN user_detail u1_ ON u0_.id = u1_.user_id
LEFT JOIN recruitment_info r2_ ON u0_.id = r2_.user_id
LEFT JOIN user u3_ ON u0_.master_account_id = u3_.id
LEFT JOIN applicants_partners a4_ ON u0_.id = a4_.applicant_id
LEFT JOIN partner p5_ ON a4_.partner_id = p5_.id
WHERE u0_.type <> 'PARTNER'
AND u0_.is_delete = 0
ORDER BY u0_.id DESC;
In my symfony, i got total :
$total = $queryBuilder->getQuery()->getSingleScalarResult();
It worked well but it took ~ 2.5s.
So, I would like to improve performance of it. I changed it into :
SELECT COUNT(u0_.id) AS sclr_0
FROM user u0_
LEFT JOIN user_detail u1_ ON u0_.id = u1_.user_id
LEFT JOIN recruitment_info r2_ ON u0_.id = r2_.user_id
LEFT JOIN user u3_ ON u0_.master_account_id = u3_.id
LEFT JOIN applicants_partners a4_ ON u0_.id = a4_.applicant_id
LEFT JOIN partner p5_ ON a4_.partner_id = p5_.id
WHERE u0_.type <> 'PARTNER'
AND u0_.is_delete = 0
GROUP BY u0_.id
ORDER BY u0_.id DESC;
The change here is remove DISTINCT and add GROUP BY.
Then I apply in symfony by count array result:
$result = $queryBuilder->getQuery()->getArrayResult();
$total = count($result);
So the total is correct but this time, it took ~ 20s , OMG. When I tried run only raw query in Sequel Pro tool, it only took ~ 40ms. Maybe there is a problem in getArrayResult() ? . Please help me, thank you.
As Akina mentions it in his comment, you can simplify this query by removing all the left join, the order by and even the distinct.
Your query will become something like this :
SELECT COUNT(u0_.id) AS sclr_0
FROM user u0_
WHERE u0_.type <> 'PARTNER'
AND u0_.is_delete = 0;
And yes, you have to use getSingleScalarResult() on your Doctrine Query instance.

mysql query very slow when adding subquery

I have below query , only 800 record taking 5 minits to run, can you some help please
SELECT
vtiger_salesorder.salesorderid,vtiger_salesorder.salesorder_no,vtiger_salesorder.sostatus,
(SELECT se.s_date
FROM
softMax_events as se
INNER JOIN vtiger_salesorder as bm ON bm.salesorderid = se.orderNum
where (bm.sostatus = 'Order' AND se.orderNum = vtiger_salesorder.salesorderid) AND se.appointTyp='60'
group by bm.salesorderid Limit 0,1) As sdate
FROM
vtiger_salesorder
Inner Join vtiger_crmentity ON vtiger_salesorder.salesorderid = vtiger_crmentity.crmid
WHERE (vtiger_salesorder.sostatus = 'Order')
and ( vtiger_crmentity.deleted<>'1')
Try this query, hope so this will help you,
SELECT vtiger_salesorder.salesorderid,vtiger_salesorder.salesorder_no,vtiger_salesorder.sostatus,se.s_date
FROM vtiger_salesorder
Inner Join vtiger_crmentity ON vtiger_salesorder.salesorderid = vtiger_crmentity.crmid
INNER JOIN softMax_events se ON se.orderNum = salesorderid
WHERE (vtiger_salesorder.sostatus = 'Order') AND/OR
se.orderNum = vtiger_salesorder.salesorderid AND se.appointTyp='60'
and ( vtiger_crmentity.deleted<>'1')
Edit:
I noticed that there is some relation between vtiger_salesorder and softMax_events and you can use a join for both table, and in this way you can remove that inner query, i tried it you may test it. this will help you for sure after a bit modification.

SQL query wrong result

i have this query:
SELECT `completed`.`ID` AS `ID`,`completed`.`level` AS `level`,`completed`.`completed_in` AS `completed_in`, COUNT(1) AS `right_answers_num`
FROM `completed`
INNER JOIN `history` ON `history`.`ID` = `completed`.`ID`
INNER JOIN `questions` ON `questions`.`ID` = `history`.`question`
WHERE `completed`.`student_id` = '1' AND `questions`.`answer` = `history`.`answer`
GROUP BY `completed`.`ID`
ORDER BY `completed`.`completed_in` DESC
what i need is to get info of each test in completed table (id,level,completed_in,right_answer_num)
the problem with that query is that if there is no one right answer(history.answer = questions.answer) then it doesn't return the row, while it should return the row(id,level,completed_in) and the right_answer_num(counter) should be zero..
please help me,, thanks ahead.
SELECT
completed.ID AS ID,
completed.level AS level,
completed.completed_in AS completed_in,
COUNT(questions.answer) AS right_answers_num
FROM completed
INNER JOIN history ON history.ID = completed.ID
LEFT JOIN questions ON questions.ID = history.question AND questions.answer = history.answer
WHERE
completed.student_id = '1'
GROUP BY
completed.ID
ORDER BY completed.completed_in DESC
use a LEFT OUTER JOIN intead of an INNER JOIN.
The second inner join is what's causing rows with no record in the questions table to be omitted. An inner join will only return rows that have data in all corresponding tables. Change the second inner join to a left join like so:
SELECT
completed.ID AS ID,
completed.level AS level,
completed.completed_in AS completed_in,
COUNT(questions.answer) AS right_answers_num
FROM completed
INNER JOIN history ON history.ID = completed.ID
LEFT JOIN questions ON questions.ID = history.question
WHERE completed.student_id = 1
GROUP BY completed.ID
ORDER BY completed.completed_in DESC

Counting rows from a big mysql query (zend)

I a developing in zend and have a rather large mysql query. The query works fine and i get the list I expect. I am doing this using Select->Where.... below is the query.
SELECT DISTINCT `d`.* FROM `deliverable` AS `d` INNER JOIN `groups` AS `g1` ON d.id = g1.deliverable_id INNER JOIN `groupmembers` AS `gm1` ON g1.id = gm1.group_id LEFT JOIN `connection` AS `c` ON d.id = c.downstreamnode_id LEFT JOIN `deliverable` AS `d1` ON c.upstreamnode_id = d1.id INNER JOIN `deliverable` AS `d2` ON CASE WHEN d1.id IS NULL THEN d.id ELSE d1.id END = d2.id INNER JOIN `groups` AS `g` ON d2.id = g.deliverable_id INNER JOIN `groupmembers` AS `gm` ON g.id = gm.group_id WHERE (g1.group_type = 100) AND (gm1.member_id = 1) AND (c.downstreamnode_id IS NULL OR d.restrict_access = 1) AND (g.group_type = 100 OR g.group_type = 110) AND (gm.member_id = 1) AND (d.deliverable_type = 110 OR d.deliverable_type = 100) GROUP BY CASE WHEN c.downstreamnode_id IS NULL THEN d.id ELSE c.downstreamnode_id END
Only problem is when I try to count the rows in a mysql query I only get 1 returned. below is the query
SELECT DISTINCT count(*) AS `rowCount` FROM `deliverable` AS `d` INNER JOIN `groups` AS `g1` ON d.id = g1.deliverable_id INNER JOIN `groupmembers` AS `gm1` ON g1.id = gm1.group_id LEFT JOIN `connection` AS `c` ON d.id = c.downstreamnode_id LEFT JOIN `deliverable` AS `d1` ON c.upstreamnode_id = d1.id INNER JOIN `deliverable` AS `d2` ON CASE WHEN d1.id IS NULL THEN d.id ELSE d1.id END = d2.id INNER JOIN `groups` AS `g` ON d2.id = g.deliverable_id INNER JOIN `groupmembers` AS `gm` ON g.id = gm.group_id WHERE (g1.group_type = 100) AND (gm1.member_id = 1) AND (c.downstreamnode_id IS NULL OR d.restrict_access = 1) AND (g.group_type = 100 OR g.group_type = 110) AND (gm.member_id = 1) AND (d.deliverable_type = 110 OR d.deliverable_type = 100) GROUP BY CASE WHEN c.downstreamnode_id IS NULL THEN d.id ELSE c.downstreamnode_id END
i generate this from by using the same 'select' that generated the first query but I reset the columns and add count in.
$this->getAdapter()->setFetchMode(Zend_Db::FETCH_ASSOC);
$select
->reset( Zend_Db_Select::COLUMNS)
->columns(array('count('.$column.') as rowCount'));
$rowCount = $this->getAdapter()->fetchOne($select);
This method works fine for all my other queries only this one i am having trouble with. I suspect it has something to do the 'CASE' I have in there but it is strange because I am getting the correct rows the the first query. Any ideas. Thanks.
FYI below are two queries that I have working successfully.
SELECT DISTINCT `po`.* FROM `post` AS `po` INNER JOIN `postinfo` AS `p` ON po.postinfo_id = p.id WHERE (p.creator_id = 1) ORDER BY `p`.`date_created` DESC
SELECT DISTINCT count(*) AS `rowCount` FROM `post` AS `po` INNER JOIN `postinfo` AS `p` ON po.postinfo_id = p.id WHERE (p.creator_id = 1) ORDER BY `p`.`date_created` DESC
In this one I have 4 rows returned in the first query and 'int 4' returned for the second one. Does anyone know why it doesnt work for the big query?
Move your DISTINCT.
SELECT COUNT(DISTINCT `po`.*) AS `rowCount` ...
Ok figured it out It was the GROUP BY that was causing only 1 result to be returned. Thanks Interrobang for you help I am sure that using DISTINCT incorrectly will have caused me a headache in the future.
Try using SQL_CALC_FOUND_ROWS in your query?
http://dev.mysql.com/doc/refman/5.0/en/information-functions.html#function_found-rows
Using SQL_CALC_FOUND_ROWS is mysql-specific, but it's pretty nice for getting a full record count even when your initial query contains a limit. Once you get the count, don't include SQL_CALC_FOUND_ROWS in subsequent queries for extra records since that will cause extra load on your query.
Your initial query would be:
SELECT SQL_CALC_FOUND_ROWS DISTINCT `d`.* FROM `deliverable` AS `d` INNER JOIN `groups` ...
You'll have to do a subsequent call after your initial query executes to get the count by doing a SELECT FOUND_ROWS().
If you do a little searching, you'll find someone who extended Zend_Db_Select to include this ability.