Laravel Join tables and group by sum query too slow - mysql

I am using Laravel query builder to get desired results from database. The following query if working perfectly but taking too much time to get results. Can you please help me with this?
select
`amz_ads_sp_campaigns`.*,
SUM(attributedUnitsOrdered7d) as order7d,
SUM(attributedUnitsOrdered30d) as order30d,
SUM(attributedSales7d) as sale7d,
SUM(attributedSales30d) as sale30d,
SUM(impressions) as impressions,
SUM(clicks) as clicks,
SUM(cost) as cost,
SUM(attributedConversions7d) as attributedConversions7d,
SUM(attributedConversions30d) as attributedConversions30d
from
`amz_ads_sp_product_targetings`
inner join `amz_ads_sp_report_product_targetings` on `amz_ads_sp_product_targetings`.`campaignId` = `amz_ads_sp_report_product_targetings`.`campaignId`
inner join `amz_ads_sp_campaigns` on `amz_ads_sp_report_product_targetings`.`campaignId` = `amz_ads_sp_campaigns`.`campaignId`
where
(
`amz_ads_sp_product_targetings`.`user_id` = ?
and `amz_ads_sp_product_targetings`.`profileId` = ?
)
group by
`amz_ads_sp_product_targetings`.`campaignId`
Result of Explain SQL
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE amz_ads_sp_report_product_targetings ALL campaignId NULL NULL NULL 50061 Using temporary; Using filesort
1 SIMPLE amz_ads_sp_campaigns ref campaignId campaignId 8 pr-amz-ppc.amz_ads_sp_report_product_targetings.ca... 1
1 SIMPLE amz_ads_sp_product_targetings ref campaignId campaignId 8 pr-amz-ppc.amz_ads_sp_report_product_targetings.ca... 33 Using where

Your query could benefit from several indices to cover the WHERE clause as well as the join conditions:
CREATE INDEX idx1 ON amz_ads_sp_product_targetings (
user_id, profileId, campaignId);
CREATE INDEX idx2 ON amz_ads_sp_report_product_targetings (
campaignId);
CREATE INDEX idx3 ON amz_ads_sp_campaigns (campaignId);
The first index idx1 covers the entire WHERE clause, which might let MySQL throw away many records on the initial scan of the amz_ads_sp_product_targetings table. It also includes the campaignId column, which is needed for the first join. The second and third indices cover the join columns of each respective table. This might let MySQL do a more rapid lookup during the join process.
Note that selecting amz_ads_sp_campaigns.* is not valid unless the campaignId of that table be the primary key. Also, there isn't much else we can do speed up the query, as SUM, by its nature, requires touching every record in order to come up the result sum.

Related

Slow MySQL query, EXPLAIN shows Using temporary; Using filesort

This query:
EXPLAIN SELECT ppi_loan.customerID,
loan_number,
CONCAT(forename, ' ', surname) AS agent,
name,
broker,
(SELECT timestamp
FROM ppi_sar_status
WHERE history = 0
AND (status = 10 || status = 13)
AND ppi_sar_status.loanID = ppi_loan.loanID) AS ppi_unsure_date,
fosSent,
letterSent,
(SELECT timestamp
FROM ppi_ques_status
WHERE status = 1
AND ppi_ques_status.loanID = ppi_loan.loanID
ORDER BY timestamp DESC LIMIT 1) AS sent_date,
ppi_ques_status.timestamp
FROM ppi_loan
LEFT JOIN ppi_assignments ON ppi_assignments.customerID = ppi_loan.customerID
LEFT JOIN italk.users ON italk.users.id = agentID
LEFT JOIN ppi_ques_status ON ppi_ques_status.loanID = ppi_loan.loanID
JOIN ppi_lenders ON ppi_lenders.id = ppi_loan.lender
JOIN ppi_status ON ppi_status.customerID = ppi_loan.customerID
JOIN ppi_statuses ON ppi_statuses.status = ppi_status.status
AND ppi_ques_status.status = 1
AND ppi_ques_status.history = 0
AND (cc_type = '' || (cc_type != '' AND cc_accepted = 'no'))
AND ppi_loan.deleted = 'no'
AND ppi_loan.customerID != 10
GROUP BY ppi_loan.customerID, loan_number
Is very slow, here are all the results from the EXPLAIN query
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY ppi_ques_status ref loanID,status,history status 3 const 91086 Using where; Using temporary; Using filesort
1 PRIMARY ppi_loan eq_ref PRIMARY,customerID PRIMARY 8 ppimm.ppi_ques_status.loanID 1 Using where
1 PRIMARY ppi_lenders eq_ref PRIMARY PRIMARY 4 ppimm.ppi_loan.lender 1 Using where
1 PRIMARY ppi_assignments eq_ref customerID customerID 8 ppimm.ppi_loan.customerID 1
1 PRIMARY users eq_ref PRIMARY PRIMARY 8 ppimm.ppi_assignments.agentID 1
1 PRIMARY ppi_status ref status,customerID customerID 8 ppimm.ppi_loan.customerID 6
1 PRIMARY ppi_statuses eq_ref PRIMARY PRIMARY 4 ppimm.ppi_status.status 1 Using where; Using index
3 DEPENDENT SUBQUERY ppi_ques_status ref loanID,status loanID 8 func 1 Using where; Using filesort
2 DEPENDENT SUBQUERY ppi_sar_status ref loanID,status,history loanID 8 func 2 Using where
Why is it scanning so many rows and why "Using temporary; Using filesort"?
I can't remove any subqueries as I need all of the results that they produce
As already mentioned in a comment, the main cause of a slow query is that you seem to have single column indexes only, while you would need multi-column indexes to cover the joins, the filters, and the group by.
Also, your query has 2 other issues:
Even though you group by on 2 fields only, several other fields are listed in the select list without being subject to an aggregate function, such as min(). MySQL does allow such queries to be run under certain sql mode settings, but they are still against the sql standard and may have unexpected side effects, unless you really know what your are doing.
You have filters on the ppi_loan table in the join condition that is the left table in a left join. Due to the nature of the left join, these records will not be eliminated from the resultset, but MySQL will not join any values on them. These criteria should be moved to the where clause.
The indexes I would create:
ppi_sar_status: multi-column index on loanID, status, history fields - I would consider moving this to the join section because this table is not there
ppi_ques_status: multi-column index on loanID, status, timestamp fields - this would support both the subquery and the join. Remember, the subquery also has filesort in the explain.
ppi_loan: as a minimum a multi-column index on customerID, loan_number fields to support the group by clause, therefore avoiding the filesort as a minimum. You may consider adding the other fields in the join criteria based on their selectivity to this index.
I'm also not sure why you have the last 2 status tables in the join, since you are not retrieving any values from them. If you re using these tables to eliminate certain records, then consider using an exists() subquery instead of a join. In a join MySQL needs to fetch data from all joined tables, whereas in an exists() subquery it would only check if at least 1 record exists in the resultset without retrieving any actual data from the underlying tables.

How optimize this query

select mobile_no,mobile_source_type_id,voter_id,district_id,
constituency_id,tehsil_id,local_election_body_id,panchayat_id,
booth_id,is_dnd
from mobile_numbers2
where mobile_no not in (
SELECT mobile_number
from mobile_numbers
)
For this Query it's taking more time.
By using Explain query . It showing below message, How optimize this query.
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY mobile_numbers2 ALL NULL NULL NULL NULL 7783355 Using where
2 DEPENDENT SUBQUERY mobile_numbers index idx_mobile_numbers_mobile_number,idx_mobile_no idx_mobile_numbers_mobile_number 48 NULL 49256693 Using where; Using index
since I don't have your database at my disposal, I'm unable to test this query so it may need some tweaking, but this might be faster:
select
mobile_no as mobile_no1,
mobile_numbers.mobile_number as mobile_no2,
mobile_source_type_id,
voter_id,
district_id,
constituency_id,
tehsil_id,
local_election_body_id,
panchayat_id,
booth_id,
is_dnd
from mobile_numbers2
left join mobile_numbers on mobile_numbers.mobile_number = mobile_numbers2.mobile_no
where mobile_no2 IS NULL
A couple of notes:
Subqueries, particularly when paired with 'IN()' are slow
if I were maintaining your database, I would create a single 'mobile_numbers' table and I would reference it from all other tables using it's 'id' column, this would make things cleaner/faster in general.

Complex MySQL Select Left Join Optimization Indexing

I have a very complex query that is running and finding locations of members joining the subscription details and sorting by distance.
Can someone provide instruction on the correct indexes and cardinality I should add to make this load faster.
Right now on 1 million records it takes 75 seconds and I know it can be improved.
Thank you.
SELECT SQL_CALC_FOUND_ROWS (((acos(sin((33.987541*pi()/180)) * sin((users_data.lat*pi()/180))+cos((33.987541*pi()/180)) * cos((users_data.lat*pi()/180)) * cos(((-118.472153- users_data.lon)* pi()/180))))*180/pi())*60*1.1515) as distance,subscription_types.location_limit as location_limit,users_data.user_id,users_data.last_name,users_data.filename,users_data.user_id,users_data.phone_number,users_data.city,users_data.state_code,users_data.zip_code,users_data.country_code,users_data.quote,users_data.subscription_id,users_data.company,users_data.position,users_data.profession_id,users_data.experience,users_data.account_type,users_data.verified,users_data.nationwide,IF(listing_type = 'Company', company, last_name) as name
FROM `users_data`
LEFT JOIN `users_reviews` ON users_data.user_id=users_reviews.user_id AND users_reviews.review_status='2'
LEFT JOIN users_locations ON users_locations.user_id=users_data.user_id
LEFT JOIN subscription_types ON users_data.subscription_id=subscription_types.subscription_id
WHERE users_data.active='2'
AND subscription_types.searchable='1'
AND users_data.state_code='CA'
AND users_data.country_code='US'
GROUP BY users_data.user_id
HAVING distance <= '50'
OR location_limit='all'
OR users_data.nationwide='1'
ORDER BY subscription_types.search_priority ASC, distance ASC
LIMIT 0,10
EXPLAIN
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE users_reviews system user_id,review_status NULL NULL NULL 0 const row not found
1 SIMPLE users_locations system user_id NULL NULL NULL 0 const row not found
1 SIMPLE users_data ref subscription_id,active,state_code,country_code state_code 47 const 88241 Using where; Using temporary; Using filesort
1 SIMPLE subscription_types ALL PRIMARY,searchable NULL NULL NULL 4 Using where; Using join buffer
You query is not that complex. You have only one join, on a table subscription_types which is certainly a little table with no more than a few hundred rows.
Where are your indexes ? The best way to improve your query is to create indexes on the field you are filtering, like active, country_code, state_code and searchable
Have you create the foreign key on users_data.subscription_id ? You need an index on that too.
ForceIndex is useless, let the RDBMS determine the best indexes to chose.
Left Join is useless too, because the line subscription_types.searchable='1' will remove the unmatch correspondance
The order on search_priority implies that you need indexes on this columns too
The filtering in the HAVING can make the indexes not used. You don't need to put these filters in the HAVING. If I understand your table schema, this is not really the aggregate that is filtered.
Your table contains 1 million rows, but how much rows are returned, without the limit? With the right indexes, the query should execute under a second.
SELECT ...
FROM `users_data`
INNER JOIN subscription_types
ON users_data.subscription_id = subscription_types.subscription_id
WHERE users_data.active='2'
AND users_data.country_code='US'
AND users_data.state_code='NY'
AND subscription_types.searchable='1'
AND (distance <= '50' OR location_limit='all' OR users_data.nationwide='1')
GROUP BY users_data.user_id
ORDER BY subscription_types.search_priority ASC, distance ASC
LIMIT 0,10

MySQL Query optimization in PHP

I have three tables, all of them can have possibly millions of rows. I have an actions table and a reactions table, that holds reactions related to actions. Then there is a emotes table linked to reactions. What I would like to do with this particular query, is finding the most clicked emote for a certain action. The difficulty for me is that the query includes three tables instead of only two.
Table actions (postings):
PKY id
...
Table reactions (comments, emotes etc.):
PKY id
INT action_id (related to actions table)
...
Table emotes:
PKY id
INT react_id (related to reactions table)
INT emote_id (related to a hardcoded list of available emotes)
...
The SQL query I came up with basically seems to work, but it takes 12 seconds if the tables contain millions rows. The SQL query looks like this:
select emote_id, count(*) as cnt from emotes
where react_id in (
select id from reactions where action_id=2942715
)
group by emote_id order by cnt desc limit 1
MySQL explain says the following:
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY emotes index NULL react_id_2 21 NULL 4358594 Using where; Using index; Using temporary; Using f...
2 DEPENDENT SUBQUERY reactions unique_subquery PRIMARY,action_id PRIMARY 8 func 1 Using where
...I am grateful for any tips for improving on the query. Note that I will NOT call this query every time a list of actions is being built, but only when emotes are being added. Therefore it's no problem if the query takes maybe 0.5 seconds to finish. But 12 is too long!
what about this
SELECT
emote_id,
count(*) as cnt
FROM emotes a
INNER JOIN reactions r
ON r.id = a.react_id
WHERE action_id = 2942715
GROUP BY emote_id
ORDER BY cnt DESC
LIMIT 1

Help on MySQL table indexing when GROUP BY is used in a query

Thank you for your attention.
There are two INNODB tables:
Table authors
id INT
nickname VARCHAR(50)
status ENUM('active', 'blocked')
about TEXT
Table books
author_id INT
title VARCHAR(150)
I'm running a query against these tables, to get each author and a count of books he has:
SELECT a. * , COUNT( b.id ) AS book_count
FROM authors AS a, books AS b
WHERE a.status != 'blocked'
AND b.author_id = a.id
GROUP BY a.id
ORDER BY a.nickname
This query is very slow (takes about 6 seconds to execute). I have an index on books.author_id and it works perfectly, but I do not know how to create an index on authors table, so that this query could use it.
Here is how current EXPLAIN looks:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE a ALL PRIMARY,id_status_nickname NULL NULL NULL 3305 Using where; Using temporary; Using filesort
1 SIMPLE b ref key_author_id key_author_id 5 a.id 2 Using where; Using index
I've looked at MySQL manual on optimizing queries with group by, but could not figure out how I can apply it on my query.
I'll appreciate any help and hints on this - what must be the index structure, so that MySQL could use it?
Edit
I have tried:
(id, status, nickname)
(status, nickname)
Both resulted in the same situation.
I assume that the id_status_nickname is a composite index (id,status,nickname). In your query you filter the rows by saying a.status != blocked. This has following issues:
You dont have an index that can be used for this. (id,status,nickname) cannot be used because status is not the prefix of that index
Assuming you have an index on status, it cannot be used when using !=. you have to change that to status='active'
Also, status being an enum field with just two values the cardinality will be low. So mysql may endup not using the index at all.
You can try this: create index as (status,id,nickname) and use status='active'. My guess is that since you are using '=' and status is the prefix of the index it should select this index and then use it for group by and then order by.Hope this helps.
UPDATE:
Looks like it is not possible to avoid filesort when the WHERE clause does not have the field used in ORDER BY.
I would try an index on (status, nickname). That should get rid of the necessity of "Using filesort".