I have two inner joins in my SQL query:
SELECT `M`.`msg_id`,
`U`.`username`,
`U`.`seo_username`
FROM `newdb2`.`users` AS `U`
INNER JOIN (SELECT subscriber_to_id
FROM subscriptions
WHERE subscriber_id = 434) AS subscriber
ON id = subscriber_to_id
INNER JOIN `newdb2`.`messages` AS `M`
ON (`M`.`uid_fk` = `U`.`id`)
ORDER BY id DESC LIMIT 10
When I execute this query I see that is really slow.
How can I modify thiş query to make it faster?
Quick fixes for things like this are adding indexes which allows your database server to quickly look up columns you are searching on. For more info on how to add indexes to columns, see the manual.
In this query, those columns are:
subscriptions.subscriber_id
subscriptions.subscriber_to_id
users.id
messages.uid_fk
The ORDER BY id should be OK as I assume your id column has a primary key index on it already, but ordering queries will slow it down too.
Subselect queries will also slow the query down. In this particular query, I can't see the alias subscriber (containing the results of your subquery, which is inner joined on) used anywhere, so remove that join completely.
Related
I have following tables on Production with the respective counts of records,
people count= '565367'
donors count= '556325'
telerec_recipients count= '115147'
person_addresses count= '563183'
person_emails count= '106958'
person_phones count= '676474'
person_sms count= '22275'
On the UI end I want to display some data by applying some joins or left joins, and at the end by grouping, ordering and paginating I'm generating a json response.
So for achieving this I tried 2 methods, one by creating the view file of the join queries and apply where, group by and order by clauses inside controller, and second by directly firing the laravel syntax of joins in my controller.
Because I have a large data in all the tables, The join query works good but when it comes to group by statement it takes much time to execute.
Help me out in order to optimize or faster this process.
My Example Query is:
SELECT people.id as person_id,donors.donor_id,telerec_recipients.id as telerec_recipient_id,people.first_name,people.last_name,people.birth_date,
person_addresses.address_id,person_emails.email_id,person_phones.phone_id,person_sms.sms_id
FROM `people`
inner join `donors`
on `people`.`id` = `donors`.`person_id`
left join `telerec_recipients`
on `people`.`id` = `telerec_recipients`.`person_id`
left join `person_addresses`
on `people`.`id` = `person_addresses`.`person_id`
left join `person_emails`
on `people`.`id` = `person_emails`.`person_id`
left join `person_phones`
on `people`.`id` = `person_phones`.`person_id`
left join `person_sms`
on `people`.`id` = `person_sms`.`person_id`
GROUP BY `people`.`id`, `telerec_recipients`.`id`
ORDER BY `last_name` ASC LIMIT 25 offset 0;
Result of the explain:
Donors Table Indexes
The problem is that MySQL uses the wrong index on the donors table, you need to instruct MySQL using force index index hint to use the primary key instead.
Explanation
From the output of the explain it is clear that sg really goes wrong with donors table, since you can see using temporary, using filesort in the extra column.
The key column tells you that MySQL uses a unique index on the donors.donor_id field. However, donor_id field is not used anywhere in the query for joining / grouping / filtering purposes. donors.person_id field is used in the join. Since the primary key of donors table is on the person_id field, myql should use that for the join. force index index hint tells MySQL that you firmly believe that a certain index should be used.
Note 1:
You have several duplicate indexes. Remove all duplicates, they are only slowing your system own.
Note 2:
I think you query is against the sql standards because you have fields in the select list that are neither in the group by list, nor are subject of an aggregate function such as sum(), nor are functionally dependent on the grouping fields. In MySQL under certain sql mode settings such queries are allowed to run, but they may not produce entirely the same output as you would like.
this query has multiple JOIN including aggregate functions
executing this query for approximately 6000 users took 20 seconds.
is there any other method to run this query faster?
SELECT users.id, SUM(orders.totalCost) AS bought, COUNT(comment.id) AS commentsCount, COUNT(topics.id) AS topicsCount, COUNT(users_login.id) AS loginCount, COUNT(users_download.id) AS downloadsCount
FROM users
LEFT JOIN orders ON users.id=orders.userID AND orders.status=1
LEFT JOIN comment ON users.id=comment.userID
LEFT JOIN topics ON users.id=topics.userID
LEFT JOIN users_login ON users.id=users_login.userID
LEFT JOIN users_download ON users.id=users_download.userID
WHERE users.id='$userID'
GROUP BY users.id
ORDER BY `bought` DESC
The result of running explain
The EXPLAIN output shows you are doing full-table scans on everything except users. You need to create secondary (non-unique) indexes on userID on all the other tables in the join. That will speed up queries on individual users.
However, if you're going to process all users in one pass then do a single select without a WHERE users.id= clause. Your aggregation returns only one row per user and you should create a single resultset containing all the rows and iterate over that, instead of reissuing the query once per user. In this case the secondary indexes may still help as counts can be determined from the index alone without looking at the tables themselves.
I have this simple join that works great but is HORRIBLY slow I think because the tech table is very large. There are many instances of uid as it tracks timestamp of the uid thus the distinct. What is the best way to speed this query up?
SELECT DISTINCT tech.uid,
listing.empno,
listing.firstname,
listing.lastname
FROM tech,
listing
WHERE tech.uid = listing.empno
ORDER BY listing.empno ASC
First add an Index to tech.UID and listing.EmpNo on their respective tables.
After you are sure there are indexes you can try to re-write your query like this:
SELECT DISTINCT tech.uid, listing.EmpNo, listing.FirstName, listing.LastName
FROM listing INNER JOIN tech ON tech.uid = listing.EmpNo
ORDER BY listing.EmpNo ASC;
If it's still not fast enough, put the word EXPLAIN before the query to get some hints about the execution plan of the query.
EXPLAIN SELECT DISTINCT tech.uid, listing.EmpNo, listing.FirstName, listing.LastName
FROM listing INNER JOIN tech ON tech.uid = listing.EmpNo
ORDER BY listing.EmpNo ASC;
Posts the Explain results so we can get better insight.
Hope it helps,
This is very simple query. Only thing you can do in SQL - you may add indexes on fields used in JOIN/WHERE and ORDER BY clauses (tech.uid, listing.empno), if there are no indexes.
If there are JOIN fields with NULL values - they may ruin your performance. You should filter them in WHERE clause (WHERE tech.uid is not null and listing.empno not null). If there are many rows with JOIN on NULL field - that data may produce cartesian result (not sure how is this called in english) with may contain enormous count of rows.
You may change MySQL configuration. There are many options useful for performance tuning, like key_buffer_size, sort_buffer_size, tmp_table_size, max_heap_table_size, read_buffer_size etc.
I'm having a serious problem with MySQL (innoDB) 5.0.
A very simple SQL query is executed with a very unexpected query plan.
The query:
SELECT
SQL_NO_CACHE
mbCategory.*
FROM
MBCategory mbCategory
INNER JOIN ResourcePermission as rp
ON rp.primKey = mbCategory.categoryId
where mbCategory.groupId = 12345 AND mbCategory.parentCategoryId = 0
limit 20;
MBCategory - contains 216583 rows
ResourcePermission - contains 3098354 rows.
In MBCategory I've multiple indexes (columns order as in index):
Primary (categoryId)
A (groupId,parentCategoryId,categoryId)
B (groupId,parentCategoryId)
In ResourcePermission I've multiple indexes (columns order as in index):
Primary - on some column
A (primKey).
When I look into query plan Mysql changes tables sequence and selects rows from ResourcePermission at first and then it joins the MBCategory table (crazy idea) and it takes ages. So I added STRAIGHT_JOIN to force the innodb engine to use correct table sequence:
SELECT
STRAIGHT_JOIN SQL_NO_CACHE
mbCategory.*
FROM
MBCategory
mbCategory
INNER JOIN ResourcePermission as rp
ON rp.primKey = mbCategory.categoryId
where mbCategory.groupId = 12345 AND mbCategory.parentCategoryId = 0
limit 20;
But here the second problem materialzie:
In my opinion mysql should use index A (primKey) on the join operation instead it performs Range checked for each record (index map: 0x400) and it again takes ages !
Force index doesn't help, mysql still performing Range checked for each record .
There are only 23 rows in the MBCategory which fulfill where criteria, and after join there are only 75 rows.
How can I make mysql to choose correct index on this operation ?
Ok,
elementary problem.
I owe myself a beer.
The system I'm recently tunning is not a system I've developted - I've been assigned to it by my management to improve performance (originall team doesn't have knowledge on this topic).
After fee weeks of improving SQL queries, indexes, number of sql queries that are beeing executed by application I didn't check one of the most important things in this case !!
COLUMN TYPES ARE DIFFERENT !
Developer who have written than kind of code should get quite a big TALK.
Thanks for help !
I had the same problem with a different cause. I was joining a large table, and the ON clause used OR to compare the primary key (ii.itemid) to two different columns:
SELECT *
FROM share_detail sd
JOIN box_view bv ON sd.container_id = bv.id
JOIN boxes b ON b.id = bv.shared_id
JOIN item_index ii ON ii.itemid = bv.shared_id OR b.parent_itemid = ii.itemid;
Fortunately, it turned out the parent_itemid comparison was redundant, so I was able to remove it. Now the index is being used as expected. Otherwise, I was going to try splitting the item_index join into two separate joins.
I have the following query which both tables are huge. The query were very slow and I need your idea to optimize this query or do you have any other solution?
SELECT c.EstablishmentID,
(SELECT COUNT(ID)
FROM cleanpoi
WHERE EstablishmentID=c.EstablishmentID OR EstablishmentID
IN (SELECT ChildEstablishmentID
FROM crawlerchildren
WHERE ParentEstablishmentID=c.EstablishmentID)
) POI
FROM crawler c
GROUP BY c.EstablishmentID
BTW, I have the appropriate indexes applied.
UPDATE:
Okay, I have attached the explain result.
Try it by using JOIN
SELECT c.EstablishmentID, COUNT(d.ID)
FROM crawler c
LEFT JOIN cleanpoi d
ON c.establishmentid = d.establishmentID
LEFT JOIN
(
SELECT DISTINCT ChildEstablishmentID
FROM crawlerchildren
) e ON e.ParentEstablishmentID = c.EstablishmentID
GROUP BY c.EstablishmentID
IN() and NOT IN() subqueries are poorly optimized:
MySQL executes the subquery as a dependent subquery for each row in the outer query. This is a frequent cause of serious performance problems in MySQL 5.5 and older versions. The query probably should be rewritten as a JOIN or a LEFT OUTER JOIN, respectively.
Non-deterministic GROUP BY:
The SQL retrieves columns that are neither in an aggregate function nor the GROUP BY expression, so these values will be non-deterministic in the result.
GROUP BY or ORDER BY on different tables:
This will force the use of a temporary table and filesort, which can be a huge performance problem and can consume large amounts of memory and temporary space on disk.