How do I improve performance this query while also get all the information needed..
SELECT
tr.id, tr.request_status, tr.note, tr.created_date,
c.name AS customer_name, c.mobile_phONe,
u.full_name AS created_by_name, tt.name AS ticket_type_name
FROM
ticket_request tr
LEFT JOIN
ticket_type tt ON tt.id = tr.ticket_type_id
LEFT JOIN
users u ON u.id = tr.created_by
LEFT JOIN
customer c ON c.id = tr.customer_id
WHERE
tr.is_deleted != 1
AND tr.user_id IN (SELECT u.id FROM users u WHERE u.status = '1')
GROUP BY
tr.id
ORDER BY
tr.created_date DESC
LIMIT 0,20
Currently, this query runs in 7-10 seconds.
ticket_request table has about 100k rows
customers table has about 300k rows
users table and ticket_type don't have that much (about 1k rows)
The speedup technique below is to dispense with the LIMIT first, and only after that, do all the JOINs.
SELECT tr3.id, tr3.request_status, tr3.note, tr3.created_date,
c.name AS customer_name, c.mobile_phONe,
u2.full_name AS created_by_name,
tt.name AS ticket_type_name
FROM
(
SELECT tr1.id
FROM ticket_request tr1
JOIN users u1 ON u1.id = tr1.created_by
WHERE u1.status = '1'
AND tr1.is_deleted != 1
ORDER BY tr1.created_date DESC
LIMIT 0,20
) AS tr2
JOIN ticket_request AS tr3 ON tr3.id = tr2.id
JOIN user AS u2 ON u2.id = tr3.created_by
LEFT JOIN ticket_type tt ON tt.id = tr3.ticket_type_id
LEFT JOIN customer c ON c.id = tr3.customer_id
ORDER BY tr3.created_date
The JOINs, after the one in the "derived" table tr2, are touching only 20 rows; this is much of the speedup.
This may be equally good:
SELECT d.id, d.request_status, d.note, d.created_date,
c.name AS customer_name, c.mobile_phONe, d.created_by_name,
tt.name AS ticket_type_name
FROM
(
SELECT tr.id AS tr_id, tr.request_status, tr.note, tr.created_date,
tr.ticket_type_id, tr.customer_id
u.full_name AS created_by_name
FROM ticket_request tr
JOIN users u ON u.id = tr.created_by
WHERE u.status = '1'
AND tr.is_deleted != 1
ORDER BY tr.created_date DESC
LIMIT 0,20
) AS d
LEFT JOIN ticket_type tt ON tt.id = d.ticket_type_id
LEFT JOIN customer c ON c.id = d.customer_id
ORDER BY d.created_date
I'm assuming that you are using MySQL. If not, this answer can be slightly modified to fit another database, but the concept should remain the same. You can add indices to all the ID columns which are involved in the right hand side of your left joins with the ticket_request column:
ALTER TABLE ticket_type ADD INDEX (id);
ALTER TABLE users ADD INDEX (id);
ALTER TABLE customer ADD INDEX (id); -- important
To explain why an index would help, consider the first LEFT JOIN between your ticket_request table and the ticket_type table. Without an index, for each record in ticket_request the database would have to potentially scan the entire ticket_type table to find records which match the join condition. This is costly from a performance point of view. But with an index, the database can complete this operation much faster, since it "knows" where to look exactly (or almost exactly) for the matching records.
Though you mentioned that only the customer table is very large, you can still add indices to the other tables. In the future, they might get larger too. Most likely the join involving customer is the bottleneck in your query.
SELECT
tr.id, tr.request_status, tr.note, tr.created_date,
c.name AS customer_name, c.mobile_phONe,
u.full_name AS created_by_name, tt.name AS ticket_type_name
FROM
ticket_request tr
LEFT JOIN
ticket_type tt ON tt.id = tr.ticket_type_id and tr.is_deleted != 1
LEFT JOIN
users u ON u.id = tr.created_by
JOIN
users u1 ON u1.id = tr.user_id and u1.status = '1'
LEFT JOIN
customer c ON c.id = tr.customer_id
GROUP BY
tr.id
ORDER BY
tr.created_date DESC
LIMIT 0,20
try this it will work with improved performance and tweak as per your requirement
The biggest opportunity for optimization here is with LIMIT 0,20
GROUP BY tr.id make no sense and should be removed.
create index ticket_request_ix_is_deleted_created_date on ticket_request (is_deleted,created_date) and change tr.is_deleted != 1 to tr.is_deleted = 0.
Or
create index ticket_request_ix_created_date on ticket_request (created_date)
Other than indexing , On application level you can use Memcached ( in case you are using php) like stuffs. This will also give you great performance.
Related
I want to optimize this query becouse it takes to much time to return records
SELECT
u.*,
s.legal_name AS structure_name,
ui.id AS userinfo_id,
ui.structure_id AS structure_id,
ui.lrn_user,
ui.gender,
ui.fiscal_code,
ui.prov,
ui.phone,
ui.school_name,
ui.school_codice_meccanografico,
us.status, us.date AS status_date,
CONCAT(u.lastname,' ',u.firstname) AS fullname,
CONCAT(u.firstname,' ',u.lastname) AS display_name,
uu.username AS created_by_name,
g.group_names,
IF(u.website_id = 0,'Sito Web principale', w.name) AS website_name
FROM fcf_users AS u
LEFT JOIN (
SELECT
gu.user_id,
GROUP_CONCAT(gg.name SEPARATOR ', ') AS group_names
FROM fcf_user_user_groups gu
JOIN fcf_user_groups gg ON gg.id = gu.group_id
GROUP BY user_id
) g ON g.user_id = u.id
LEFT JOIN fcf_users_userinfo AS ui ON ui.user_id = u.id
LEFT JOIN fcf_users_user_statuses AS us ON us.user_id = u.id
LEFT JOIN fcf_structures_structures AS s ON s.id = ui.structure_id
LEFT JOIN fcf_users AS uu ON uu.id = u.created_by
LEFT JOIN fcf_websites AS w ON w.id = u.website_id
WHERE
u.id IN (SELECT user_id FROM fcf_user_user_groups WHERE group_id = '8')
AND u.id IN (SELECT user_id FROM fcf_user_user_groups WHERE group_id = '8')
AND ui.lrn_user = '0'
ORDER BY fullname ASC
LIMIT 0,25
If anyone can help, thanks
Turn it inside-out. That is, first use a 'derived' table to locate 25 users you want. Then gather the rest of the info.
What you have gathers all the info (including all the JOIN work) for all the users, then sorts and peels off 25.
It will be something like:
SELECT -- lots of stuff
FROM ( SELECT u.id,
CONCAT(u.lastname,' ',u.firstname) AS fullname
FROM fcf_users AS u
JOIN fcf_user_user_groups AS ug ON ...
JOIN fcf_users_userinfo AS ui ON ui.user_id = u.id
WHERE ug.group_id = '8'
AND ui.lrn_user = '0'
ORDER BY u.lastname, u.firstname -- now sargeable
LIMIT 25
) AS u25
JOIN .... -- whatever tables are needed to get the rest of the columns
ORDER BY u25.fullname -- yes, again, but now using the CONCAT
-- no limit here
Also:
u: INDEX(lastname, firstname, id)
user_user_group is a "many-t0=many mapping" table? If so, follow the indexing advice here: http://mysql.rjweb.org/doc.php/index_cookbook_mysql#many_to_many_mapping_table
Ditto for any other many:many tables.
Note how I put into the derived table only the tables needed to achieve the LIMIT.
I have products , and categories table, and a pivot table named product_catalog, I need to update the product_catalog table so that I can remove the categories which have less than five products. Those products which are in these redundant categories should move to their parent categories. I have written a query for this but problem is that this product_catalog table has 55213277 records in it and it takes lot of time to run .
Basically it is a nested query and we have to run this query for as many times unless there is no category left having less than five products.
Here is my sql query I tested.
Can you propose me an optimized solution.
UPDATE product_catalogT AS C
INNER JOIN
(SELECT
COUNT(*) AS tp, catalog_id cid, g.parent_id pid
FROM
product_catalog AS p
LEFT JOIN catalog AS g ON p.catalog_id = g.id
Where g.parent_id <> 0
GROUP BY catalog_id
HAVING tp < 5)
AS A ON C.catalog_id = A.cid
SET
C.catalog_id = A.pid
Here's a little less writing, but for performance we'd need to see your tables, indexes, and the EXPLAIN, as mentioned.
UPDATE product_catalogT C
JOIN
( SELECT p.catalog_id
FROM product_catalog p
JOIN catalog g
ON p.catalog_id = g.id
Where g.parent_id <> 0
GROUP
BY catalog_id
HAVING COUNT(*) < 5
) A
ON C.catalog_id = A.cid
SET C.catalog_id = A.pid
Also, I might mention that this seems like a rather strange request
SELECT p.id,
p.title,
p.slug,
p.content,
(SELECT url
FROM gallery
WHERE postid = p.id
LIMIT 1) AS url,
t.name
FROM posts AS p
INNER JOIN termrel AS tr
ON ( tr.object = p.id )
INNER JOIN termtax AS tx
ON ( tx.id = tr.termtax_id )
INNER JOIN terms AS t
ON ( t.id = tx.term_id )
WHERE tx.taxonomy_id = 3
AND p.post_status IS NULL
ORDER BY t.name ASC
This query took about 0.2407s to execute. How to make it fast?
Correlated subqueries can have subpar performance as they are executed row by row.
To solve this move your correlated subquery into a regular subquery/derived table and join to it. It will then not have execute row by row for the entire returned result set as it will be executed BEFORE the select statement.
mysql specific links that confirm correaleated subqueries are not optimal choices in mysql.
How to optimize
Answer indicating msql notoriously bad at optimizing correlated subqueries
I use sql-server, but I'm sure the principle is the same for mysql, so I hope this at least points you in the right direction. You would need to partition/return your one result per loan, maybe some could chime in on mysql specific syntax and I could update my answer
select
p.id
,p.title
,p.slug
,p.content
,t.name
,mySubQuery.value
from
posts as p
inner join termrel as tr
on ( tr.object = p.id )
inner join termtax as tx
on ( tx.id = tr.termtax_id )
inner join terms as t
on ( t.id = tx.term_id )
left join (
-- use MYSQL function to partition the reslts and only return 1, I use sql-server, not sure of the RDMS specific syntax
select
id
,url
from
gallery
limit 1
) as mySubquery
on mySubquery.id = p.id
where
tx.taxonomy_id = 3
and p.post_status is null
order by
t.name asc
I have three tables - tblpollquestions, tblpollanswers and tblpollresponses.
I want to select a random question that a user hasn't responded to yet, with the respective answers.
The SQL below returns exactly what I need, but I'm concerned that it takes three SELECTs to do it. There must surely be a more efficient way?
SELECT
poll.id,
poll.question,
a.answer
FROM tblpollquestions poll
INNER JOIN tblpollanswers a ON a.question_id = poll.id
INNER JOIN (
SELECT id FROM tblpollquestions WHERE id NOT IN(
SELECT question_id FROM tblpollresponses WHERE user_id = 1
) ORDER BY RAND() LIMIT 1
) as t ON t.id = poll.id
This could be made a bit better by switching NOT IN(SELECT...) into LEFT JOIN
SELECT
poll.id,
poll.question,
a.answer
FROM
tblpollquestions poll
INNER JOIN
tblpollanswers a
ON
a.question_id = poll.id
INNER JOIN (
SELECT
q.id
FROM
tblpollquestions AS q
LEFT JOIN
tblpollresponses AS r
ON
q.id = r.question_id
AND r.user_id = 1
WHERE
r.question_id IS NULL
ORDER BY RAND() LIMIT 1
) as t ON t.id = poll.id
ORDER BY RAND() can also be slow if there are many rows in tblpollquestions table. See this presentation from Bill Karwin (slide 142 and onwards) for some other ideas on selecting a random row.
http://www.slideshare.net/billkarwin/sql-antipatterns-strike-back
Is seems fine to me, although I would change it slightly:
SELECT
poll.id,
poll.question,
a.answer
FROM tblpollquestions poll
INNER JOIN tblpollanswers a ON a.question_id = poll.id
WHERE poll.id = (
SELECT id FROM tblpollquestions WHERE NOT EXISTS (
SELECT * FROM tblpollresponses WHERE user_id = 1 AND question_id = tblpollquestions.id )
ORDER BY RAND() LIMIT 1)
Written that way should do a better job of using indexes, and not checking the join conditions for every single tblpollanswers.
Make sure you have a UNIQUE index (or primary key) on tblpollresponses for (user_id, question_id) (in that order). If you need it for other queries, you can add an additional UNIQUE index with the columns in the reverse order.
Edit: Actually putting it in the where might not be so good http://jan.kneschke.de/projects/mysql/order-by-rand/ You will need to explain the query and compare.
Use left join like this:
SELECT ques.id, ques.question, ans.answer FROM tblpollquestions ques
INNER JOIN tblpollanswers ans ON(ans.question_id = ques.id)
left join tblpollresponses res on(res.question_id=ques.id and user_id = 1)
where res.question_id is null ORDER BY RAND() LIMIT 1;
I changed your table aliases to make better sense.
I'm trying to do a select statement and it works except that it's not limiting the number of results for each user (U.id) to 5.
SELECT F.id,F.created,U.username,U.fullname,U.id,I.id,I.cached_image
FROM favorites AS F
INNER JOIN users AS U
ON F.faver_profile_id = U.id
INNER JOIN items AS I
ON F.notice_id = I.id
WHERE faver_profile_id IN ('.$users.')
GROUP BY I.id HAVING COUNT(U.id) <= 5
ORDER BY F.faver_profile_id, F.created DESC
I'm grouping by I.id to eliminate duplicates. From my research it looks like you can only use HAVING COUNT if your also grouping by that column, but I cannot group by U.id or I'd lose results rows.
Instead of HAVING, can you slap a LIMIT 5 in there?
Edit: OP cannot LIMIT entire query,
and, AFAIK, MySQL does not support LIMIT in subqueries,
so you can create a temporary table with your five (5) user ids:
create table temp_table ( id INT );
insert into temp_table (id) SELECT U.id FROM users U LIMIT 5;
SELECT F.id,F.created,U.username,U.fullname,U.id,I.id,I.cached_image
FROM favorites AS F
INNER JOIN temp_table AS Ut
ON F.faver_profile_id = Ut.id
INNER JOIN items AS I
ON F.notice_id = I.id
WHERE faver_profile_id IN ('.$users.')
GROUP BY I.id
ORDER BY F.faver_profile_id, F.created DESC;
drop table temp_Table;
Let us know how that works.