Optimize Query with JOINS and Subqueries - mysql

I want to speed up one of my slower queries.
The problem is that I can't access the outer colum value within a subquery.
What I have:
SELECT r.id AS room_id, r.room_name, coalesce(d.score,0) AS total_messages, d.latest
FROM cf_rooms_time_frames tf
INNER JOIN cf_rooms r on r.id = tf.room_id
INNER JOIN(
SELECT cf.room_id, count(*) as score, max(cf.id) as latest
FROM cf_rooms_messages cf
WHERE EXISTS(
SELECT NULL FROM cf_rooms_time_frames tf
WHERE tf.start <= cf.id AND ( tf.end IS NULL OR tf.end >= cf.id )
AND tf.room_id = cf.room_id AND tf.uid = 8
)
GROUP BY cf.room_id
ORDER BY latest
DESC ) d on d.room_id = r.id
WHERE tf.uid = 8
ORDER BY coalesce(latest, score) DESC LIMIT 0, 20
What I want:
SELECT r.id AS room_id, r.room_name, coalesce(d.score,0) AS total_messages, d.latest
FROM cf_rooms_time_frames tf
INNER JOIN cf_rooms r on r.id = tf.room_id
INNER JOIN(
SELECT cf.room_id, count(*) as score, max(cf.id) as latest
FROM cf_rooms_messages cf
/* line added here */
WHERE cf.room_id = tf.room_id
/* */
AND EXISTS(
SELECT NULL FROM cf_rooms_time_frames tf
WHERE tf.start <= cf.id AND ( tf.end IS NULL OR tf.end >= cf.id )
AND tf.room_id = cf.room_id AND tf.uid = 8
)
GROUP BY cf.room_id
ORDER BY latest
DESC ) d on d.room_id = r.id
WHERE tf.uid = 8
ORDER BY coalesce(latest, score) DESC LIMIT 0, 20
I think the markup explains what the query does.
It searches for "chatrooms" for a given user and orders them by the last message, gets the number of total message which ids are in a given range ( timeframes ), and the last message id.
I don't know why, but the first query returns all rows within the chatmessage table ( cf ) if I can trust EXPLAIN. It delivers the correct results but is kind of slow on a huge table.
I tested the second one with a "hardcoded" room_id and this one was very fast and doesn't "touched" the whole table.

Related

Optimizing MySQL query with multiple joins and Sub query

I am using the following query to get data from 10 table, It is working fine but quite slow, Is there any way to Optimizing the query.
Query: SELECT emi.emi_due_date,users.usr_mobile,users.usr_id,concat_ws(" ",users.usr_fname,users.usr_mname,users.usr_lname) as borrower,users.usr_status,users.usr_curnt_city, users.usr_email,emi.loan_id,emi.emi_show_date,sum(emi.emi_amount)-sum(ifnull(emi.settled_amount,0)) as due_amount,cb.cb_type,blr.bloan_collection_executive_id,blr.pp_allow,blr.bloan_legal_team_id,blr.bloan_legal_team_status,concat_ws(" ",cp.cp_fname,cp.cp_lname) as cp_name,cp.cp_mobile,cp.cp_firm_name,cp.cp_type,bg.guarantor_name,bg.guarantor_contact,pl.ecs_date,pd.p2p_date,
(SELECT instrument FROM borrower_payment_master WHERE loan_id = emi.loan_id order by id desc limit 0,1) as last_pmode,
(SELECT IFNULL(DATE_FORMAT(emi_show_date - INTERVAL 1 MONTH,"%m-%Y"),"") FROM emi AS e WHERE e.loan_id=emi.loan_id and e.emi_status < 2 ORDER by e.emi_show_date ASC limit 1) as paid_till,
(select payment_date from borrower_payment_master as bp where bp.loan_id=emi.loan_id order by bp.id desc limit 1) as last_emi_paid FROM emi AS emi
INNER JOIN borrower_loan_reg_requests AS blr ON emi.loan_id=blr.bloan_id
INNER JOIN users AS users ON users.usr_id=blr.bloan_user_id
INNER JOIN borrower_loan_disbursed_funds AS blf ON blf.df_bloan_id=emi.loan_id
LEFT JOIN channel_partners AS cp ON cp.cp_id=users.usr_cp_referral_id
LEFT JOIN borrower_posted_loans AS pl ON pl.pl_bloan_id=emi.loan_id
LEFT JOIN collection_bucket AS cb ON cb.cb_loan_id=emi.loan_id AND cb.cb_status = 1
LEFT JOIN borrower_guarantors AS bg ON bg.guarantor_borrower_id=users.usr_id
LEFT JOIN p2p_dates AS pd ON pd.p2p_loan_id=emi.loan_id AND pd.p2p_status = 1
WHERE emi.emi_status<2 AND emi.emi_amount != 0
AND (SELECT count(*) FROM borrower_payment_master as pm WHERE pm.loan_id = emi.loan_id
AND MONTH(pm.payment_date) = "'.date('m').'" AND YEAR(pm.payment_date) = "'.date('Y').'") = 0
AND (select s.settlement_date as sdate from settlement as s WHERE emi.loan_id=s.loan_id limit 1) !=""
group by emi.loan_id order by emi.loan_id desc

Odd behavior combining multiple tables and using COALESCE

I have a big query that I have been struggling with and tweaking for awhile.
SELECT
tastingNotes.userID, tastingNotes.beerID, tastingNotes.noteID,
tastingNotes.note, user.userName,
COALESCE(sum(tasteNoteRate.score),0) as `score`
FROM
tastingNotes
INNER JOIN `user` on tastingNotes.userID = `user`.userID
LEFT JOIN tasteNoteRate on tastingNotes.noteID = tasteNoteRate.noteID
WHERE tastingNotes.beerID = 'C5RJc0'
GROUP BY tastingNotes.noteID
ORDER BY score DESC
LIMIT 0,50;
I am using the COALESCE(sum(tasteNoteRate.score),0) to give results returned a value of zero if they do not have a score yet.
The odd behavior was that when I should have had two results it only returned one note with a score of zero.
When I then gave one a score they then both showed up, one with its score and then the second with zero.
Try
SELECT q.noteID, q.userID, q.beerID, q.note, q.score, u.userName
FROM (
SELECT n.noteID, n.userID, n.beerID, n.note, COALESCE(SUM(r.score), 0) score
FROM tastingNotes n LEFT JOIN tasteNoteRate r
ON n.noteID = r.noteID
WHERE n.beerID = 'C5RJc0'
GROUP BY n.noteID, n.userID, n.beerID, n.note
) q JOIN `user` u ON q.userID = u.userID
ORDER BY score DESC
LIMIT 50
SQLFiddle

Join with Max and pre-Max row

I have two tables 'sites' and 'index_log'. Table 'sites' is a information about sites(userid, name, description ...) Table index_log have columns date, index_count and siteid. So I want to get last and previous index_log row for each site, where userid = 10. This's my variant:
SELECT ff.id,
ff.siteurl,
ff.last_count,
ff.last_date,
il2.index_count as previous_count,
MAX(il2.date) as previous_date
FROM (
SELECT s.siteurl,
s.id,
il.index_count as last_count,
MAX(il.date) as last_date
FROM sites s
LEFT JOIN index_logs il ON il.siteid = s.id
WHERE s.userid = 10
GROUP BY s.id
) as ff
LEFT JOIN index_logs il2 ON il2.siteid = ff.id AND il2.date < ff.last_date
GROUP BY ff.id
But in this variant index_count column(last and previous) do not match with max date row. I hope for your help.
this will gives you last 2 log entries per sites.id
; with CTE as
(
SELECT s.siteurl, s.id, il.index_count, il.date,
RN = ROW_NUMBER() OVER (PARTITION BY s.id ORDER BY il.date DESC)
FROM sites s
JOIN index_logs il ON il.siteid = s.id
WHERE s.userid = 10
)
SELECT *
FROM CTE
WHERE RN <= 2

2 requests in only one using mysql

I have two requests
UPDATE :
I need to do something like that :
SELECT poste_nom, ups_type_contrat,
(SELECT `entpro_date`
FROM ENT_PRO
WHERE entpro_user_id = 2
ORDER BY `entpro_id` DESC
LIMIT 1) ,
serv_nom,
serv_id_resp,
user_credit_cpf,
user_indice_salarial,
FLOOR( DATEDIFF( CURDATE( ) , user_dateentree ) /365 ) AS dateEntree
FROM USER
INNER JOIN USER_POSTE_SERVICE
ON USER.user_id= USER_POSTE_SERVICE.ups_poste_id
INNER JOIN POSTE
ON USER_POSTE_SERVICE. ups_poste_id = POSTE.poste_id
INNER JOIN SERVICE
ON USER_POSTE_SERVICE.ups_id_serv = SERVICE.serv_id
WHERE user_id = 2
ORDER BY user_nom ASC
Is it possible to gather two requests in only one ?
From what I understood you want to simple merge the result of your sub-query to your main SELECT, if so you could try it this way:
SELECT poste_nom,
ups_type_contrat,
ENT_PRO_RESULT.entpro_date,
serv_nom,
serv_id_resp,
user_credit_cpf,
user_indice_salarial,
FLOOR( DATEDIFF( CURDATE( ) , user_dateentree ) /365 ) AS dateEntree
FROM USER
LEFT JOIN (SELECT entpro_date,
entpro_user_id
FROM ENT_PRO
ORDER BY entpro_id DESC
LIMIT 1) ENT_PRO_RESULT
ON USER.user_id = ENT_PRO_RESULT.entpro_user_id
INNER JOIN USER_POSTE_SERVICE
ON USER.user_id = USER_POSTE_SERVICE.ups_poste_id
INNER JOIN POSTE
ON USER_POSTE_SERVICE.ups_poste_id = POSTE.poste_id
INNER JOIN SERVICE
ON USER_POSTE_SERVICE.ups_id_serv = SERVICE.serv_id
WHERE user_id = 2
ORDER BY user_nom ASC
I've joined it on:
ON USER.user_id = ENT_PRO_RESULT.entpro_user_id
So you only need to specify the:
WHERE user_id = 2
And the sub-query will use the current row user id for the LEFT JOIN.

Join between sub-queries in SQLAlchemy

In relation to the answer I accepted for this post, SQL Group By and Limit issue, I need to figure out how to create that query using SQLAlchemy. For reference, the query I need to run is:
SELECT t.id, t.creation_time, c.id, c.creation_time
FROM (SELECT id, creation_time
FROM thread
ORDER BY creation_time DESC
LIMIT 5
) t
LEFT OUTER JOIN comment c ON c.thread_id = t.id
WHERE 3 >= (SELECT COUNT(1)
FROM comment c2
WHERE c.thread_id = c2.thread_id
AND c.creation_time <= c2.creation_time
)
I have the first half of the query, but I am struggling with the syntax for the WHERE clause and how to combine it with the JOIN. Any one have any suggestions?
Thanks!
EDIT: First attempt seems to mess up around the .filter() call:
c = aliased(Comment)
c2 = aliased(Comment)
subq = db.session.query(Thread.id).filter_by(topic_id=122098).order_by(Thread.creation_time.desc()).limit(2).offset(2).subquery('t')
subq2 = db.session.query(func.count(1).label("count")).filter(c.id==c2.id).subquery('z')
q = db.session.query(subq.c.id, c.id).outerjoin(c, c.thread_id==subq.c.id).filter(3 >= subq2.c.count)
this generates the following SQL:
SELECT t.id AS t_id, comment_1.id AS comment_1_id
FROM (SELECT count(1) AS count
FROM comment AS comment_1, comment AS comment_2
WHERE comment_1.id = comment_2.id) AS z, (SELECT thread.id AS id
FROM thread
WHERE thread.topic_id = :topic_id ORDER BY thread.creation_time DESC
LIMIT 2 OFFSET 2) AS t LEFT OUTER JOIN comment AS comment_1 ON comment_1.thread_id = t.id
WHERE z.count <= 3
Notice the sub-query ordering is incorrect, and subq2 somehow is selecting from comment twice. Manually fixing that gives the right results, I am just unsure of how to get SQLAlchemy to get it right.
Try this:
c = db.aliased(Comment, name='c')
c2 = db.aliased(Comment, name='c2')
sq = (db.session
.query(Thread.id, Thread.creation_time)
.order_by(Thread.creation_time.desc())
.limit(5)
).subquery(name='t')
sq2 = (
db.session.query(db.func.count(1))
.select_from(c2)
.filter(c.thread_id == c2.thread_id)
.filter(c.creation_time <= c2.creation_time)
.correlate(c)
.as_scalar()
)
q = (db.session
.query(
sq.c.id, sq.c.creation_time,
c.id, c.creation_time,
)
.outerjoin(c, c.thread_id == sq.c.id)
.filter(3 >= sq2)
)