MySQL query optimization - mysql

Just wondering what's a better way to write this query. Cheers.
SELECT r.user_id AS ID, m.prenom, m.nom
FROM `0_rank` AS l
LEFT JOIN `0_right` AS r ON r.rank_id = l.id
LEFT JOIN `0_user` AS m ON r.user_id = m.id
WHERE r.section_id = $section_id
AND l.rank = '$rank_name' AND depart_id IN
(SELECT depart_id FROM 0_depart WHERE user_id = $user_id AND section_id = $section_id)
GROUP BY r.user_id
Here are the table structures:
0_rank: id | section_id | rank_name |
other_stuffs
0_user: id | prenom | nom | other_stuffs
0_right: id | section_id | user_id |
rank_id | other_stuffs
0_depart: id | section_id | user_id | depart_id
| other_stuffs
The idea is to use the same in a function like:
public function usergroup($section_id,$rank_name,$user_id) {
// mysql query goes here to get a list of appropriate users
}
Update: I think I have not been able to express myself clearly earlier. Here is the most recent query that seems to be working.
SELECT m.id, m.prenom, m.nom,
CAST( GROUP_CONCAT( DISTINCT d.depart ) AS char ) AS deps,
CAST( GROUP_CONCAT( DISTINCT x.depart ) AS char ) AS depx
FROM `0_rank` AS l
LEFT JOIN `0_right` AS r ON r.rank_id = l.id
LEFT JOIN `0_member` AS m ON r.user_id = m.id
LEFT JOIN `0_depart` AS d ON m.id = d.user_id
LEFT JOIN `0_depart` AS x ON x.user_id = $user_id
WHERE r.section = $section_id
AND l.rank = '$rank_name'
GROUP BY r.user_id ORDER BY prenom, nom
Now I want to get only those result, where all entries of deps are present in entries in depx.
In other term, every user is associated with some departs. $user_id is also an user is associated with some departs.
I want to get those users whose departs are common to the departs of $user_id.
Cheers.

Update
I'm not sure without being able to see the data but I believe this query will give you the results you want the fastest.
SELECT m.id, m.prenom, m.nom,
CAST( GROUP_CONCAT( DISTINCT d.depart ) AS char ) AS deps,
FROM `0_rank` AS l
LEFT JOIN `0_right` AS r ON r.rank_id = l.id and r.user_id = $user_id
LEFT JOIN `0_member` AS m ON r.user_id = m.id
LEFT JOIN `0_depart` AS d ON m.id = d.user_id
WHERE r.section = $section_id
AND l.rank = '$rank_name'
GROUP BY r.user_id ORDER BY prenom, nom
Let me know if this works.
Try this:
(By converting the functionality of the IN (SELECT...) to an inner join, you get exactly the same results but it might be the optimizer will make better choices.)
SELECT r.user_id AS ID, m.prenom, m.nom
FROM `0_rank` AS l
LEFT JOIN `0_right` AS r ON r.rank_id = l.id and r.section_id = 2
LEFT JOIN `0_user` AS m ON r.user_id = m.id
INNER JOIN `0_depart` AS x ON l.section_id = x.section_id and x.user_id = $user_id AND x.section_id = $section_id
WHERE l.rank = 'mod'
GROUP BY r.user_id
I also moved the constraints on 0_right to the join statement because I think that is clearer -- presumably this change won't matter to the optimizer.

I know nothing about your DB structure but your subselect looks like it can be replaced with a simple INNER JOIN against whatever table has the depart column. MySQL is well known for its poor subquery optimization.

Without knowing the structures or indexes, I would first add "STRAIGHT_JOIN" if the critical criteria is in-fact from the 0-rank table. Then, ensure 0_rank has an index on "rank". Next, ensure the 0_right has an index on rank_id at a minimum, but rank_id, section to take advantage of BOTH your criteria. Index on 0_member on id.
Additionally, do you mean left-join (ie: record only required in the 0_rank or 0_member) on the respective 0_right and 0_member tables instead of a normal join (where BOTH tables must match on their IDs).
Finally, ensure index on the depart table on user_id.
SELECT STRAIGHT_JOIN
r.user_id AS ID,
m.prenom,
m.nom
FROM
0_rank AS l
LEFT JOIN `0_right` AS r
ON l.id = r.rank_id
AND r.section = 2
LEFT JOIN `0_member` AS m
ON r.user_id = m.id
WHERE
l.rank = 'mod'
AND depart IN (SELECT depart
FROM 0_depart
WHERE user_id = 2
AND user_sec = 2)
GROUP BY
r.user_id
---- revised post from feedback.
From the parameters you are listing, you are always including the User ID... If so, I would completely restructure it to get whatever info is for that user. Each user should apparently can be associated to multiple departments and may or may NOT match the given rank / department / section you are looking for... I would START the query with the ONE USER because THAT will guarantee a single entry, THEN tune-down to the other elements...
select STRAIGHT_JOIN
u.id,
u.prenom,
u.nom,
u.other_stuffs,
rank.rank_name
from
0_user u
left join 0_right r
on u.id = r.user_id
AND r.section_id = $section_id
join 0_rank rank
on r.rank_id = rank.id
AND rank.rank_name = '$rank_name'
left join 0_dept dept
on u.id = dept.user_id
where
u.id = $user_id
Additionally, I have concern about your table relationships and don't see a legit join to the department table...
0_user
0_right by User_ID
0_rank by right.rank_id
0_dept has section which could join to rank or right, but nothing to user_id directly

Run explain on the query - it will help you find where the caveats are:
EXPLAIN SELECT r.user_id AS ID, m.prenom, m.nom
FROM 0_rank AS l
LEFT JOIN `0_right` AS r ON r.rank_id = l.id
LEFT JOIN `0_member` AS m ON r.user_id = m.id
WHERE r.section = 2
AND l.rank = 'mod' AND depart IN
(SELECT depart FROM 0_depart WHERE user_id = 2 AND user_sec = 2)
GROUP BY r.user_id\G

Related

Substitute "OR EXISTS" in MySql query so i can get better perfomance results

This query is taking forever to finish in MySql 8, doing some research i found out that the "EXISTS" in this code can be extremely slow in some queries.
When i remove the "OR EXISTS" sub-query part, it runs in less than a second.
So i need to substitute the "OR EXISTS" in this query so i can get all the users i need:
SELECT u.name,
u.email,
u.cpf,
u.register,
r.name AS role_name,
s.name AS sector_name,
b.name AS branch_name,
u.status
FROM users u
INNER JOIN roles r ON r.id = u.role_id
INNER JOIN sectors s ON s.id = u.sector_id
INNER JOIN branches b ON b.id = u.branch_id
WHERE u.status = 2 OR EXISTS (
SELECT *
FROM user_recovery ur
WHERE ur.user_id = u.id
AND ur.status_recovery = 1
)
Is there a way to do it without the "OR EXISTS"?
Or can enforce a full scan
try
you can't get rid of the eXISTS clause because it increases the number of returned rows.
Add a INDEX on user status and user_recovery userid,status_recovery and on the on Clause columns.
SELECT u.name,
u.email,
u.cpf,
u.register,
r.name AS role_name,
s.name AS sector_name,
b.name AS branch_name,
u.status
FROM users u
INNER JOIN roles r ON r.id = u.role_id
INNER JOIN sectors s ON s.id = u.sector_id
INNER JOIN branches b ON b.id = u.branch_id
WHERE u.status = 2
UNION
SELECT u.name,
u.email,
u.cpf,
u.register,
r.name AS role_name,
s.name AS sector_name,
b.name AS branch_name,
u.status
FROM users u
INNER JOIN roles r ON r.id = u.role_id
INNER JOIN sectors s ON s.id = u.sector_id
INNER JOIN branches b ON b.id = u.branch_id
WHERE EXISTS (
SELECT 1
FROM user_recovery ur
WHERE ur.user_id = u.id
AND ur.status_recovery = 1
)
"I'll see your UNION; and raise you a derived table."
SELECT u.name,
u.email,
u.cpf,
u.register,
r.name AS role_name,
s.name AS sector_name,
b.name AS branch_name,
u.status
FROM ( SELECT id
FROM users
WHERE status = 2
UNION DISTINCT -- or UNION ALL; see below
SELECT user_id
FROM user_recovery
WHERE status_recovery = 1 -- see new index
) AS u1
JOIN users AS u USING(id) -- self-join to pick up other columns
JOIN roles r ON r.id = u.role_id
JOIN sectors s ON s.id = u.sector_id
JOIN branches b ON b.id = u.branch_id;
Indexes:
user_recovery: INDEX(status_recovery, user_id) -- in this order
users: INDEX(status, id) -- in this order
(I assume `id` is the PRIMARY KEY in each table)
The general rule here is... When you have a bunch of JOINs, but a single table that controls which rows, but that is messy or slow (eg UNION in this case, GROUP BY or LIMIT in other cases),
Optimize finding the ids (user.id aka user_id) is the optimal way.
Then JOIN back to the original table (if needed), plus the other tables.
In doing all that, it became apparent that a new index for user_recovery might be beneficial.
(If UNION ALL won't produce any dups, switch to it for a little more speed.)

mysql query takes to much time

I want to optimize this query becouse it takes to much time to return records
SELECT
u.*,
s.legal_name AS structure_name,
ui.id AS userinfo_id,
ui.structure_id AS structure_id,
ui.lrn_user,
ui.gender,
ui.fiscal_code,
ui.prov,
ui.phone,
ui.school_name,
ui.school_codice_meccanografico,
us.status, us.date AS status_date,
CONCAT(u.lastname,' ',u.firstname) AS fullname,
CONCAT(u.firstname,' ',u.lastname) AS display_name,
uu.username AS created_by_name,
g.group_names,
IF(u.website_id = 0,'Sito Web principale', w.name) AS website_name
FROM fcf_users AS u
LEFT JOIN (
SELECT
gu.user_id,
GROUP_CONCAT(gg.name SEPARATOR ', ') AS group_names
FROM fcf_user_user_groups gu
JOIN fcf_user_groups gg ON gg.id = gu.group_id
GROUP BY user_id
) g ON g.user_id = u.id
LEFT JOIN fcf_users_userinfo AS ui ON ui.user_id = u.id
LEFT JOIN fcf_users_user_statuses AS us ON us.user_id = u.id
LEFT JOIN fcf_structures_structures AS s ON s.id = ui.structure_id
LEFT JOIN fcf_users AS uu ON uu.id = u.created_by
LEFT JOIN fcf_websites AS w ON w.id = u.website_id
WHERE
u.id IN (SELECT user_id FROM fcf_user_user_groups WHERE group_id = '8')
AND u.id IN (SELECT user_id FROM fcf_user_user_groups WHERE group_id = '8')
AND ui.lrn_user = '0'
ORDER BY fullname ASC
LIMIT 0,25
If anyone can help, thanks
Turn it inside-out. That is, first use a 'derived' table to locate 25 users you want. Then gather the rest of the info.
What you have gathers all the info (including all the JOIN work) for all the users, then sorts and peels off 25.
It will be something like:
SELECT -- lots of stuff
FROM ( SELECT u.id,
CONCAT(u.lastname,' ',u.firstname) AS fullname
FROM fcf_users AS u
JOIN fcf_user_user_groups AS ug ON ...
JOIN fcf_users_userinfo AS ui ON ui.user_id = u.id
WHERE ug.group_id = '8'
AND ui.lrn_user = '0'
ORDER BY u.lastname, u.firstname -- now sargeable
LIMIT 25
) AS u25
JOIN .... -- whatever tables are needed to get the rest of the columns
ORDER BY u25.fullname -- yes, again, but now using the CONCAT
-- no limit here
Also:
u: INDEX(lastname, firstname, id)
user_user_group is a "many-t0=many mapping" table? If so, follow the indexing advice here: http://mysql.rjweb.org/doc.php/index_cookbook_mysql#many_to_many_mapping_table
Ditto for any other many:many tables.
Note how I put into the derived table only the tables needed to achieve the LIMIT.

Error with ORDER BY clause using UNION in MySQL

I have the following query in MySQL:
(SELECT ue.id, ue.userid, ue.status, ue.timestart, ue.timeend, e.courseid,
e.id AS enrolid, ra.roleid
FROM user_enrolments ue
JOIN enrol e ON e.id = ue.enrolid
JOIN course c ON c.id = e.courseid
JOIN user u ON u.id = ue.userid
JOIN context ct ON ct.instanceid = c.id
LEFT JOIN role_assignments ra ON ra.userid = u.id AND
ra.contextid = ct.id AND
ra.itemid = e.id
WHERE e.customint1 = 1 AND u.deleted = 0 AND
ct.contextlevel = 50 AND (ue.status = 0 OR ue.status = 1))
UNION
(SELECT de.enrolid AS id, de.userid, de.status, de.date_ini, de.date_fin,
de.courseid, de.enrolid, de.roleid
FROM deleted_enrols de
JOIN user u ON u.id = de.userid
WHERE userid = ANY (SELECT userid FROM local_users WHERE clientid = 1))
ORDER BY u.firstname, u.lastname, c.fullname LIMIT 0, 100
If I delete ORBER BY and LIMIT, this query works fine... but the ORDER BY clause gives an error:
Table 'u' from one of the SELECTs cannot be used in global ORDER clause
If I delete the parentheses of both SELECT querys, the error is different:
Table 'u' from one of the SELECTs cannot be used in field list
I have also tried with UNION ALL, but it does not work either.
Any suggestion or clue? Thanks in advance for your time...
The results of your UNION do not include any fields from table 'u', so those results cannot be sorted by table 'u' fields.
You could perhaps perform the UNION and then re-join the results to table 'u', and then use that to sort the results by table 'u' fields. A similar issue exists for sorting on
course.fullname, so that would need to be joined back in, too.
SELECT x.id, x.userid, x.status, x.timestart, x.timeend, x.courseid, x.enrolid, x.roleid
FROM ((SELECT ue.id, ue.userid, ue.status, ue.timestart, ue.timeend, e.courseid,
e.id AS enrolid, ra.roleid
FROM user_enrolments ue
JOIN enrol e ON e.id = ue.enrolid
JOIN course c ON c.id = e.courseid
JOIN user u ON u.id = ue.userid
JOIN context ct ON ct.instanceid = c.id
LEFT JOIN role_assignments ra ON ra.userid = u.id
AND ra.contextid = ct.id
AND ra.itemid = e.id
WHERE e.customint1 = 1 AND u.deleted = 0
AND ct.contextlevel = 50 AND (ue.status = 0 OR ue.status = 1))
UNION
(SELECT de.enrolid AS id, de.userid, de.status, de.date_ini, de.date_fin,
de.courseid, de.enrolid, de.roleid
FROM deleted_enrols de
JOIN user u ON u.id = de.userid
WHERE userid = ANY (SELECT userid FROM local_users WHERE clientid = 1))
) x
JOIN user z ON z.id = x.userid
JOIN course d ON d.id = x.courseid
ORDER BY z.firstname, z.lastname, d.fullname LIMIT 0, 100
Assuming you want to sort the whole lot, try parentheses round the whole query with the ORDER BY done afterwards:
select id, userid, status, timestart, timeend, courseid, enrolid, roleid from
((SELECT ue.id, ue.userid, ue.status, ue.timestart, ue.timeend, e.courseid,
e.id AS enrolid, ra.roleid, u.firstname, u.lastname, c.fullname
FROM user_enrolments ue
JOIN enrol e ON e.id = ue.enrolid
JOIN course c ON c.id = e.courseid
JOIN user u ON u.id = ue.userid
JOIN context ct ON ct.instanceid = c.id
LEFT JOIN role_assignments ra ON ra.userid = u.id AND
ra.contextid = ct.id AND
ra.itemid = e.id
WHERE e.customint1 = 1 AND u.deleted = 0 AND
ct.contextlevel = 50 AND (ue.status = 0 OR ue.status = 1))
UNION
(SELECT de.enrolid AS id, de.userid, de.status, de.date_ini, de.date_fin,
de.courseid, de.enrolid, de.roleid, u.firstname, u.lastname, ' ' as fullname
FROM deleted_enrols de
JOIN user u ON u.id = de.userid
WHERE userid = ANY (SELECT userid FROM local_users WHERE clientid = 1))) s1
ORDER BY firstname, lastname, fullname LIMIT 0, 100
(obviously fullname in the second SELECT statement would be populated however seems sensible)
You need to include the data to be ordered by in the selects of the unioned queries; an ORDER BY following a UNION is handled as if it were SELECT * FROM (unions) ORDER BY ... so anything not coming out of the union cannot be used for ordering.
Ironically, a query similar to that is the key to getting what you want though, with something like
SELECT x, y, z
FROM (
SELECT x, y, z, somethingIdontactuallywant
FROM blah
UNION
SELECT a, b, c, somethingIdontactuallywant
FROM blah2
) AS u
ORDER BY u.somethingIdontactuallywant
As mysql documentation on union says:
This kind of ORDER BY cannot use column references that include a
table name (that is, names in tbl_name.col_name format). Instead,
provide a column alias in the first SELECT statement and refer to the
alias in the ORDER BY. (Alternatively, refer to the column in the
ORDER BY using its column position. However, use of column positions
is deprecated.)
Also, if a column to be sorted is aliased, the ORDER BY clause must
refer to the alias, not the column name.
So, do not refer to any table names and use columns that are actually in the resultset of the union.

SQL query improvement (extra column in table)

I have this query in SQL that I KNOW it is horribly written. Could you guys help me write it in a decent, normal person manner?
Thanks.
select distinct R.*, X.LIKED
from Recipe R
left join (select distinct R.* , '1' as LIKED
from Recipe R, Likes L
where R.id = L.idRecipe
and L.email = 'dvader#deathstar.galacticempire') X
on R.id = X.id
looks like you need all from recipe with marks on liked by vader#deathstar.galacticempire
select R.*, likedR.LIKED
from Recipe R
left join (select distinct R.id , '1' as LIKED
from Recipe R
inner join Likes L on R.id = L.idRecipe
where
L.email = 'dvader#deathstar.galacticempire') likedR
on R.id = likedR.id
Thanks everyone for your help.
I was able to do what i wanted with this query:
select distinct R.*, X.LIKED, U.imgUrl
from User U, Recipe R
left join
(select distinct R.* , '1' as LIKED
from Recipe R, Likes L
where R.id = L.idRecipe and L.email = 'dvader#deathstar.ge') X
on R.id = X.id
where R.email = U.email
This will bring all the info i need in one table plus 1 extra column with either a 1 or a null if the entry of dvader is in another table, Using joins.

How can I optimise an extremely slow MySQL query that uses COUNT DISTINCT

I have a very slow MySQL query that I would like to optimise.
The query is taking 66.2070 seconds to return 5 results from tables containing around 200 rows.
The database tables store users, experiments (A/B tests), goals (page URLs), visits (page visits) and conversions (clicks a goal's URL). The visit and conversion tables both have a combination column that records if version A or B of a page was visited or a conversion came from version A or B. Combinations are stored in the db as 1 or 2.
I'm trying to get a list of a user's experiments with the number of visits and conversions for each combination.
For some relationships I'm using composite primary keys, which does make the joins more complicated. I doubt it but could this be the cause of the problem?
How can I rewrite this query to make it run in a reasonable time, at least less than a second?
Here's my database schema:
and her's my query:
SELECT e.id AS id,
e.name AS name,
e.status AS status,
e.created AS created,
Count(DISTINCT v1.id) AS visits1,
Count(DISTINCT v2.id) AS visits2,
Count(DISTINCT c1.id) AS conversions1,
Count(DISTINCT c2.id) AS conversions2
FROM experiment e
LEFT JOIN visit v1
ON ( v1.experiment_id = e.id
AND v1.user_id = e.user_id
AND v1.combination = 1 )
LEFT JOIN visit v2
ON ( v2.experiment_id = e.id
AND v2.user_id = e.user_id
AND v2.combination = 2 )
LEFT JOIN goal g
ON ( g.experiment_id = e.id
AND g.user_id = e.user_id
AND g.principal = 1 )
LEFT JOIN conversion c1
ON ( c1.experiment_id = e.id
AND c1.user_id = e.user_id
AND c1.goal_id = g.id
AND c1.combination = 1 )
LEFT JOIN conversion c2
ON ( c2.experiment_id = e.id
AND c2.user_id = e.user_id
AND c2.goal_id = g.id
AND c2.combination = 2 )
WHERE e.user_id = 25
GROUP BY e.id
ORDER BY e.created DESC
LIMIT 5
The resulting table should look something like this:
You should do the aggregations before doing the joins, to avoid getting large intermediate results. I think the logic is
SELECT e.id, e.name, e.status, e.created,
v.visits1, v.visits2, g.conversions1, g.conversions2
FROM experiment e LEFT JOIN
(SELECT experiment_id, user_id,
SUM(combination = 1) as visits1,
SUM(combination = 2) as visits2
FROM visits
WHERE combination IN (1, 2)
GROUP BY experiment_id, user_id
) v
ON v.experiment_id = e.id AND
v.user_id = e.user_id LEFT JOIN
(SELECT g.experiment_id, g.user_id,
SUM(c.combination = 1) as conversions1,
SUM(c.combination = 2) as conversions2
FROM goal g LEFT JOIN
conversion c
ON c.experiment_id = g.experiment_id AND
c.user_id = g.user_id AND
c.goal_id = g.id
WHERE g.principal = 1
GROUP BY g.experiment_id, g.user_id
) g
ON g.experiment_id = e.id AND
g.user_id = e.user_id LEFT JOIN
WHERE e.user_id = 25
ORDER BY e.created DESC
LIMIT 5 ;
There are further optimizations for this. For instance, an index on experiment(user_id, created, id).
For your question about the drawback of using composite keys I found this:
Drawback of composite keys
I can't currently test ur database but use the EXPLAIN syntax in mysql to find out what is wrong with the perfomance of ur query:
MySQL docs about EXPLAIN and optimizing ur query with EXPLAIN