This query takes about 9 seconds and returns 2 records.
SELECT
s.description, s.improvement,
s.first_name, s.last_name, s.finding, s.action, s.share,
s.learned, s.timestamp, d.title as department_title,
group_concat(DISTINCT g.title SEPARATOR " | ") as strategic_goals,
group_concat(DISTINCT m.statement SEPARATOR " | ") as mission_references,
group_concat(DISTINCT meas.statement SEPARATOR " | ") as measure_statement,
group_concat(DISTINCT o.statement SEPARATOR " | ") as outcome_statement,
group_concat(DISTINCT i.title SEPARATOR " | ") as ilo_title,
group_concat(DISTINCT cv.title SEPARATOR " | ") as core_value_title,
y1.year as current_year_title, y2.year as previous_year_title,
u.file_name as file_name
FROM summary s
LEFT JOIN year y1 ON s.current_year_id = y1.id
INNER JOIN year y2 ON s.previous_year_id = y2.id
INNER JOIN strategic_goal_entries sge ON s.id = sge.summary_id
INNER JOIN goal g ON sge.goal_id = g.id
INNER JOIN outcome o ON s.id = o.summary_id
LEFT JOIN measure meas ON o.id = meas.outcome_id
INNER JOIN department d ON s.department_id = d.id
LEFT JOIN uploads u ON s.id = u.summary_id
INNER JOIN mission_entries me ON s.id = me.summary_id
LEFT JOIN mission m ON me.mission_id = m.id
LEFT JOIN ilo_entries ie ON s.id = ie.summary_id
LEFT JOIN ilo i ON ie.ilo_id = i.id
INNER JOIN core_value_entries cve ON s.id = cve.summary_id
INNER JOIN core_value cv ON cve.core_value_id = cv.id
INNER JOIN executive_of_department eod ON s.department_id = eod.department_id
WHERE eod.executive_id = 3
GROUP BY s.id
I added the rest of the primary keys. Query went from 9 seconds to 2 seconds.
Then I set fields that refer to other table's primary keys, as index fields. The query went from 2 seconds to 20 seconds, did I assign too many indexes?
What are some ways I could speed this up?
1. After setting indexes, etc. you can speed up a little by limiting the JOIN conditions. For example, instead of
LEFT JOIN uploads u ON s.id = u.summary_i
do
LEFT JOIN uploads u ON (s.id = u.summary_i AND s.id = 3)
The WHERE clause is evaluated after all tables have been joined. Preciese your JOIN condition when you can.
2. Did you try constructing a view from your select and perform a SELECT * FROM myView WHERE myView.id = 3 ?
UPDATE : read this blog comment.
Related
We have a home grown document management system and our system is running very slow, particularly on the search. It worked fine at first, but it has gotten progressively slower over time. Its now taking anywhere from 30 to 150 seconds to return results depending upon criteria. This is our search query. We’ve been staring at this thing left and right and can’t see any place to tune this more. All of the joined fields are indexed on their respective tables.
SELECT DISTINCT f.*, ts.*, fo.*, ft.*, p.*, u.*, c.*, co.*, ct.*, fs.*, fd.*, r.*, rt.*, si.*, s.* FROM (
SELECT DISTINCT f.* FROM files f
JOIN folders fo ON(fo.id = f.belongs_to_folder_id)
JOIN projects p ON(p.id = f.belongs_to_project_id)
LEFT OUTER JOIN file_statuses fs ON(fs.id = f.file_status_id)
LEFT OUTER JOIN submittal_items_files sif ON(sif.file_id = f.id)
LEFT OUTER JOIN submittal_items si ON(si.id = sif.submittal_item_id)
LEFT OUTER JOIN submittals s ON(s.id = si.belongs_to_submittal_id)
LEFT OUTER JOIN record_types rt ON(rt.id = f.record_type_id)
LEFT OUTER JOIN companies co ON(co.id = f.company_id)
LEFT JOIN folders_actions_groups ag ON (
f.belongs_to_folder_id = ag.folder_id AND
ag.action_id = 10010
)
LEFT JOIN files_actions_groups fg ON (fg.file_id = f.id)
JOIN users_groups ug ON ((ug.group_id = ag.group_id OR ug.group_id = fg.group_id) AND ug.user_id = 411)
WHERE (
(f.file_generated_name LIKE CONCAT('%', 'the', '%')) OR
(f.record_id LIKE CONCAT('%', 'the', '%')) OR
(f.record_title LIKE CONCAT('%', 'the', '%')) OR
(f.additional_info LIKE CONCAT('%', 'the', '%')) OR
(si.item_number LIKE CONCAT('%', 'the', '%')) OR
(s.element_number LIKE CONCAT('%', 'the', '%'))
) AND f.path LIKE CONCAT('Some Text', '%') AND
f.file_status_id = 3 AND
f.file_revision = 1 AND
f.discipline_id = 1 AND
f.record_type_id = 2 AND
f.triage_status_id = 2 AND
f.deleted = 0
ORDER BY f.created DESC, f.id DESC
LIMIT 100
) AS f
LEFT OUTER JOIN users u ON(f.created_by_user_id = u.id)
LEFT OUTER JOIN contacts c ON(c.user_id = u.id)
LEFT OUTER JOIN companies co ON(co.id = f.company_id)
LEFT OUTER JOIN company_types ct ON(ct.id = co.company_type_id)
JOIN triage_statuses ts ON(f.triage_status_id = ts.id)
JOIN folders fo ON(fo.id = f.belongs_to_folder_id)
JOIN folder_types ft ON(ft.id = fo.folder_type_id)
JOIN projects p ON(p.id = f.belongs_to_project_id)
LEFT OUTER JOIN file_statuses fs ON(fs.id = f.file_status_id)
LEFT OUTER JOIN file_disciplines fd ON(fd.id = f.discipline_id)
LEFT OUTER JOIN revisions r ON(r.id = f.file_revision)
LEFT OUTER JOIN record_types rt ON(rt.id = f.record_type_id)
LEFT OUTER JOIN submittal_items_files sif ON(sif.file_id = f.id)
LEFT OUTER JOIN submittal_items si ON(si.id = sif.submittal_item_id)
LEFT OUTER JOIN submittals s ON(s.id = si.belongs_to_submittal_id)
LEFT OUTER JOIN files_actions_groups ffg ON(ffg.file_id = f.id)
LEFT OUTER JOIN groups g ON(g.id = ffg.group_id)
ORDER BY f.created DESC, f.id DESC
This might be an obvious answer, but have you indexed your database? If you're new to indexing, here's a pretty good rule: just put a unique index on all the columns named "id", such as folders.id or projects.id, then put a standard index on all the columns that reference a foreign id, such as folder.belongs_to_folder_id or folder.record_type_id
Another thing I would change is to try and select only the columns you will actually use rather than your huge list of f.*, ts.*, fo.*, ft.*, p.*, u.*, c.*, co.*, ct.*, fs.*, etc...
You also have TONS of joins, which are very expensive in terms of processing time. Do you really need all those joined tables?
I have a mysql query as stated below, it returns exactly the results I want for one row, but doesn't return any other rows where I expect there to be 8 in my test data (there are 8 unique test ids). I was inspired by this answer but obviously messed up my implementation, does anyone see where I'm going wrong?
SELECT
c.first_name,
c.last_name,
n.test_name,
e.doc_name,
e.email,
e.lab_id,
a.test_id,
a.date_req,
a.date_approved,
a.accepts_terms,
a.res_value,
a.reason,
a.test_type,
a.date_collected,
a.date_received,
k.kind_name,
sum(case when metabolite_name = "Creatinine" then t.res_val end) as Creatinine,
sum(case when metabolite_name = "Glucose" then t.res_val end) as Glucose,
sum(case when metabolite_name = "pH" then t.res_val end) as pH
FROM test_requisitions AS a
INNER JOIN personal_info AS c ON (a.user_id = c.user_id)
INNER JOIN test_types AS d ON (a.test_type = d.test_type)
INNER JOIN kinds AS k ON (k.id = d.kind_id)
INNER JOIN test_names AS n ON (d.name_id = n.id)
INNER JOIN docs AS e ON (a.doc_id = e.id)
INNER JOIN test_metabolites AS t ON (t.test_id = a.test_id)
RIGHT JOIN metabolites AS m ON (m.id = t.metabolite_id)
GROUP BY a.test_id
ORDER BY (a.date_approved IS NOT NULL),(a.res_value IS NOT NULL), a.date_req, c.last_name ASC;
Most of your joins are inner joins. The last is a right outer join. As written, the query keeps all the metabolites, but not necessarily all the tests.
I would suggest that you change them all to left outer joins, because you want to keep all the rows in the first table:
FROM test_requisitions AS a
LEFT JOIN personal_info AS c ON (a.user_id = c.user_id)
LEFT JOIN test_types AS d ON (a.test_type = d.test_type)
LEFT JOIN kinds AS k ON (k.id = d.kind_id)
LEFT JOIN test_names AS n ON (d.name_id = n.id)
LEFT JOIN docs AS e ON (a.doc_id = e.id)
LEFT JOIN test_metabolites AS t ON (t.test_id = a.test_id)
LEFT JOIN metabolites AS m ON (m.id = t.metabolite_id)
I would also suggest that your aliases be related to the table, so tr for test_requisition, pi for personal_info, and so on.
I'm having problems adding a conditional field to my query.
The query returns the jobs that are published on my website (many joines, nothing very speical). The thing is I need to add another field in order to know if the current user already applied for this job (in order to remove the APPLY button)...
I have a jobApplication table, which has the user_id and job_id tables. I thought about using another join but that can't work. I tried to create it with an IF and SELECT but didn't quite managed to make it work :/...
this is my query:
SELECT *, j.job_id as jid, c.name as city_name
FROM jobs j
JOIN areas a ON a.area_id = j.job_area
JOIN positions p ON p.position_id = j.job_position
JOIN fields f ON f.id = j.job_field
JOIN cities c ON j.job_city = c.id
JOIN jobTypes jt ON j.job_type = jt.job_id
JOIN companies comp ON j.job_company = comp.company_id
LEFT JOIN jobApplications ja ON <missing>
WHERE j.job_field = '$field' AND j.job_position = '$position'
AND j.job_area = '$area'
ORDER BY j.creationDate DESC"
any attempt I made to add a condition select broke the query, any chance someone can give me a better direction?
Thanks!!
It's hard to say for sure without knowing more about your schema, but you are on the right track with the left join. You will need to left join onto the applicants table using the applicant Id for the current user and to the job ID from the jobs table.
SELECT
*,
j.job_id as jid,
c.name as city_name
FROM jobs j
JOIN areas a ON a.area_id = j.job_area
JOIN positions p ON p.position_id = j.job_position
JOIN fields f ON f.id = j.job_field
JOIN cities c ON j.job_city = c.id
JOIN jobTypes jt ON j.job_type = jt.job_id
JOIN companies comp ON j.job_company = comp.company_id
LEFT JOIN jobApplications ja ON ja.applicantId = '$applicantId' and ja.jobId = j.jobId
WHERE
j.job_field = '$field'
AND j.job_position = '$position'
AND j.job_area = '$area'
ORDER BY j.creationDate DESC"
select
s.id, s.description, s.improvement, s.previous_year_id,
s.current_year_id, s.first_name, s.last_name, s.username,
s.finding, s.action, s.share, s.learned, s.timestamp,
d.title as department_title,
group_concat(g.title SEPARATOR \' | \') as strategic_goals,
y1.year as current_year_title, y2.year as previous_year_title,
u.summary_id, u.file_name as file_name
from
summary s, year y1, year y2, strategic_goal_entries sge,
goal g, department d, uploads u
where
s.id = sge.summary_id
and
s.current_year_id = y1.id
and
s.previous_year_id = y2.id
and
sge.goal_id = g.id
and
s.id = u.summary_id
and
s.department_id = d.id
and
s.department_id = '4'
group by
s.id
This only returns records from the summary table that has a relating record in the uploads table (s.id = uploads.summary_id) that contain a value within the uploads.summary_id field
I want to return all records, whether or not it has a file associated with it.
Any help is appreciated.
Suggest refactoring this SQL query to use ANSI joins. To achive your goal, you'd want a LEFT JOIN instead:
SELECT /*your columns*/
from summary s
INNER JOIN year y1 ON s.current_year_id = y1.id
INNER JOIN year y2 ON s.previous_year_id = y2.id
INNER JOIN strategic_goal_entries sge ON s.id = sge.summary_id
INNER JOIN goal g ON sge.goal_id = g.id
INNER JOIN department d ON s.department_id = d.id
LEFT JOIN uploads u ON s.id = u.summary_id
WHERE s.department_id = '4'
group by s.id
Just wondering what's a better way to write this query. Cheers.
SELECT r.user_id AS ID, m.prenom, m.nom
FROM `0_rank` AS l
LEFT JOIN `0_right` AS r ON r.rank_id = l.id
LEFT JOIN `0_user` AS m ON r.user_id = m.id
WHERE r.section_id = $section_id
AND l.rank = '$rank_name' AND depart_id IN
(SELECT depart_id FROM 0_depart WHERE user_id = $user_id AND section_id = $section_id)
GROUP BY r.user_id
Here are the table structures:
0_rank: id | section_id | rank_name |
other_stuffs
0_user: id | prenom | nom | other_stuffs
0_right: id | section_id | user_id |
rank_id | other_stuffs
0_depart: id | section_id | user_id | depart_id
| other_stuffs
The idea is to use the same in a function like:
public function usergroup($section_id,$rank_name,$user_id) {
// mysql query goes here to get a list of appropriate users
}
Update: I think I have not been able to express myself clearly earlier. Here is the most recent query that seems to be working.
SELECT m.id, m.prenom, m.nom,
CAST( GROUP_CONCAT( DISTINCT d.depart ) AS char ) AS deps,
CAST( GROUP_CONCAT( DISTINCT x.depart ) AS char ) AS depx
FROM `0_rank` AS l
LEFT JOIN `0_right` AS r ON r.rank_id = l.id
LEFT JOIN `0_member` AS m ON r.user_id = m.id
LEFT JOIN `0_depart` AS d ON m.id = d.user_id
LEFT JOIN `0_depart` AS x ON x.user_id = $user_id
WHERE r.section = $section_id
AND l.rank = '$rank_name'
GROUP BY r.user_id ORDER BY prenom, nom
Now I want to get only those result, where all entries of deps are present in entries in depx.
In other term, every user is associated with some departs. $user_id is also an user is associated with some departs.
I want to get those users whose departs are common to the departs of $user_id.
Cheers.
Update
I'm not sure without being able to see the data but I believe this query will give you the results you want the fastest.
SELECT m.id, m.prenom, m.nom,
CAST( GROUP_CONCAT( DISTINCT d.depart ) AS char ) AS deps,
FROM `0_rank` AS l
LEFT JOIN `0_right` AS r ON r.rank_id = l.id and r.user_id = $user_id
LEFT JOIN `0_member` AS m ON r.user_id = m.id
LEFT JOIN `0_depart` AS d ON m.id = d.user_id
WHERE r.section = $section_id
AND l.rank = '$rank_name'
GROUP BY r.user_id ORDER BY prenom, nom
Let me know if this works.
Try this:
(By converting the functionality of the IN (SELECT...) to an inner join, you get exactly the same results but it might be the optimizer will make better choices.)
SELECT r.user_id AS ID, m.prenom, m.nom
FROM `0_rank` AS l
LEFT JOIN `0_right` AS r ON r.rank_id = l.id and r.section_id = 2
LEFT JOIN `0_user` AS m ON r.user_id = m.id
INNER JOIN `0_depart` AS x ON l.section_id = x.section_id and x.user_id = $user_id AND x.section_id = $section_id
WHERE l.rank = 'mod'
GROUP BY r.user_id
I also moved the constraints on 0_right to the join statement because I think that is clearer -- presumably this change won't matter to the optimizer.
I know nothing about your DB structure but your subselect looks like it can be replaced with a simple INNER JOIN against whatever table has the depart column. MySQL is well known for its poor subquery optimization.
Without knowing the structures or indexes, I would first add "STRAIGHT_JOIN" if the critical criteria is in-fact from the 0-rank table. Then, ensure 0_rank has an index on "rank". Next, ensure the 0_right has an index on rank_id at a minimum, but rank_id, section to take advantage of BOTH your criteria. Index on 0_member on id.
Additionally, do you mean left-join (ie: record only required in the 0_rank or 0_member) on the respective 0_right and 0_member tables instead of a normal join (where BOTH tables must match on their IDs).
Finally, ensure index on the depart table on user_id.
SELECT STRAIGHT_JOIN
r.user_id AS ID,
m.prenom,
m.nom
FROM
0_rank AS l
LEFT JOIN `0_right` AS r
ON l.id = r.rank_id
AND r.section = 2
LEFT JOIN `0_member` AS m
ON r.user_id = m.id
WHERE
l.rank = 'mod'
AND depart IN (SELECT depart
FROM 0_depart
WHERE user_id = 2
AND user_sec = 2)
GROUP BY
r.user_id
---- revised post from feedback.
From the parameters you are listing, you are always including the User ID... If so, I would completely restructure it to get whatever info is for that user. Each user should apparently can be associated to multiple departments and may or may NOT match the given rank / department / section you are looking for... I would START the query with the ONE USER because THAT will guarantee a single entry, THEN tune-down to the other elements...
select STRAIGHT_JOIN
u.id,
u.prenom,
u.nom,
u.other_stuffs,
rank.rank_name
from
0_user u
left join 0_right r
on u.id = r.user_id
AND r.section_id = $section_id
join 0_rank rank
on r.rank_id = rank.id
AND rank.rank_name = '$rank_name'
left join 0_dept dept
on u.id = dept.user_id
where
u.id = $user_id
Additionally, I have concern about your table relationships and don't see a legit join to the department table...
0_user
0_right by User_ID
0_rank by right.rank_id
0_dept has section which could join to rank or right, but nothing to user_id directly
Run explain on the query - it will help you find where the caveats are:
EXPLAIN SELECT r.user_id AS ID, m.prenom, m.nom
FROM 0_rank AS l
LEFT JOIN `0_right` AS r ON r.rank_id = l.id
LEFT JOIN `0_member` AS m ON r.user_id = m.id
WHERE r.section = 2
AND l.rank = 'mod' AND depart IN
(SELECT depart FROM 0_depart WHERE user_id = 2 AND user_sec = 2)
GROUP BY r.user_id\G