I am stuck with this query below. It takes forever to complete the query.
I have been using describe to help me find best way to pull this data. But this is what I can come up with so far. But this take forever to pull just 1000 records . I would like to get some recommendations / suggestions on how to approve this issue ? I'm stuck.
I have read about "EXISTS" that I can use to improve subquery , but I am not quite understand how it would work in this situation.
I have already used primary keys as much as I could ( id , contact_id , flag_if ) => those ending with _id are all primary keys.
Thanks.
SELECT ac.id,
acc.category,
ac.value,
fg.*
FROM contact AS ac
JOIN application AS app ON app.id = ac.id
JOIN category AS acc USING (contact_id)
LEFT JOIN (SELECT af.name_short
,att.value
,att.flag_id
FROM attribute AS att
LEFT JOIN attribute_flag AS af ON af.flag_id = att.flag_id
LEFT JOIN application app on
( (att.level = 'customer' AND att.related_id = app.customer_id)
OR (att.level = 'application' and att.related_id = app.id)
)
WHERE app.id IN (001,002,003)
) fg on ac.value = fg.value
WHERE ac.id IN (001,002,003)
== UPDATE ==
here is the result from explain query'
Related
My mysql query is too slow and i don't know how to optimize it. My webapp cant load this query because take too much time to run and the webserver have a limit time to get the result.
SELECT rc.trial_id,
rc.created,
rc.date_registration,
rc.agemin_value,
rc.agemin_unit,
rc.agemax_value,
rc.agemax_unit,
rc.exclusion_criteria,
rc.study_design,
rc.expanded_access_program,
rc.number_of_arms,
rc.enrollment_start_actual,
rc.target_sample_size,
(select name from repository_institution where id = rc.primary_sponsor_id) as
primary_sponsor,
(select label from vocabulary_studytype where id = rc.study_type_id) as study_type,
(select label from vocabulary_interventionassigment where id =
rc.intervention_assignment_id) as intervention_assignment,
(select label from vocabulary_studypurpose where id = rc.purpose_id) as study_purpose,
(select label from vocabulary_studymasking where id = rc.masking_id) as study_mask,
(select label from vocabulary_studyallocation where id = rc.allocation_id) as
study_allocation,
(select label from vocabulary_studyphase where id = rc.phase_id) as phase,
(select label from vocabulary_recruitmentstatus where id = rc.recruitment_status_id) as
recruitment_status,
GROUP_CONCAT(vi.label)
FROM
repository_clinicaltrial rc
inner JOIN repository_clinicaltrial_i_code rcic ON rcic.clinicaltrial_id = rc.id JOIN
vocabulary_interventioncode vi ON vi.id = rcic.interventioncode_id
GROUP BY rc.id;
Using inner join instead join could be a solution?
Changing to JOINs vs continuous selects per every row will definitely improve. Also, since you are using MySQL, using the keyword "STRAIGHT_JOIN" tells MySQL to do the query in the order I provided. Since your "rc" table is the primary and all the others are lookups, this will make MySQL use it in that context rather than hoping some other lookup table be the basis of the rest of the joins.
SELECT STRAIGHT_JOIN
rc.trial_id,
rc.created,
rc.date_registration,
rc.agemin_value,
rc.agemin_unit,
rc.agemax_value,
rc.agemax_unit,
rc.exclusion_criteria,
rc.study_design,
rc.expanded_access_program,
rc.number_of_arms,
rc.enrollment_start_actual,
rc.target_sample_size,
ri.name primary_sponsor,
st.label study_type,
via.label intervention_assignment,
vsp.label study_purpose,
vsm.label study_mask,
vsa.label study_allocation,
vsph.label phase,
vrs.label recruitment_status,
GROUP_CONCAT(vi.label)
FROM
repository_clinicaltrial rc
JOIN repository_clinicaltrial_i_code rcic
ON rc.id = rcic.clinicaltrial_id
JOIN vocabulary_interventioncode vi
ON rcic.interventioncode_id = vi.id
JOIN repository_institution ri
on rc.primary_sponsor_id = ri.id
JOIN vocabulary_studytype st
on rc.study_type_id = st.id
JOIN vocabulary_interventionassigment via
on rc.intervention_assignment_id = via.id
JOIN vocabulary_studypurpose vsp
ON rc.purpose_id = vsp.id
JOIN vocabulary_studymasking vsm
ON rc.masking_id = vsm.id
JOIN vocabulary_studyallocation vsa
ON rc.allocation_id = vsa.id
JOIN vocabulary_studyphase vsph
ON rc.phase_id = vsph.id
JOIN vocabulary_recruitmentstatus vrs
ON rc.recruitment_status_id = vrs.id
GROUP BY
rc.id;
One final note. You are using a GROUP BY and applying to the GROUP_CONCAT() which is ok. However, proper group by says you need to group by all non-aggregate columns, which in this case is every other column in the list. You may know this, and the fact the lookups will be the same based on the "rc" associated columns, but its not good practice to do so.
Your joins and subqueries are probably not the problem. Assuming you have correct indexes on the tables, then these are fast. "Correct indexes" means that the id column is the primary key -- a very reasonable assumption.
My guess is that the GROUP BY is the performance issue. So, I would suggest structuring the query with no `GROUP BY:
select . . .
(select group_concat(vi.label)
from repository_clinicaltrial_i_code rcic
vocabulary_interventioncode vi
on vi.id = rcic.interventioncode_id
where rcic.clinicaltrial_id = rc.id
)
from repository_clinicaltrial rc ;
For this, you want indexes on:
repository_clinicaltrial_i_code(clinicaltrial_id, interventioncode_id)
vocabulary_interventioncode(id, label)
I am learning SQL and when doing some exercises this question showed up to me.
In matter of Multi-table consults I see sometimes that I can use Where clause instead of inner join, but Í try to use join because is the topic im trying to learn now. In this example I wrote this:
select p.nombre from pokemon p join pokemon_forma_evolucion pfe
on p.numero_pokedex = pfe.numero_pokedex
join forma_evolucion fo
on pfe.id_forma_evolucion = fo.id_forma_evolucion
join tipo_evolucion t
on t.tipo_evolucion = fo.tipo_evolucion
where lower(t.tipo_evolucion) = 'intercambio';
but it shows nothing. ( 0 results )
in the Exercises the correct answer by the blogger is this one and works fine showing 4 results :
select p.nombre
from pokemon p, pokemon_forma_evolucion pfe,
forma_evolucion fe, tipo_evolucion te
where p.numero_pokedex = pfe.numero_pokedex
and pfe.id_forma_evolucion = fe.id_forma_evolucion
and fe.tipo_evolucion = te.id_tipo_evolucion
and lower(te.tipo_evolucion) = 'intercambio';
I would like to know why mine is not working and how should I use the JOIN command properly.
Link of the Exercises blog with DER and access to entire "pokemon" SQL database: https://www.discoduroderoer.es/ejercicios-propuestos-y-resueltos-consultas-sql-bd-pokemon/
Thanks for the help. I am new on posting in this community but it is not the first time I used and enjoyed it.
Thanks for all
the where and join should word fine. the problem is that you are joining the table "tipo_evolucion" ON the wrong field
fild Tipo_Evolucion in table FORMA-EVOLUCION is numeric (this is the foreing key). Fild Tipo_Evolucion in table TIPO_EVOLUCION is varchar2 (this is not the primary key)
the primary key in TIPO_EVOLUCION IS id_tipo_evolucion
So your inner join should be
JOIN tipo_evolucion t ON t.id_tipo_evolucion = fo.tipo_evolucion
the query therefore is:
SELECT p.nombre
FROM pokemon p
JOIN pokemon_forma_evolucion pfe ON p.numero_pokedex = pfe.numero_pokedex
JOIN forma_evolucion fo ON pfe.id_forma_evolucion = fo.id_forma_evolucion
JOIN tipo_evolucion t ON t.id_tipo_evolucion = fo.tipo_evolucion
WHERE LOWER(t.tipo_evolucion) = 'intercambio';
My MySQL query code likes as shown below, and there are about several thousands of records in the table, by now below SQL executes about 5 minutes and more. I am looking for ways to optimize it so that it takes less time to execute. Thank you!
SELECT `m`.`id`,
`m`.`id`,
`tr`.`name`,
`m`.`m_date`,
`t1`.`t_name` AS home,
`t2`.`t_name` AS away,
`m`.`score1`,
`m`.`score2`,
`cw1`.`tid` AS tid1,
`cw2`.`tid` AS tid2,
`o1`.`odds` AS odds1,
`o2`.`odds` AS odds2,
`m`.`m_time`
FROM `jos_bl_match` AS `m`
LEFT JOIN `jos_bl_matchday` AS `md` ON (`md`.`id` = `m`.`m_id`)
LEFT JOIN `jos_bl_seasons` AS `s` ON (`s`.`s_id` = `md`.`s_id`)
LEFT JOIN `jos_bl_tournament` AS `tr` ON (`tr`.`id` = `s`.`t_id`)
LEFT JOIN `jos_bl_teams` AS `t1` ON (`m`.`team1_id` = `t1`.`id`)
LEFT JOIN `jos_bl_teams` AS `t2` ON (`m`.`team2_id` = `t2`.`id`)
LEFT JOIN `jos_vuvuzelaodds_odds` AS `o1` ON (`o1`.`m_id` = `m`.`id`)
AND `o1`.`market_id` = 1
AND `o1`.`bookmaker_id` = 1
LEFT JOIN `jos_vuvuzelaodds_odds` AS `o2` ON (`o2`.`m_id` = `m`.`id`)
AND `o2`.`market_id` = 1
AND `o2`.`bookmaker_id` = 2
LEFT JOIN `jos_cwtags_tags` AS `cw1` ON (`cw1`.`item_id` = `o1`.`m_id`)
LEFT JOIN `jos_cwtags_tags` AS `cw2` ON (`cw2`.`item_id` = `o2`.`m_id`)
WHERE `m`.`published` = 1
AND `s`.`published` = '1'
AND `tr`.`published` = '1'
AND `s`.`s_id` = 869
AND `m`.`m_played` = '1'
AND `m`.`m_date` > 2013-01-01
AND `o1`.`odds` != ''
AND `o2`.`odds` != ''
AND `cw1`.`cat_id` = 19
AND `cw2`.`cat_id` = 21
ORDER BY `m`.`m_date`,
`md`.`id`,
`s`.`s_id`,
`tr`.`id` DESC LIMIT 0, 15
"Normalize, but don't over-normalize."
Some composite indexes you may be missing...
jos_bl_match: INDEX(m_played, published, m_date)
The columns need to be in that order. That will more quickly start the filtering.
The following should speed up their JOINs:
jos_vuvuzelaodds_odds: INDEX(market_id, bookmaker_id, m_id)
jos_cwtags_tags: INDEX(cat_id, item_id)
It seems like those last two indexes could (should) be the PRIMARY KEY. Are they?
Some (perhaps all) of the LEFT JOINs may as well be INNER JOINs; did you consider that?
Please provide EXPLAIN SELECT.
Without having access to the database it is a little hard to tell. This seems to be a lot of data to only pull 15 records. Are you sure you need to pull the data this way?
Probably the best route:
Optimize your database as below.
Make a flat database view of all the games that has the fields that you need. A static table will be much faster but you would need to set up updates with triggers which is beyond the scope of this answer but the process is similar
Write queries against the
view.
You are not selecting any fields from the seasons table in your select. I used the field from the jos_bl_matchday table.
This should get you started. You can also use conditionals in your select statements
(IF value = 1, table.field, null) as yadda
instead of joining one table over and over again but you would have to experiment.
CREATE VIEW allTheGames AS SELECT
`m`.`id` as id,
`md`.`s_id` as seasonId,
`tr`.`name` as name,
`m`.`m_date` as m_date,
`t1`.`t_name` AS home,
`t2`.`t_name` AS away,
`m`.`score1` as score1,
`m`.`score2` as score2,
`cw1`.`tid` AS tid1,
`cw2`.`tid` AS tid2,
`o1`.`odds` AS odds1,
`o2`.`odds` AS odds2,
`m`.`m_time` as m_time
FROM `jos_bl_match` AS `m`
LEFT JOIN `jos_bl_matchday` AS `md` ON (`md`.`id` = `m`.`m_id`)
LEFT JOIN `jos_bl_tournament` AS `tr` ON (`tr`.`id` = `s`.`t_id`)
LEFT JOIN `jos_bl_teams` AS `t1` ON (`m`.`team1_id` = `t1`.`id`)
LEFT JOIN `jos_bl_teams` AS `t2` ON (`m`.`team2_id` = `t2`.`id`)
LEFT JOIN `jos_vuvuzelaodds_odds` AS `o1` ON (`o1`.`m_id` = `m`.`id`) AND `o1`.`market_id` = 1 AND `o1`.`bookmaker_id` = 1
LEFT JOIN `jos_vuvuzelaodds_odds` AS `o2` ON (`o2`.`m_id` = `m`.`id`) AND `o2`.`market_id` = 1 AND `o2`.`bookmaker_id` = 2
LEFT JOIN `jos_cwtags_tags` AS `cw1` ON (`cw1`.`item_id` = `o1`.`m_id`)
LEFT JOIN `jos_cwtags_tags` AS `cw2` ON (`cw2`.`item_id` = `o2`.`m_id`)
WHERE `m`.`published` = 1 AND `s`.`published` = '1' AND `tr`.`published` = '1'
AND `m`.`m_played` = '1'
AND `o1`.`odds1` != '' AND `o2`.`odds2` != ''
Then query it with:
select * from allTheGames
WHERE season_id = 869 AND m_date > 2013-01-01 AND tid1 = 19 AND tid2 = 21
Steps to optimize:
Figure out exactly which data you want out of this query and why:
Is this a custom report? A web page? do you need to have all of this data at once or would it make more sense to have the user drill down?
How often is the query run? Once a minute? Once a day?
How many records are in each table? Your view should reflect this "game object"
Your database:
Run "explain" against this query http://dev.mysql.com/doc/refman/5.7/en/explain.html.
It will show all of the work the database is doing, the query execution plan, and how many records it is looking in to do it. This usually happens quickly: it does not actually execute the query.
See if you can put the database on SSD drives or even RAM
Your table structure:
Make sure that you have indexes on all of the fields that you are searching on. There are many ways to optimize this.
Use "explain" to be sure MySQL is able to use indexes.
If there are really only 2 markets in the jos_vuvuzelaodds_odds table consider making 2 fields.
Good luck!
I was just wondering if I could get some pointers as to where I may be going wrong.
I have been using mysql statements and I am in the process of switching to PDO statements to use with MySQL.
I have been able to get my head around SELECT statements, but I am having a bit of trouble trying to get the insert statement to work.
I have been Googling and tried a couple of different ways to get this to work but to no avail.
This is what I have so far:
$sqlu = $conn->prepare("UPDATE ".PERSON." p
JOIN contact c ON c.personID = p.adbkid
JOIN address a ON a.personID = p.adbkid
JOIN misc m ON m.personID = p.adbkid
JOIN variables v ON v.personID = p.adbkid
SET lastname = :ln
WHERE p.pid = :id");
$sqlu->bindParam(':ln', $ln, PDO::PARAM_STR);
$sqlu->bindParam(':id', $id, PDO::PARAM_STR);
$sqlu->execute();
I have also tried it without using bindParam and using as follows:
$sqlu->execute(array('ln' => $ln, 'id' => $id));
I have also used a '?' instead of ':' and then bound the parameter or used it in the array.
When I hit the update button, I have echoed the query so I can see what is being passed through and this is what I get:
PDOStatement Object ( [queryString] => UPDATE person p JOIN contact c ON c.personID = p.adbkid JOIN address a ON a.personID = p.adbkid JOIN misc m ON m.personID = p.adbkid JOIN variables v ON v.personID = p.adbkid SET lastname = :ln WHERE p.pid = :id ) 1
I just can't see where I am going wrong. Like I say, I have Googled this and come across some answers on here too and I seem to be stuck as to where to go next.
This is a personal project I am working on and I am not looking for someone to figure this out for me, I am just looking for some pointers so I can try to fix and learn myself.
Thanks in advance.
One possibility is that the records do not have matching records in all the tables. You could try using left join. But why are you doing joins at all? Does this work?
UPDATE ".PERSON." p
SET lastname = :ln
WHERE p.pid = :id;
This assumes that lastname is in the Person table, but that seems like a reasonable assumption.
I have gone through everything and it would appear that the echoing of the query is showing the query with the placeholders and it is actually updating the database.
I have a query in MySQL and I am making a crystal report by using this.
Now inside the query i have a column called scan_mode and it is coming from gfi_transaction table. This scan_mode I am using in report to suppress some sections. But some times this value is coming null for some transaction ids.
So now I want to take this scan_mode as separate query so that it will work.
Can any one please help how I can modify the below query to take only scan_mode column.
SELECT
cc.cost_center_code AS cccde,
cc.name AS ccnme,gf.scan_mode,
cc.cost_center_id AS ccid,
site.name AS siteme,
crncy.currency_locale AS currency_locale,
cntry.language AS LANGUAGE,
cntry.country_name AS cntrynm,
crncy.decimal_digits AS rnd,
gf.transaction_no AS Serial_No,
brnd.name AS brand_name,
rsn.description AS reason,
gf.comment AS COMMENT,
ts.status_description AS STATUS,
DATE_FORMAT(gf.created_date,'%d/%m/%Y') AS created_date,
gf.created_by AS created_by,
IFNULL(gf.approval_no,'Not authorized') AS Trans_no,
gf.approved_date AS approval_dt,
gf.approved_by AS approved_by,gf.status AS status1,
IFNULL(loc.cost_center_code,cc.cost_center_code) AS cur_location,
gf.document_ref_no,gf.document_ref_type,
,DATE_FORMAT(document_ref_date1,'%d/%m/%Y')) AS invoice_no
FROM
gfi_transaction gf
INNER JOIN gfi_instruction gfn ON (gf.transaction_id=gfn.transaction_id)
INNER JOIN gfi_document_instruction doc ON (gf.ref_transaction_no = doc.document_instruction_id)
INNER JOIN reason rsn ON (gf.reason_id = rsn.reason_id)
INNER JOIN gfi_status ts ON (gf.status = ts.gfi_status_id)
INNER JOIN transaction_type tt ON (gf.transaction_type_id = tt.transaction_type_id)
INNER JOIN brand brnd ON(gf.brand_id=brnd.brand_id)
-- cc details
INNER JOIN cost_center cc ON (brnd.parent_brand = cc.brand_id OR gf.brand_id = cc.brand_id)
INNER JOIN site site ON(cc.site_id = site.site_id)
INNER JOIN country cntry ON (site.country_id = cntry.country_id)
INNER JOIN currency crncy ON (cntry.currency_id=crncy.currency_id)
LEFT OUTER JOIN alshaya_location_details loc ON
(gf.brand_id = loc.brand_id AND loc.cost_center_id = gf.cost_centre_id)
LEFT OUTER JOIN alshaya_location_details locto ON
(locto.cost_center_id = gf.from_cost_center_id)
WHERE
gf.transaction_id='{?TransID}'
AND rsn.transaction_type_id IN (10,11,14)
wow, that's a big query. I ran across a similar problem in a query i was building and found the if syntax to be a solution to my problem. This was also answered in this question: MYSQL SELECT WITHIN IF Statement
$psdb->query = "SELECT count, s.classid,
if (k.sic != k.siccode, k.siccode, s.siccode) as siccode,
if (k.sic != k.siccode, k.sicdesc, s.sicdesc) as sicdesc,
if (k.sic != k.siccode, k.sicslug, s.sicslug) as sicslug
FROM ...
It looks like scan_mode column comes from "gfi_transaction" table which seems to be primary table in your query. If you get null for this column then it means your table itself have NULL value for this column. Taking that separately in a query wont solve your problem. Try replacing null with a default value and handle it in code. You can add default value instead of NULL by using ifnull(scan_mode, 'default')