mysql: fast group by with join - mysql

I have this query with many left joins and a inner join with dates.
I need to group by id_art (from articles_art) and date_dat (dates_dat). The problem is that is really slow. it takes 3second for 1000records.
dates_dat is indexed in dates_dat table and id_art is a primary key of articles_art.
What can I do to optimize this query?
SELECT
id_art, image2_art, video_art, website,
text.title_int, text.intro_int, text.text_int, text.extra_int,
dat.date_dat, dat.date2_dat,
group_concat(tim.time_tim),
prd.name_prd,
group_concat(cat.name_cat),
trg.name_trg,
spa.name_spa,
spa2.name_spa
FROM
articles_art AS art
LEFT JOIN internText_int AS text ON text.idart_int = art.id_art
INNER JOIN dates_dat AS dat ON art.id_art = dat.idart_dat
LEFT JOIN spaces_spa As spa ON spa.id_spa = dat.idspa_dat
LEFT JOIN spaces_spa As spa2 ON spa.id_spa = dat.idspa2_dat
LEFT JOIN times_tim AS tim ON tim.iddat_tim = dat.id_dat
LEFT JOIN articles_products_artprd AS artprd ON artprd.idart_artprd = art.id_art
LEFT JOIN products_prd AS prd ON prd.id_prd = artprd.idprd_artprd
LEFT JOIN cater_cev AS cev ON cev.idart_cev = dat.idart_dat
LEFT JOIN categories_cat AS cat ON cat.id_cat = cev.idcat_cev
LEFT JOIN targets_trg AS trg ON trg.id_trg = art.idtrg_art
WHERE
prd.id_prd in (1,2)
AND validated_art = 1
AND text.idlin_int in (1,4)
GROUP BY
id_art, date_dat

Look like you can put an index on these columns
prd.id_prd
validated_art
text.idlin_int
Test this first then if this does not work put indexes on column conditions on the ON clause

If data latency isn't an issue, can you hive the data off (perhaps overnight?) into a single normalised table? That way you query a single table without all those JOINS. You could even apply indexes to help speed things up further.

Related

Mysql view are very slow

My mysql view is really slow in thousands of data how can we improve this functionality?
While fetch this view in 10000s data then it takes more than 30 sec. how could we revise this view table?
SELECT
i.jo_in_id,
j.*,
i.jo_in_week_number,
i.jo_in_client_ref_no,
i.cl_id AS jo_in_cl_id,
c.cl_short_name,
c.cl_business_name,
m.me_first_name,
m.me_last_name,
m.me_mobile,
sk.sk_name,
sk.sk_ticketed,
ti.ti_id,
ta.ta_name,
u.un_id,
u.un_from,
u.un_to,
v.ve_name,
mp.vmp_name,
r.vr_name
FROM
jo_2_no j2n,
jo_in_numbers i,
jobs j
LEFT JOIN venues_new v ON j.ve_id = v.ve_id
LEFT JOIN venues_meeting_place mp ON j.vmp_id = mp.vmp_id
LEFT JOIN venues_rooms r ON j.vr_id = r.vr_id
LEFT JOIN clients c ON j.cl_id = c.cl_id
LEFT JOIN members m ON j.me_id = m.me_id
LEFT JOIN skills sk ON j.sk_id = sk.sk_id
LEFT JOIN tasks ta ON j.ta_id = ta.ta_id
LEFT JOIN crew_tickets ti ON j.sk_id = ti.sk_id AND j.me_id = ti.me_id AND j.jo_time_off < ti.ti_expire
LEFT JOIN unavailability u ON j.me_id = u.me_id AND ((j.jo_time_on BETWEEN u.un_from AND u.un_to) OR (j.jo_time_on BETWEEN u.un_from AND u.un_to))
WHERE
j.jo_id = j2n.jo_id
AND j2n.jo_in_numbers_id = i.jo_in_id
AFTER user EXPLAIN SELECT following is the output
In your EXPLAIN, I see that your joined tables ti and u are read with table-scans (type: ALL). This is probably the biggest problem for your performance.
You should make sure you have the following indexes created:
ALTER TABLE crew_tickets ADD KEY (sk_id, me_id, ti_expire);
ALTER TABLE unavailability ADD KEY (me_id, un_from, un_to);
That should help the joins to those tables work with index lookups instead of table-scans. I think they'll be accessed as covering indexes, too.
Also, please don't use the outdated "comma-joins." Especially do not mix both styles. It will bite you when you get surprised by the order of precedence between comma-joins and JOIN operators. See examples in Can someone help explain why not using a SQL JOIN is bad practice and wrong? or Error on JOIN mysql.
Write your joins this way:
FROM jo_2_no j2n
INNER JOIN jo_in_numbers i ON j2n.jo_in_numbers_id = i.jo_in_id
INNER JOIN jobs j ON j.jo_id = j2n.jo_id
LEFT JOIN venues_new v ON j.ve_id = v.ve_id
LEFT JOIN venues_meeting_place mp ON j.vmp_id = mp.vmp_id
LEFT JOIN venues_rooms r ON j.vr_id = r.vr_id
LEFT JOIN clients c ON j.cl_id = c.cl_id
LEFT JOIN members m ON j.me_id = m.me_id
LEFT JOIN skills sk ON j.sk_id = sk.sk_id
LEFT JOIN tasks ta ON j.ta_id = ta.ta_id
LEFT JOIN crew_tickets ti ON j.sk_id = ti.sk_id
AND j.me_id = ti.me_id AND j.jo_time_off < ti.ti_expire
LEFT JOIN unavailability u ON j.me_id = u.me_id
AND j.jo_time_on BETWEEN u.un_from AND u.un_to
I removed the redundant term in the join condition for u. The optimizer might eliminate that logic, but why make it work so hard?

Can we write condition in ON clause of left join which may or may not satisfy?

I have a table for audit trail which saves all actions performed records through out the project like add update and delete.
there I maintain a column which saves primary keys of multiple tables on which action is performed. This is integer column
my query is like this
select * from
user usr1
left join activity_history
on activity_history.userID= usr1.sequenceID
left join candidate can1 on can1.userID = usr1.sequenceID
and can1.userID = activity_history.activity_sequenceID
left join institute ins1 on ins1.userID= usr1.sequenceID
and ins1.userID = activity_history.activity_sequenceID
left join candidate_institutes caninst on caninst.candidateID = can1.candidateID and caninst.instituteID= ins1.instituteID
left join exam exam1 on exam1.instituteID = ins1.instituteID
and exam1.examID = activity_history.activity_sequenceID
left join proctor pro1 on pro1.userID = usr1.sequenceID
and pro1.proctorID = activity_history.activity_sequenceID
left join appointment appt1 on appt1.examID = exam1.examID
and appt1.sequenceID = activity_history.activity_sequenceID
/* COMMENTED CODE-----
on( activity_history.activity_sequenceID=can1.userID
OR activity_history.activity_sequenceID=ins1.userID
OR activity_history.activity_sequenceID=exam1.examID and activity_history.userID= usr1.sequenceID
OR activity_history.activity_sequenceID=pro1.userID
OR activity_history.activity_sequenceID=appt1.sequenceID
)
*/
order by activity_history.sequenceID desc
activity_sequenceID is the column where I am storing keys of other tables
I need to map that. Is this a right way to join these tables or Commented part could be the rite way ?
Or is there any other way to join these tables.
I am confused because I am writing a condition in but activity_histry table may or may not have records of that particular table .

SQL Join for Five Tables

I am actually trying to Join Five Tables in MySQL Database based upon RegionID which is a unique key present in all tables. I am new to it and using the following left join statement:
SELECT `tbllistings`.*,
`tblgallery`.*,
`tbltowns`.*,
`tblregions`.*,
`tblcontent`.*,
`tbllistings`.*,
`tblgallery`.*,
`tbltowns`.*,
`tblregions`.*,
`tblcontent`.*
FROM tbllistings
LEFT JOIN `webspace_db`.`tblregions`
ON `tbllistings`.`intregionid` = `tblregions`.`intregionid`
LEFT JOIN `webspace_db`.`tblcontent`
ON `tblregions`.`intregionid` = `tblcontent`.`intregionid`
LEFT JOIN `webspace_db`.`tblgallery`
ON `tblregions`.`intregionid` = `tblgallery`.`regionid`
LEFT JOIN `webspace_db`.`tbltowns`
ON `tblregions`.`intregionid` = `tbltowns`.`intregionid`
The problem that I am facing is each value is showing up more then 20+ times and I am unsure why is this happening. What I actually want is to simply join all fields from all tables based on RegionID.
Any help and suggestions are highly welcomed. Thanks a lot.
Use tbllistings.intregionid in every ON clause.
SELECT `tbllistings`.*,
`tblgallery`.*,
`tbltowns`.*,
`tblregions`.*,
`tblcontent`.*,
`tbllistings`.*,
`tblgallery`.*,
`tbltowns`.*,
`tblregions`.*,
`tblcontent`.*
FROM tbllistings
LEFT JOIN `webspace_db`.`tblregions`
ON `tblregions`.`intregionid` = `tbllistings`.`intregionid`
LEFT JOIN `webspace_db`.`tblcontent`
ON `tblcontent`.`intregionid` = `tbllistings`.`intregionid`
LEFT JOIN `webspace_db`.`tblgallery`
ON `tblgallery`.`regionid` = `tbllistings`.`intregionid`
LEFT JOIN `webspace_db`.`tbltowns`
ON `tbltowns`.`intregionid` = `tbllistings`.`intregionid`

Double query a table from two fields (INNER JOIN?)

I hope I can explain myself.
I have a many to many table (asignaciones) which points to alumnos and invest tables. Both of those tables has a institucionID which points to instituciones table.
I need to get (in one query) both instituciones from alumnos and invest. I have this but is not complete. I guess if because of the AND in the last inner join:
SELECT
alumnos.alumnosID,
invest.investigadoresID,
asignaciones.alumnosID AS alumnosID1,
asignaciones.investigadoresID AS investigadoresID1,
instituciones.institucion
FROM alumnos
INNER JOIN asignaciones ON alumnos.alumnosID = asignaciones.alumnosID
INNER JOIN invest ON asignaciones.investigadoresID = invest.investigadoresID
INNER JOIN instituciones ON alumnos.institucionesID = instituciones.institucionesID AND invest.institucionesID = instituciones.institucionesID
This lacks the second institucion. I am getting just one
Any hints on this is really appreciated
This Query schold solve your problem:
SELECT
alumnos.alumnosID,
invest.investigadoresID,
asignaciones.alumnosID AS alumnosID1,
asignaciones.investigadoresID AS investigadoresID1,
instituciones.institucion
instituciones1.institucion
FROM alumnos
INNER JOIN asignaciones ON alumnos.alumnosID = asignaciones.alumnosID
INNER JOIN invest ON asignaciones.investigadoresID = invest.investigadoresID
INNER JOIN instituciones instituciones ON alumnos.institucionesID = instituciones.institucionesID
INNER JOIN instituciones instituciones1 ON invest.institucionesID = instituciones1.institucionesID
You have to join the last table twice.

Optimising a SQL Query where multiple left join duplicated twice

If someone could offer advice on improving the below query this would be most useful. I am unsure how I can make improvement when i have a left join twice in many instance to seperate tables. For example I have a location left join to the user table and a location left join to the image gallery table. I was unsure if i could optimise the sql from this point of view. It is very slow at the moment. I have ensured all columns are indexed on all joins and where statements.
SELECT im.alias_title, im.title,im.guid_id, im.description, im.hits, im.show_comment, im.can_print,
im.can_download, im.can_share, im.created_on, im.date_taken, im.approved, im.visible,
ad.address_line_1, ad.address_line_2, ad.town_village_city, ad.state_province_county, ad.postal_code, ad.other_address_detail, co.country,
geo.latitude, geo.longitude, geo.zoom, geo.yaw, geo.pitch,
c.make, c.model,
us.first_name, us.surname, uf.user_id, uf.real_name, uf.user_name, uf.gender, uf.description, uf.description, uf.buddy_icon_url, uf.first_taken_date, uf.first_date,
uf.time_zone_label, uf.time_zone_offset,
adf.address_line_1 as user_address_line_1, adf.address_line_2 as user_address_line_2, adf.town_village_city as user_town_village_city, adf.state_province_county as user_state_province_county,
adf.postal_code as user_postal_code, adf.other_address_detail as user_other_address_detail, cof.country as user_country,
geof.latitude as user_geolocation_latitude, geof.longitude as user_geolocation_longitude, geof.zoom as user_geolocation_zoom, geof.yaw as user_geolocation_yaw, geof.pitch as user_geolocation_pitch,
im.alias_title = in_image_alias_title AS image_selected -- image selected
FROM image im
LEFT JOIN address ad ON im.address_id = ad.id
LEFT JOIN country co ON ad.country_id = co.id
LEFT JOIN geolocation geo ON im.geolocation_id = geo.id
LEFT JOIN camera c ON im.camera_id = c.id
INNER JOIN user us ON im.user_id = us.id
LEFT JOIN user_flickr uf ON us.id = uf.id
LEFT JOIN address adf ON uf.address_id =adf.id
LEFT JOIN country cof ON ad.country_id = cof.id
LEFT JOIN geolocation geof ON uf.geolocation_id = geof.id
WHERE (im.alias_title = in_image_alias_title OR im.user_id = user_id)
AND im.approved = in_image_approved
AND im.visible = in_image_visible
AND (im.advertise_to <= NOW() OR im.advertise_to IS NULL)
ORDER BY image_selected DESC;
After the discussion / chat room, and learning more of what you were trying to do...
Build a compound index on the components associated with your where clause so all parts can be applied, not just the best of the first key element. Also, by removing the "alias_title" from the where clause (since you were getting the user ID based on the alias title to begin with), it was a redundant clause taking up more consideration in the query.
I would index on (user_id, approved, visible, advertise_to )
The results will come back and be small in the scheme of things, so your ultimate "order by" clause will have no problem with its final sort output.