Mysql view are very slow - mysql

My mysql view is really slow in thousands of data how can we improve this functionality?
While fetch this view in 10000s data then it takes more than 30 sec. how could we revise this view table?
SELECT
i.jo_in_id,
j.*,
i.jo_in_week_number,
i.jo_in_client_ref_no,
i.cl_id AS jo_in_cl_id,
c.cl_short_name,
c.cl_business_name,
m.me_first_name,
m.me_last_name,
m.me_mobile,
sk.sk_name,
sk.sk_ticketed,
ti.ti_id,
ta.ta_name,
u.un_id,
u.un_from,
u.un_to,
v.ve_name,
mp.vmp_name,
r.vr_name
FROM
jo_2_no j2n,
jo_in_numbers i,
jobs j
LEFT JOIN venues_new v ON j.ve_id = v.ve_id
LEFT JOIN venues_meeting_place mp ON j.vmp_id = mp.vmp_id
LEFT JOIN venues_rooms r ON j.vr_id = r.vr_id
LEFT JOIN clients c ON j.cl_id = c.cl_id
LEFT JOIN members m ON j.me_id = m.me_id
LEFT JOIN skills sk ON j.sk_id = sk.sk_id
LEFT JOIN tasks ta ON j.ta_id = ta.ta_id
LEFT JOIN crew_tickets ti ON j.sk_id = ti.sk_id AND j.me_id = ti.me_id AND j.jo_time_off < ti.ti_expire
LEFT JOIN unavailability u ON j.me_id = u.me_id AND ((j.jo_time_on BETWEEN u.un_from AND u.un_to) OR (j.jo_time_on BETWEEN u.un_from AND u.un_to))
WHERE
j.jo_id = j2n.jo_id
AND j2n.jo_in_numbers_id = i.jo_in_id
AFTER user EXPLAIN SELECT following is the output

In your EXPLAIN, I see that your joined tables ti and u are read with table-scans (type: ALL). This is probably the biggest problem for your performance.
You should make sure you have the following indexes created:
ALTER TABLE crew_tickets ADD KEY (sk_id, me_id, ti_expire);
ALTER TABLE unavailability ADD KEY (me_id, un_from, un_to);
That should help the joins to those tables work with index lookups instead of table-scans. I think they'll be accessed as covering indexes, too.
Also, please don't use the outdated "comma-joins." Especially do not mix both styles. It will bite you when you get surprised by the order of precedence between comma-joins and JOIN operators. See examples in Can someone help explain why not using a SQL JOIN is bad practice and wrong? or Error on JOIN mysql.
Write your joins this way:
FROM jo_2_no j2n
INNER JOIN jo_in_numbers i ON j2n.jo_in_numbers_id = i.jo_in_id
INNER JOIN jobs j ON j.jo_id = j2n.jo_id
LEFT JOIN venues_new v ON j.ve_id = v.ve_id
LEFT JOIN venues_meeting_place mp ON j.vmp_id = mp.vmp_id
LEFT JOIN venues_rooms r ON j.vr_id = r.vr_id
LEFT JOIN clients c ON j.cl_id = c.cl_id
LEFT JOIN members m ON j.me_id = m.me_id
LEFT JOIN skills sk ON j.sk_id = sk.sk_id
LEFT JOIN tasks ta ON j.ta_id = ta.ta_id
LEFT JOIN crew_tickets ti ON j.sk_id = ti.sk_id
AND j.me_id = ti.me_id AND j.jo_time_off < ti.ti_expire
LEFT JOIN unavailability u ON j.me_id = u.me_id
AND j.jo_time_on BETWEEN u.un_from AND u.un_to
I removed the redundant term in the join condition for u. The optimizer might eliminate that logic, but why make it work so hard?

Related

SQL Join for Five Tables

I am actually trying to Join Five Tables in MySQL Database based upon RegionID which is a unique key present in all tables. I am new to it and using the following left join statement:
SELECT `tbllistings`.*,
`tblgallery`.*,
`tbltowns`.*,
`tblregions`.*,
`tblcontent`.*,
`tbllistings`.*,
`tblgallery`.*,
`tbltowns`.*,
`tblregions`.*,
`tblcontent`.*
FROM tbllistings
LEFT JOIN `webspace_db`.`tblregions`
ON `tbllistings`.`intregionid` = `tblregions`.`intregionid`
LEFT JOIN `webspace_db`.`tblcontent`
ON `tblregions`.`intregionid` = `tblcontent`.`intregionid`
LEFT JOIN `webspace_db`.`tblgallery`
ON `tblregions`.`intregionid` = `tblgallery`.`regionid`
LEFT JOIN `webspace_db`.`tbltowns`
ON `tblregions`.`intregionid` = `tbltowns`.`intregionid`
The problem that I am facing is each value is showing up more then 20+ times and I am unsure why is this happening. What I actually want is to simply join all fields from all tables based on RegionID.
Any help and suggestions are highly welcomed. Thanks a lot.
Use tbllistings.intregionid in every ON clause.
SELECT `tbllistings`.*,
`tblgallery`.*,
`tbltowns`.*,
`tblregions`.*,
`tblcontent`.*,
`tbllistings`.*,
`tblgallery`.*,
`tbltowns`.*,
`tblregions`.*,
`tblcontent`.*
FROM tbllistings
LEFT JOIN `webspace_db`.`tblregions`
ON `tblregions`.`intregionid` = `tbllistings`.`intregionid`
LEFT JOIN `webspace_db`.`tblcontent`
ON `tblcontent`.`intregionid` = `tbllistings`.`intregionid`
LEFT JOIN `webspace_db`.`tblgallery`
ON `tblgallery`.`regionid` = `tbllistings`.`intregionid`
LEFT JOIN `webspace_db`.`tbltowns`
ON `tbltowns`.`intregionid` = `tbllistings`.`intregionid`

How to filter not null values in mysql query?

I have the following mysql query. Joins are done only on FKs.
select le.lexicalentryid, def.senseid, def.definitionid, frwf.writtenForm, trdef.writtentext, frwf.wordformid, frwf.phoneticForm, le.partofspeech, le.lexiconid
from formrepresentation_wordform frwf
left join wordform wf on wf.wordformid = frwf.wordformid
left join lexicalentry le on le.lexicalentryid = wf.lexicalentryid
left join sense se on se.lexicalentryid = le.lexicalentryid
left join definition def on def.senseid = se.senseid
left join textrepresentation_definition trdef on trdef.definitionid = def.definitionid
where frwf.languageidentifier like '%deu%'
I get the following results (I show only the first 4)
I would like to get rid of the null values. They appear because for the same definitionid I have a single written form. When this happens I know I have to take the missing words from other table so now, I have to clear off the nulls.
I tried to add to the where clause frwf.writtenForm is not null, but with no success (all the writtenform column became blank)
I have managed to create the query:
SELECT frl.writtenForm, frl.languageIdentifier, le.partOfSpeech, wf.idx, frw.phoneticForm, se._index, trd.writtenText
FROM FormRepresentation_Lemma frl
JOIN LexicalEntry le ON (frl.lemmaId = le.lemmaId)
JOIN WordForm wf ON (le.lexicalEntryId = wf.lexicalEntryId)
JOIN FormRepresentation_WordForm frw ON (frw.wordFormId = wf.wordFormId)
JOIN Sense se ON (le.lexicalEntryId = se.lexicalEntryId)
JOIN Definition de ON (se.senseId = de.senseId)
JOIN TextRepresentation_Definition trd ON (de.definitionId = trd.definitionId)
WHERE frl.languageidentifier like '%deu%'
It was this table that I didn't take into account FormRepresentation_Lemma that contains link entries for null values
you need an inner join
select le.lexicalentryid, def.senseid, def.definitionid, frwf.writtenForm, trdef.writtentext, frwf.wordformid, frwf.phoneticForm, le.partofspeech, le.lexiconid
from formrepresentation_wordform frwf
inner join wordform wf on wf.wordformid = frwf.wordformid
left join lexicalentry le on le.lexicalentryid = wf.lexicalentryid
left join sense se on se.lexicalentryid = le.lexicalentryid
left join definition def on def.senseid = se.senseid
left join textrepresentation_definition trdef on trdef.definitionid = def.definitionid
where frwf.languageidentifier like '%deu%'
and possibly the other joins too. Inner join states that both tables must have a matching entry to be fetched.

Optimising a SQL Query where multiple left join duplicated twice

If someone could offer advice on improving the below query this would be most useful. I am unsure how I can make improvement when i have a left join twice in many instance to seperate tables. For example I have a location left join to the user table and a location left join to the image gallery table. I was unsure if i could optimise the sql from this point of view. It is very slow at the moment. I have ensured all columns are indexed on all joins and where statements.
SELECT im.alias_title, im.title,im.guid_id, im.description, im.hits, im.show_comment, im.can_print,
im.can_download, im.can_share, im.created_on, im.date_taken, im.approved, im.visible,
ad.address_line_1, ad.address_line_2, ad.town_village_city, ad.state_province_county, ad.postal_code, ad.other_address_detail, co.country,
geo.latitude, geo.longitude, geo.zoom, geo.yaw, geo.pitch,
c.make, c.model,
us.first_name, us.surname, uf.user_id, uf.real_name, uf.user_name, uf.gender, uf.description, uf.description, uf.buddy_icon_url, uf.first_taken_date, uf.first_date,
uf.time_zone_label, uf.time_zone_offset,
adf.address_line_1 as user_address_line_1, adf.address_line_2 as user_address_line_2, adf.town_village_city as user_town_village_city, adf.state_province_county as user_state_province_county,
adf.postal_code as user_postal_code, adf.other_address_detail as user_other_address_detail, cof.country as user_country,
geof.latitude as user_geolocation_latitude, geof.longitude as user_geolocation_longitude, geof.zoom as user_geolocation_zoom, geof.yaw as user_geolocation_yaw, geof.pitch as user_geolocation_pitch,
im.alias_title = in_image_alias_title AS image_selected -- image selected
FROM image im
LEFT JOIN address ad ON im.address_id = ad.id
LEFT JOIN country co ON ad.country_id = co.id
LEFT JOIN geolocation geo ON im.geolocation_id = geo.id
LEFT JOIN camera c ON im.camera_id = c.id
INNER JOIN user us ON im.user_id = us.id
LEFT JOIN user_flickr uf ON us.id = uf.id
LEFT JOIN address adf ON uf.address_id =adf.id
LEFT JOIN country cof ON ad.country_id = cof.id
LEFT JOIN geolocation geof ON uf.geolocation_id = geof.id
WHERE (im.alias_title = in_image_alias_title OR im.user_id = user_id)
AND im.approved = in_image_approved
AND im.visible = in_image_visible
AND (im.advertise_to <= NOW() OR im.advertise_to IS NULL)
ORDER BY image_selected DESC;
After the discussion / chat room, and learning more of what you were trying to do...
Build a compound index on the components associated with your where clause so all parts can be applied, not just the best of the first key element. Also, by removing the "alias_title" from the where clause (since you were getting the user ID based on the alias title to begin with), it was a redundant clause taking up more consideration in the query.
I would index on (user_id, approved, visible, advertise_to )
The results will come back and be small in the scheme of things, so your ultimate "order by" clause will have no problem with its final sort output.

mysql: fast group by with join

I have this query with many left joins and a inner join with dates.
I need to group by id_art (from articles_art) and date_dat (dates_dat). The problem is that is really slow. it takes 3second for 1000records.
dates_dat is indexed in dates_dat table and id_art is a primary key of articles_art.
What can I do to optimize this query?
SELECT
id_art, image2_art, video_art, website,
text.title_int, text.intro_int, text.text_int, text.extra_int,
dat.date_dat, dat.date2_dat,
group_concat(tim.time_tim),
prd.name_prd,
group_concat(cat.name_cat),
trg.name_trg,
spa.name_spa,
spa2.name_spa
FROM
articles_art AS art
LEFT JOIN internText_int AS text ON text.idart_int = art.id_art
INNER JOIN dates_dat AS dat ON art.id_art = dat.idart_dat
LEFT JOIN spaces_spa As spa ON spa.id_spa = dat.idspa_dat
LEFT JOIN spaces_spa As spa2 ON spa.id_spa = dat.idspa2_dat
LEFT JOIN times_tim AS tim ON tim.iddat_tim = dat.id_dat
LEFT JOIN articles_products_artprd AS artprd ON artprd.idart_artprd = art.id_art
LEFT JOIN products_prd AS prd ON prd.id_prd = artprd.idprd_artprd
LEFT JOIN cater_cev AS cev ON cev.idart_cev = dat.idart_dat
LEFT JOIN categories_cat AS cat ON cat.id_cat = cev.idcat_cev
LEFT JOIN targets_trg AS trg ON trg.id_trg = art.idtrg_art
WHERE
prd.id_prd in (1,2)
AND validated_art = 1
AND text.idlin_int in (1,4)
GROUP BY
id_art, date_dat
Look like you can put an index on these columns
prd.id_prd
validated_art
text.idlin_int
Test this first then if this does not work put indexes on column conditions on the ON clause
If data latency isn't an issue, can you hive the data off (perhaps overnight?) into a single normalised table? That way you query a single table without all those JOINS. You could even apply indexes to help speed things up further.

Generate FULL JOIN with LinqToSQL

I have this linq query :
(from rapportBase in New_RapportReferencementBases
join rapportExtensionAll in New_RapportReferencementExtensionBases on rapportBase.New_RapportReferencementId equals rapportExtensionAll.New_RapportReferencementId into jointureRapportExtension
from rapportExtension in jointureRapportExtension.DefaultIfEmpty()
join packExtensionAll in New_PackExtensionBases on rapportExtension.New_PackId equals packExtensionAll.New_PackId into jointurePackExtension
from packExtension in jointurePackExtension.DefaultIfEmpty()
join packBaseAll in New_PackBases on packExtension.New_PackId equals packBaseAll.New_PackId into jointurePackBase
from packBase in jointurePackBase.DefaultIfEmpty()
join domaineBaseAll in New_DomaineBases on packExtension.New_DomaineId equals domaineBaseAll.New_DomaineId into jointureDomaineBase
from domaineBase in jointureDomaineBase.DefaultIfEmpty()
join domaineExtensionAll in New_DomaineExtensionBases on domaineBase.New_DomaineId equals domaineExtensionAll.New_DomaineId into jointureDomaineExtension
from domaineExtension in jointureDomaineExtension.DefaultIfEmpty()
join compteBaseAll in AccountBases on domaineExtension.New_AccountId equals compteBaseAll.AccountId into jointureCompteBase
from compteBase in jointureCompteBase.DefaultIfEmpty()
join compteExtensionAll in AccountExtensionBases on compteBase.AccountId equals compteExtensionAll.AccountId into jointureCompteExtension
from compteExtension in jointureCompteExtension.DefaultIfEmpty()
select rapportBase)
which generate :
SELECT [t0].[New_RapportReferencementId], [t0].[CreatedOn], [t0].[CreatedBy], [t0].[ModifiedOn], [t0].[ModifiedBy], [t0].[OwningUser], [t0].[OwningBusinessUnit], [t0].[statecode] AS [Statecode], [t0].[statuscode] AS [Statuscode], [t0].[DeletionStateCode], [t0].[VersionNumber], [t0].[ImportSequenceNumber], [t0].[OverriddenCreatedOn], [t0].[TimeZoneRuleVersionNumber], [t0].[UTCConversionTimeZoneCode]
FROM [New_RapportReferencementBase] AS [t0]
LEFT OUTER JOIN [New_RapportReferencementExtensionBase] AS [t1] ON [t0].[New_RapportReferencementId] = [t1].[New_RapportReferencementId]
LEFT OUTER JOIN [New_PackExtensionBase] AS [t2] ON [t1].[New_PackId] = ([t2].[New_PackId])
LEFT OUTER JOIN [New_PackBase] AS [t3] ON [t2].[New_PackId] = [t3].[New_PackId]
LEFT OUTER JOIN [New_DomaineBase] AS [t4] ON [t2].[New_DomaineId] = ([t4].[New_DomaineId])
LEFT OUTER JOIN [New_DomaineExtensionBase] AS [t5] ON [t4].[New_DomaineId] = [t5].[New_DomaineId]
LEFT OUTER JOIN [AccountBase] AS [t6] ON [t5].[New_AccountId] = ([t6].[AccountId])
LEFT OUTER JOIN [AccountExtensionBase] AS [t7] ON [t6].[AccountId] = [t7].[AccountId]
But I want to generate :
SELECT [t0].[New_RapportReferencementId], [t0].[CreatedOn], [t0].[CreatedBy], [t0].[ModifiedOn], [t0].[ModifiedBy], [t0].[OwningUser], [t0].[OwningBusinessUnit], [t0].[statecode] AS [Statecode], [t0].[statuscode] AS [Statuscode], [t0].[DeletionStateCode], [t0].[VersionNumber], [t0].[ImportSequenceNumber], [t0].[OverriddenCreatedOn], [t0].[TimeZoneRuleVersionNumber], [t0].[UTCConversionTimeZoneCode]
FROM [New_RapportReferencementBase] AS [t0]
FULL OUTER JOIN [New_RapportReferencementExtensionBase] AS [t1] ON [t0].[New_RapportReferencementId] = [t1].[New_RapportReferencementId]
FULL OUTER JOIN [New_PackExtensionBase] AS [t2] ON [t1].[New_PackId] = ([t2].[New_PackId])
FULL OUTER JOIN [New_PackBase] AS [t3] ON [t2].[New_PackId] = [t3].[New_PackId]
FULL OUTER JOIN [New_DomaineBase] AS [t4] ON [t2].[New_DomaineId] = ([t4].[New_DomaineId])
FULL OUTER JOIN [New_DomaineExtensionBase] AS [t5] ON [t4].[New_DomaineId] = [t5].[New_DomaineId]
FULL OUTER JOIN [AccountBase] AS [t6] ON [t5].[New_AccountId] = ([t6].[AccountId])
FULL OUTER JOIN [AccountExtensionBase] AS [t7] ON [t6].[AccountId] = [t7].[AccountId]
In other word, i want to generate full outer join for this query and not just left.
someone know how to do this in a simple way ?
thanks
There isn't a Full Outer Join in Linq. You have to do two left joins and concat them together. Here's some pseudocode that looks like linq:
var foj =
(from l in left
join r in right on l.Id equals r.Id into g
from r in g.DefaultIfEmpty()
select new { l, r })
.Concat
(from r in right
join l in left on r.Id equals l.Id into g
from l in g.DefaultIfEmpty()
where l == null
select new { l, r });
Probably be better to push this logic into a stored procedure if you're planning on using Linq to Sql.
See this post from the VB Team:
http://blogs.msdn.com/vbteam/archive/2008/02/12/converting-sql-to-linq-part-9-full-outer-join-bill-horst.aspx
Looks a little verbose.
Otherwise, there may be a way to achieve what you want using Foreign Keys and the Union operator.
Sadly I saw that before but i wanted to be sure.
The thing we did is to do it in a stored procedure and access it via linqtosql method.
then we map table result, like linqtosql do, using grouping to obtain collection of enties link to another entity.
for example, if we have an account with some contacts link to it. We request all contact with for each of them the account with a full join and them group on account id to obtain list of contact of each.
It work very well but it would be better if linqtosql was able to generate full join....