optimizing a query using union left joins and inner joins - mysql

Question

I have a query that I want to optimize. Both unions executed separatley run fine, however as soon as I include inner join it takes up to 57 seconds. How do I solve this. My query is as follows
SELECT
p.PROJID,
p.StartDate,
o.ORDERNO,
p.PROJCODE,
p.PROJECT,
cat.type AS CATEGORY,
p.AREA,
p.STATE,
p.COUNTRY,
p.VALUE,
p.PROCESSOR,
p.PROJINFO,
p.NES,
p.SPECSALE,
p.OFFICE,
p.DEPTCODE,
p.INTERNLCHG,
p.INTERCOCHG,
p.LORM,
p.PERCENT,
d.COMPANY,
CONCAT(
d.LASTNAME,
", ",
d.FIRSTNAME
) AS Contact
FROM
(
(
SELECT
*
FROM
(
SELECT
`clients`.`CLIENTID` AS `CLIENTIDA`,
`clients`.`COMPANY` AS `COMPANY`
FROM
`hdb`.`clients`
UNION
SELECT
`accounts`.`id` AS `CLIENTIDA`,
`accounts`.`name` AS `COMPANY`
FROM
`sugarcrm`.`accounts`
) AS a
INNER JOIN (
SELECT
`hdb`.`contacts`.`CONTACTID` AS `CONTACTID`,
`hdb`.`contacts`.`CLIENTID` AS `CLIENTIDC`,
`hdb`.`contacts`.`FIRSTNAME` AS `FIRSTNAME`,
`hdb`.`contacts`.`LASTNAME` AS `LASTNAME`
FROM
`hdb`.`contacts`
UNION
SELECT
`sugarcrm`.`contacts`.`id` AS `CONTACTID`,
`sugarcrm`.`accounts_contacts`.`account_id` AS `CLIENTIDC`,
`sugarcrm`.`contacts`.`first_name` AS `FIRSTNAME`,
`sugarcrm`.`contacts`.`last_name` AS `LASTNAME`
FROM
`sugarcrm`.`contacts`
LEFT JOIN `sugarcrm`.`accounts_contacts` ON `sugarcrm`.`contacts`.`id` = `sugarcrm`.`accounts_contacts`.`contact_id`
) AS c ON a.CLIENTIDA = c.CLIENTIDC
) AS d
)
INNER JOIN (
(
projects AS p
INNER JOIN orders AS o ON p.ORDERNO = o.ORDERNO
)
INNER JOIN category AS cat ON p.category_id = cat.category_id
) ON d.CONTACTID = o.CONTACTID
Explain on this provides the following:
1, PRIMARY, cat, ALL, PRIMARY, , , , 10,
1, PRIMARY, p, ref, FK_orders_projects,FK_category_projects_idx, FK_category_projects_idx, 5, hdb.cat.category_id, 400, Using where
1, PRIMARY, o, eq_ref, PRIMARY, PRIMARY, 4, hdb.p.ORDERNO, 1,
1, PRIMARY, <derived2>, ALL, , , , , 18878, Using where
2, DERIVED, <derived3>, ALL, , , , , 7087,
2, DERIVED, <derived5>, ALL, , , , , 18879, Using where
5, DERIVED, contacts, ALL, , , , , 8261,
6, UNION, contacts, ALL, , , , , 10251,
6, UNION, accounts_contacts, ref, idx_contid_del_accid, idx_contid_del_accid, 111, sugarcrm.contacts.id, 1, Using index
, UNION RESULT, <union5,6>, ALL, , , , , ,
3, DERIVED, clients, ALL, , , , , 2296,
4, UNION, accounts, ALL, , , , , 4548,
, UNION RESULT, <union3,4>, ALL, , , , , ,
The original query without the union takes 0.125 seconds
SELECT p.PROJID, p.StartDate, o.ORDERNO, p.PROJCODE, p.PROJECT, cat.type AS CATEGORY, p.AREA, p.STATE, p.COUNTRY,
p.VALUE, p.PROCESSOR, p.PROJINFO, p.NES, p.SPECSALE, p.OFFICE, p.DEPTCODE, p.INTERNLCHG, p.INTERCOCHG, p.LORM,
p.PERCENT, a.COMPANY, CONCAT(c.LASTNAME, ", ", c.FIRSTNAME) AS Contact
FROM (clients AS a
INNER JOIN contacts AS c ON a.CLIENTID =c.CLIENTID)
INNER JOIN ((projects AS p INNER JOIN orders AS o ON p.ORDERNO = o.ORDERNO)
INNER JOIN category AS cat ON p.category_id = cat.category_id) ON c.CONTACTID = o.CONTACTID
ORDER BY p.PROJID, a.COMPANY;
explain on this provides following:
1, SIMPLE, cat, ALL, PRIMARY, , , , 10, Using temporary; Using filesort
1, SIMPLE, p, ref, FK_orders_projects,FK_category_projects_idx, FK_category_projects_idx, 5, hdb.cat.category_id, 400, Using where
1, SIMPLE, o, eq_ref, PRIMARY,FK_contacts_orders, PRIMARY, 4, hdb.p.ORDERNO, 1,
1, SIMPLE, c, eq_ref, PRIMARY,FK_clients_contacts, PRIMARY, 52, hdb.o.CONTACTID, 1,
1, SIMPLE, a, eq_ref, PRIMARY, PRIMARY, 52, hdb.c.CLIENTID, 1,
Query with view:
SELECT
p.PROJID,
p.StartDate,
o.ORDERNO,
p.PROJCODE,
p.PROJECT,
cat.type AS CATEGORY,
p.AREA,
p.STATE,
p.COUNTRY,
p.VALUE,
p.PROCESSOR,
p.PROJINFO,
p.NES,
p.SPECSALE,
p.OFFICE,
p.DEPTCODE,
p.INTERNLCHG,
p.INTERCOCHG,
p.LORM,
p.PERCENT,
a.COMPANY,
CONCAT(
c.LASTNAME,
", ",
c.FIRSTNAME
) AS Contact
FROM
(
view_accounts_sugar_hdb AS a
INNER JOIN view_contacts_sugar_hdb AS c ON a.CLIENTID = c.CLIENTID
)
INNER JOIN (
(
projects AS p
INNER JOIN orders AS o ON p.ORDERNO = o.ORDERNO
)
INNER JOIN category AS cat ON p.category_id = cat.category_id
) ON c.CONTACTID = o.CONTACTID
ORDER BY
p.PROJID,
a.COMPANY;
This takes over 340 secs.

Answer 1

This one was definitely uglier than the last one I helped you on :)... Anyhow, the same principles apply. For future, please do try to understand what I'm doing here. Write the JOIN relationships FIRST to know where your data is coming from. Also, take a look at my indentations... I am showing at each level for readability.
Orders -> Projects -> Categories...
then, the path for those in the normal clients table via
Orders -> Contacts -> Clients
finally to the SugarCRM contacts...
Orders -> Accounts_Contacts -> Accounts
So, now that you have the relationships set (and aliased), this follows similar of last answer implementing LEFT-JOIN to normal contact/client vs CRM Contacts/Accounts.
The field list is simple on the Order, Products and Category tables as those are pretty direct. That leaves just the "who/client" information where the LEFT-JOIN comes in. If the Normal client is null, use the CRM version fields, otherwise use the normal client fields...
SELECT
P.PROJID,
P.StartDate,
O.ORDERNO,
P.PROJCODE,
P.PROJECT,
cat.`type` AS CATEGORY,
P.AREA,
P.STATE,
P.COUNTRY,
P.VALUE,
P.PROCESSOR,
P.PROJINFO,
P.NES,
P.SPECSALE,
P.OFFICE,
P.DEPTCODE,
P.INTERNLCHG,
P.INTERCOCHG,
P.LORM,
P.PERCENT,
CASE when HCLIENT.ClientID IS NULL
then SCLIENT.`name`
ELSE HCLIENT.Company end as Company,
CASE when HCLIENT.ClientID IS NULL
then CONCAT( SCT.LAST_NAME, ", ", SCT.FIRST_NAME )
ELSE CONCAT( HCT.LASTNAME, ", ", HCT.FIRSTNAME ) end as Contact
FROM
orders O
JOIN projects P
ON O.OrderNo = P.OrderNo
JOIN category AS cat
ON p.category_id = cat.category_id
LEFT JOIN hdb.contacts HCT
ON O.ContactID = HCT.ContactID
LEFT JOIN hdb.clients HCLIENT
ON HCT.ClientID = HCLIENT.ClientID
LEFT JOIN sugarcrm.contacts SCT
ON O.ContactID = SCT.ID
LEFT JOIN sugarcrm.accounts_contacts SAC
ON SCT.ID = SAC.contact_id
LEFT JOIN sugarcrm.accounts SCLIENT
ON SCT.account_id = SCLIENT.ID
I'd be interested in the performance improvement too.