How can i eliminate duplicate data when joining 2 queries via union? - mysql

Question

i have 2 queries that pull same fields with only diff is what the subquery in the where clause returns.
first query pulls based on the code and insurance_group_name
SELECT
cpt_codes.code,
ii.insurance_group_name,
ii.insurance_group_id,
CAST(AVG(dcc_total_view.allowedAmount) AS decimal(10, 2)) `AVG Allowed by Group Name`,
CAST(AVG(rc_total_view.runningCollectedReceivedByProvider) AS decimal(10, 2)) `AVG Collected by Group Name`,
GROUP_CONCAT(DISTINCT file_acceptance) AS `FAN`
FROM ds
LEFT OUTER JOIN dd
ON ds.dos_detail_id = dd.id
LEFT OUTER JOIN ii
ON ds.ii_id = ii.id
LEFT OUTER JOIN dcc
ON ds.id = dcc.dos_id
LEFT OUTER JOIN cpt_codes
ON dcc.cpt_id = cpt_codes.id
LEFT OUTER JOIN bi
ON ds.billing_id = bi.id
LEFT OUTER JOIN cs
ON bi.claim_status = cs.id
LEFT OUTER JOIN rc_total_view
ON ds.payment_information_id = rc_total_view.id
LEFT OUTER JOIN dcc_total_view
ON ds.id = dcc_total_view.dos_id
WHERE (cpt_codes.code, ii.insurance_group_name) IN (SELECT
cpt_codes.code,
ii.insurance_group_name
FROM ds
LEFT OUTER JOIN dd
ON ds.dos_detail_id = dd.id
LEFT OUTER JOIN ii
ON ds.ii_id = ii.id
LEFT OUTER JOIN dcc
ON ds.id = dcc.dos_id
LEFT OUTER JOIN cpt_codes
ON dcc.cpt_id = cpt_codes.id
LEFT OUTER JOIN bi
ON ds.billing_id = bi.id
LEFT OUTER JOIN cs
ON bi.claim_status = cs.id
WHERE cs.status = 'Pending'
AND date_of_service BETWEEN '2017/01/01' AND '2018/08/31'
AND (dcc.rev_code = '0490'
OR dcc.rev_code = '0360'))
AND date_of_service BETWEEN '2017/01/01' AND '2018/08/31'
AND (dcc.rev_code = '0490'
OR dcc.rev_code = '0360')
AND dcc_total_view.allowedAmount > 0
GROUP BY code ,insurance_group_name
this would return a line like this
code
insurance_group_name
insurance_group_id
AVG Allowed by Group Name
AVG Collected by Group Name
FAN
20553
AE
215825
1440.97
889.48
DELA081518,MOLDGILR919,SHIC060618,MANS072718,DELA053018,DELS072518
the Second one pulls based on insurance_group_id
SELECT
cpt_codes.code,
ii.insurance_group_name,
ii.insurance_group_id,
CAST(AVG(dcc_total_view.allowedAmount) AS decimal(10, 2)) `AVG Allowed by Group Name`,
CAST(AVG(rc_total_view.runningCollectedReceivedByProvider) AS decimal(10, 2)) `AVG Collected by Group Name`,
GROUP_CONCAT(DISTINCT file_acceptance) AS `FAN`
FROM ds
LEFT OUTER JOIN dd
ON ds.dos_detail_id = dd.id
LEFT OUTER JOIN ii
ON ds.ii_id = ii.id
LEFT OUTER JOIN dcc
ON ds.id = dcc.dos_id
LEFT OUTER JOIN cpt_codes
ON dcc.cpt_id = cpt_codes.id
LEFT OUTER JOIN bi
ON ds.billing_id = bi.id
LEFT OUTER JOIN cs
ON bi.claim_status = cs.id
LEFT OUTER JOIN rc_total_view
ON ds.payment_information_id = rc_total_view.id
LEFT OUTER JOIN dcc_total_view
ON ds.id = dcc_total_view.dos_id
WHERE (cpt_codes.code, ii.insurance_group_id) IN (SELECT
cpt_codes.code,
ii.insurance_group_id
FROM ds
LEFT OUTER JOIN dd
ON ds.dos_detail_id = dd.id
LEFT OUTER JOIN ii
ON ds.ii_id = ii.id
LEFT OUTER JOIN dcc
ON ds.id = dcc.dos_id
LEFT OUTER JOIN cpt_codes
ON dcc.cpt_id = cpt_codes.id
LEFT OUTER JOIN bi
ON ds.billing_id = bi.id
LEFT OUTER JOIN cs
ON bi.claim_status = cs.id
WHERE cs.status = 'Pending'
AND date_of_service BETWEEN '2017/01/01' AND '2018/08/31'
AND (dcc.rev_code = '0490'
OR dcc.rev_code = '0360'))
AND date_of_service BETWEEN '2017/01/01' AND '2018/08/31'
AND (dcc.rev_code = '0490'
OR dcc.rev_code = '0360')
AND dcc_total_view.allowedAmount > 0
GROUP BY code ,insurance_group_id
this would return a line like
code
insurance_group_name
insurance_group_id
AVG Allowed by Group Name
AVG Collected by Group Name
FAN
20553
AE
10198703100012
250.25
150.36
MOLDGILR919
when doing union on these 2 i find some data is duplicated where the FAN # that the second query returns is included in the output from the first as you can see the FAN# is part of the ones that make up the results of the first script.
how can i join together the results of these 2 queries and make sure if the FAN# in the second is already included in results from the first for a given code (like 20553 in the example) then that row from the second will not be included in the final output ?
I hope i have included enough info and detail to help me out here, I really appreciate any help to figure this out.

Answer 1

Put the output of your first statemen in a temporary table, lets name that #temp1, and the output of your second query in a temporary table named #temp2.
After that you can do:
SELECT *
FROM #temp1
UNION ALL
SELECT *
FROM #temp2
WHERE code NOT IN (SELECT code FROM #temp1)
EDIT: Because of the FAN problem..
I think you should change your queries, in this way:
Do not use GROUP_CONCAT(...) to combine the results of different FAN's in one row, and add file_acceptance to the GROUP BY-clause
This way you will get indivdual records for every FAN, annd, after the UNION ALL you can GROUP_CONCAT() then again.
This might need some more think/tweak work because of the AVG() fields....