I have this query that takes more than 10 seconds,
The most bulky table is requetes, the clause where depends on filter:
SELECT SQL_CALC_FOUND_ROWS
requetes.id AS thread_id,
requetes.id AS thread_id2,
CONCAT_WS("/", SUBSTR(requetes.id, 5, 2), SUBSTR(requetes.id, 3, 2), SUBSTR(requetes.id, 1, 2)) AS thread_create_date,
DATE_FORMAT(create_date, "%d/%m/%Y %H:%i") AS create_date,
IF(UNIX_TIMESTAMP(requetes.date_objectif_requete) = 0, NULL, DATE_FORMAT(requetes.date_objectif_requete, "%d/%m/%Y %H:%i:%S")) AS thread_target_date,
IF(UNIX_TIMESTAMP(requetes.date_objectif_requete) = 0, NULL, DATE_FORMAT(requetes.date_objectif_requete, "%Y%m%d")) AS int_thread_target_date,
requetes.prenom AS customer_first_name,
requetes.nom AS customer_last_name,
requetes.subject AS subject_action,
requetes.ident_e AS code_customer,
CONCAT_WS(" ", UPPER(requetes.nom), requetes.prenom, requetes.pseudo) AS complete_customer_name,
requetes.type_rs AS type_rs,
themes_sous.id AS thread_type_id,
requetes.cp AS zip_code,
requetes.libelle_pc_origine AS libelle_pc_origine,
themes_sous.libelle AS thread_type_label,
themes_sous.action_page AS thread_type_target_url,
agents.prenom AS user_first_name,
agents.nom AS user_last_name,
req_agent.id_agent AS code_user,
CONCAT_WS(" ", UPPER(agents.nom), agents.prenom) AS complete_user_name,
(1 - statut_inverse) AS done,
archive AS archived,
num_messages,
themes.libelle AS libelle_categ
FROM requetes
LEFT OUTER JOIN themes_sous
ON requetes.id_ss_theme = themes_sous.id
LEFT OUTER JOIN req_agent
ON requetes.id = req_agent.id_requete
LEFT OUTER JOIN agents
ON req_agent.id_agent = agents.id
LEFT OUTER JOIN themes
ON themes.id = themes_sous.id_theme
WHERE requetes.client_identifier=4
AND themes_sous.client_identifier=4
AND requetes.statut_inverse = 1 AND DATE_FORMAT(requetes.date_objectif_requete, "%Y%m%d") <> 20171215 AND NOT (DATE_FORMAT(requetes.date_objectif_requete, "%Y%m%d") > 20171215) AND ( req_agent.id_agent_origine = "5087050" ) AND requetes.date_objectif_requete IS NOT NULL AND requetes.date_objectif_requete <> "0000-00-00 00:00:00"
ORDER BY requetes.id LIMIT 0, 25;
The explain of this query is :
explain of the query
config of the most bulky table
thanks for your help
Related
I have this MYSQL Query:
select
sc.supplier,
sq.query_value,
sc.cnv_name as name,
sc.cnv_id,
sq.query_type_id,
qt.query_type_desc,
date_format(sq.query_date, '%d/%m/%Y') as query_date,
qs.query_status,
sq.sla_count,
sq.sla_state,
sq.assigned_to,
IfNull(teams.team_count, 0) team_count,
if(
sq.sla_state is null
OR sq.sla_state = '',
'',
if(
sq.sla_state = 0,
'Query Handler',
if(
sq.sla_state = 1,
'Team Leader',
if(
sq.sla_state = 2,
'Manager',
if(
sq.sla_state = 3,
'Senior Manager',
'Senior Manager (GSCOP BREACH)'
)
)
)
)
) as sla_state_desc,
if(
team_count <= 0
OR team_count is null,
'',
if(team_count > 1, '(multiple)', pt.team_desc)
) as team
from
supconversation sc
left outer join supconvers_querydtls sq on sq.cnv_id = sc.cnv_id
left outer join querytype qt on qt.query_type_id = sq.query_type_id
left outer join querystatus qs on qs.query_status_id = sq.query_status_id
left outer join (
select
count(*) team_count,
tu.user_id
from
teamusers tu,
supconvers_querydtls sq2
where
tu.user_id = sq2.assigned_to
) teams on (teams.user_id = sq.assigned_to)
left outer join teamusers tu on tu.user_id = sq.assigned_to
left outer join portalteam pt on pt.team_id = tu.team_id
where
sc.supplier = 'SUPTEST1'
and sc.cnv_type = 'Q'
order by
query_date desc
What I am trying to do is check if a user has a record in teamusers. The user can be a part of more than 1 team. This is stored in the teamusers table with the columns team_id, user_id. If a user has more than 1 record in teamusers then return team value as (multiple). Otherwise, return the team_desc found in the portalteam table. The portalteam table consists of the columns team_id,team_desc.
This query seems to work but it's returning duplicate rows. I imagine this is because there are multiple records for some users in teamusers. How would I go about fixing this issue?
If you really get duplicated rows then you could try using a select DISTINCT
select distinct
sc.supplier,
sq.query_value,
sc.cnv_name as name,
sc.cnv_id,
sq.query_type_id,
qt.query_type_desc,
date_format(sq.query_date, '%d/%m/%Y') as query_date,
qs.query_status,
sq.sla_count,
sq.sla_state,
sq.assigned_to,
IfNull(teams.team_count, 0) team_count,
if(
sq.sla_state is null
OR sq.sla_state = '',
'',
if(
sq.sla_state = 0,
'Query Handler',
if(
sq.sla_state = 1,
'Team Leader',
if(
sq.sla_state = 2,
'Manager',
if(
sq.sla_state = 3,
'Senior Manager',
'Senior Manager (GSCOP BREACH)'
)
)
)
)
) as sla_state_desc,
if(
team_count <= 0
OR team_count is null,
'',
if(team_count > 1, '(multiple)', pt.team_desc)
) as team
from
supconversation sc
.....
.....
For my query, the two derived tables at the bottom are causing a crazy slow up for this query. The query, as is, takes about 45-55 seconds to execute.. NOW, when i remove just one of those derived tables (it does not matter which one) the query goes down to 0.1 - 0.3 seconds. My questions; Is there an issue with having multiple derived tables? Is there a better way to execute this? My indexes all seem to be correct, I will also include the explain from this query.
select t.name as team, u.name as "REP NAME",
count(distinct activity.id) as "TOTAL VISITS",
count(distinct activity.account_id) as "UNIQUE VISITS",
count(distinct placement.id) as "COMMITMENTS ADDED",
CASE WHEN
count(distinct activity.account_id) = 0 THEN (count(distinct
placement.id) / 1)
else (cast(count(distinct placement.id) as decimal(10,2)) /
cast(count(distinct activity.account_id) as decimal(10,2)))
end as "UNIQUE VISIT TO COMMITMENT %",
case when o.mode='basic' then count(distinct placement.id) else
count(distinct(case when placement.commitmentstatus='fullfilled'
then placement.id else 0 end))
end as "COMMITMENTS FULFILLED",
case when o.mode='basic' then 1 else
(CASE WHEN
count(distinct placement.id) = 0 THEN (count(distinct(case when
placement.commitmentstatus='fullfilled' then placement.id else 0
end)) / 1)
else (cast(count(distinct(case when
placement.commitmentstatus='fullfilled' then placement.id else 0
end)) as decimal(10,2)) / cast(count(distinct placement.id) as
decimal(10,2)))
end) end as "COMMITMENT TO FULFILLMENT %"
from lpmysqldb.users u
left join lpmysqldb.teams t on t.team_id=u.team_id
left join lpmysqldb.organizations o on o.id=t.org_id
left join (select * from lpmysqldb.activity where
org_id='555b918ae4b07b6ac5050852' and completed_at>='2018-05-01' and
completed_at<='2018-06-01' and tag='visit' and accountname is not
null and (status='active' or status='true' or status='1')) as
activity on activity.user_id=u.id
left join (select * from lpmysqldb.placements where
orgid='555b918ae4b07b6ac5050852' and placementdate>='2018-05-01' and
placementdate<='2018-06-01' and (status IN ('1','active','true') or
status is null)) as placement on placement.userid=u.id
where u.org_id='555b918ae4b07b6ac5050852'
and (u.status='active' or u.status='true' or u.status='1')
and istestuser!='1'
group by u.org_id, t.name, u.id, u.name, o.mode
order by count(distinct activity.id) desc
Thank you for assistance!
I have edited below with changing the two bottom joins from joining on subqueries to joining on the table directly. Still yielding the same result.
This is a SLIGHTLY restructured query of your same. Might be simplified as the last two subqueries are all pre-aggregated for your respective counts and count distincts so you can use those column names directly instead of showing all the count( distinct ) embedded throughout the query.
I also tried to simplify the division by multiplying a given count by 1.00 to force decimal-based precision as result.
select
t.name as team,
u.name as "REP NAME",
Activity.DistIdCnt as "TOTAL VISITS",
Activity.UniqAccountCnt as "UNIQUE VISITS",
Placement.DistIdCnt as "COMMITMENTS ADDED",
Placement.DistIdCnt /
CASE WHEN Activity.UniqAccountCnt = 0
THEN 1.00
ELSE Activity.UniqAccountCnt * 1.00
end as "UNIQUE VISIT TO COMMITMENT %",
case when o.mode = 'basic'
then Placement.DistIdCnt
else Placement.DistFulfillCnt
end as "COMMITMENTS FULFILLED",
case when o.mode = 'basic'
then 1
else ( Placement.DistFulfillCnt /
CASE when Placement.DistIdCnt = 0
then 1.00
ELSE Placement.DistIdCnt * 1.00
END TRANSACTION )
END as "COMMITMENT TO FULFILLMENT %"
from
lpmysqldb.users u
left join lpmysqldb.teams t
on u.team_id = t.team_id
left join lpmysqldb.organizations o
on t.org_id = o.id
left join
( select
user_id,
count(*) as AllRecs,
count( distinct id ) DistIdCnt,
count( distinct account_id) as UniqAccountCnt
from
lpmysqldb.activity
where
org_id = '555b918ae4b07b6ac5050852'
and completed_at>='2018-05-01'
and completed_at<='2018-06-01'
and tag='visit'
and accountname is not null
and status IN ( '1', 'active', 'true')
group by
user_id ) activity
on u.id = activity.user_id
left join
( select
userid,
count(*) AllRecs,
count(distinct id) as DistIdCnt,
count(distinct( case when commitmentstatus = 'fullfilled'
then id
else 0 end )) DistFulfillCnt
from
lpmysqldb.placements
where
orgid = '555b918ae4b07b6ac5050852'
and placementdate >= '2018-05-01'
and placementdate <= '2018-06-01'
and ( status is null OR status IN ('1','active','true')
group by
userid ) as placement
on u.id = placement.userid
where
u.org_id = '555b918ae4b07b6ac5050852'
and u.status IN ( 'active', 'true', '1')
and istestuser != '1'
group by
u.org_id,
t.name,
u.id,
u.name,
o.mode
order by
activity.DistIdCnt desc
FINALLY, your inner queries are querying for ALL users. If you have a large count of users that are NOT active, you MIGHT exclude those users from each inner query by adding those join/criteria there too such as...
( ...
from
lpmysqldb.placements
JOIN lpmysqldb.users u2
on placements.userid = u2.id
and u2.status IN ( 'active', 'true', '1')
and u2.istestuser != '1'
where … ) as placement
Not sure whether the actually query matters but, I have a MySQL Stored Procedure where I commented out the other parts of the proc except the following query...
INSERT INTO temp_attribution (`attribute_type`, `domain`, `id`, `name`, `score`, `rank`, `partner_match`, `person_match`, `sponsor_match`, `date_match`)
SELECT 'Campaign' AS attribute_type, domain, id, name, score, (#proc_counter := #proc_counter + 1) AS rank,
partner_match, person_match, sponsor_match, date_match
FROM (
SELECT m_c.domain, m_c.campaign_id AS id, m_c.name, m_c.client_id, m_c.sent_date,
proc_sponsors AS invoice_sponsor, bs.sponsor AS campaign_sponsor,
proc_email AS invoice_email, aes_decrypt(m_r.email, in_encrypt_key) as campaign_email,
if (m_c.client_id = proc_client_id COLLATE latin1_general_ci, 'Yes', 'No') AS partner_match,
if (aes_encrypt(proc_email, in_encrypt_key) = m_r.email, 'Exact Email', 'Email Domain') AS person_match,
if (LOCATE(CONVERT(bs.sponsor USING utf8mb4), proc_sponsors) > 0, 'Sponsor',
if (CONVERT(bs.vendor USING utf8mb4) = proc_vendor, 'Vendor', 'No') ) AS sponsor_match,
if (datediff(proc_invoice_date, m_c.sent_date) BETWEEN 0 AND 92, 'Within Three', 'Within Six') AS date_match,
(
if (m_c.client_id = proc_client_id COLLATE latin1_general_ci, 45, 10) + 30 +
if (LOCATE(CONVERT(bs.sponsor USING utf8mb4), proc_sponsors) > 0, 10,
if (CONVERT(bs.vendor USING utf8mb4) = proc_vendor, 5, 0) ) +
if (datediff(proc_invoice_date, m_c.sent_date) BETWEEN 0 AND 92, 15, 5)
) AS score
FROM campaign_table m_c
INNER JOIN recipient_table m_r ON m_c.domain = m_r.domain AND m_c.campaign_id = m_r.campaign_id
LEFT JOIN booking_sponsor bs ON m_c.domain = bs.domain AND m_c.campaign_id = bs.campaign_id
WHERE datediff(proc_invoice_date, m_c.sent_date) BETWEEN 0 AND 185
AND ( aes_encrypt(proc_email, in_encrypt_key) = m_r.email OR m_r.email_domain = proc_email_domain )
) T ORDER BY score DESC, sent_date DESC LIMIT 5;
The fields starting with 'proc_' are actually variables declared at the beginning of the procedure and this only takes 0.385 seconds to initialise whereas the entire proc takes 15 seconds.
On a separate query window, I copied the relevant query and substituted variables starting with 'proc_' to test speed and optimise, like so...
INSERT INTO temp_attribution (`attribute_type`, `domain`, `id`, `name`, `score`, `rank`, `partner_match`, `person_match`, `sponsor_match`, `date_match`)
SELECT 'Campaign' AS attribute_type, domain, id, name, score, (#proc_counter := #proc_counter + 1) AS rank,
partner_match, person_match, sponsor_match, date_match
FROM (
SELECT m_c.domain, m_c.campaign_id AS id, m_c.name, m_c.client_id, m_c.sent_date,
'VENDOR SPONSOR VALUE' AS invoice_sponsor, bs.sponsor AS campaign_sponsor,
'johnsmith#domain.com' AS invoice_email, aes_encrypt('johnsmith#domain.com', 'secret_key') as campaign_email,
if (m_c.client_id = m_c.client_id COLLATE latin1_general_ci, 'Yes', 'No') AS partner_match,
if (aes_encrypt('johnsmith#domain.com', 'secret_key'), 'Exact Email', 'Email Domain') AS person_match,
if (LOCATE(CONVERT(bs.sponsor USING utf8mb4), 'VENDOR SPONSOR VALUE') > 0, 'Sponsor',
if (CONVERT(bs.vendor USING utf8mb4) = 'VENDOR', 'Vendor', 'No') ) AS sponsor_match,
if (datediff('2016-10-14', m_c.sent_date) BETWEEN 0 AND 92, 'Within Three', 'Within Six') AS date_match,
(
if (m_c.client_id = m_c.client_id COLLATE latin1_general_ci, 45, 10) + 30 +
if (LOCATE(CONVERT(bs.sponsor USING utf8mb4), 'VENDOR SPONSOR VALUE') > 0, 10,
if (CONVERT(bs.vendor USING utf8mb4) = 'VENDOR', 5, 0) ) +
if (datediff('2016-10-14', m_c.sent_date) BETWEEN 0 AND 92, 15, 5)
) AS score
FROM campaign_table m_c
INNER JOIN recipient_table m_r ON m_c.domain = m_r.domain AND m_c.campaign_id = m_r.campaign_id
LEFT JOIN booking_sponsor bs ON m_c.domain = bs.domain AND m_c.campaign_id = bs.campaign_id
WHERE datediff('2016-10-14', m_c.sent_date) BETWEEN 0 AND 185
AND ( aes_encrypt('johnsmith#domain.com', 'secret_key') = m_r.email OR m_r.email_domain = 'domain.com' )
) T ORDER BY score DESC, sent_date DESC LIMIT 5;
Now, magically without doing anything else, the query runs within two seconds. How is that possible?
Figured it out. Some of the declared variable type was different compared to the column being compared, so I guess MySQL could not compare them in the most efficient way possible.
I have a rather complex DELETE query that I need to break down into batches, but I can't do that until I can determine how many rows would be deleted before performing the query. Here is the query that I want to get a count on:
DELETE parts, binaries FROM parts LEFT JOIN binaries ON binaries.ID = parts.binaryID LEFT JOIN releasenfo rn ON rn.binaryID = binaries.ID WHERE binaries.procstat IN (4, 6) AND (rn.binaryid IS NULL OR (rn.binaryid IS NOT NULL AND rn.nfo IS NOT NULL))) OR binaries.dateadded < '2013-01-04 22:01:17' - INTERVAL 36 HOUR;
Here is what I thought might work, but it's not valid:
Select COUNT(*) FROM (DELETE parts, binaries FROM parts LEFT JOIN binaries ON binaries.ID = parts.binaryID LEFT JOIN releasenfo rn ON rn.binaryID = binaries.ID WHERE binaries.procstat IN (4, 6) AND (rn.binaryid IS NULL OR (rn.binaryid IS NOT NULL AND rn.nfo IS NOT NULL))) OR binaries.dateadded < '2013-01-04 22:01:17' - INTERVAL 36 HOUR) AS countme;
Can someone please point me in the right direction?
To put this into context, I want to use it with the following php pseudo code:
$limit = 10000; // Do in batches to give user status
$totalRows = $db->query("Count query goes here");
while($rowsLeft > 0)
{
echo "$rowsLeft remaining to be deleted\n";
$db->query("DELETE ...... LIMIT ".$limit);
$rowsLeft -= $limit;
}
You can simply try this:
SELECT * FROM parts p
LEFT JOIN binaries b ON b.ID = p.binaryID AND b.procstat IN (4, 6)
LEFT JOIN releasenfo rn ON rn.binaryID = b.ID
WHERE rn.binaryid IS NULL OR (rn.binaryid IS NOT NULL AND rn.nfo IS NOT NULL) OR
b.dateadded < '2013-01-04 22:01:17' - INTERVAL 36 HOUR;
DELETE p, b FROM parts p
LEFT JOIN binaries b ON b.ID = p.binaryID AND b.procstat IN (4, 6)
LEFT JOIN releasenfo rn ON rn.binaryID = b.ID
WHERE rn.binaryid IS NULL OR (rn.binaryid IS NOT NULL AND rn.nfo IS NOT NULL) OR
b.dateadded < '2013-01-04 22:01:17' - INTERVAL 36 HOUR;
I want to:
if the value qty_out! = 0, then set qty_out = 0
Help me to display results
SELECT
SUBSTR(stok_control.tgl_faktur,9,2) AS 'tanggal',
SUBSTR(stok_control.tgl_faktur,6,2) AS 'bulan',
CONCAT(
(SELECT(
IFNULL(stok_control.faktur_beli,''))
)
,
(SELECT(
IFNULL(stok_control.faktur_jual,''))
)
) AS 'faktur',
bahan.id AS 'kode_bahan',
bahan.nm_bahan AS 'nama_bahan',
stok_control.qty_in AS 'masuk',
(
SELECT IF(stok_control.qty_out != '',0,0)
FROM
stok_control
WHERE
stok_control.faktur_beli !=''
) AS 'keluar'
FROM
stok_control
LEFT JOIN bahan ON stok_control.id_bahan=bahan.id
#eggyal, I have changed your code to be:
SELECT
SUBSTR(sc.tgl_faktur, 9, 2) AS 'tanggal',
SUBSTR(sc.tgl_faktur, 6, 2) AS 'bulan',
CONCAT(
IFNULL(sc.faktur_beli, ''),
IFNULL(sc.faktur_jual, '')
) AS 'faktur',
b.id AS 'kode_bahan',
b.nm_bahan AS 'nama_bahan',
(SELECT IFNULL(sc.qty_in,0)) AS 'masuk',
(SELECT
(
IF(faktur_beli <> '',
0,
(SELECT sc.qty_out)
)
)
) AS 'qty_out'
FROM
stok_control sc LEFT JOIN bahan b ON sc.id_bahan = b.id
I've tried to reference an existing but still error...
I want to:
if the value qty_out! = 0, then set qty_out = 0
Do you mean that you want qty_out to be 0 irrespective of its original value?
SELECT SUBSTR(sc.tgl_faktur, 9, 2) AS tanggal,
SUBSTR(sc.tgl_faktur, 6, 2) AS bulan,
CONCAT(
IFNULL(sc.faktur_beli, ''),
IFNULL(sc.faktur_jual, '')
) AS faktur,
b.id AS kode_bahan,
b.nm_bahan AS nama_bahan,
sc.qty_in AS masuk,
0 AS qty_out
FROM stok_control sc LEFT JOIN bahan b ON sc.id_bahan = b.id