The whole point is to get all artist records, and any matching records in 5 other tables.
There are maybe 50,000 artist records.
I plan on creating a view out of this statement and then using it to keep a sugarcrm db up to date.
Any help is hugely appreciated.
here is the SQL statement,
SELECT
/*foreign keys*/
band_profiles.id AS 'BP_ARTIST_ID',
bandsintown_artists.id AS 'BID_ARTIST_ID',
contacts.id AS 'CONTACTS_MGMT_ID',
facebook_stats.id AS 'FB_STATS_ID',
outbound_links.id AS'OUTB_LINKS_ID',
/*high level*/
band_profiles.name AS 'ACCOUNT_NAME',
band_profiles.description AS 'ACCOUNT_DESCRIPTION',
band_profiles.bandsintown_status AS 'BIT_STATUS',
band_profiles.created_at AS 'DR_CREATED_DATETIME',
band_profiles.updated_at AS 'DR_UPDATED_DATETIME',
/*account mgmt fields*/
contacts.description AS 'ACCOUNT_MGMT_CONTACT',
contacts.contact_type AS 'ACCOUNT_MGMT_TYPE',
contacts.custom_contact_type AS 'ACCOUNT_MGMT_TYPE_C',
/*account web & social*/
band_profiles.website_url AS 'WEBSITE_URL',
band_profiles.twitter_url AS 'ACCOUNT_TWITTER_URL',
band_profiles.facebook_url AS 'ACCOUNT_FACEBOOK_URL',
band_profiles.facebook_page_id AS 'ACCOUNT_FACEBOOK_ID',
band_profiles.facebook_like_count AS 'ACCOUNT_FACEBOOK_LIKES',
facebook_stats.like_count AS 'FB_TOTAL_LIKES',
facebook_stats.share_count AS 'FB_TOTAL_SHARES',
facebook_stats.comment_count AS 'FB_TOTAL_COMMENTS',
facebook_stats.click_count AS 'FB_TOTAL_CLICKS',
outbound_links.target_url AS 'OUTBOUND_LINK',
outbound_links.link_type AS 'OUTB_LINK_TYPE',
bandsintown_artists.facebook_tour_dates_url AS 'ACCOUNT_FB_TOUR_DATES'
FROM band_profiles
LEFT JOIN bandsintown_artists
ON band_profiles.id = bandsintown_artists.band_profile_id
LEFT JOIN contacts
ON band_profiles.id = contacts.id
LEFT JOIN facebook_stats
ON band_profiles.id = facebook_stats.band_profile_id
LEFT JOIN outbound_links
on band_profiles.id = outbound_links.band_profile_id
GROUP BY band_profiles.id
LIMIT 10
this is the error code I am getting
Error Code: 2013. Lost connection to MySQL server during query 600.000 sec
If you are the sysadm, you can increase the mysql timeout values. see http://www.rackspace.com/knowledge_center/article/how-to-change-the-mysql-timeout-on-a-server
Depending on the number of deleted entries in your db, you might see significant memory/time savings in this query by pruning deleted entries in your JOIN queries
also limit your final output by band_profiles.deleted = 0
LEFT JOIN bandsintown_artists
ON (band_profiles.id = bandsintown_artists.band_profile_id AND bandsintown_artists.deleted = 0)
LEFT JOIN contacts
ON (band_profiles.id = contacts.id AND contacts.deleted = 0)
LEFT JOIN facebook_stats
ON (band_profiles.id = facebook_stats.band_profile_id AND facebook_stats.deleted = 0)
LEFT JOIN outbound_links
on (band_profiles.id = outbound_links.band_profile_id AND outbound_links.deleted = 0)
WHERE band_profiles.deleted = 0
Related
I have a database with the following relationship:
relationship
These tables link a customer to an address:
Cliente = client
Clientenendereco = Many to many table between Endereco (table) and Cliente (table)
Endereco = address
Endereco_rua = address_street
Endereco_cidade = address_city
Endereco_estado = address_state
The cardinality is:
Cliente NxN Endereco
Endereco 1x1 Endereco_rua
Endereco_rua 1x1 Endereco_cidade
Endereco_cidade 1x1 Endereco_state
And the query I use is:
SELECT cli.cliente_id, cli.cliente_cpf, cli.cliente_nome, TIMESTAMPDIFF(YEAR,
cli.cliente_datanasc, CURDATE()) AS idade, ende.endereco_cep, ende.endereco_numero,
ben.beneficio_numero,
ben.beneficio_codigo, ben.beneficio_valor, ban.banco_cod, ab.banco_agencia_numero,
cbb.conta_banco_numero, ben.beneficio_margem_emprestimo, ben.beneficio_margem_cartao
FROM cliente AS cli
RIGHT JOIN telefonencliente AS tnc FORCE INDEX(`fk_telefone_has_cliente_cliente1_idx`)
ON tnc.cliente_id = cli.cliente_id
RIGHT JOIN telefone AS tel
ON tel.telefone_id = tnc.telefone_id
INNER JOIN beneficio AS ben
ON ben.cliente_id = cli.cliente_id
LEFT JOIN clientenendereco AS ce
ON ce.cliente_id = cli.cliente_id
LEFT JOIN endereco AS ende
ON ende.endereco_id = ce.endereco_id
LEFT JOIN endereco_rua AS er
ON er.endereco_rua_id = ende.endereco_rua_id
LEFT JOIN endereco_cidade AS ec
ON ec.endereco_cidade_id = er.cidade_id
LEFT JOIN endereco_estado AS ee
ON ee.endereco_estado_id = ec.estado_id
LEFT JOIN banco_agencia AS ab
ON ab.banco_agencia_id = ben.fk_banco_agencia_id
LEFT JOIN conta_banco AS cbb
ON cbb.conta_banco_id = ben.fk_conta_banco_id
LEFT JOIN banco AS ban
ON ban.banco_id = ben.fk_banco_id
WHERE ee.endereco_estado_id = 1 AND cli.cliente_id IS NOT NULL AND cli.cliente_id NOT
IN(SELECT ci.cliente_id FROM cliente_inativo AS ci WHERE ci.cliente_id = cli.cliente_id)
AND
cli.cliente_id NOT IN (SELECT ma.cliente_id FROM mailing_alocado AS ma WHERE
ma.cliente_id = cli.cliente_id) AND TIMESTAMPDIFF(YEAR, cli.cliente_datanasc,
CURDATE()) >=18 AND TIMESTAMPDIFF(YEAR, cli.cliente_datanasc, CURDATE()) <=68 AND
(ben.beneficio_datainicio BETWEEN '1990-01-01' AND '2007-12-31')
GROUP BY cli.cliente_id
LIMIT 20000
When I use "ee.endereco_estado_id = 1" the query takes 2 minutes to return the records
On the other hand if I remove "ee.endereco_estado_id = 1" the query takes 0.068 seconds
Here with Explain:
With "ee.endereco_estado_id = 1"
Without "ee.endereco_estado_id = 1"
I noticed that without the "ee.endereco_estado_id = 1" It starts with the table "ben", which is good, however, with the "ee.endereco_estado_id = 1" It starts with the table "ee" which takes a long time.
I don't know what to do anymore, because I need the records that are in state number 1, please help me I don't know what I can do or where is my error.
Tables:
endereco ~ 3milions rows
endereco_rua ~ 4milions rows
endereco_cidade ~ 21thousand rows
endereco_estado ~ 26 rows
Let's try turning the query inside out to avoid the GROUP BY, which is probably causing some of the performance hangup.
SELECT ... ((lots of columns))
FROM (
SELECT cliente_id
FROM client
((with the minimal JOINs and WHEREs to get the ids))
ORDER BY .. LIMIT .. -- if needed
) AS ids
JOIN client AS cli ON cli.cliente_id = ids.cliente_id
JOIN (( all the other tables and WHEREs ))
ORDER BY .. -- if needed; note: the inner sort will be lost
I assume you indexes on the various columns in the WHERE clauses? If not, please provide SHOW CREATE TABLE and EXPLAIN SELECT...
Please do not use LEFT when the 'right' table is not optional. Example: ee.endereco_estado_id = 1
Please say which JOINs are 1:1 vs 1:many vs many:1. I need to understand whether the joining "explodes" the number of rows (only to have the GROUP BY implode them). My suggested rewrite assumes that is happening.
my sql query is normally executed in mysql workbench (despite the 15-second wait), but when I try to run the query from the side of another application (phpmyadmin, etc...) it gives me several errors, such as:
Timeout
Error in SELECT clause: expression near '"'.
Missing FROM clause.
Below is the query code:
SELECT investors_positivador.cod_cliente, investors_saldo_financeiro.nome_cliente, investors_base_assessores.squad, investors_base_assessores.nome_investor, investors_saldo_financeiro.saldo_d0,
SUM(investors_posicao_geral.financeiro) AS vencimentos_ate_data, investors_posicao_geral.vencimento,
CASE WHEN investors_diversificacao.produto = "FUNDO" THEN
ROUND(SUM( IF( investors_diversificacao.produto = "FUNDO", investors_diversificacao.net / 6, 0 ) ), 2)
ELSE
"0,00"
END AS fundos_ate_data
,investors_guia_fundos.liquidez_total
,investors_positivador.contatar_liquidity_map
,investors_positivador.id
FROM investors_positivador INNER JOIN
investors_saldo_financeiro ON investors_positivador.cod_cliente = investors_saldo_financeiro.cod_cliente LEFT OUTER JOIN
investors_base_assessores ON investors_positivador.cod_assessor = investors_base_assessores.cod_assessor LEFT OUTER JOIN
investors_posicao_geral ON investors_positivador.cod_cliente = investors_posicao_geral.cod_cliente
LEFT OUTER JOIN investors_diversificacao ON investors_diversificacao.cod_cliente = investors_positivador.cod_cliente AND investors_diversificacao.net != "0,00"
LEFT OUTER JOIN investors_guia_fundos ON investors_guia_fundos.cnpj = investors_diversificacao.cnpj
WHERE investors_base_assessores.nome_investor = "Daiane Costa" AND investors_saldo_financeiro.saldo_d0 > 0 AND investors_posicao_geral.financeiro > 0
GROUP BY cod_cliente
For the waiting time, you could try wrapping your query in a transaction. That may help in optimizing your query
I have created view which has more than 20 thousand records and executing a query on same view, its taking too long time to fetch only 20 records using limit.
If I removed ORDER BY clause then same query gives result within fraction of sec.
Even I tried that, applied order by condition on view so that I can execute query without ORDER BY clause but still query taking 1.1 min to 1.8 min to return result.
Here is the query :
SELECT full_name, project_no, customer_name, description, status, comments,
FROM (view_project)
WHERE labor_code IS NOT NULL
LIMIT 20
Definition of view :
select a.id AS id,a.tms_id AS tms_id,a.date AS date_from,a.hour AS hour,a.hour_ot AS hour_ot,a.hour_dt AS hour_dt,a.function AS function,a.billable AS billable,a.qb_class AS qb_class,a.comments AS comments,b.extra_fields AS extra_fields,a.labor_code AS labor_code,a.cost AS cost,a.bill AS bill,a.pr_cost AS pr_cost,a.pr_bill AS pr_bill,b.user_id AS user_id,b.date_from AS date_from_b,b.date_to AS date_to,c.full_name AS full_name,c.username AS username,d.name AS name,d.project_no AS project_no,d.id AS project_id,e.description AS description,f.name AS customer_name,concat(b.date_from,'|',b.date_to) AS week_range,a.data_row AS data_row,if((isnull(g.tms_row_status) or (g.tms_row_status = '')),'No Status',g.tms_row_status) AS status,ifnull(h.submit_entity,'') AS submit_entity,a.tms_row_uuid AS tms_row_uuid from (((((((labor_time_tracking a left join timesheet b on((a.tms_id = b.id))) left join user c on((b.user_id = c.id))) left join project d on((a.project_id = d.id))) left join supplier f on((d.customer_id = f.id))) left join item e on((convert(a.labor_code using utf8) = e.uuid))) left join labor_time_tracking_rows g on((a.tms_row_uuid = g.tms_row_uuid))) left join timesheet_submission h on(((a.tms_id = h.tms_id) and (a.project_id = h.project_id) and (a.data_row = h.tms_row_id)))) order by b.id desc
I currently have this working using a Sub-query, but as the DB grows this will become HUGELY inefficient. I'm wondering if there is a more efficient way to do what I need to do without sub-queries?
I need to have my final output look like so:
Question, Answer, Responses, Charts included in Response Count
Did this work?, N/A, 26, 30
Did this work?, Yes, 4, 30
This is my current query:
SELECT
bq_text,
ba_a,
bq_id,
COUNT(ba_a) AS ba_aC,
(SELECT COUNT(*) FROM board_done_sheet WHERE sd_b_id = bs.bs_id AND sd_sub = 1) AS sd_chartnumC
FROM board_done_sheet AS sh
LEFT JOIN board_done bd
ON (bd.bd_id = sh.sd_bd_id)
LEFT JOIN boardsubs bs
ON (bd.bd_b_id = bs.bs_id)
LEFT JOIN b_q_answers ba
ON (sh.sd_s_id = ba.ba_s_id)
LEFT JOIN bsquestions bq
ON (bq.bq_id = ba.ba_q_id)
LEFT JOIN multiples m
ON (ba.ba_m_id = m.m_id)
LEFT JOIN users u
ON (u.us_id = bd.bd_d_id)
LEFT JOIN profiles p
ON (p.p_u_id = bd.bd_d_id)
LEFT JOIN users rev
ON (rev.us_id = bd.bd_rev)
WHERE sd_sub = '1' AND bq_text <> 'Date' AND bq_id = 380
GROUP BY bs_id, bq_text, ba_a
That works perfectly, the problem is it has to use sub-queries which as time goes by will get less efficient.
I'm just wondering if there is a better more efficient way to do that summed field without it.
Presumably the subquery you're concerned about is the one in your toplevel SELECT.
That is easy to refactor so it won't get repeated.
Just JOIN it to the rest of the table. You'll want this sort of thing:
SELECT
bq_text, ...
COUNT(ba_a) AS ba_aC,
countup.countup AS sd_chartnumC
FROM board_done_sheet AS sh
LEFT JOIN board_done bd
ON (bd.bd_id = sh.sd_bd_id)
...
LEFT JOIN users rev
ON (rev.us_id = bd.bd_rev)
JOIN (
SELECT COUNT(*) AS countup , sd_b_id
FROM board_done_sheet
WHERE sd_sub = 1
GROUP BY sd_b_id
) AS countup ON countup.sd_b_id = bs.bs_id
WHERE sd_sub = '1'
AND bq_text <> 'Date'
AND bq_id = 380
GROUP BY bs_id, bq_text, ba_a
The countup subquery generates a summary table of counts and ids, and then joins it to the other tables.
A JOIN cascade of this complexity may become inefficient for other reasons as your table grows if you don't structure your indexes correctly.
I am trying to run that filters certain data from the result, including making sure that none of the records that have 'A_null_choice' as a value for it's moduleName.
$sql = "SELECT am.moduleID, m.moduleName, am.type, s.regNum, s.award
FROM tbl_awardModules am
LEFT OUTER JOIN tbl_module m ON am.moduleID = m.moduleID
INNER JOIN tbl_awardLevels al ON am.awardLevelID = al.awardLevelID
INNER JOIN tbl_award a ON al.awardID = a.awardID
INNER JOIN tbl_student s ON a.awardID = s.award
WHERE am.type <> 'C' AND regNum = '$student_id' AND m.moduleName NOT LIKE 'A_null_Choice'";
The rest of the query runs fine and there are no errors etc, it just keeps showing the records with that value.
Any insights will be greatly appreciated, thanks