I have the following SQL but its execution this very slow, takes about 45 seconds, the table has 15 million record, how can I improve?
SELECT A.*, B.ESPECIE
FROM
(
SELECT
A.CODIGO_DOCUMENTO,
A.DOC_SERIE,A.DATA_EMISSAO,
A.DOC_NUMERO,
A.CF_NOME,
A.CF_SRF,
A.TOTAL_DOCUMENTO,
A.DOC_MODELO
FROM MOVIMENTO A
WHERE
A.CODIGO_EMPRESA = 1
AND A.CODIGO_FILIAL = 5
AND A.DOC_TIPO_MOVIMENTO = 1
AND A.DOC_MODELO IN ('65','55')
AND (A.CF_NOME LIKE '%TEXT_SEARCH%'
OR A.CF_CODIGO LIKE 'TEXT_SEARCH%'
OR A.CF_SRF LIKE 'TEXT_SEARCH%'
OR A.DOC_SERIE LIKE 'TEXT_SEARCH%'
OR A.DOC_NUMERO LIKE 'TEXT_SEARCH%')
ORDER BY A.DATA_EMISSAO DESC , A.CODIGO_DOCUMENTO DESC
LIMIT 0, 100
) A
LEFT JOIN MODELODOCUMENTOFISCAL B ON A.DOC_MODELO = B.CODMODELO
For this query, I would start with an index on MOVIMENTO(CODIGO_EMPRESA, CODIGO_FILIAL, DOC_MODELO) and MODELODOCUMENTOFISCAL(CODMODELO).
That should speed the query.
If it doesn't you may need to consider a full text search to handle the LIKE clauses. I do note that you only have a wildcard at the beginning of one of the patterns. Is that intentional?
Related
I running into some time issues using my simple select:
SELECT *
FROM ltowert
WHERE bat = 3
AND id >= (SELECT id
FROM ltowert
WHERE bat = 3 AND ident = 'v0'
ORDER BY id DESC
LIMIT 1)
ORDER BY ident;
It takes nearly 12 seconds (depending on the index, etc..)
If I run the subquery (0.00075 sec) and put the result in the statement:
SELECT *
FROM ltowert
WHERE bat = 3 AND id >= 20979399
ORDER BY ident;
it runs in just 0.00095 sec, in addition: 0.0017 sec
So it seems, using the subquery avoid the Optimizer to use the index ?
How can I fix it and get quicker results ?
Thanks for any answers.
JR
I test your proposal, but I run into a problem of understanding. Maybe, it's my issue:
Your idea works, but the result is wrong: Result is only 1 row, but I expect 11 rows. I expect all row from l1, where id >= l2.id.
So I changed to:
SELECT *
FROM ltowert l1
INNER JOIN (
SELECT id
FROM ltowert
WHERE bat = 3 AND ident = 'v0'
order by id desc limit 1
) as l2 on l1.id>=l2.id
order by l1.ident;
This return much rows. l1.bat = 0 which is not the request.
Change to:
A. ") as l2 on l1.id>=l2.id AND l1.bat = 3"
or
B. WHERE l1.bat = 3
runs into a timeout Error 2013.
What is my mistake ?
Additional, due to my 2 step tests, I do a test using a variable:
SELECT #ref_id:=id FROM ltowert WHERE bat = 3 AND ident = 'v0' order by id desc limit 1;
SELECT * FROM ltowert l1 where bat = 3 AND l1.id >= #ref_id order by l1.ident;
It is fast, and give the right results.
Is there a disadvantage to this usage? (I know: 2 separate statements were not maintenance friendly)
You need both of these:
INDEX(bat, ident, id)
INDEX(bat, id)
If they don't suffice, please provide
SHOW CREATE TABLE ltowert;
EXPLAIN SELECT ...;
so this query took 40 seconds to load, any way to speed it up? the framework that I used are Laravel and I need to show this data using ajax and this query took 40 seconds to load, I need to speed it up, I know the r.10 - r.13 the one that caused it I tested it out and found out that 3 are taking almost 20 secs to load
SELECT
`ms_paket_berjalan`.`kdPaketBerjalan` as `r1`,
`ms_pelanggan`.`namaLengkap` as `r2`,
`ms_rtrwnet`.`kabupaten` as `r3`,
`ms_paket_berjalan`.`jenis_pelanggan` as `r4`,
`ms_rtrwnet`.`nama` as `r5`,
DATE_FORMAT(ms_paket_berjalan.tglpasang, "%d/%m/%Y") as r6,
`ms_paket_berjalan`.`noTelp` as `r7`,
`ms_paket_berjalan`.`terverifikasi` as `r8`,
`ms_paket_berjalan`.`hargaPaket` as `r9`,
(
SELECT
COUNT(verifikasi_pelanggan.kdVerifikasi)
FROM
verifikasi_pelanggan
WHERE
verifikasi_pelanggan.kdPaketBerjalan = r1
) as r10,
(
SELECT
verifikasi_pelanggan.statusVerifikasi
FROM
verifikasi_pelanggan
WHERE
verifikasi_pelanggan.kdPaketBerjalan = r1
ORDER BY
kdVerifikasi DESC
LIMIT
1
) as r11,
(
SELECT
verifikasi_pelanggan.tanggalBa
FROM
verifikasi_pelanggan
WHERE
verifikasi_pelanggan.kdPaketBerjalan = r1
ORDER BY
kdVerifikasi DESC
LIMIT
1
) as r12,
(
SELECT
COUNT(verifikasi_pelanggan.noBa)
FROM
verifikasi_pelanggan
WHERE
verifikasi_pelanggan.kdPaketBerjalan = r1
) as r13
FROM
`ms_paket_berjalan`
inner join `ms_pelanggan` on `ms_pelanggan`.`kdPelanggan` = `ms_paket_berjalan`.`kdPelanggan`
inner join `ms_rtrwnet` on `ms_rtrwnet`.`kdRtRwNet` = `ms_pelanggan`.`kdRtRwNet`
inner join `tr_resellerisp` on `tr_resellerisp`.`kdRtRwNet` = `ms_rtrwnet`.`kdRtRwNet`
left join `verifikasi_pelanggan` on `ms_paket_berjalan`.`kdPaketBerjalan` = `verifikasi_pelanggan`.`kdPaketBerjalan`
WHERE
`tr_resellerisp`.`kdIsp` = '11'
and `ms_paket_berjalan`.`break` = '0'
and `ms_paket_berjalan`.`deleted` = '0'
and `ms_pelanggan`.`deleted` = '0'
and `ms_pelanggan`.`suspended` = '0'
and `tr_resellerisp`.`persetujuan` = '1'
and `ms_paket_berjalan`.`terverifikasi` = '0'
and (
`ms_rtrwnet`.`nama` like '%%'
or `ms_pelanggan`.`namaLengkap` like '%%'
)
GROUP BY
`ms_paket_berjalan`.`kdPaketBerjalan`
ORDER BY
`ms_paket_berjalan`.`tglpasang` DESC
LIMIT
20 offset 0
If you don't have an index on your verifikasi_pelanggan table, you need to.
To add an index run something like:
ALTER TABLE `verifikasi_pelanggan`
ADD INDEX `index4` (`kdPaketBerjalan` ASC, `kdVerifikasi` DESC)
Just do that once and things should speed up.
EDIT: Your query without an index will scan the entire verifikasi_pelanggan multiple times for every result. Since it has an ORDER BY clause, it isn't even limited to 20, it will need to scan for every result and THEN sort and then limit to 20 results.
An INDEX essentially pre-sorts the table. It can find matches on the indexed columns almost instantly, rather than scanning and reading every row to see if it is a match.
Just indexing on kdPaketBerjalan should help a lot, but since you order by kdVerifikasi it looks like you might have multiple matches on kdPaketBerjalan, which would require sorting. Including kdVerifikasi in the index will find it instantly.
ayetler Table = 6500 rows
fihristayet Table = 13279 rows
Sql query :
SELECT ayetler.ayet_tr
FROM fihristayet,ayetler
WHERE (fihristayet.fihristAyet_ayet=ayetler.ayet_no AND
fihristayet.fihristAyet_sure_id=ayetler.sure_id)
AND fihristayet.fihristAyet_fihrist_id=354
I have 15 rows return. Why records arrive after 40 seconds
First, learn to use proper, explicit, standard JOIN syntax. Never use commas in the FROM clause. So, your query should look like this:
SELECT a.ayet_tr
FROM fihristayet f JOIN
ayetler a
ON f.fihristAyet_ayet = a.ayet_no AND
f.fihristAyet_sure_id = a.sure_id
WHERE f.fihristAyet_fihrist_id = 354;
Then you want indexes on fihristayet(fihristAyet_fihrist_id, fihristAyet_ayet, fihristAyet_sure_id) and ayetler(ayet_no, sure_id).
This query takes 18 seconds
SELECT `wd`.`week` AS `start_week`, `wd`.`hold_code`, COUNT(wd.hold_code) AS hold_code_count
FROM `weekly_data` AS `wd`
JOIN aol_reporting_hold_codes hc ON hc.hold_code = wd.hold_code AND chart = 'GR'
WHERE `wd`.`days` <= 6
AND `wd`.`hold_code` IS NOT NULL
AND NOT `wd`.`hold_code` = ''
AND `wd`.`week` >= '201717'
AND `wd`.`itemgroup` IN ('BOTDTO', 'BOTDWG', 'C&FORG', 'C&FOTO', 'MF-SUB', 'MI-SUB', 'PROPRI', 'PROPTO', 'STRSTO', 'STRSUB')
AND `production_type` = 2
AND `contract` = "1234"
AND `project` = 8
GROUP BY `start_week`, `wd`.`hold_code`
This query takes 4 seconds
SELECT `wd`.`week` AS `start_week`, `wd`.`hold_code`, COUNT(wd.hold_code) AS hold_code_count
FROM `weekly_data` AS `wd`
JOIN aol_reporting_hold_codes hc ON hc.hold_code = wd.hold_code AND chart = 'GR'
WHERE `wd`.`days` <= 6
AND `wd`.`hold_code` IS NOT NULL
AND NOT `wd`.`hold_code` = ''
AND `wd`.`week` >= '201717'
AND `wd`.`itemgroup` IN ('BOTDWG', 'C&FORG', 'C&FOTO', 'MF-SUB', 'MI-SUB', 'PROPRI', 'PROPTO', 'STRSTO', 'STRSUB')
AND `production_type` = 2
AND `contract` = "1234"
AND `project` = 8
GROUP BY `start_week`, `wd`.`hold_code`
All I have done is removed one item from the IN clause. I can remove any one of the items. It runs in 4 seconds as long as there are 9 items or less. It takes 18 seconds to run as soon as I increase to 10 items.
I thought MySQL limited length of command by size i.e. 1MB
More than just the EXPLAIN, use EXPLAIN FORMAT=JSON and get the "Optimizer trace" for the query. I suspect the length of the IN leads to picking a different query plan.
There is virtually no limit to the number of items in IN. I have seen as many as 70K.
That aside, you may be able to speed up even the 4-sec version...
I suggest having this index. Grrr... I can't tell which columns are in which tables. So, if these are all in one table, then make such an index:
INDEX(production_type, contract, project) -- in any order
If those are all in wd, then tack on a 4th column - any of week, itemgroup, days.
Be cautious about COUNT(wd.hold_code).
COUNT(x) checks x for being non-NULL; is that what you want? If not, then simply say COUNT(*).
When JOINing, then GROUP BY, you get an "explode-implode". The number of intermediate rows is big; that is when the COUNT is performed.
It seems wrong to both COUNT(hold_code) and GROUP BY hold_code. What are you trying to do?
For further discussion, please provide SHOW CREATE TABLE and EXPLAIN.
Please note MySql IN clause limit is established with max_allowed_packet value. You may check with NOT IN if results are faster. Also I suggest put values to be checked with IN clause under a buffer string instead of comma separated values and then give a try.
The following query is overloading my system. It seems to be a problem with the rand(). I have seen other posts dealing with similar issues, but can't quite get them working in this problem. It is being run on a 10M+ row table. I know that the order by rand() is the issue, but after reading there seems to be an issue of the autoincrement (items.ID) increments by 2 not 1.
SELECT stores.phone, stores.storeID, stores.name, stores.ZIP,
stores.state,stores.city, storeID, GEOCODES.lon, GEOCODES.lat
FROM items
LEFT JOIN stores on stores.storeID = items.store_ID
LEFT JOIN GEOCODES on GEOCODES.address = CONCAT(stores.address1,', ',stores.ZIP)
WHERE stores.phone IS NOT NULL
GROUP BY items.store_ID
ORDER BY RAND( )
LIMIT 200
The other article that I was trying to follow was How can i optimize MySQL's ORDER BY RAND() function?, but can't seem to figure out how to adapt it to this query. Please note that this is done in PHP.
if I were you I would LIMIT first and then ORDER BY RAND() on the limited query.. that way you arent pulling everything out and randomizing it.. I have used this exact method to speed up my queries exponentially
SELECT *
FROM
( SELECT stores.phone, stores.storeID, stores.name, stores.ZIP,
stores.state,stores.city, storeID, GEOCODES.lon, GEOCODES.lat
FROM items
LEFT JOIN stores on stores.storeID = items.store_ID
LEFT JOIN GEOCODES on GEOCODES.address = CONCAT(stores.address1,', ',stores.ZIP)
WHERE stores.phone IS NOT NULL
GROUP BY items.store_ID
LIMIT 200
) t
ORDER BY RAND( )
Some proof:
CREATE table digits as (-- a digit table with 1 million rows)
1000000 row(s) affected Records: 1000000 Duplicates: 0 Warnings: 0
1.869 sec
SELECT * FROM digits ORDER BY RAND() LIMIT 200
200 row(s) returned
0.465 sec / 0.000 sec
SELECT * FROM (SELECT * FROM digits LIMIT 200)t ORDER BY RAND()
200 row(s) returned
0.000 sec / 0.000 sec
Using RAND() in your query has serious performance implications, avoiding it will speed up your query a lot.
Also since you're using php, randomizing the ordering using shuffle() w/ php may be a significantly quicker alternative to mysql.
See: http://php.net/manual/en/function.shuffle.php