Following is the query that i am using:
SELECT *
FROM (
SELECT
(
CASE WHEN product_name like '%word1%' THEN 1 ELSE 0 END +
CASE WHEN product_name like '%word2%' THEN 1 ELSE 0 END +
CASE WHEN product_name like '%word3%' THEN 1 ELSE 0 END
) AS numMatches
FROM products as p
) as derived
WHERE numMatches > 0
ORDER BY numMatches DESC
LIMIT 30,10
I added an index (BTREE) on product_name, there are 3 million records in the column, the query is executing in 3-5 seconds.
Explain says 'Using where; Using filesort' so i can figure out its not using the index.
No, it's not using the index.
For that, you would have to compare with 'word1%', 'word2%', etc.. but doesn't work when you use the joker at the beginning.
But, If your mysql version is relatively modern you can use fulltext indexes, which would serve for your query.
https://dev.mysql.com/doc/refman/5.6/en/innodb-fulltext-index.html
Related
I have got a SQL query that I tried to optimize and I could reduce through various means the time from over 5 seconds to about 1.3 seconds, but no further. I was wondering if anyone would be able to suggest further improvements.
The Explain diagram shows a full scan:
explain diagram
The Explain table will give you more details:
explain tabular
The query is simplified and shown below - just for reference, I'm using MySQL 5.6
select * from (
select
#row_num := if(#yacht_id = yacht_id and #charter_type = charter_type and #start_base_id = start_base_id and #end_base_id = end_base_id, #row_num +1, 1) as row_number,
#yacht_id := yacht_id as yacht_id,
#charter_type := charter_type as charter_type,
#start_base_id := start_base_id as start_base_id,
#end_base_id := end_base_id as end_base_id,
model, offer_type, instant, rating, reviews, loa, berths, cabins, currency, list_price, list_price_per_day,
discount, client_price, client_price_per_day, days, date_from, date_to, start_base_city, end_base_city, start_base_country, end_base_country,
service_binary, product_id, ext_yacht_id, main_image_url
from (
select
offer.yacht_id, offer.charter_type, yacht.model, offer.offer_type, offer.instant, yacht.rating, yacht.reviews, yacht.loa,
yacht.berths, yacht.cabins, offer.currency, offer.list_price, offer.list_price_per_day,
offer.discount, offer.client_price, offer.client_price_per_day, offer.days, date_from, date_to,
offer.start_base_city, offer.end_base_city, offer.start_base_country, offer.end_base_country,
offer.service_binary, offer.product_id, offer.start_base_id, offer.end_base_id,
yacht.ext_yacht_id, yacht.main_image_url
from website_offer as offer
join website_yacht as yacht
on offer.yacht_id = yacht.yacht_id,
(select #yacht_id:='') as init
where date_from > CURDATE()
and date_to <= CURDATE() + INTERVAL 3 MONTH
and days = 7
order by offer.yacht_id, charter_type, start_base_id, end_base_id, list_price_per_day asc, discount desc
) as filtered_offers
) as offers
where row_number=1;
Thanks,
goppi
UPDATE
I had to abandon some performance improvements and replaced the original select with the new one. The select query is actually dynamically built by the backend based on which filter criteria are set. As such the where clause of the most inner select can expland quite a lot. However, this is the default select if no filter is set and is the version that takes significantly longer than 1 sec.
explain in text form - doesn't come out pretty as I couldn't figure out how to format a table, but here it is:
1 PRIMARY ref <auto_key0> <auto_key0> 9 const 10
2 DERIVED ALL 385967
3 DERIVED system 1 Using filesort
3 DERIVED offer ref idx_yachtid,idx_search,idx_dates idx_dates 5 const 385967 Using index condition; Using where
3 DERIVED yacht eq_ref PRIMARY,id_UNIQUE PRIMARY 4 yachtcharter.offer.yacht_id 1
4 DERIVED No tables used
Sub selects are never great,
You should sign up here: https://www.eversql.com/
Run that and it will give you all the right indexes and optimsiations you need for this query.
There's still some optimization you can use. Considering the subquery returns 5000 rows only you could use an index for it.
First rephrase the predicate as:
select *
from website_offer
where date_from >= CURDATE() + INTERVAL 1 DAY -- rephrased here
and date(date_to) <= CURDATE() + INTERVAL 3 MONTH
and days = 7
order by yacht_id, charter_type, list_price_per_day asc, discount desc
limit 5000
Then, if you add the following index the performance could improve:
create index ix1 on website_offer (days, date_from, date_to);
I try to create a view which join from 4 tables (tb_user is 200 row, tb_transaction is 250.000 row, tb_transaction_detail is 250.000 row, tb_ms_location is 50 row),
when i render with datatables serverside, it's take 13 secons. even when I filtering it.
I don't know why it's take too long...
here my sql query
CREATE VIEW `vw_cashback` AS
SELECT
`tb_user`.`nik` AS `nik`,
`tb_user`.`full_name` AS `nama`,
`tb_ms_location`.`location_name` AS `lokasi`,
`tb_transaction`.`date_transaction` AS `tanggal_setor`,
sum(CASE WHEN `tb_transaction_detail`.`vehicle_type`=1 THEN 1 ELSE 0 END) AS `mobil`,
sum(CASE WHEN `tb_transaction_detail`.`vehicle_type`=2 THEN 1 ELSE 0 END) AS `motor`,
sum(CASE WHEN `tb_transaction_detail`.`vehicle_type`=3 THEN 1 ELSE 0 END) AS `truck`,
sum(CASE WHEN `tb_transaction_detail`.`vehicle_type`=4 THEN 1 ELSE 0 END) AS `speda`,
sum(`tb_transaction_detail`.`total`) AS `total_global`,
(sum(`tb_transaction_detail`.`total`) * 0.8) AS `total_user`,
(sum(`tb_transaction_detail`.`total`) * 0.2) AS `total_tgr`,
((sum(`tb_transaction_detail`.`total`) * 0.2) / 2) AS `total_cashback`,
(curdate() - cast(`tb_user`.`created_at` AS date)) AS `status`
FROM `tb_user`
JOIN `tb_transaction` ON `tb_user`.`id` = `tb_transaction`.`user_id`
JOIN `tb_transaction_detail` ON `tb_transaction`.`id` = `tb_transaction_detail`.`transaction_id`
JOIN `tb_ms_location` ON `tb_ms_location`.`id` = `tb_transaction`.`location_id`
GROUP BY
`tb_user`.`id`,
`tb_transaction`.`date_transaction`,
`tb_user`.`nik`,
`tb_user`.`full_name`,
`tb_user`.`created_at`,
`tb_ms_location`.`location_name`
thanks
The unfiltered query must be slow, because it takes all records from all tables, joins and aggregates them.
But you say the view is still slow when you filter. The question is: How do you filter? As you are aggregating by user, location and transaction date, it should be one of these. However, you don't have the user ID or the transaction ID in your result list. This doesn't feel natural and I'd suggest you add them, so a query like
select * from vw_cashback where user_id = 5
or
select * from vw_cashback where transaction_id = 12345
would be possible.
As is, you'd have to filter by location name or user nik / name. So if you want it thus, then create Indexes for the lookup:
CREATE idx_location_name ON tb_ms_location(location_name, id)
CREATE idx_user_name ON tb_user(full_name, id)
CREATE idx_user_nik ON tb_user(nik, id)
The latter two can even be turned into covering indexs (i.e. indexes containing all columns used in the query) that may still speed up the process:
CREATE idx_user_name ON tb_user(nik, id, full_name, created_at);
CREATE idx_user_nik ON tb_user(full_name, id, nik, created_at);
As for the access via index, you also may want covering indexes:
CREATE idx_location_id ON tb_ms_location(id, location_name)
CREATE idx_user_id ON tb_user(id, nik, full_name, created_at);
I have the following SQL but its execution this very slow, takes about 45 seconds, the table has 15 million record, how can I improve?
SELECT A.*, B.ESPECIE
FROM
(
SELECT
A.CODIGO_DOCUMENTO,
A.DOC_SERIE,A.DATA_EMISSAO,
A.DOC_NUMERO,
A.CF_NOME,
A.CF_SRF,
A.TOTAL_DOCUMENTO,
A.DOC_MODELO
FROM MOVIMENTO A
WHERE
A.CODIGO_EMPRESA = 1
AND A.CODIGO_FILIAL = 5
AND A.DOC_TIPO_MOVIMENTO = 1
AND A.DOC_MODELO IN ('65','55')
AND (A.CF_NOME LIKE '%TEXT_SEARCH%'
OR A.CF_CODIGO LIKE 'TEXT_SEARCH%'
OR A.CF_SRF LIKE 'TEXT_SEARCH%'
OR A.DOC_SERIE LIKE 'TEXT_SEARCH%'
OR A.DOC_NUMERO LIKE 'TEXT_SEARCH%')
ORDER BY A.DATA_EMISSAO DESC , A.CODIGO_DOCUMENTO DESC
LIMIT 0, 100
) A
LEFT JOIN MODELODOCUMENTOFISCAL B ON A.DOC_MODELO = B.CODMODELO
For this query, I would start with an index on MOVIMENTO(CODIGO_EMPRESA, CODIGO_FILIAL, DOC_MODELO) and MODELODOCUMENTOFISCAL(CODMODELO).
That should speed the query.
If it doesn't you may need to consider a full text search to handle the LIKE clauses. I do note that you only have a wildcard at the beginning of one of the patterns. Is that intentional?
I have one table for User detail in MySql with about 500000 records in it. I have also created fulltext index on firstname, lastname field on this table. but when I am trying to search any single latter/alphabet (e.g. a to z, single character), it is responding very slow in first time. It's taking about 5-6 seconds to respond. after that, it's come down to 800 milliseconds. EXPLAIN command seems ok as It shows "fulltext" in type column, but I couldn't find why it is reacting very slow.
my query is looks like as follows.
SELECT SQL_NO_CACHE usr.id, usr.uname, ifnull(usr.fullname,'') fullname,
ifnull(ct.City, '') city,
MATCH(usr.fname,usr.lname) AGAINST('a*' IN BOOLEAN MODE) ordfld
FROM usertable usr
LEFT JOIN citymas ct ON ct.CityID = upm.CityID
WHERE usr.UserStatus IN(10,11)
AND usr.id <> 1
AND MATCH(usr.fname,usr.lname) AGAINST('a*' IN BOOLEAN MODE) > 0
ORDER BY ( CASE WHEN usr.fullname = 'a' THEN 1
WHEN usr.fname rlike 'a%' THEN 2
WHEN usr.lname LIKE 'a%' THEN 3
WHEN usr.fname like '%a' THEN 6
WHEN usr.lname LIKE '%a' THEN 7
WHEN usr.fullname LIKE '%a%' THEN 8
ELSE 10 END ),
ordfld DESC,
( CASE WHEN ifnull(usr.cityid,0) = 234 THEN '0' ELSE '1' END ), usr.fullname
LIMIT 20
and explain show me following
1, 'SIMPLE', 'usr', 'fulltext', 'PRIMARY,IX_usertable_fname_lname', 'IX_usertable_fname_lname', 0, NULL , 1, 'Using where; Using filesort'
1, 'SIMPLE', 'ct' , 'eq_ref' , 'PRIMARY' , 'PRIMARY' , 3, 'usr.cityid', 1, NULL
above query is taking too much time, it is responding between 800-900ms.
Any guess?
EDIT :
does ft_min_word_len matter? when I changed it in my localhost and rebuild index again, same query returns within 500ms. If I would like to change this on Amazon RDS, How Do I do this?
I'm struggling to optimise this SQL code, can someone please help? Ideally I need the data outputted exactly the same to avoid any php coding.
I basically want to be able to group tariff_freemins by a set range. The below code works but it takes over 2 seconds to run which for a website isn't really quick enough. Is there an optimised version of the code. The table has an index on tariff_freemins setup.
SELECT 'Free Mins' as category,
'tariff_freemins' as fieldname,
t.range as `element`,
count(*) as phone_count
from ( select deal_count,
case
when tariff_freemins between 0 and 200 then 'a0TO200'
when tariff_freemins between 200 and 400 then 'b200TO400'
when tariff_freemins between 400 and 600 then 'c400TO600'
when tariff_freemins between 600 and 1000 then 'd600TO1000'
when tariff_freemins between 1000 and 2000 then 'e1000TO2000'
when tariff_freemins >2001 then 'hUnlimited'
end as `range`
from options
) t group by t.range
The output should be something like the below:
"category" "fieldname" "element" "phone_count"
"Data Allowance" "tariff_dataallowance" "a0TO1" "289716"
"Data Allowance" "tariff_dataallowance" "b1TO2" "64472"
"Data Allowance" "tariff_dataallowance" "c2TO5" "114685"
"Data Allowance" "tariff_dataallowance" "d5TO11" "33305"
"Data Allowance" "tariff_dataallowance" "e11TO20" "36798"
"Data Allowance" "tariff_dataallowance" "f20TO50" "5839"
"Data Allowance" "tariff_dataallowance" "hUnlimited" "51114"
UPDATE ////////////////////
Secondly, I'm using many of the above mySQL queries to generate a master table. Each of these have separate group by's and are joined using a UNION ALL (See below). As you can see there is a where clause which states "Where type = 'Contract'". I can only assume there is a way to using the same filtered options table on all of the group by queries? How would this look? Is there a better way to optimise this? Thanks!
SELECT 'Phone Cost' as category,'offer_phonecost' as fieldname, offer_phonecost_range as `element`, count(*) as phone_count from
options
WHERE 1 AND type = 'Contract' group by offer_phonecost_range UNION ALL
SELECT 'Monthly Cost' as category,'offer_offerental' as fieldname, offer_offerental_range as `element`, count(*) as phone_count from
options
WHERE 1 AND type = 'Contract' GROUP BY offer_offerental_range UNION ALL
SELECT 'Data Allowance' as category,'tariff_dataallowance' as fieldname, tariff_dataallowance_range as `element`, count(*) as phone_count from
options
WHERE 1 AND type = 'Contract' GROUP BY tariff_dataallowance_range
Create this index:
create index opt_tariff on options( tariff_freemins );
and rewrite the query to this:
select 'Free Mins' as category,
'tariff_freemins' as fieldname,
case
when tariff_freemins between 0 and 200 then 'a0TO200'
when tariff_freemins between 200 and 400 then 'b200TO400'
when tariff_freemins between 400 and 600 then 'c400TO600'
when tariff_freemins between 600 and 1000 then 'd600TO1000'
when tariff_freemins between 1000 and 2000 then 'e1000TO2000'
when tariff_freemins >2001 then 'hUnlimited'
end as `element`,
sum(cnt) as phone_count
from (
select tariff_freemins, count(*) As cnt
from options
group by tariff_freemins
) x
group by element
;
The inner subquery uses the index to optimize GROUP BY and it is fast.
The outer query cannot be optimized - it cannot be in MySql, since MySql does not support expression based indexes.
However the outer query performs an aggregation on much smaller set of data, that is already preagregated by the inner subquery, and the whole query should be faster than your's version.
Queries like this can be easily optimized in other RDBMS like Oracle, PostgreSQL and SQL-Server, using expression based indexes.