I'm trying to optimize an algoritmic query by using a view. I created the view like:
Simplified version:
CREATE ALGORITHM = MERGE VIEW view_name AS (SELECT field1, fiedl2, field 3,
FUNC1(1, 2) AS score1,
FUNC2(1, 2) AS score2,
FUNC3(1, 2) AS score3,
FUNC4(1, 2) AS score4
FROM table
WHERE field1 = 1
); # only 0.003 seconds - approximate 2000 records
But when I try to do a SELECT query like:
SELECT * FROM view_name
ORDER BY (score1 * 1 + score2 * 2 + score3 * 2 + score4 * 4) DESC
LIMIT 0,20; # 9 to 11 seconds
The query takes about 10 seconds to get up with the result.
I already tried a couple of other solutions, but nothing has succeeded, see:
MySQL query slow because of ORDER BY with Stored Functions
If anyone has a clue, suggestion or an answer that would be awesome!
Thanks!
Related
I running into some time issues using my simple select:
SELECT *
FROM ltowert
WHERE bat = 3
AND id >= (SELECT id
FROM ltowert
WHERE bat = 3 AND ident = 'v0'
ORDER BY id DESC
LIMIT 1)
ORDER BY ident;
It takes nearly 12 seconds (depending on the index, etc..)
If I run the subquery (0.00075 sec) and put the result in the statement:
SELECT *
FROM ltowert
WHERE bat = 3 AND id >= 20979399
ORDER BY ident;
it runs in just 0.00095 sec, in addition: 0.0017 sec
So it seems, using the subquery avoid the Optimizer to use the index ?
How can I fix it and get quicker results ?
Thanks for any answers.
JR
I test your proposal, but I run into a problem of understanding. Maybe, it's my issue:
Your idea works, but the result is wrong: Result is only 1 row, but I expect 11 rows. I expect all row from l1, where id >= l2.id.
So I changed to:
SELECT *
FROM ltowert l1
INNER JOIN (
SELECT id
FROM ltowert
WHERE bat = 3 AND ident = 'v0'
order by id desc limit 1
) as l2 on l1.id>=l2.id
order by l1.ident;
This return much rows. l1.bat = 0 which is not the request.
Change to:
A. ") as l2 on l1.id>=l2.id AND l1.bat = 3"
or
B. WHERE l1.bat = 3
runs into a timeout Error 2013.
What is my mistake ?
Additional, due to my 2 step tests, I do a test using a variable:
SELECT #ref_id:=id FROM ltowert WHERE bat = 3 AND ident = 'v0' order by id desc limit 1;
SELECT * FROM ltowert l1 where bat = 3 AND l1.id >= #ref_id order by l1.ident;
It is fast, and give the right results.
Is there a disadvantage to this usage? (I know: 2 separate statements were not maintenance friendly)
You need both of these:
INDEX(bat, ident, id)
INDEX(bat, id)
If they don't suffice, please provide
SHOW CREATE TABLE ltowert;
EXPLAIN SELECT ...;
I have got a SQL query that I tried to optimize and I could reduce through various means the time from over 5 seconds to about 1.3 seconds, but no further. I was wondering if anyone would be able to suggest further improvements.
The Explain diagram shows a full scan:
explain diagram
The Explain table will give you more details:
explain tabular
The query is simplified and shown below - just for reference, I'm using MySQL 5.6
select * from (
select
#row_num := if(#yacht_id = yacht_id and #charter_type = charter_type and #start_base_id = start_base_id and #end_base_id = end_base_id, #row_num +1, 1) as row_number,
#yacht_id := yacht_id as yacht_id,
#charter_type := charter_type as charter_type,
#start_base_id := start_base_id as start_base_id,
#end_base_id := end_base_id as end_base_id,
model, offer_type, instant, rating, reviews, loa, berths, cabins, currency, list_price, list_price_per_day,
discount, client_price, client_price_per_day, days, date_from, date_to, start_base_city, end_base_city, start_base_country, end_base_country,
service_binary, product_id, ext_yacht_id, main_image_url
from (
select
offer.yacht_id, offer.charter_type, yacht.model, offer.offer_type, offer.instant, yacht.rating, yacht.reviews, yacht.loa,
yacht.berths, yacht.cabins, offer.currency, offer.list_price, offer.list_price_per_day,
offer.discount, offer.client_price, offer.client_price_per_day, offer.days, date_from, date_to,
offer.start_base_city, offer.end_base_city, offer.start_base_country, offer.end_base_country,
offer.service_binary, offer.product_id, offer.start_base_id, offer.end_base_id,
yacht.ext_yacht_id, yacht.main_image_url
from website_offer as offer
join website_yacht as yacht
on offer.yacht_id = yacht.yacht_id,
(select #yacht_id:='') as init
where date_from > CURDATE()
and date_to <= CURDATE() + INTERVAL 3 MONTH
and days = 7
order by offer.yacht_id, charter_type, start_base_id, end_base_id, list_price_per_day asc, discount desc
) as filtered_offers
) as offers
where row_number=1;
Thanks,
goppi
UPDATE
I had to abandon some performance improvements and replaced the original select with the new one. The select query is actually dynamically built by the backend based on which filter criteria are set. As such the where clause of the most inner select can expland quite a lot. However, this is the default select if no filter is set and is the version that takes significantly longer than 1 sec.
explain in text form - doesn't come out pretty as I couldn't figure out how to format a table, but here it is:
1 PRIMARY ref <auto_key0> <auto_key0> 9 const 10
2 DERIVED ALL 385967
3 DERIVED system 1 Using filesort
3 DERIVED offer ref idx_yachtid,idx_search,idx_dates idx_dates 5 const 385967 Using index condition; Using where
3 DERIVED yacht eq_ref PRIMARY,id_UNIQUE PRIMARY 4 yachtcharter.offer.yacht_id 1
4 DERIVED No tables used
Sub selects are never great,
You should sign up here: https://www.eversql.com/
Run that and it will give you all the right indexes and optimsiations you need for this query.
There's still some optimization you can use. Considering the subquery returns 5000 rows only you could use an index for it.
First rephrase the predicate as:
select *
from website_offer
where date_from >= CURDATE() + INTERVAL 1 DAY -- rephrased here
and date(date_to) <= CURDATE() + INTERVAL 3 MONTH
and days = 7
order by yacht_id, charter_type, list_price_per_day asc, discount desc
limit 5000
Then, if you add the following index the performance could improve:
create index ix1 on website_offer (days, date_from, date_to);
My query goes from 15 seconds to 0.05 seconds when I remove the ORDER BY in the following query:
simplified version:
SELECT field1, fiedl2, field 3,
FUNC1(1, 2) AS score1,
FUNC2(1, 2) AS score2,
FUNC3(1, 2) AS score3,
FUNC4(1, 2) AS score4
FROM table
WHERE field1 = 1
ORDER BY (score1 * 1 + score2 * 2 + score3 * 2 + score4 * 4) DESC;
I have a couple of stored functions that calculate sub-scores. Except I have to order the result based on the total score. In the ORDER BY I use * 2 to add some weight to the subscores to influence the total score.
I use MySQL 5.6.13
Has anybody has an idea how I can make the ORDER BY, but not slow it down?
Like is it possible to and store the score# fields and sum them up?
Thanks!
The difference in time is because MySql needs to create a sorted temp table and fill it with data. As your query is running much slower when using order by, the problem might be in disk where temp data is stored. You haven't mentioned how many rows you are returning from this query. You may also try manually perform the steps that MySql is probably doing, so create a temp table with primary key (order_by_result int, n int auto_increment) and insert into it your select results:
Insert into t(order_by_result, n, ...)
select (score1 * 1 + score2 * 2 + score3 * 2 + score4 * 4),null,...
and check hpw fast it runs - you may also check this way if the problem lies in your storage.
You could add a total_score column to the table, and define a trigger to update it automatically whenever a row is added or updated. Then index the column, and ORDER BY total_score should be fast.
I think the best solution is to precalculate the values of the functions and store them in the database. This way, the problem of calculating the values of the function on the fly will be transformed into a very simple ordering query.
As lowleveldesing has said, this type of queries forces mysql to calculate the product (score1 * 1 + score2 * 2 + score3 * 2 + score4 * 4) for all the register before giving you any output.
I Have a sitatation like the following:
In MySql Table Entries:
img_id name
1 aa.jpg
2 aab.mpeg
3 aabc.jpg
4 aabd.jpg
5 aabn.jpg
6 aabf.jpg
7 aadf.jpg
8 aacf.jpg
I want the count after splitting thew above values........
like
".jpg"=>7
".mpeg"=>1
SELECT RIGHT(Name, LOCATE('.', REVERSE(Name)) - 1) Format,
COUNT(*) TotalCOunt
FROM TableName
GROUP BY RIGHT(Name, LOCATE('.', REVERSE(Name)) - 1)
SQLFiddle Demo
Consider normalizing the table. In the long run, this will perform slow.
I'm having a problem with a mysql query, i have a table which contains both the amount of an order (the first int in the table) and it's description. Unfortunately i cant change the database structure. I need to split these in a query, i have the following query;
Load table:
3 pallets 120 x 100 x 100
12 pallet 120 x 100 x 84
SELECT *,
IF (load * 1! = 0, LEFT (load, 1), '1 ') AS cargo_quantity,
IF (load * 1! = 0, SUBSTR (load, 2), load) AS cargo_description
FROM transport
The query works when the amount is below 10, when the number is 10 or higher the query will fall short. I could really use some help solving this problem, any suggestions?
If you are sure that the first "column" of your entry is always seperated by " " then you can do this:
SELECT *,
LEFT(loadcolumn, LOCATE(" ",loadcolumn)) AS cargo_quantity,
SUBSTR(loadcolumn, LOCATE(" ",loadcolumn)+1) AS cargo_description
FROM transport
and here a sqlfiddle with a working solution:
Note that "load" seems to be a mysql keyword. I am not sure, so in my sqlfiddle I renamed it.
EDIT:
here is the improved solution if your "first column" does not exist:
SELECT *,
IF (concat('',(LEFT(loadcolumn, LOCATE(" ",loadcolumn))) * 1) =
LEFT(loadcolumn, LOCATE(" ",loadcolumn)),
LEFT(loadcolumn, LOCATE(" ",loadcolumn)), '1') AS cargo_quantity,
IF (concat('',(LEFT(loadcolumn, LOCATE(" ",loadcolumn))) * 1) =
LEFT(loadcolumn, LOCATE(" ",loadcolumn)),
SUBSTR(loadcolumn, LOCATE(" ",loadcolumn)+1), loadcolumn) AS cargo_description
FROM transport
improved sqlfiddle