Optimize MySQL query with many subselects - mysql

I have to maintain an application with SQL commands (article management).
One query looks like this:
SET #from_timestamp = '2013-07-01 00:00:00';
SET #serialnumber = '%aaaa01%';
SELECT
test1.article,
test1.serialnumber,
test2.`timestamp` as test2_timestamp,
test2.additionalinfo,
test1.`timestamp` AS test1_timestamp,
test1.text0 AS test1_text0,
test1.text1 AS test1_text1,
test1.text2 AS test1_text2,
test1.text3 AS test1_text,
test3.text1 as test3_text1,
test3.`timestamp` AS test3_timestamp,
test3.`status`,
TIMESTAMPDIFF(MINUTE, test1.`timestamp`, test3.`timestamp`) as DeltaT
FROM (
SELECT
max(`timestamp`) AS TIMESTAMP,
article,
serialnumber,
text1,
text2,
text3,
text4
FROM exampleTable
WHERE
test = 'test1' AND
`timestamp` >= #from_timestamp AND
os0 LIKE #serialnumber
GROUP BY serialnumber
) AS test1
LEFT JOIN (
SELECT
max(`timestamp`) AS TIMESTAMP,
serialnumber,
status,
text1
FROM exampleTable
WHERE
test = 'test3' AND
`timestamp` >= #from_timestamp
GROUP BY serialnumber
) AS test3
ON test1.serialnumber = test3.serialnumber
LEFT JOIN (
SELECT
max(`timestamp`) AS TIMESTAMP,
serialnumber,
article,
text1
FROM exampleTable
WHERE
LENGTH(serialnumber) = 13 AND
test = 'test2' AND
`status` = -1 AND
`timestamp` >= #from_timestamp
GROUP BY serialnumber
) AS test2
ON test1.serialnumber = test2.serialnumber
ORDER BY test1.`timestamp`
It is working with only one table where product data is saved. But as expected, this query is really slow. Is there any optimization possible?
Edit: Attached image of the explain sql result
Edit2: The table has many data (last COUNT(*) provides 39589279 ;-) )
Edit3: Attached image of the explain table result
Edit4: Please don't blame me for table definition, it is very old developed by other guys and it is not possible to change it without breaking changes.

First of all, you qre querying three times the same table and join afterwards. The first simplification would be to use only one select :
SELECT
article,
serialnumber,
text1,
text2,
text3,
text4,
max(case when
test = 'test1' AND
os0 LIKE #serialnumber
then `timestamp` else null end) AS test1_timestamp,
max(case when
LENGTH(serialnumber) = 13 AND
test = 'test2' AND
`status` = -1
then `timestamp` else null end) AS test2_timestamp,
max(case when
test = 'test3'
then `timestamp` else null end) AS test3_timestamp,
FROM exampleTable
WHERE
test in ('test1', 'test2', 'test3') AND
`timestamp` >= #from_timestamp
GROUP BY serialnumber
This will already give you a great improvement.
Secondly, you have to put an index on the timestamp column to improve speed.

Related

How to write the a new select query with backward compatibility after adding a new column to a MySQL database

I have a MySQL table running for 4 months and I have a select statement in that table, like below.
SELECT
CONCAT(
YEAR(FROM_UNIXTIME(creation_time)),
'-',
IF(
MONTH(FROM_UNIXTIME(creation_time)) < 10,
CONCAT('0', MONTH(FROM_UNIXTIME(creation_time))),
MONTH(FROM_UNIXTIME(creation_time))
)
) AS Period,
(
COUNT(CASE
WHEN system_name = 'System' THEN 1
ELSE NULL
END)
) AS "Some data",
FROM table_name
GROUP BY
Period
ORDER BY
Period DESC
Lately, I've added a new feature and a column, let's say is_rerun. This value is just added and not exist previously. Now, i would like to write a query with the current statement which checks the system_name and also the is_rerun field and if this field exists and value is 1 then return 1 and if the column not exist or it its value is zero, then return null.
I tried IF EXISTS re_run THEN 1 ELSE NULL, but no luck. I can also insert values for the previous runs but i don't want to do that. Is there any solution. Thanks.
SELECT
CONCAT(
YEAR(FROM_UNIXTIME(creation_time)),
'-',
IF(
MONTH(FROM_UNIXTIME(creation_time)) < 10,
CONCAT('0', MONTH(FROM_UNIXTIME(creation_time))),
MONTH(FROM_UNIXTIME(creation_time))
)
) AS Period,
(
COUNT(CASE
WHEN system_name = 'System' AND IF EXISTS is_rerun THEN 1
ELSE NULL
END)
) AS "Some data",
FROM table_name
GROUP BY
Period
ORDER BY
Period DESC
As a starter: you have a group by query, so you need to put is_rerun in an aggregate function.
Based on your description, I think that something like case(case when is_rerun = 1 then 1 end) should do the work: it returns 1 if any is_rerun in the group is 1, else null.
Or if you can live with 0 instead of null, then you can use a simpler expression: max(is_rerun = 1).
Note that your query could be largely simplified as for the date formating logic and the conditional count. I would phrase it as:
select
date_format(from_unixtime(creation_time),'%Y-%m') period,
sum(system_name = 'System') some_data,
max(is_rerun = 1) is_rerun
from mytable
group by period
order by period desc

Sql query with three conditions

I have a database with a table having content as below :
message_number message_type message_chat
0 IN Hi
1 OB Hello
2 IN Help
3 IN Want to find this thing
4 OB Sure
5 OB Please let me know
I have written 5 rows since i want to incorporate all possible cases that i want in my query in the example table that i showed.
Now in my query output, i want something like :
message_in message_out
Hi Hello
Help NULL
Want to find this string Sure
NULL Please let me know
So the cases that i want to consider are :
suppose if message_number=0 and message_number=1 both have message_type value as IN then put message_chat_in as message_chat(at message_number=0) and message_chat out as NULL and the iterate over message_number=1
if message_number =0 have message_type=IN and message_number =1 have message_type=OB, then show message_chat(at message_number=0) as message_chat_in and message_chat(at message_number=1) as message_out and dont iterate over message_number=1;
hope i have clarified the condition though i have included all three condition in the expected output.How should my sqlquery look like?
Edit : I am using mysql version 5.5.8
Try the following query
SELECT
q1.message_number in_num,
q1.message_chat in_chat,
q2.message_number out_num,
q2.message_chat out_chat
FROM
(
SELECT *,#i1:=IFNULL(#i1,0)+1 num
FROM Chat
ORDER BY message_number
) q1
LEFT JOIN
(
SELECT *,#i2:=IFNULL(#i2,0)+1 num
FROM Chat
ORDER BY message_number
) q2
ON q2.num=q1.num+1 AND q2.message_type<>q1.message_type
WHERE q1.message_type='IN'
UNION ALL
SELECT
q1.message_number in_num,
q1.message_chat in_chat,
q2.message_number out_num,
q2.message_chat out_chat
FROM
(
SELECT *,#i3:=IFNULL(#i3,0)+1 num
FROM Chat
ORDER BY message_number
) q1
RIGHT JOIN
(
SELECT *,#i4:=IFNULL(#i4,0)+1 num
FROM Chat
ORDER BY message_number
) q2
ON q2.num=q1.num+1 AND q2.message_type<>q1.message_type
WHERE q2.message_type='OB'
AND q1.message_type IS NULL
ORDER BY IFNULL(in_num,out_num)
SQL Fiddle - http://sqlfiddle.com/#!9/95a515/1
The second variant
SET #i1 = 0;
SET #i2 = 0;
SET #i3 = 0;
SET #i4 = 0;
-- the same query
SQL Fiddle - http://sqlfiddle.com/#!9/95a515/2
Or
SELECT 0,0,0,0 INTO #i1,#i2,#i3,#i4;
-- the same query
SQL Fiddle - http://sqlfiddle.com/#!9/95a515/5
why not using a analytic function here? I would do it with Lead() like this:
with inc as (
--Do the incorporation in this block. could be subquery too
--but its easier to read this way.
select
case when message_type = 'IN'
then message_chat
end as message_in
,case when LEAD(message_type) OVER (Order by message_number) = 'OB' --get the next message by number if it is type OB
then LEAD(message_chat) OVER (order by message_number)
end as message_out
from input
)
select *
from inc
where coalesce(message_in, message_out) is not null --filter out rows where with in & out is null
Ok, since there is no analytical functions in MySQL less than 8 the code may not be easy to follow:
with data_rn as
(
-- this isolate consecutive rows with the same message_type
select d1.*, count(d2.message_number) rn
from data d1
left join data d2 on d1.message_number > d2.message_number and d1.message_type != d2.message_type
group by d1.message_number
),
data_rn2 as
(
-- this marks the rows where new rows has to be added (i.e. when rn2 != 0)
select d1.*, count(d2.message_number) rn2
from data_rn d1
left join data_rn d2 on d1.rn = d2.rn and d1.message_type = d2.message_type and d1.message_number > d2.message_number
group by d1.message_number
),
data_added as
(
-- this add new rows
select message_number, message_type, message_chat
from data_rn2
union all
select message_number - 0.5, 'OB', NULL from data_rn2 where message_type = 'IN' and rn2 != 0
union all
select message_number - 0.5, 'IN', NULL from data_rn2 where message_type = 'OB' and rn2 != 0
order by message_number
), data_added_rn as
(
-- this compute new row numbering
select d1.*, ceil((count(d2.message_number)+1)/2) rn
from data_added d1
left join data_added d2 on d1.message_number > d2.message_number
group by d1.message_number
)
-- this will do the final formating
select max(case when message_type = 'IN' then message_chat end) message_in,
max(case when message_type = 'OB' then message_chat end) message_out
from data_added_rn
group by rn
demo
I have tried to comment each section appropriately.

MYSQL Query : How to get values per category?

I have huge table with millions of records that store stock values by timestamp. Structure is as below:
Stock, timestamp, value
goog,1112345,200.4
goog,112346,220.4
Apple,112343,505
Apple,112346,550
I would like to query this table by timestamp. If the timestamp matches,all corresponding stock records should be returned, if there is no record for a stock for that timestamp, the immediate previous one should be returned. In the above ex, if I query by timestamp=1112345 then the query should return 2 records:
goog,1112345,200.4
Apple,112343,505 (immediate previous record)
I have tried several different ways to write this query but no success & Im sure I'm missing something. Can someone help please.
SELECT `Stock`, `timestamp`, `value`
FROM `myTable`
WHERE `timestamp` = 1112345
UNION ALL
SELECT `Stock`, `timestamp`, `value`
FROM `myTable`
WHERE `timestamp` < 1112345
ORDER BY `timestamp` DESC
LIMIT 1
select Stock, timestamp, value from thisTbl where timestamp = ? and fill in timestamp to whatever it should be? Your demo query is available on this fiddle
I don't think there is an easy way to do this query. Here is one approach:
select tprev.*
from (select t.stock,
(select timestamp from t.stock = s.stock and timestamp <= <whatever> order by timestamp limit 1
) as prevtimestamp
from (select distinct stock
from t
) s
) s join
t tprev
on s.prevtimestamp = tprev.prevtimestamp and s.stock = t.stock
This is getting the previous or equal timestamp for the record and then joining it back in. If you have indexes on (stock, timestamp) then this may be rather fast.
Another phrasing of it uses group by:
select tprev.*
from (select t.stock,
max(timestamp) as prevtimestamp
from t
where timestamp <= YOURTIMESTAMP
group by t.stock
) s join
t tprev
on s.prevtimestamp = tprev.prevtimestamp and s.stock = t.stock

temporary table has a count for each other table

I'm trying to make a statistics page in my php script. in order to select the count from each table I need more than 30 Queries like this
SELECT COUNT(order_id) as `uncompleted_orders` FROM `orders` WHERE `order_status` != 0
and then I need to run another query like this:
SELECT COUNT(order_id) as `completed_orders` FROM `orders` WHERE `order_status` = 1
I've tried this approach, but it didn't work:
SELECT COUNT(order_id) as `uncompleted_orders` FROM `sd_orders` WHERE `order_status` != 4;
SELECT COUNT(order_id) as `completed_orders` FROM `sd_orders` WHERE `order_status` = 4;
Is there any way to creat a new temp table in MySQL contains the count for other tables?
You could try something like this:
SELECT
(
SELECT COUNT(order_id) FROM `sd_orders` WHERE `order_status` != 4
) as `uncompleted_orders`,
(
SELECT COUNT(order_id) FROM `sd_orders` WHERE `order_status` = 4
) as `completed_orders`
You will have a result set with one row and a field for each count.
Without more information it's impossible to generalise, but there are many constructs that can help you here.
First, your example is actually from one table, and not two. This means that you can do the following...
SELECT
COUNT(CASE WHEN order_status = 4 THEN order_id END) AS complete_orders,
COUNT(CASE WHEN order_status <> 4 THEN order_id END) AS incomplete_orders
FROM
sd_orders
This works because COUNT(<something>) doesn't include an NULLs in the results. And by not including an ELSE clause, anything that doesn't match returns NULL. Another way people accomplish the same result is SUM(CASE WHEN ? THEN 1 ELSE 0 END).
Second, where you do actually have multiple tables, you can combine the results in several different ways...
-- Where you want one value from each table...
--------------------------------------------------------------------------------
SELECT
(SELECT COUNT(*) FROM table1 WHERE fieldx = ?) AS value1,
(SELECT COUNT(*) FROM table2 WHERE fieldy = ?) AS value2
-- Where you want one row of values from each table...
--------------------------------------------------------------------------------
SELECT
table1_summary.value1 AS table1_value1,
table1_summary.value2 AS table1_value2,
table2_summary.value1 AS table2_value1,
table2_summary.value2 AS table2_value2
FROM
(
SELECT
COUNT(CASE WHEN fieldx = ? THEN id END) AS value1,
COUNT(CASE WHEN fieldx <> ? THEN id END) AS value2
FROM
table1
)
AS table1_summary
CROSS JOIN
(
SELECT
COUNT(CASE WHEN fieldy = ? THEN id END) AS value1,
COUNT(CASE WHEN fieldy <> ? THEN id END) AS value2
FROM
table2
)
AS table2_summary
-- Where you want many rows, but of the same fields, from each table...
--------------------------------------------------------------------------------
SELECT
*
FROM
(
SELECT
'Table1' AS source_table,
fielda AS some_grouping,
COUNT(CASE WHEN fieldx = ? THEN id END) AS value1,
COUNT(CASE WHEN fieldx <> ? THEN id END) AS value2
FROM
table1
GROUP BY
fielda
UNION ALL
SELECT
'Table2' AS source_table,
fieldb AS some_grouping,
COUNT(CASE WHEN fieldy = ? THEN id END) AS value1,
COUNT(CASE WHEN fieldy <> ? THEN id END) AS value2
FROM
table2
GROUP BY
fieldb
)
AS summary
ORDER BY
source_table,
some_grouping,
value1,
value2
As you can see, there are a lot of ways to do this. How you approach it totally depends on your data and your needs.

Update multiple rows from same table in mysql

Update a single column over multiple rows depending on the data from the same table.
update table1 set status=newtime
from (
select
case
when TIME_FORMAT( TIMEDIFF( ADDTIME( time_val, '120:00:00' ), NOW() ), '%Hh %im %ss')<0 then '4'
else '0'
end as newtime,
id as new_id
FROM table1
where id2='2'
and status='0'
)
where id=new_id
This is my query. Thanks in advance.
Edit:
This is an alternate query to achieve this. But it also gives me an error
update table1 set status=
(select
case when timeleft<0 then '4' else '0' end as something,
new_id
from
(
select
TIME_FORMAT( TIMEDIFF( ADDTIME( time_val, '120:00:00' ), NOW() ), '%Hh %im %ss') as newtime,
id as new_id
FROM
table1
where id2='2' and
status='0'
)
}
where id=new_id
"#1248 - Every derived table must have its own alias".
I cannot use alias as I am fetching two columns from the query. Any help would be great.
UPDATE statements have no FROM clause in MySQL syntax. However, you can JOIN table against the subquery.
UPDATE
table1 t1
JOIN (
select
case
when TIME_FORMAT( TIMEDIFF( ADDTIME( time_val, '120:00:00' ), NOW() ), '%Hh %im %ss')<0 then '4'
else '0'
end as newtime,
id as new_id
FROM table1
WHERE id2='2' AND status='0'
) tsub ON t1.id = tsub.new_id
SET status = tsub.newtime
It looks to me like you don't need to do any subquerying or joining at all. This should do what you want:
UPDATE table1
SET status = CASE WHEN TIME_FORMAT(TIMEDIFF(ADDTIME(time_val, '120:00:00'), NOW()), '%Hh %im %ss') < 0 THEN '4' ELSE '0' END
WHERE id2 = '2' AND status = '0'
In the query you wrote, your subquery will get back the new time_val and the id number of the row to update, for any rows that match the criteria id2 = '2' AND status = '0'. You will then update all those rows (that matched the above criteria) and set the status to the new time_val.
Instead of selecting them first, cut out the middle man and just update all rows that match that criteria with the new value. Your query will be faster and more straightforward.
Besides the simplified version (provided by #Travesty3), it seems you are using a whole bunch of date and time functions to test for a simple thing:
UPDATE table1
SET status = '4'
WHERE id2 = '2'
AND status = '0'
AND time_val < NOW() - INTERVAL 120 HOUR
We can update table with multiple row by same table or two different table in this manner, just posting a snippet of mysql code from my procedure
Update documentcolumns as tb1, documentcolumns as tb2 set tb1.documentColumnPos = tb2.documentColumnPos Where tb1.userID = user_id and tb2.userID is NULL and tb1.columnNameDefID= tb2.columnNameDefID and tb1.tabtype = tab_type and tb2.tabtype = tab_type;,