This type of question is asked every now and then. The queries provided works, but it affects performance.
I have tried the JOIN method:
SELECT *
FROM nbk_tabl
INNER JOIN (
SELECT ITEM_NO, MAX(REF_DATE) as LDATE
FROM nbk_tabl
GROUP BY ITEM_NO) nbk2
ON nbk_tabl.REF_DATE = nbk2.LDATE
AND nbk_tabl.ITEM_NO = nbk2.ITEM_NO
And the tuple one (way slower):
SELECT *
FROM nbk_tabl
WHERE REF_DATE IN (
SELECT MAX(REF_DATE)
FROM nbk_tabl
GROUP BY ITEM_NO
)
Is there any other performance friendly way of doing this?
EDIT: To be clear, I'm applying this to a table with thousands of rows.
Yes, there is a faster way.
select *
from nbk_table
order by ref_date desc
limit <n>
Where is the number of rows that you want to return.
Hold on. I see you are trying to do this for a particular item. You might try this:
select *
from nbk_table n
where ref_date = (select max(ref_date) from nbk_table n2 where n.item_no = n2.item_no)
It might optimize better than the "in" version.
Also in MySQL you can use user variables (Suppose nbk_tabl.Item_no<>0):
select *
from (
select nbk_tabl.*,
#i := if(#ITEM_NO = ITEM_NO, #i + 1, 1) as row_num,
#ITEM_NO := ITEM_NO as t_itemNo
from nbk_tabl,(select #i := 0, #ITEM_NO := 0) t
order by Item_no, REF_DATE DESC
) as x where x.row_num = 1;
Related
So I have a bit of an unusual request. I'm working with a table with billions of rows.
The table has a column 'id' which is not unique, and has a column 'data'
What I want to do is run a count on the number of rows grouped by the 'id', but limit the counting to only 150 entries. I only need to know if there are 150 rows by any given id.
This is in an effort to optimize the query and performance.
It doesn't have to be a count. I only need to know if a given id has 150 entries, without have MySQL continue counting entries during the query. If that makes sense.
I know how to count, and I know how to group, and I know how to do both, but the count will come back with a number in the millions which is wasted processing time and the query needs to run on hundred of thousands of ids.
You can't really optimize performance for this -- I don't think.
select id, (count(*) >= 150)
from t
group by id;
If you happen to have a separate table with one row per id and an index on t(id), then this might be faster:
select ids.id,
((select count(*)
from t
where t.id = ids.id
) >= 150
)
from ids;
Unfortunately, MySQL does not support double nesting for correlated subqueries, so this is not possible:
select ids.id,
((select count(*)
from (select 1
from t
where t.id = ids.id
limit 150
) t
) >= 150
)
from ids;
If so, this might be faster.
EDIT:
If you have an index on id and only want ids that have 150 or more, then variables might be faster:
select id,
(#rn := if(#id = id, #rn + 1,
if(#id := id, 1, 1)
)
) as rn
from (select id
from t
order by id
) t cross join
(select #id := 0, #rn := 0) params
having rn = 150;
The thinking here is that using the index to order the table, materializing, and scanning again is probably faster than group by. I don't think row_number() would have the same performance characteristics.
EDIT II:
A slight variation on the above can be used to get all ids with a flag:
select id, (max(id) = 150)
from (select id,
(#rn := if(#id = id, #rn + 1,
if(#id := id, 1, 1)
)
) as rn
from (select id
from t
order by id
) t cross join
(select #id := 0, #rn := 0) params
having rn in (1, 150)
) t
group by id;
EDIT III:
Ahh! If you have a separate table of ids, then this might be the best approach:
select ids.id,
(select id
from t
where t.id = ids.id
limit 1 offset 149
) is not null
from ids;
This will fetch the 150th row from the index. If it not there, then no row is returned.
I don't think that this is possible. You will have to scan the entire table to know which ids have at least 150 entries.
So:
select id
from mytable
group by id
having count(*) >= 150
With an index on id, this should be as efficient as it can be.
I have met a situation that I have a list of IDs of a Store table and need to fetch the latest 10 files from each store.
SELECT *
FROM tblFiles
WHERE storeId in (IDs)
ORDER BY createdDate DESC
LIMIT 10
But, this limits the whole results. I found an answer to a similar SO question. But, the answer recommends using loop for each ID. This results in multiple DB hit.
Another option is to fetch all records and group them in the code. But, this will be heavy if there are large no.of records.
It'll be nice if it can be handled at the query level. Any help will be appreciated.
NB: The tables used here are dummy ones.
Pre-MySQL 8.0, the simplest method is probably variables:
select f.*
from (select f.*,
(#rn := if(#s = storeId, #rn + 1,
if(#s := storeId, 1, 1)
)
) as rn
from (select f.*
from tblfiles f
where storeId in (IDs)
order by storeId, createdDate desc
) f cross join
(select #s := 0, #rn := 0) params
) f
where rn <= 10;
In MySQL 8+ or MariaDB 10.3+, you would simply use window functions:
select f.*
from (select f.*,
row_number() over (partition by storeid order by createdDate desc) as seqnum
from tblfiles f
) f
where seqnum <= 10;
In older versions of MySQL and MariaDB, the innermost subquery may not be needed.
use select in where
SELECT * from tblFiles where storeId in (SELECT id from store ORDER BY datefield/id field desc limit 10)
You could workaround it with an UNIONed query, where each subquery searches for a particular id and enforces a LIMIT clause, like :
(SELECT *
FROM tblFiles
WHERE storeId = ?
ORDER BY createdDate DESC
LIMIT 10)
UNION
(SELECT *
FROM tblFiles
WHERE storeId = ?
ORDER BY createdDate DESC
LIMIT 10)
...
With this solution only one db hit will happen, and you are guarantee to get the LIMIT on a per id basis. Such a SQL can easily be generated from within php code.
Nb : the maximum allowed of UNIONs in a mysql query is 61.
I have a table like this:
01-Jul-17 100
02-Jul-17 100
03-Jul-17 300
04-Jul-17 300
05-Jul-17 500
06-Jul-17 500
07-Jul-17 300
08-Jul-17 400
09-Jul-17 100
10-Jul-17 100
What I want to output is (in this order) by eliminating the continuous duplicates but not all duplicates:
100
300
500
300
400
100
I cannot select Distinct, as it will eliminate the second instances of 300, 100. Is there a way to achieve this result in MySQL?
Thanks!
You want to get the previous value. If the dates really have no gaps or duplicates, just do:
select t.*
from t left join
t tprev
on t.col1 = date_add(tprev.col1, interval 1 day)
where tprev.col2 is null or tprev.col2 <> t.col2;
EDIT:
If the dates don't meet these conditions, then you can use variables:
select t.*
from (select t.*,
(#rn := if(#v = col2, #rn + 1,
if(#v := col2, 1, 1)
)
) as rn
from t cross join
(select #v := 0, #rn := 0) params
order by t.col1
) t
where rn = 1;
Note that MySQL does not guarantee the order of evaluation of expressions in the SELECT. So variables should not be assigned in one expression and then used in another -- they should be assigned in a single expression.
One way to handle this problem is by using session variables to track the changes of the values as ordered by your date column. In the query below, we keep track of the value, ordered by date, and assign a row number to each group of identical value. Then, only the first value in each group is retained. Note that this approach is robust to any number of duplicates. It is also robust with respect to there being gaps in your dates, so long as each record can be ordered by date.
SET #rn = 1;
SET #val = NULL;
SELECT t.val
FROM
(
SELECT
#rn:=CASE WHEN #val = val THEN #rn+1 ELSE 1 END rn,
#val:=val AS val,
dt
FROM yourTable
ORDER BY dt
) t
WHERE t.rn = 1
ORDER BY t.dt;
Output:
Demo here:
Rextester
You can make use of lag and lead functions.
select y from (select y , lag(y,1,0) over (order by x) as prev_y from t1) where y <> prev_y;
Is it possible to get specific row in query using like SUM?
Example:
id tickets
1 10 1-10 10=10
2 35 11-45 10+35=45
3 45 46-90 10+35+45=90
4 110 91-200 10+35+45+110=200
Total: 200 tickets(In SUM), I need to get row ID who have ticket with number like 23(Output would be ID: 2, because ID: 2 contains 11-45tickets in SUM)
You can do it by defining a local variable into your select query (in form clause), e.g.:
select id, #total := #total + tickets as seats
from test, (select #total := 0) t
Here is the SQL Fiddle.
You seem to want the row where "23" fits in. I think this does the trick:
select t.*
from (select t.*, (#total := #total + tickets) as running_total
from t cross join
(select #total := 0) params
order by id
) t
where 23 > running_total - tickets and 23 <= running_total;
SELECT
d.id
,d.tickets
,CONCAT(
TRIM(CAST(d.RunningTotal - d.tickets + 1 AS CHAR(10)))
,'-'
,TRIM(CAST(d.RunningTotal AS CHAR(10)))
) as TicketRange
,d.RunningTotal
FROM
(
SELECT
id
,tickets
,#total := #total + tickets as RunningTotal
FROM
test
CROSS JOIN (select #total := 0) var
ORDER BY
id
) d
This is similar to Darshan's answer but there are a few key differences:
You shouldn't use implicit join syntax, explicit join has more functionality in the long run and has been a standard for more than 20 years
ORDER BY will make a huge difference on your running total when calculated with a variable! if you change the order it will calculate differently so you need to consider how you want to do the running total, by date? by id? by??? and make sure you put it in the query.
finally I actually calculated the range as well.
And here is how you can do it without using variables:
SELECT
d.id
,d.tickets
,CONCAT(
TRIM(d.LowRange)
,'-'
,TRIM(
CAST(RunningTotal AS CHAR(10))
)
) as TicketRange
,d.RunningTotal
FROM
(
SELECT
t.id
,t.tickets
,CAST(COALESCE(SUM(t2.tickets),0) + 1 AS CHAR(10)) as LowRange
,t.tickets + COALESCE(SUM(t2.tickets),0) as RunningTotal
FROM
test t
LEFT JOIN test t2
ON t.id > t2. id
GROUP BY
t.id
,t.tickets
) d
Couldn't really explain my problem with words, but with an example I can show it clearly:
I have a table like this:
id num val
0 3 10
1 5 12
2 7 12
3 11 15
And I want to go through all the rows, and calculate the increase of the "num", and multiply that difference with the "val" value. And when I calculated all of these, I want to add these results together.
This is the mathematical equation, that I want to run on the table:
Result = (3-0)*10 + (5-3)*12 + (7-5)*12 + (11-7)*15
138 = Result
Thank you.
You can do with mysql variables, but you will still get one record for each entry.
select
#lastTotal := #lastTotal + ( (yt.num - #lastNum) * yt.val ) thisLineTotal,
#lastNum := yt.num as saveForNextRow,
yt.id
from
yourTable yt,
( select #lastTotal := 0,
#lastNum := 0 ) sqlvars
order by
id
This SHOULD give you what you want to confirm the calculations to each record basis.
Now, to get the one record and one column result, you can wrap it such as
select
pq.thisLineTotal
from
(above entire query ) as pq
order by
pq.id DESC
limit 1
Assuming the IDs are consecutive as your sample data suggests, just join the table to itself:
select sum((t1.num-ifnull(t2.num,0))*t1.val) YourValue
from YourTable t1
left join YourTable t2
on t2.id = t1.id - 1;
http://www.sqlfiddle.com/#!2/40b9f/12
This will give you the total. Make sure to order in the order you wish - I have ordered by id
SET #runtot:=0;
SET #prevval:=0;
select max(rt) as total FROM (
SELECT
q.val,
q.num,
(#runtot := #runtot + (q.num- #prevval) * q.val) AS rt,
(#prevval := q.num) AS pv
FROM thetable q
ORDER by ID) tot
If you want to see the details of the calculation, leave out the outer select as so:
SET #runtot:=0;
SET #prevval:=0;
SELECT
q.val,
q.num,
(#runtot := #runtot + (q.num- #prevval) * q.val) AS rt,
(#prevval := q.num) AS pv
FROM thetable q
ORDER by ID
If it is possible to have negative numbers for your column values, using max(rt) won't work for the total. You should then use:
SET #runtot:=0;
SET #prevval:=0;
select #runtot as total FROM (
SELECT
q.val,
q.num,
(#runtot := #runtot + (q.num- #prevval) * q.val) AS rt,
(#prevval := q.num) AS pv
FROM thetable q
ORDER by ID) tot LIMIT 1