I want to fetch records in batches from MySQL database. Initially I fetch records of 1st 10 rows, then next 10 rows, until I found every matched row.
I found one solution of limiting query, but It didn’t help me.
Pageable p = PageRequest.of(offset, limit)
Here, if the offset is 0, and the limit is 10 then I got 10 records correctly. But, then if I applied value for offset other than 0, no rows were returned.
How can I use offset here? Or is there any other solution for this query? Please suggest if any.
public static PageRequest of(int page, int size)
Parameters:
page - zero-based page index, must not be negative.
size - the size of the page to be returned, must be greater than 0.
Here the first parameter is index of page not offset and the second parameter is size of the page not limit. Ref
Example: Suppose you have 30 documents. If you use page size as 10 then there will be 3 page total.
PageRequest.of(0, 10) -> offset 0 limit 10 -> will fetch first 10 document.
PageRequest.of(1, 10) -> offset 10 limit 10 -> will fetch 10 document after first 10 document.
PageRequest.of(2, 10) -> offset 20 limit 10 -> will fetch 10 document after first 20 document.
Related
I'm trying to write a query that will only return a recordset between 2 positions in a table
For example, using Excel to demonstrate, populate a recordset between positions 3 and 7
I thought about using BETWEEN with the ID field, but there may be times that a record has been deleted and the ID field auto-incrementation will be out.
Ultimately, I want to use the query to populate a list for reporting:
Page 1 shows Records 1 to 10
Page 2 shows Records 11 to 20
Page 3 shows Records 21 to 30 etc
Why not use limit and offset?
select t.*
from t
order by id
limit 10 offset 0;
Then for subsequent pages, you would have:
limit 10 offset 9;
limit 10 offset 19;
I have to extract quite big chunk of data from database (MYSQL MariaDB) and hereby I decided to take it by small chunks (limit = 1000 rows ). In total I need to extract ~9228 rows (it sounds not much but fetch time reach 100 - 120 seconds when I try to get all 9228 lines at once with one query).
When I fetch first 8 data chunks (1000 rows each), everything is good ~0.4 seconds per query. But when I try to extract the last 228 lines everything goes really slow - 80 seconds if I use LIMIT 1000 OFFSET 9000 or 50 seconds when I use the exact number of rows for LIMIT LIMIT 228 OFFSET 9000. But the query which is used to get total number of lines takes 30 seconds, so the two queries in total 80 seconds again.
My sql query to get data looks the following:
SELECT events.eventid, functions.triggerid FROM events
INNER JOIN functions ON events.objectid = functions.triggerid
WHERE
events.name LIKE 'DISCONNECT MSK-AP%'
OR events.name LIKE 'AP MSK-AP%' # '%MSK-AP%' is much slower than OR
AND events.value = 1
AND events.clock >= '1588280400'
AND events.clock <= '1590958799'
GROUP BY events.eventid
ORDER BY events.eventid DESC
LIMIT 1000 OFFSET 0; # SO OFFSET COULD BE 0, 1000, 2000, ... 8000, 9000
My sql query to get total number of lines (it is slow 30 seconds!) is as follows:
SELECT COUNT(distinct(events.eventid)) FROM events
INNER JOIN functions ON events.objectid = functions.triggerid
WHERE
events.name LIKE 'DISCONNECT MSK-AP%'
OR events.name LIKE 'AP MSK-AP%'
AND events.value = 1
AND events.clock >= '1588280400'
AND events.clock <= '1590958799';
My Database version:
protocol_version 10
slave_type_conversions
version 5.5.60-MariaDB
version_comment MariaDB Server
version_compile_machine x86_64
version_compile_os Linux
Why the last query to get the last chunk is so slow in comparison with other and what can I do to solve the issue? Can temporary database table help in the case?
Why I am not sure that the answer to question fits my case:
Why does MYSQL higher LIMIT offset slow the query down?
Because the problems does not correlate with OFFSET SIZE, e.g. :
LIMIT 100 OFFSET 9100; - 0.25 seconds BUT
LIMIT 100 OFFSET 9200; - 114 seconds!
So the problem appears when offset + limit is close or larger than total lines number (9228) !
OFFSET sucks performance.
A better way is to "remember where you left off".
Discussion: http://mysql.rjweb.org/doc.php/pagination
Why slower than reading all?
When using OFFSET, the query first counts off the number of rows given by the OFFSET, then delivers the number of rows given by the LIMIT. So, it gets slower and slower. The final offset takes about the same amount of time as reading the entire table.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
How can I speed up a MySQL query with a large offset in the LIMIT clause?
In our application we are showing records from MySQL on a web page. Like in most such applications we use paging. So the query looks like this:
select * from sms_message
where account_group_name = 'scott'
and received_on > '2012-10-11' and
received_on < '2012-11-30'
order by received_on desc
limit 200 offset 3000000;
This query takes 104 seconds. If I only change offset to low one or remove it completely, it's only half a second. Why is that?
There is only one compound index, and it's account_group_name, received_on and two other columns. Table is InnoDB.
UPDATE:
Explain returns:
1 SIMPLE sms_message ref all_filters_index all_filters_index 258 const 2190030 Using where
all_filters_index is 4-columns filter mentioned above.
Yes this is true, time increases as offset value increases and the reason is because offset works on the physical position of rows in the table which is not indexed. So to find a row at offset x, the database engine must iterate through all the rows from 0 to x.
i have a table with about 100 rows , and i want every time to get rows between number and number , like this
if `i=1` i want to get the rows `0 1 2 3 4`
if `i=2` i want to get the rows `5 6 7 8 9`
if `i=3` i want to get the rows `10 11 12 13 14`
and maybe the last value of i will just take 3 or 4 rows not 5
i think the solution will be something like this
select * from question limit (i-1)*5 , i*5-1
but doesn't work , cos i don't know how to use variable in a query , and when i tried it for i=1 it doesn't work and i got a syntax error
The first parameter to LIMIT is the zero-based starting row index. The second one is the number of rows. So, if you're always looking for five rows:
SELECT * FROM question LIMIT (i-1)*5 , 5
(You'll have to calculate (i-1)*5 with whatever language you're using to build the query, and pass an actual number to MySQL).
Im having some difficulties trying to order my list of a resultset. Im trying to create pagination. I have a table called Track. I use the following sql to retrieve a list of identies used by my pagination:
SELECT trackid FROM track ORDER BY filelength limit 10
Wich results in:
1
2
3
4
5
6
7
8
9
10
Then after that i execute the following statement:
SELECT * FROM track ORDER BY filelength ASC LIMIT 10
and i receive the following:
250 Track 250 0
251 Track 251 0
252 Track 252 0
253 Track 253 0
254 Track 254 0
255 Track 255 0
256 Track 256 0
257 Track 257 0
258 Track 258 0
259 Track 259 0
In the second resultset i see that the resultset gives different tracks, i would expect the same id's from first resultset, but it seems to start at 250. Why is this happening?
I have about 20000 rows in this table, and all of the column filelength columns are 0.
Im running a Mysql server 5.1.30 - community version
You said it yourself:
and all of the column filelength columns are 0.
So ORDER BY filelength is just noise, and arbitrary rows will be returned. The database engine will pick any rows in an implementation specific way, because all of them equally qualify and no particular order is specified.
There is no such thing as a "natural order" in a table of a relational database. You seem to be assuming that.
Use meaningful columns in your ORDER BY to select rows in a defined sequence. Like:
SELECT trackid FROM track ORDER BY trackid LIMIT 10;
To achieve this:
I basicly want a resultset that only contains the id's of the second
query (optionally without the limit)
Try:
SELECT trackid
FROM track
WHERE trackid BETWEEN 250 AND 259
ORDER BY trackid;
You're pulling from two different tables. playitem is not track:
SELECT trackid FROM playitem ORDER BY filelength limit 10
SELECT * FROM track ORDER BY filelength ASC LIMIT 10
Well if you are trying to use this for pagination. In SQL the LIMIT clause can take two arguments. With two arguments, the first argument specifies the offset of the first row to return, and the second specifies the maximum number of rows to return. The offset of the initial row is 0 (not 1)
For example, the first one would be:
LIMIT 0, 10
The next one would be:
LIMIT 10, 10
The next one would be:
LIMIT 20, 10
And so on...