I am trying to implement pagination using LIMIT and OFFSET with MySQL. On each database query, I would like to check if the last record within the table is reached. What is the best approach on this? Should I use another query for COUNT and then compare it with the OFFSET?
You can use sql_calc_found_rows, which makes MySQL calculate how many rows would have been returned if there had been no limit:
SELECT SQL_CALC_FOUND_ROWS ... FROM ... WHERE ... LIMIT x,y
^^^^^^^^^^^^^^^^^^^
then you can use
SELECT found_rows();
to retrieve that count. It's still two separate query calls, but the found_rows() call is very cheap, because the heavy lifting was already done by the main SELECT.
You can do two separate queries with a COUNT(), but then you're replicating work and throwing away most of it.
Related
I have a large database in which I use LIMIT in order not to fetch all the results of the query every time (It is not necessary). But I have an issue: I need to count the number of results. The dumbest solution is the following and it works:
We just get the data that we need:
SELECT * FROM table_name WHERE param > 3 LIMIT 10
And then we find the length:
SELECT COUNT(1) FROM table_name WHERE param > 3 LIMIT 10
But this solution bugs me because unlike the query in question, the one that I work with is complex and you have to basically run it twice to achieve the result.
Another dumb solution for me was to do:
SELECT COUNT(1), param, anotherparam, additionalparam FROM table_name WHERE param > 3 LIMIT 10
But this results in only one row. At this point I will be ok if it would just fill the count row with the same number, I just need this information without wasting computation time.
Is there a better way to achieve this?
P.S. By the way, I am not looking to get 10 as the result of COUNT, I need the length without LIMIT.
You should (probably) run the query twice.
MySQL does have a FOUND_ROWS() function that reports the number of rows matched before the limit. But using this function is often worse for performance than running the query twice!
https://www.percona.com/blog/2007/08/28/to-sql_calc_found_rows-or-not-to-sql_calc_found_rows/
...when we have appropriate indexes for WHERE/ORDER clause in our query, it is much faster to use two separate queries instead of one with SQL_CALC_FOUND_ROWS.
There are exceptions to every rule, of course. If you don't have an appropriate index to optimize the query, it could be more costly to run the query twice. The only way to be sure is to repeat the tests shown in that blog, using your data and your query on your server.
This question is very similar to: How can I count the numbers of rows that a MySQL query returned?
See also: https://mariadb.com/kb/en/found_rows/
This is probably the most efficient solution to your problem, but it's best to test it using EXPLAIN with a reasonably sized dataset.
MySQL Data - Best way to implement paging?
SELECT * FROM SALES
WHERE name like 'Sl%'
ORDER BY name DESC
LIMIT 1,2;
Pagination or Where Clause executed First. I am going to run this in huge Database
Thanks
Pagination or Where Clause executed First
The limit (pagination) always applies last. Otherwise the database would just be taking a few random records, and then attempting to apply your where clause to them and possibly returning no records at all from your query. That would not make any sense.
LIKE operation kills or hangs for long time. If LIMIT is applied first, i am happy to put the LIKE in a HUGE table
If your table is huge, then you need to make sure your where clause is always running against an index.
I have this query :
SELECT *,(SELECT count(id) FROM riverLikes
WHERE riverLikes.river_id = River.id) as likeCounts
FROM River
WHERE user_id IN (1,2,3)
LIMIT 10
my question is my sub-query runs only 10 time ( foreach row that are fetched ) or it run for every row in the "River" table ?
my "River" has lots of records and i like to have the best performance to get the rivers .
thanks.
In general, calculated data (either subqueries or functions), is calculated for the rows that matter, being rows that are returned, or rows for which the outcome of the calculation is relevant to further filtering or grouping.
In addition, the query optimizer may do all kinds of magic, and it is unlikely that it will run the subquery many times as such. It can be transformed in such a way that all relevant information is fetched at once.
And even if it didn't do that, it all takes place within the same operation in the database SQL engine, so executing this subselect 10 times is way, way faster than executing that subselect as a separate select 10 times, because the SQL engine only has to parse and prepare it once, and doesn't suffer from roundtrip times.
A simple select like that could easily take 30 milliseconds or so when executed from PHP, so quick math would suggest that it'd take 300ms extra to have this subselect in a 10-row query, but that's not the case, because the lion's share of those 30ms is overhead of communication between PHP and the database.
Because of the reasons mentioned above, this subselect is possibly way faster than a join, and it's a common misconception that a join is (almost) always faster.
So, to get back to your example, the subquery won't be executed for all rows in River, but will only be executed, probably in optimized form, for those 10 records of Rivers 1, 2 and 3.
In most production-ready RDBMS's subquery will be run only for rows which included in result set, i.e. only 10 times in your case. I think it is true for mysql too.
EDIT:
For assurance run
EXPLAIN <your query>
And view execution plan of your query
The subquery in the select statement runs one time per row returned, in your sample 10 times
I´m running a cost-time query in MySQL and Rails. This query is created dynamically and it also manages pagination with LIMIT and OFFSET. This is a summarized example:
SELECT fields
FROM tables
WHERE conditions
ORDER BY order DESC LIMIT ? OFFSET ?
I would also like to get the total count of elements, but I would like to avoid run the query twice for performance purposes. I don´t think is possible, but maybe you surprise me :)
Currently, I have something like:
objects = Object.find_by_sql(query)
totalCount = objects.count
But, of course, this is always returning the limit count.
Because you're using pagination and offsetting, you're not going to get a complete result. You can either run the two separate queries, or you can pull the complete dataset and then filter for pagination. The first option is likely to be faster, especially as your dataset grows.
To improve performance you'd getter better results looking at a caching strategy at various points. Without knowing when the data changes I can't offer any specific advice.
Edit 1: Expanding for Clarification
It might help to explain why this is the case. When you put into place the limit and offset manually, Rails knows nothing about the data not returned by the query. So without having that data available, it's definitionally impossible to make Rails aware of the total count.
That said, a simple COUNT aggregation should always be very fast to execute. If you're seeing speed issues with the execution of that query you'll want to make sure you have the right indexes in place, and that Rails is rendering the conditions in an optimal format.
MySQL? Why not to use SQL_CALC_FOUND_ROWS with FOUND_ROWS() ?
With two queries: (the second query will not hit the database)
SELECT SQL_CALC_FOUND_ROWS * FROM users LIMIT 0,5;
SELECT COUNT(FOUND_ROWS()) AS rows_count FROM users;
But one advise: you must test it. This might be slower or faster than two queries, it depends on some factors, like caching, engine, indexes, etc...
https://dev.mysql.com/doc/refman/5.7/en/information-functions.html#function_found-rows
Is this possible to get total number of rows count with offset limit
To Count the records just add one query as a column to your dynamically created query.
Check this:
SELECT fields,(SELECT count(*) FROM tables WHERE conditions) as obj_count
FROM tables
WHERE conditions
ORDER BY order DESC LIMIT ? OFFSET ?
Using MySQL session variables(starting with symbol #) we can write more efficient query.
SELECT fields,#count as obj_count
FROM tables,(SELECT #count:=count(*) FROM tables WHERE conditions) as sub
WHERE conditions
ORDER BY order DESC LIMIT ? OFFSET ?
This is a bit late, but try using objects.length. Length will count what you already have in the array.
So I have a table that has a little over 5 million rows. When I use SQL_CALC_FOUND_ROWS the query just hangs forever. When I take it out the query executes within a second withe LIMIT ,25. My question is for pagination reasons is there an alternative to getting the number of total rows?
SQL_CALC_FOUND_ROWS forces MySQL to scan for ALL matching rows, even if they'd never get fetched. Internally it amounts to the same query being executed without the LIMIT clause.
If the filtering you're doing via WHERE isn't too crazy, you could calculate and cache various types of filters to save the full-scan load imposed by calc_found_rows. Basically run a "select count(*) from ... where ...." for most possible where clauses.
Otherwise, you could go Google-style and just spit out some page numbers that occasionally have no relation whatsoever with reality (You know, you see "Goooooooooooogle", get to page 3, and suddenly run out of results).
Detailed talk about implementing Google-style pagination using MySQL
You should choose between COUNT(*) AND SQL_CALC_FOUND_ROWS depending on situation. If your query search criteria uses rows that are in index - use COUNT(*). In this case Mysql will "read" from indexes only without touching actual data in the table while SQL_CALC_FOUND_ROWS method will load rows from disk what can be expensive and time consuming on massive tables.
More information on this topic in this article #mysqlperformanceblog.