MySQL Data - Best way to implement paging?
SELECT * FROM SALES
WHERE name like 'Sl%'
ORDER BY name DESC
LIMIT 1,2;
Pagination or Where Clause executed First. I am going to run this in huge Database
Thanks
Pagination or Where Clause executed First
The limit (pagination) always applies last. Otherwise the database would just be taking a few random records, and then attempting to apply your where clause to them and possibly returning no records at all from your query. That would not make any sense.
LIKE operation kills or hangs for long time. If LIMIT is applied first, i am happy to put the LIKE in a HUGE table
If your table is huge, then you need to make sure your where clause is always running against an index.
Related
I have a large database in which I use LIMIT in order not to fetch all the results of the query every time (It is not necessary). But I have an issue: I need to count the number of results. The dumbest solution is the following and it works:
We just get the data that we need:
SELECT * FROM table_name WHERE param > 3 LIMIT 10
And then we find the length:
SELECT COUNT(1) FROM table_name WHERE param > 3 LIMIT 10
But this solution bugs me because unlike the query in question, the one that I work with is complex and you have to basically run it twice to achieve the result.
Another dumb solution for me was to do:
SELECT COUNT(1), param, anotherparam, additionalparam FROM table_name WHERE param > 3 LIMIT 10
But this results in only one row. At this point I will be ok if it would just fill the count row with the same number, I just need this information without wasting computation time.
Is there a better way to achieve this?
P.S. By the way, I am not looking to get 10 as the result of COUNT, I need the length without LIMIT.
You should (probably) run the query twice.
MySQL does have a FOUND_ROWS() function that reports the number of rows matched before the limit. But using this function is often worse for performance than running the query twice!
https://www.percona.com/blog/2007/08/28/to-sql_calc_found_rows-or-not-to-sql_calc_found_rows/
...when we have appropriate indexes for WHERE/ORDER clause in our query, it is much faster to use two separate queries instead of one with SQL_CALC_FOUND_ROWS.
There are exceptions to every rule, of course. If you don't have an appropriate index to optimize the query, it could be more costly to run the query twice. The only way to be sure is to repeat the tests shown in that blog, using your data and your query on your server.
This question is very similar to: How can I count the numbers of rows that a MySQL query returned?
See also: https://mariadb.com/kb/en/found_rows/
This is probably the most efficient solution to your problem, but it's best to test it using EXPLAIN with a reasonably sized dataset.
I´m running a cost-time query in MySQL and Rails. This query is created dynamically and it also manages pagination with LIMIT and OFFSET. This is a summarized example:
SELECT fields
FROM tables
WHERE conditions
ORDER BY order DESC LIMIT ? OFFSET ?
I would also like to get the total count of elements, but I would like to avoid run the query twice for performance purposes. I don´t think is possible, but maybe you surprise me :)
Currently, I have something like:
objects = Object.find_by_sql(query)
totalCount = objects.count
But, of course, this is always returning the limit count.
Because you're using pagination and offsetting, you're not going to get a complete result. You can either run the two separate queries, or you can pull the complete dataset and then filter for pagination. The first option is likely to be faster, especially as your dataset grows.
To improve performance you'd getter better results looking at a caching strategy at various points. Without knowing when the data changes I can't offer any specific advice.
Edit 1: Expanding for Clarification
It might help to explain why this is the case. When you put into place the limit and offset manually, Rails knows nothing about the data not returned by the query. So without having that data available, it's definitionally impossible to make Rails aware of the total count.
That said, a simple COUNT aggregation should always be very fast to execute. If you're seeing speed issues with the execution of that query you'll want to make sure you have the right indexes in place, and that Rails is rendering the conditions in an optimal format.
MySQL? Why not to use SQL_CALC_FOUND_ROWS with FOUND_ROWS() ?
With two queries: (the second query will not hit the database)
SELECT SQL_CALC_FOUND_ROWS * FROM users LIMIT 0,5;
SELECT COUNT(FOUND_ROWS()) AS rows_count FROM users;
But one advise: you must test it. This might be slower or faster than two queries, it depends on some factors, like caching, engine, indexes, etc...
https://dev.mysql.com/doc/refman/5.7/en/information-functions.html#function_found-rows
Is this possible to get total number of rows count with offset limit
To Count the records just add one query as a column to your dynamically created query.
Check this:
SELECT fields,(SELECT count(*) FROM tables WHERE conditions) as obj_count
FROM tables
WHERE conditions
ORDER BY order DESC LIMIT ? OFFSET ?
Using MySQL session variables(starting with symbol #) we can write more efficient query.
SELECT fields,#count as obj_count
FROM tables,(SELECT #count:=count(*) FROM tables WHERE conditions) as sub
WHERE conditions
ORDER BY order DESC LIMIT ? OFFSET ?
This is a bit late, but try using objects.length. Length will count what you already have in the array.
When I add LIMIT 1 to a MySQL query, does it stop the search after it finds 1 result (thus making it faster) or does it still fetch all of the results and truncate at the end?
Depending on the query, adding a limit clause can have a huge effect on performance. If you want only one row (or know for a fact that only one row can satisfy the query), and are not sure about how the internal optimizer will execute it (for example, WHERE clause not hitting an index and so forth), then you should definitely add a LIMIT clause.
As for optimized queries (using indexes on small tables) it probably won't matter much in performance, but again - if you are only interested in one row than add a LIMIT clause regardless.
Limit can affect the performance of the query (see comments and the link below) and it also reduces the result set that is output by MySQL. For a query in which you expect a single result there is benefits.
Moreover, limiting the result set can in fact speed the total query time as transferring large result sets use memory and potentially create temporary tables on disk. I mention this as I recently saw a application that did not use limit kill a server due to huge result sets and with limit in place the resource utilization dropped tremendously.
Check this page for more specifics: MySQL Documentation: LIMIT Optimization
The answer, in short, is yes. If you limit your result to 1, then even if you are "expecting" one result, the query will be faster because your database wont look through all your records. It will simply stop once it finds a record that matches your query.
If there is only 1 result coming back, then no, LIMIT will not make it any faster. If there are a lot of results, and you only need the first result, and there is no GROUP or ORDER by statements then LIMIT will make it faster.
If you really only expect one single result, it really makes sense to append the LIMIT to your query. I don't know the inner workings of MySQL, but I'm sure it won't gather a result set of 100'000+ records just to truncate it back to 1 at the end..
I am trying to implement pagination using LIMIT and OFFSET with MySQL. On each database query, I would like to check if the last record within the table is reached. What is the best approach on this? Should I use another query for COUNT and then compare it with the OFFSET?
You can use sql_calc_found_rows, which makes MySQL calculate how many rows would have been returned if there had been no limit:
SELECT SQL_CALC_FOUND_ROWS ... FROM ... WHERE ... LIMIT x,y
^^^^^^^^^^^^^^^^^^^
then you can use
SELECT found_rows();
to retrieve that count. It's still two separate query calls, but the found_rows() call is very cheap, because the heavy lifting was already done by the main SELECT.
You can do two separate queries with a COUNT(), but then you're replicating work and throwing away most of it.
I'm sure the answer is somehow logical but here goes.
I have three big tables joined on three columns, each column is part of the primary key.
I want to get a distinct select on column1.
It works if I get the whole result at once, i.e. i export it in to a file.
But if I paginate it like phpadmin would do LIMIT 1000, 0 I get some column1 values twice, e.g. val1 on page 1 and val1 on the last page. This also means I'm not getting some values back which I should have had.
If I add a ORDER BY column1 everything is ok again, but I loose speed on the last pages, or that is what I've been told.
I guess it has something to do with the way mysql is handling the pagination and returns the result without actually knowing the whole result, but it still bugs my.
Can anyone elaborate on that.
The reason for paginating the query is because I don't like to lock the tables for longer periods at a time.
Does anyone have a insight how to achieve this and at the same time get all the data?
It doesn't make sense to implement paging using LIMIT without an ORDER BY.
Yes, you're right that it's faster without the ORDER BY, because the server is free to return arbitrary results in any order and the results don't have to be consistent from one query to the next.
If you want correct and consistent results, you must have the ORDER BY. If you are concerned about performance consider adding on index for the column you are ordering by.
From the manual page LIMIT optimization:
If you use LIMIT row_count with ORDER BY, MySQL ends the sorting as soon as it has found the first row_count rows of the sorted result, rather than sorting the entire result. If ordering is done by using an index, this is very fast.
The reason for paginating the query is because I don't like to lock the tables for longer periods at a time. Does anyone have a insight how to achieve this and at the same time get all the data?
If you're trying to perform some operation on every row then your approach won't work if data can be added or removed. This is because it will push all the following rows and some rows will be moved onto different pages. Adding a row will push some rows onto the next page, meaning that you see one row twice. Removing a row from an earlier page will cause you to skip a row.
Instead you could use one of these approaches:
Use some id to keep track of how far you have progressed. Select the next n rows with higher id.
Record which rows you have handled by storing a boolean in a column. Select any n rows that you haven't handled yet.