I want select X records from database (in PHP script), then sleep 60 seconds after continue the next 60 results...
SO:
SELECT * FROM TABLE WHERE A = 'B' LIMIT 60
SELECT SLEEP(60);
....
SELECT * FROM TABLE WHERE A = 'B' LIMIT X **where X is the next 60 results, then**
SELECT SLEEP(60);
AND etc...
How can I achievement this?
There is no such thing as "the next 60 records". SQL tables represent unordered sets. Without an order by, a SQL statement can return a result set in any order -- and even in different orders on different executions.
Hence, you first need something to guarantee the ordering . . . that is, an order by with keys that uniquely identify each row.
You can then use offset/limit to accomplish what you want. Or, you could put the code into a stored procedure and use a while loop. Or, you could do this on the application side.
In PHP:
<?php
// obtain the database connection, there's a heap of examples on the net, assuming you're using a library like mysqlite
$offset = 0;
while (true) {
if ($offset == 0) {
$res = $db->query('SELECT * FROM TABLE WHERE A = 'B' LIMIT 60');
} else {
$res = $db->query('SELECT * FROM TABLE WHERE A = 'B' LIMIT ' . $offset . ',60');
}
$rows = $db->fetch_assoc($res);
sleep(60);
if ($offset >= $some_arbitrary_number) {
break;
}
$offset += 60;
}
What you're doing is gradually incrementing the limit field by 60 until you reach a limit. The easiest way to do it is in a control while loop using true for the condition and break when you reach your invalid condition.
Related
I have been experimenting with Redis and MongoDB lately and it would seem that there are often cases where you would store an array of id's in either MongoDB or Redis. I'll stick with Redis for this question since I am asking about the MySQL IN operator.
I was wondering how performant it is to list a large number (300-3000) of id's inside the IN operator, which would look something like this:
SELECT id, name, price
FROM products
WHERE id IN (1, 2, 3, 4, ...... 3000)
Imagine something as simple as a products and categories table which you might normally JOIN together to get the products from a certain category. In the example above you can see that under a given category in Redis ( category:4:product_ids ) I return all the product ids from the category with id 4, and place them in the above SELECT query inside the IN operator.
How performant is this?
Is this an "it depends" situation? Or is there a concrete "this is (un)acceptable" or "fast" or "slow" or should I add a LIMIT 25, or doesn't that help?
SELECT id, name, price
FROM products
WHERE id IN (1, 2, 3, 4, ...... 3000)
LIMIT 25
Or should I trim the array of product id's returned by Redis to limit it to 25 and only add 25 id's to the query rather than 3000 and LIMIT-ing it to 25 from inside the query?
SELECT id, name, price
FROM products
WHERE id IN (1, 2, 3, 4, ...... 25)
Any suggestions/feedback is much appreciated!
Generally speaking, if the IN list gets too large (for some ill-defined value of 'too large' that is usually in the region of 100 or smaller), it becomes more efficient to use a join, creating a temporary table if need so be to hold the numbers.
If the numbers are a dense set (no gaps - which the sample data suggests), then you can do even better with WHERE id BETWEEN 300 AND 3000.
However, presumably there are gaps in the set, at which point it may be better to go with the list of valid values after all (unless the gaps are relatively few in number, in which case you could use:
WHERE id BETWEEN 300 AND 3000 AND id NOT BETWEEN 742 AND 836
Or whatever the gaps are.
I have been doing some tests, and as David Fells says in his answer, it is quite well optimized. As a reference, I have created an InnoDB table with 1,000,000 registers and doing a select with the "IN" operator with 500,000 random numbers, it takes only 2.5 seconds on my MAC; selecting only the even registers takes 0.5 seconds.
The only problem that I had is that I had to increase the max_allowed_packet parameter from the my.cnf file. If not, a mysterious “MYSQL has gone away” error is generated.
Here is the PHP code that I use to make the test:
$NROWS =1000000;
$SELECTED = 50;
$NROWSINSERT =15000;
$dsn="mysql:host=localhost;port=8889;dbname=testschema";
$pdo = new PDO($dsn, "root", "root");
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$pdo->exec("drop table if exists `uniclau`.`testtable`");
$pdo->exec("CREATE TABLE `testtable` (
`id` INT NOT NULL ,
`text` VARCHAR(45) NULL ,
PRIMARY KEY (`id`) )");
$before = microtime(true);
$Values='';
$SelValues='(';
$c=0;
for ($i=0; $i<$NROWS; $i++) {
$r = rand(0,99);
if ($c>0) $Values .= ",";
$Values .= "( $i , 'This is value $i and r= $r')";
if ($r<$SELECTED) {
if ($SelValues!="(") $SelValues .= ",";
$SelValues .= $i;
}
$c++;
if (($c==100)||(($i==$NROWS-1)&&($c>0))) {
$pdo->exec("INSERT INTO `testtable` VALUES $Values");
$Values = "";
$c=0;
}
}
$SelValues .=')';
echo "<br>";
$after = microtime(true);
echo "Insert execution time =" . ($after-$before) . "s<br>";
$before = microtime(true);
$sql = "SELECT count(*) FROM `testtable` WHERE id IN $SelValues";
$result = $pdo->prepare($sql);
$after = microtime(true);
echo "Prepare execution time =" . ($after-$before) . "s<br>";
$before = microtime(true);
$result->execute();
$c = $result->fetchColumn();
$after = microtime(true);
echo "Random selection = $c Time execution time =" . ($after-$before) . "s<br>";
$before = microtime(true);
$sql = "SELECT count(*) FROM `testtable` WHERE id %2 = 1";
$result = $pdo->prepare($sql);
$result->execute();
$c = $result->fetchColumn();
$after = microtime(true);
echo "Pairs = $c Exdcution time=" . ($after-$before) . "s<br>";
And the results:
Insert execution time =35.2927210331s
Prepare execution time =0.0161771774292s
Random selection = 499102 Time execution time =2.40285992622s
Pairs = 500000 Exdcution time=0.465420007706s
You can create a temporary table where you can put any number of IDs and run nested query
Example:
CREATE [TEMPORARY] TABLE tmp_IDs (`ID` INT NOT NULL,PRIMARY KEY (`ID`));
and select:
SELECT id, name, price
FROM products
WHERE id IN (SELECT ID FROM tmp_IDs);
Using IN with a large parameter set on a large list of records will in fact be slow.
In the case that I solved recently I had two where clauses, one with 2,50 parameters and the other with 3,500 parameters, querying a table of 40 Million records.
My query took 5 minutes using the standard WHERE IN. By instead using a subquery for the IN statement (putting the parameters in their own indexed table), I got the query down to TWO seconds.
Worked for both MySQL and Oracle in my experience.
IN is fine, and well optimized. Make sure you use it on an indexed field and you're fine.
It's functionally equivalent to:
(x = 1 OR x = 2 OR x = 3 ... OR x = 99)
As far as the DB engine is concerned.
EDIT: Please notice this answer was written in 2011, and see the comments of this answer discussing the latest MySQL features.
When you provide many values for the IN operator it first must sort it to remove duplicates. At least I suspect that. So it would be not good to provide too many values, as sorting takes N log N time.
My experience proved that slicing the set of values into smaller subsets and combining the results of all the queries in the application gives best performance. I admit that I gathered experience on a different database (Pervasive), but the same may apply to all the engines. My count of values per set was 500-1000. More or less was significantly slower.
I need help optimizing the below querys for a recurrent calendar i've built.
if user fail to accomplish all task where date
This is the query i use inside a forech which fetched all dates that the current activity is active.
This is my current setup, which works, but is very slow.
Other string explained:
$today=date("Y-m-d");
$parts = explode($sepparator, $datespan);
$dayForDate2 = date("l", mktime(0, 0, 0, $parts[1], $parts[2], $parts[0]));
$week2 = strtotime($datespan);
$week2 = date("W", $week2);
if($week2&1) { $weektype2 = "3"; } # Odd week 1, 3, 5 ...
else { $weektype2 = "2"; } # Even week 2, 4, 6 ...
Query1:
$query1 = "SELECT date_from, date_to, bok_id, kommentar
FROM bokningar
WHERE bokningar.typ='2'
and date_from<'".$today."'";
function that makes the foreach move ahead one day at the time...
function date_range($first, $last, $step = '+1 day', $output_format = 'Y-m-d' )
{
$dates = array();
$current = strtotime($first);
$last = strtotime($last);
while( $current <= $last ) {
$dates[] = date($output_format, $current);
$current = strtotime($step, $current);
}
return $dates;
}
foreach:
foreach (date_range($row['date_from'], $row['date_to'], "+1 day", "Y-m-d")
as $datespan)
if ($datespan < $today)
Query 2:
$query2 = "
SELECT bok_id, kommentar
FROM bokningar b
WHERE b.typ='2'
AND b.bok_id='".$row['bok_id']."'
AND b.weektype = '1'
AND b.".$dayForDate2." = '1'
AND NOT EXISTS
(SELECT t.tilldelad, t.bok_id
FROM tilldelade t
WHERE t.tilldelad = '".$datespan."'
AND t.bok_id='".$row['bok_id']."')
OR b.typ='2'
AND b.bok_id='".$row['bok_id']."'
AND b.weektype = '".$weektype2."'
AND b.".$dayForDate2." = '1'
AND NOT EXISTS
(SELECT t.tilldelad, t.bok_id
FROM tilldelade t
WHERE t.tilldelad = '".$datespan."'
AND t.bok_id='".$row['bok_id']."')";
b.weektype is either 1,2 or 3 (every week, every even week, every uneven week)
bokningar needs INDEX(typ, date_from)
Instead of computing $today, you can do
and date_from < CURDATE()
Are you running $query2 for each date? How many days is that? You may be able to build a table of dates, then JOIN it to bokningar to do all the SELECTs in a single SELECT.
When doing x AND y OR x AND z, first add parenthes to make it clear which comes first AND or OR: (x AND y) OR (x AND z). Then use a simple rule in Boolean arithmetic to transform it into a more efficient expression: x AND (y OR z) (where the parens are necessary).
The usual pattern for EXISTS is EXISTS ( SELECT 1 FROM ... ); there is no need to list columns.
If I am reading it correctly, the only difference is in testing b.weektype. So the WHERE can be simply
WHERE b.weektype IN ('".$weektype2."', '1')
AND ...
There is no need for OR, since it is effectively in IN().
tilldelade needs INDEX(tilldelad, bok_id), in either order. This should make the EXISTS(...) run faster.
Finally, bokningar needs INDEX(typ, bok_id, weektype) in any order.
That is a lot to change and test. See if you can get those things done. If it still does not run fast enough, start a new Question with the new code. Please include SHOW CREATE TABLE for both tables.
I'm just a beginner at mysql so in school we got task to do. It goes like this. Display / print 10% of all books from books in falling order. So i tried to use limit, but it doesn't work. What can i do? My code i've tried to use:
select title, price from book
order by price desc
limit (select count(*)*0.1 from book);
thank you for your answers!
limit values have to be hard-coded constants. You can't use variables on them, e.g. select ... limit #somevar is a syntax error. You also can't use sub-queries or other dynamic values either. So you're stuck with either fetching the row count ahead of time and stuff it into the query string as a "hard-coded" value:
$ten_percent = get_from_database('select count(*) / 10 from book');
$sql = "SELECT .... LIMIT $ten_percent";
Or you simply fetch everything and then abort your loop once you've reached 10%:
$sql = "SELECT ....";
$result = mysql_query($sql) or die(mysql_error());
$total_rows = mysql_num_rows($result);
$fetched = 0;
while($row = mysql_fetch_assoc()) {
$fetched++;
if ($fetched >= ($total_rows / 10)) {
break; // abort the loop at 10%
}
... do stuff with $row
}
I wonder if there is a way to accomplish:
SELECT * FROM table
by using LIMIT and OFFSET like so:
SELECT * FROM table LIMIT all OFFSET 0
Can I write SQL statement using LIMIT and OFFSET but still getting ALL result?
* of course I can use an IF statement but I rather avoid it if possible
From the MySQL documentation:
To retrieve all rows from a certain offset up to the end of the result
set, you can use some large number for the second parameter. This
statement retrieves all rows from the 96th row to the last:
SELECT * FROM tbl LIMIT 95,18446744073709551615;
So getting all rows might look as follows:
SELECT * FROM tbl LIMIT 0,18446744073709551615;
Yes, it is possible by providing NULL:
SELECT * FROM tab LIMIT NULL OFFSET NULL
db<>fiddle PostgreSQL demo
7.6. LIMIT and OFFSET
LIMIT ALL is the same as omitting the LIMIT clause, as is LIMIT with a NULL argument.
Snowflake LIMIT / FETCH
The values NULL, empty string (''), and $$$$ are also accepted and are treated as “unlimited”; this is useful primarily for connectors and drivers (such as the JDBC driver) if they receive an incomplete parameter list when dynamically binding parameters to a statement.
SELECT * FROM demo1 ORDER BY i LIMIT NULL OFFSET NULL;
SELECT * FROM demo1 ORDER BY i LIMIT '' OFFSET '';
SELECT * FROM demo1 ORDER BY i LIMIT $$$$ OFFSET $$$$;
I used this code in nodeJS with MySQL and it's run well, It's may help you.
Why you use it?
It's reliable because it's a string that will append with query.
If you want to set limit then you can put the limitation with the variable otherwise pass 0 with variable.
var noOfGroupShow=0; //0: all, rest according to number
if (noOfGroupShow == 0) {
noOfGroupShow = '';
}
else {
noOfGroupShow = ' LIMIT 0, '+noOfGroupShow;
}
var sqlGetUser = "SELECT `user_name`,`first_name`,`last_name`,`image`,`latitude`, `longitude`,`phone`,`gender`,`country`,`status_message`,`dob` as user_date_of_birth FROM `tb_user` WHERE `user_id`=?"+noOfGroupShow;
This may not be the best way to do it, but its the first that comes to mind...
SELECT * FROM myTable LIMIT 0,1000000
Replace 1000000 with some adequately large number that you know will always be larger than the total number of records in the table.
As the record will grow up, use mysql_num_rows to dynamically find total amount of records, instead of using some large number.
$cektotalrec=mysql_query("SELECT * FROM TABLE");
$numoffset=mysql_num_rows($cektotalrec);
$numlimit="0";
then:
$final="SELECT * FROM table ".$numlimit.", ".$numoffset"";
Maybe not the cleanest solution but setting limit to a very high number could work. Offset needs to be 0.
Why not use a IF statement where you add the limit and offset to the query as a statement is true?
You might receive an error if you set the limit to a very high number as defined by mysql doc. Thereofre, you should try to limit it 9999999999999, going higher can give you an error unless you set up server to go higher.
You might want to use LIMIT in a function, therefore it is not a bad idea. If you use it in a function, you might want it to be Limit All at one point and limit 1 at another point.
Below, I list an example where you might want your application to have no limit.
function get_navigation($select = "*", $from= "pages", $visible= 1, $subject_id = 2, $order_by = "position", $sort_by = "asc", $offset=0, $limit = 9551615){
global $connection;
$query = " SELECT {$select} ";
$query .= " FROM {$from} ";
$query .= " WHERE visible = {$visible} ";
$query .= " AND subject_id = {$subject_id} ";
$query .= " ORDER BY {$order_by} {$sort_by} ";
$query .= " LIMIT {$offset}, {$limit} ";
mysqli_query($connection, $query);
$navigation_set = mysqli_query($connection, $query);
confirm_query($navigation_set);
return $navigation_set;
}
define ("SELECT", "*");
define ("FROM", "pages");
define ("VISIBLE", 1);
define ("SUBJECT_ID", 3);
define ("ORDER_BY", "position");
define ("SORT_BY", "ASC");
define ("LIMIT", "0");
$navigation_set = get_navigation(SELECT, FROM, VISIBLE, SUBJECT_ID, ORDER_BY, SORT_BY);
My Current query is:
SELECT DISTINCT DATE(vote_timestamp) AS Date, COUNT(*) AS TotalVotes FROM `votes`
WHERE vote_target_id='83031'
GROUP BY DATE(vote_timestamp) ORDER BY DATE(vote_timestamp) DESC LIMIT 30
(line breaks separated for readability)
Where vote_timestamp is a time for each "vote", Count(*) is the count for that day, and vote_target_id is the specific target of the vote.
Currently, this works for all days in which the target has at least one "vote", but I would like it to also return TotalVotes as 0 for days where there are no votes, rather than having no row at all.
Can this (and how?) be done in MySQL or PHP? (either is fine, as it is futher processed by PHP so either code can be used).
Thank you
The problem is how to generate records for days that have no rows. This SO question has some approaches.
Looking at that solution, it looks like for me it's much simpler to do this quick fix that sorta works.
$result = mysql_query($sql) or die('Internal Database Error');
if (mysql_num_rows($result) == 0) { return false; }
while($row = mysql_fetch_assoc( $result )) {
$votes[$row['Date']] = $row['TotalVotes'];
}
// fill 0s with php rather than using mysql
$dates = array_keys($votes);
for ($t = strtotime($dates[count($dates)-1]); $t <= time(); $t +=86400) {
$date = date('Y',$t).'-'.date('m',$t).'-'.date('d',$t);
if (!array_key_exists($date,$votes)) {
$votes[$date] = 0;
}
}
thanks though,