Select by date doesn't seem to work - mysql

As I am not the world's greatest SQLer, I am working up to something big, step by step.
I have a table:
mysql> describe taps;
+---------------+-----------+------+-----+-------------------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+-----------+------+-----+-------------------+-------+
| tag_id | int(11) | YES | | NULL | |
| time_stamp | timestamp | NO | | CURRENT_TIMESTAMP | |
| event_id | int(11) | YES | | NULL | |
| event_station | int(11) | YES | | NULL | |
| device_id | text | YES | | NULL | |
| device_type | text | YES | | NULL | |
+---------------+-----------+------+-----+-------------------+-------+
6 rows in set (0.00 sec)
And would like to select all entries for a given date (today, 12th Feb, '17).
I am trying
mysql> select * from taps WHERE (event_id=4)
AND ((time_stamp >= 1486857600000) AND (time_stamp <= 1486944000000));
Empty set, 2 warnings (0.00 sec)
IMPORTANT: I have simplified things, because I want to compare with variables, which have values which I have obtained from another table, which are also of type timestamp.
Hmmm, warnings ....
mysql> SHOW WARNINGS;
+---------+------+----------------------------------------------------------------------------+
| Level | Code | Message |
+---------+------+----------------------------------------------------------------------------+
| Warning | 1292 | Incorrect datetime value: '1486857600000' for column 'time_stamp' at row 1 |
| Warning | 1292 | Incorrect datetime value: '1486944000000' for column 'time_stamp' at row 1 |
+---------+------+----------------------------------------------------------------------------+
2 rows in set (0.00 sec)
So, I tried casting
select * from taps WHERE (event_id=4)
AND ((time_stamp >= CAST(1486857600000 AS DATETIME))
AND (time_stamp <= CAST(1486944000000 AS DATETIME)));
Empty set (0.00 sec)
Which I don't understand, as I am the table does have some entries today.
mysql> select * from taps order by time_stamp limit 3;
+--------+---------------------+----------+---------------+-----------+-------------+
| tag_id | time_stamp | event_id | event_station | device_id | device_type |
+--------+---------------------+----------+---------------+-----------+-------------+
| 44 | 2017-02-12 15:10:25 | NULL | 16 | NULL | NULL |
| 37 | 2017-02-12 15:10:27 | NULL | 14 | NULL | NULL |
| 50 | 2017-02-12 15:10:28 | NULL | 15 | NULL | NULL |
+--------+---------------------+----------+---------------+-----------+-------------+
3 rows in set (0.00 sec)
What am I doing wrongly? And what should my query be?

MySQL has very confusing terminology for date/time stuff (see here). A timestamp is essentially a datetime with a timezone, because the value is stored as UTC, but reported in the local timezone.
This timestamp is not to be confused with a Unix Timestamp, which is just the number of seconds since 1970-01-01 (or in some cases milliseconds).
In your case, try this:
select t.*
from taps t
where event_id = 4 and
time_stamp >= '2017-02-11 07:00:00' and
time_stamp < '2017-02-12 07:00:00';
You could use from_unixtimestamp(). However, people generally find date formats much easier to read.
Note: I changed the last condition to a strict inequality. This gives you 24 hours with no duplication of time, in case something happens at exactly 2017-02-12 07:00:00.

In SQL, literal TIMESTAMP values are normally supplied as strings, in the format 'YYYY-MM-DD HH:MM:SS'. MySQL does allow some latitude in the actual format. More recent versions of MySQL allow for fractional seconds. See the MySQL Reference Manual for a more complete description.
https://dev.mysql.com/doc/refman/5.6/en/datetime.html
As an example, MySQL would recognize any of these:
'2017-02-12 09:30:45'
'17-02-12 09:30:45'
170212093045
If we have a requirement to supply/specify literal values that represent integer milliseconds since '1970-01-01 00:00:00', we can use a SQL expression to convert those to values that can be compared to TIMESTAMP. As a demonstration:
SELECT '1970-01-01' + INTERVAL 1486857600000 / 1000 SECOND AS ts
If we need to supply integer millisecond values as a literal in a condition in a WHERE clause, then we can use expressions like the one above.
The query in the question could do something like this to compare the value in a TIMESTAMP column
AND time_stamp >= '1970-01-01' + INTERVAL 1486857600000 / 1000 SECOND
AND time_stamp < '1970-01-01' + INTERVAL 1486944000000 / 1000 SECOND

Related

Why do I get a full table scan in one case, but not the other?

EDIT: It is on mysql version 5.5.62-38.14-log, that I have the problem, BTW, although the examples were run on 5.7.27-0ubuntu0.18.04.1 on my local machine. I have changed the UNIX_TIMESTAMP() in my queries to TIMESTAMP(), but no change.
Can somebody help see the light, please? I have a relatively simple table:
mysql> CREATE TABLE `game_instance` (
-> `game_instance_id` bigint(20) NOT NULL AUTO_INCREMENT,
-> `game_id` int(11) NOT NULL,
-> `currency_code` varchar(15) DEFAULT NULL,
-> `start_datetime` timestamp,
-> `status` varchar(20) NOT NULL DEFAULT '' COMMENT 'COMING, NMB = No More Bets, RESOLVED, TB= Taking Bets',
-> `created_timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
-> `end_datetime` datetime DEFAULT NULL,
-> `external_ref` varchar(50) DEFAULT NULL,
-> `game_room_id` int(11) DEFAULT NULL,
-> PRIMARY KEY (`game_instance_id`,`start_datetime`),
-> KEY `GI_IDX4` (`external_ref`),
-> KEY `GI_IDX5` (`game_id`,`status`),
-> KEY `game_instance_status` (`status`),
-> KEY `game_instance_end_datetime` (`end_datetime`),
-> KEY `game_instance_start_datetime` (`start_datetime`)
-> ) ENGINE=InnoDB AUTO_INCREMENT=118386942 DEFAULT CHARSET=latin1;
Query OK, 0 rows affected (0.14 sec)
mysql> explain select * from game_instance where start_datetime >= unix_timestamp(CONCAT(DATE_SUB(CURDATE(), INTERVAL 30 DAY), ' ', '00:00:00'));
+----+-------------+---------------+------------+------+------------------------------+------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------------+------------+------+------------------------------+------+---------+------+------+----------+-------------+
| 1 | SIMPLE | game_instance | NULL | ALL | game_instance_start_datetime | NULL | NULL | NULL | 1 | 100.00 | Using where |
+----+-------------+---------------+------------+------+------------------------------+------+---------+------+------+----------+-------------+
1 row in set, 3 warnings (0.00 sec)
I have an index on start_datetime, but I still get a full table scan, according to explain.
However:
mysql> create table ex1(
-> id bigint(20),
-> start_datetime timestamp,
-> primary key (id,start_datetime),
-> key (start_datetime)
-> );
Query OK, 0 rows affected (0.02 sec)
mysql> explain select * from ex1 where start_datetime>=unix_timestamp(CONCAT(DATE_SUB(CURDATE(), INTERVAL 30 DAY), ' ', '00:00:00'));
+----+-------------+-------+------------+-------+----------------+----------------+---------+------+------+----------+--------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+----------------+----------------+---------+------+------+----------+--------------------------+
| 1 | SIMPLE | ex1 | NULL | index | start_datetime | start_datetime | 4 | NULL | 1 | 100.00 | Using where; Using index |
+----+-------------+-------+------------+-------+----------------+----------------+---------+------+------+----------+--------------------------+
1 row in set, 3 warnings (0.00 sec)
The warnings are:
mysql> show warnings;
+---------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Level | Code | Message |
+---------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Warning | 1292 | Incorrect datetime value: '1563663600' for column 'start_datetime' at row 1 |
| Warning | 1292 | Incorrect datetime value: '1563663600' for column 'start_datetime' at row 1 |
| Note | 1003 | /* select#1 */ select `ex`.`ex1`.`id` AS `id`,`ex`.`ex1`.`start_datetime` AS `start_datetime` from `ex`.`ex1` where (`ex`.`ex1`.`start_datetime` >= <cache>(unix_timestamp(concat((curdate() - interval 30 day),' ','00:00:00')))) |
+---------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
3 rows in set (0.00 sec)
This seems to suggest that start_datetime is silently converted in the background, which would explain why the index is not used, but then why does it not happen in both queries? (And as a corollary, how do I convert my date string to whatever the MySQL TIMESTAMP is?)
EDIT 2:
I've run optimize on the table, as suggested in comments (I haven't run the analyze, since it seems to have done that already):
mysql> optimize table game_instance;
+-----------------------+----------+----------+-------------------------------------------------------------------+
| Table | Op | Msg_type | Msg_text |
+-----------------------+----------+----------+-------------------------------------------------------------------+
| gameiom.game_instance | optimize | note | Table does not support optimize, doing recreate + analyze instead |
| gameiom.game_instance | optimize | status | OK |
+-----------------------+----------+----------+-------------------------------------------------------------------+
2 rows in set (21 min 31.80 sec)
However, it made no difference:
mysql> explain select * from game_instance
where start_datetime >= timestamp(CONCAT(DATE_SUB(CURDATE(), INTERVAL 30 DAY), ' ', '00:00:00')) and
start_datetime <= timestamp(CONCAT(DATE_SUB(CURDATE(), INTERVAL 1 DAY), ' ', '23:59:59'));
+----+-------------+---------------+------+------------------------------+------+---------+------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------+------+------------------------------+------+---------+------+----------+-------------+
| 1 | SIMPLE | game_instance | ALL | game_instance_start_datetime | NULL | NULL | NULL | 19065747 | Using where |
+----+-------------+---------------+------+------------------------------+------+---------+------+----------+-------------+
1 row in set (0.00 sec)
This is a real problem, since the table is 19m rows (not 11m as I said earlier).
Sometimes the query planner makes decisions about whether to scan the whole table or use the index based on statistics about the number and distribution of values in the index. Sometimes it guesses that a full table scan will take less CPU and IO resources than a table lookup.
When tables have small numbers of rows, the query planner's choices often don't match intuition. Make sure you have a few thousand rows at least, before you spend a lot of time trying to make sense of EXPLAIN output.
Also, the query planner gets better at its job with each MySQL release.
Do OPTIMIZE TABLE game_instance to clean up your table, especially if you have inserted many rows.
Then do ANALYZE TABLE game_instance to recompute the statistics used by the query planner.
By the way,
where start_datetime>=unix_timestamp(CONCAT(DATE_SUB(CURDATE(), INTERVAL 30 DAY), ' ', '00:00:00'));
is precisely the same as
where start_datetime >= DATE_SUB(CURDATE(), INTERVAL 30 DAY)
MySQL knows how to use the results of date computations directly in TIMESTAMP filters, and UNIX_TIMESTAMP() yields integers, not TIMESTAMPs.
About your invalid timestamp warning, may I suggest you ask another question? Please include your time zone setting in the question.
The answer by O. Jones was correct, but let me just add some notes about what I did to find out. What I was seeing was this, which I couldn't understand:
mysql> explain extended
select * from game_instance
where
start_datetime >= timestamp(CONCAT(DATE_SUB(CURDATE(), INTERVAL 30 DAY), ' ', '00:00:00')) and
start_datetime <= timestamp(CONCAT(DATE_SUB(CURDATE(), INTERVAL 1 DAY), ' ', '23:59:59'));
+----+-------------+---------------+------+------------------------------+------+---------+------+----------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------------+------+------------------------------+------+---------+------+----------+----------+-------------+
| 1 | SIMPLE | game_instance | ALL | game_instance_start_datetime | NULL | NULL | NULL | 18741262 | 50.00 | Using where |
+----+-------------+---------------+------+------------------------------+------+---------+------+----------+----------+-------------+
1 row in set, 1 warning (0.00 sec)
So, I found that you can force MySQL to use an index, which gave me:
mysql> explain extended select * from game_instance force index (game_instance_start_datetime) where start_datetime >= timestamp(CONCAT(DATE_SUB(CURDATE(), INTERVAL 30 DAY), ' ', '00:00:00')) and start_datetime <= timestamp(CONCAT(DATE_SUB(CURDATE(), INTERVAL 1 DAY), ' ', '23:59:59'));
+----+-------------+---------------+-------+------------------------------+------------------------------+---------+------+---------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------------+-------+------------------------------+------------------------------+---------+------+---------+----------+-------------+
| 1 | SIMPLE | game_instance | range | game_instance_start_datetime | game_instance_start_datetime | 4 | NULL | 9391936 | 100.00 | Using where |
+----+-------------+---------------+-------+------------------------------+------------------------------+---------+------+---------+----------+-------------+
1 row in set, 1 warning (0.00 sec)
IOW, using the index selects about half of all the rows in the table, and now the filtered column makes sense: it is the percentage of rows that are thrown away because they don't match the criteria, which is why MySQL doesn't use the index: it is less efficient because you'd alternate between reading the index and seeking addresses in the table.

Mysql Invalid datetime format: 1292 Incorrect datetime value

mysql> describe taps;
+-------------+-------------+------+-----+-------------------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+-------------+------+-----+-------------------+-------+
| tag_id | int(11) | NO | | NULL | |
| time_stamp | timestamp | NO | | CURRENT_TIMESTAMP | |
| device_id | varchar(45) | YES | | NULL | |
| device_type | varchar(45) | YES | | NULL | |
+-------------+-------------+------+-----+-------------------+-------+
mysql> INSERT INTO `taps` (tag_id, time_stamp) VALUES(0, 1451610061);
ERROR 1292 (22007): Incorrect datetime value: '1451610061' for column 'time_stamp' at row 1
WHY?? I have found many similar questions, but not of them seem quite this black and white.
1451610061 is a valid timestamp. I checked it at http://www.unixtimestamp.com/ and it evaluates as expected.
So, why doesn't MySql like it?
The MySQL timestamp format is 2016-02-13 15:48:29 or Y-m-d H:i:s convert your timestamp to that format first, and then MySQL will accept it.
If you insert a new record without defining the timestamp, and then select the row from that table, you will notice that that's the format that it gives to the new default record.
if if you want to convert it directly in your query use:
INSERT INTO `taps` (tag_id, time_stamp) VALUES(0, from_unixtime('1451610061'));
Using this Q&A on StackOverflow as reference.
And from_unixtime documentation
Try this ->
$item->time_stamp = date('Y-m-d H:i:s', strtotime($request->time_stamp));

Query optimization, select calling function

Consider a large table, which of the below would be faster?
Both queries will select rows where time is greater than the current time.
Calling NOW() within the WHERE clause:
SELECT *
FROM myTable
WHERE time > NOW()
Wrapping the call to NOW() in a sub query:
SELECT *
FROM myTable
LEFT JOIN (
SELECT NOW() AS currentTime
) AS currentTimeTable ON TRUE
WHERE time > currentTime
"Both queries will select rows where time is no older than 10 days."
Sorry, but your query is incorrect. I tested it in MySQL and got this:
mysql> SELECT DATE(NOW() - 10);
+------------------+
| DATE(NOW() - 10) |
+------------------+
| 2013-11-21 |
+------------------+
21.11.2013 - is the current date and not now() minus 10 days.
You should use DATE_SUB function:
mysql> SELECT DATE_SUB( NOW(), INTERVAL 10 DAY );
+-------------------------------------+
| DATE_SUB( NOW() , INTERVAL 10 DAY ) |
+-------------------------------------+
| 2013-11-11 19:40:38 |
+-------------------------------------+
Something like this:
SELECT *
FROM `myTable`
WHERE `time` > DATE_SUB(NOW(), INTERVAL 10 DAY );
Here is the analysis of both of your query types:
EXPLAIN
SELECT
*
FROM
users
WHERE
created > now();
Test data
I tried users table from my Drupal installation.
mysql> EXPLAIN SELECT *
FROM users
WHERE created < NOW();
+----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
| 1 | SIMPLE | users | range | created | created | 4 | NULL | 1 | Using where |
+----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
1 row in set (0.00 sec)
EXPLAIN
SELECT
*
FROM
users
LEFT JOIN (
SELECT NOW() AS currentTime
) AS tbl ON TRUE
WHERE created < tbl.currentTime;
+----+-------------+------------+--------+---------------+---------+---------+------+------+----------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+--------+---------------+---------+---------+------+------+----------------+
| 1 | PRIMARY | <derived2> | system | NULL | NULL | NULL | NULL | 1 | |
| 1 | PRIMARY | users | range | created | created | 4 | NULL | 1 | Using where |
| 2 | DERIVED | NULL | NULL | NULL | NULL | NULL | NULL | NULL | No tables used |
+----+-------------+------------+--------+---------------+---------+---------+------+------+----------------+
3 rows in set (0.02 sec)
Conclusion
Obviously, the 1-st query is better, as it doesn't require creating any temporary tables. Even with my little sample data, it took 0 seconds to execute the 1-st query and 0,02 secs for the 2-nd.
1) When you talk about date optimisation, using timestamps is the way to go.
2) I suggest instead of calling a function to do the math, create another column for each rows with their timestamps
3) As PeeHaa said on this thread: « When talking about which piece of code is faster (in your OP) you're talking about micro-optimization and is really something you shouldn't have to worry about. ☺
The real question is which piece of code is: better maintainable, readable, understandable. »
I understand what is very large database management but once you've optimised indexes and datatypes you most likely will be good enough.

Fetching data's using indexed column vs MySQL range criteria which is best?

TABLE User
User Table Structure .
mysql> desc User;
+--------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+----------------+
| ID | int(11) | NO | PRI | NULL | auto_increment |
| EMAIL_ID | varchar(250) | YES | | NULL | |
| IP_ADDRESS | varchar(255) | YES | | NULL | |
| CREATED_TIME | bigint(20) | YES | | NULL | |
+--------------+--------------+------+-----+---------+----------------+
4 rows in set (0.00 sec)
This table contains million's of rows and it will be gradually increasing on day by day .
Goal :
To getting past 12 month's user details from this table .
First get a user id was created before 12 month's . My query look like this .
Option 1:
Select * from User where ID > `Account created before 12 months` .
Option 2:
Select * from User where CREATED_TIME > UNIX_TIMESTAMP(`2011-01-2011 00:00:00`)*1000;
Which is efficient for fetching details . And this query will be used redudantly for audit purpose .
Try to avoid calling functions on each row. You can write your first part of the WHERE clause like this to speed up a lot (especially if you couple it with an index on the CREATED_TIME field):
Accounts.CREATED_TIME
BETWEEN UNIX_TIMESTAMP(CURRENT_DATE() - INTERVAL 1 DAY) * 1000
AND UNIX_TIMESTAMP(CURRENT_DATE() - INTERVAL 1 DAY) * 1000 + 999
Note that this will make the function calls only once and indices on the CREATED_TIME field can be used to resolve the query.

Converting MySQL warning into an error

Is there any way to convert the warning that MySQL is issuing about an invalid datetime into a hard error? I've tried using SET sql_mode='TRADITIONAL'; which apparently is supposed to turn (some) things that are warnings into errors, but it does not have any effect here. This is MySQL 5.1.56. Something that works on a session-level would be ideal, but I'll take what I can get.
mysql> describe test_table2;
+----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+-------------+------+-----+---------+-------+
| value | int(11) | YES | | NULL | |
| name | varchar(16) | YES | | NULL | |
| sometime | datetime | YES | | NULL | |
+----------+-------------+------+-----+---------+-------+
3 rows in set (0.00 sec)
mysql> select * from test_table2;
+-------+-------+---------------------+
| value | name | sometime |
+-------+-------+---------------------+
| 1 | one | 2002-09-01 10:00:00 |
| 2 | two | 2002-09-02 11:00:00 |
| 3 | three | 2002-09-03 12:00:00 |
| 4 | four | 2002-01-04 13:00:00 |
| 5 | five | 2002-01-05 14:00:00 |
+-------+-------+---------------------+
5 rows in set (0.00 sec)
mysql> select * from test_table2 where sometime = 'foo';
Empty set, 2 warnings (0.00 sec)
Warning (Code 1292): Incorrect datetime value: 'foo' for column 'sometime' at row 1
Warning (Code 1292): Incorrect datetime value: 'foo' for column 'sometime' at row 1
With SET sql_mode='TRADITIONAL', doing an INSERT with an invalid date causes an error, but doing a SELECT with an invalid date still causes a warning. You can trigger the error by passing the (possibly invalid) date value to this query first:
CREATE TEMPORARY TABLE IF NOT EXISTS date_guard (date DATE) SELECT 'foo' AS date;
where 'foo' is the date value you want to validate.
Who is supposed to see the error?
If this is a fixed string 'foo' just try converting 'foo' to a date and see if you can a valid result (i.e. not 00-00-000). Do a pre-query to check the validity of the date, and then continue after.
I have not been able to make MySQL give an error in this case (or even convert the invalid date to a NULL - it insists on making it 00-00-0000).