There is some funny conversion side-effect that I experience when converting DATETIME column to TIMESTAMP.
There are DATETIME values before conversion:
+---------------------+
| creation_date |
+---------------------+
| 2015-02-18 19:57:52 |
| 2015-02-18 19:57:18 |
| 2015-02-18 19:51:52 |
| 2015-02-18 19:51:29 |
+---------------------+
ALTER TABLE t CHANGE creation_date creation_date TIMESTAMP null DEFAULT null;
Same data after the conversion to TIMESTAMP:
+---------------------+
| creation_date |
+---------------------+
| 2015-02-18 19:58:52 |
| 2015-02-18 19:58:37 |
| 2015-02-18 19:53:52 |
| 2015-02-18 19:52:52 |
+---------------------+
No warnings were given during conversion.
I'm curious what is the reason of this change? Notice, that the delta for every row is different. For the first row it's one minute, for the second 79 seconds.
MySQL version: 5.1.73
I'm going to guess that additional (later) event records were added between your original query and your following query as hinted at by SurgeonofDeath.
Perhaps you could add some ID values to your query... and/or check that the same times identified in the original query are in fact still present in the database?
Related
I have made the following table in MySQL:
mysql> use test;
Database changed
mysql> desc NeoTec_test;
+-------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+-------------+------+-----+---------+-------+
| Product_Key | varchar(10) | NO | PRI | NULL | |
| Validation | date | YES | | NULL | |
| Expiry | date | YES | | NULL | |
+-------------+-------------+------+-----+---------+-------+
3 rows in set (0.03 sec)
mysql> select * from NeoTec_test;
+-------------+------------+------------+
| Product_Key | Validation | Expiry |
+-------------+------------+------------+
| GF427DHH5 | 2017-11-16 | 2017-11-17 |
| GFHJV75HG | 2017-11-16 | 2017-11-18 |
| GFJYFRTV5 | 2017-11-16 | 2017-11-20 |
+-------------+------------+------------+
3 rows in set (0.00 sec)
Now coming to the point, I need some help with a part of my project. I want MySQL to automatically delete the Product keys that have expired, i.e., I want to get the product keys deleted automatically on their expiry dates given under the "Expiry" Column of the table. How can I do so? I am a total newbie to MySQL events so I would appreciate the full code... Thank you! :-)
Earlier research I did was not fruitful, but I did found this, which was half helpful...:
How to delete a MySQL record after a certain time
You can use event scheduler to perform the task like below:
DELIMITER //
CREATE EVENT eventName
ON SCHEDULE EVERY 1 WEEK
STARTS 'Some Date to start'
ENDS 'End date If any'
DO
BEGIN
DELETE FROM NeoTec_test WHERE NOW() > Expiry
END//
DELIMITER ;
Thete is no functionality in mysql to automatically delete a record. You need to trigger the deletion either through a scheduler (mysql's as shown in the question you found, or an external scheduler such as cron), or via a database trigger.
The latter one is probably an overkill.
I would use a scheduler set to a convenient interval based on your business requirements to clean up the table.
As I am not the world's greatest SQLer, I am working up to something big, step by step.
I have a table:
mysql> describe taps;
+---------------+-----------+------+-----+-------------------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+-----------+------+-----+-------------------+-------+
| tag_id | int(11) | YES | | NULL | |
| time_stamp | timestamp | NO | | CURRENT_TIMESTAMP | |
| event_id | int(11) | YES | | NULL | |
| event_station | int(11) | YES | | NULL | |
| device_id | text | YES | | NULL | |
| device_type | text | YES | | NULL | |
+---------------+-----------+------+-----+-------------------+-------+
6 rows in set (0.00 sec)
And would like to select all entries for a given date (today, 12th Feb, '17).
I am trying
mysql> select * from taps WHERE (event_id=4)
AND ((time_stamp >= 1486857600000) AND (time_stamp <= 1486944000000));
Empty set, 2 warnings (0.00 sec)
IMPORTANT: I have simplified things, because I want to compare with variables, which have values which I have obtained from another table, which are also of type timestamp.
Hmmm, warnings ....
mysql> SHOW WARNINGS;
+---------+------+----------------------------------------------------------------------------+
| Level | Code | Message |
+---------+------+----------------------------------------------------------------------------+
| Warning | 1292 | Incorrect datetime value: '1486857600000' for column 'time_stamp' at row 1 |
| Warning | 1292 | Incorrect datetime value: '1486944000000' for column 'time_stamp' at row 1 |
+---------+------+----------------------------------------------------------------------------+
2 rows in set (0.00 sec)
So, I tried casting
select * from taps WHERE (event_id=4)
AND ((time_stamp >= CAST(1486857600000 AS DATETIME))
AND (time_stamp <= CAST(1486944000000 AS DATETIME)));
Empty set (0.00 sec)
Which I don't understand, as I am the table does have some entries today.
mysql> select * from taps order by time_stamp limit 3;
+--------+---------------------+----------+---------------+-----------+-------------+
| tag_id | time_stamp | event_id | event_station | device_id | device_type |
+--------+---------------------+----------+---------------+-----------+-------------+
| 44 | 2017-02-12 15:10:25 | NULL | 16 | NULL | NULL |
| 37 | 2017-02-12 15:10:27 | NULL | 14 | NULL | NULL |
| 50 | 2017-02-12 15:10:28 | NULL | 15 | NULL | NULL |
+--------+---------------------+----------+---------------+-----------+-------------+
3 rows in set (0.00 sec)
What am I doing wrongly? And what should my query be?
MySQL has very confusing terminology for date/time stuff (see here). A timestamp is essentially a datetime with a timezone, because the value is stored as UTC, but reported in the local timezone.
This timestamp is not to be confused with a Unix Timestamp, which is just the number of seconds since 1970-01-01 (or in some cases milliseconds).
In your case, try this:
select t.*
from taps t
where event_id = 4 and
time_stamp >= '2017-02-11 07:00:00' and
time_stamp < '2017-02-12 07:00:00';
You could use from_unixtimestamp(). However, people generally find date formats much easier to read.
Note: I changed the last condition to a strict inequality. This gives you 24 hours with no duplication of time, in case something happens at exactly 2017-02-12 07:00:00.
In SQL, literal TIMESTAMP values are normally supplied as strings, in the format 'YYYY-MM-DD HH:MM:SS'. MySQL does allow some latitude in the actual format. More recent versions of MySQL allow for fractional seconds. See the MySQL Reference Manual for a more complete description.
https://dev.mysql.com/doc/refman/5.6/en/datetime.html
As an example, MySQL would recognize any of these:
'2017-02-12 09:30:45'
'17-02-12 09:30:45'
170212093045
If we have a requirement to supply/specify literal values that represent integer milliseconds since '1970-01-01 00:00:00', we can use a SQL expression to convert those to values that can be compared to TIMESTAMP. As a demonstration:
SELECT '1970-01-01' + INTERVAL 1486857600000 / 1000 SECOND AS ts
If we need to supply integer millisecond values as a literal in a condition in a WHERE clause, then we can use expressions like the one above.
The query in the question could do something like this to compare the value in a TIMESTAMP column
AND time_stamp >= '1970-01-01' + INTERVAL 1486857600000 / 1000 SECOND
AND time_stamp < '1970-01-01' + INTERVAL 1486944000000 / 1000 SECOND
I have a simple table (created by django) - engine InnoDB:
+-------------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+------------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| correlation | double | NO | | NULL | |
| gene1_id | int(10) unsigned | NO | MUL | NULL | |
| gene2_id | int(10) unsigned | NO | MUL | NULL | |
+-------------+------------------+------+-----+---------+----------------+
The table has more than 411 million rows.
(The target table will have around 461M rows, 21471*21470 rows)
My main query looks like this, there might be up to 10 genes specified at most.
SELECT gene1_id, AVG(correlation) AS avg FROM genescorrelation
WHERE gene2_id IN (176829, 176519, 176230)
GROUP BY gene1_id ORDER BY NULL
This query is very slow, it takes almost 2 mins to run:
21471 rows in set (1 min 11.03 sec)
Indexes (cardinality looks strange - too small?):
Non_unique| Key_name | Seq_in_index | Column_name | Collation | Cardinality |
0 | PRIMARY | 1 | id | A | 411512194 |
1 | c_gene1_id_6b1d81605661118_fk_genes_gene_entrez | 1 | gene1_id | A | 18 |
1 | c_gene2_id_2d0044eaa6fd8c0f_fk_genes_gene_entrez | 1 | gene2_id | A | 18 |
I just run select count(*) on that table and it took 22 mins:
select count(*) from predictions_genescorrelation;
+-----------+
| count(*) |
+-----------+
| 411512002 |
+-----------+
1 row in set (22 min 45.05 sec)
What could be wrong?
I suspect that mysql configuration is not set up right.
During the import of data I experienced problem with space, so that might also affected the database, although I ran check table later - it took 2hours and stated OK.
Additionally - the cardinality of the indexes look strange. I have set up smaller database locally and there values are totally different (254945589,56528,17).
Should I redo indexes?
What params should I check of MySQL?
My tables are set up as InnoDB, would MyISAM make any difference?
Thanks,
matali
https://www.percona.com/blog/2006/12/01/count-for-innodb-tables/
SELECT COUNT(*) queries are very slow without WHERE clause or without SELECT COUNT(id) ... USE INDEX (PRIMARY).
to speedup this:
SELECT gene1_id, AVG(correlation) AS avg FROM genescorrelation
WHERE gene2_id IN (176829, 176519, 176230)
GROUP BY gene1_id ORDER BY NULL
you should have composite key on (gene2_id, gene1_id, correlation) in that order. try
About index-cardinality: stats of Innodb tables are approximate, not accurate (sometimes insane). there even was (IS?) a bug-report https://bugs.mysql.com/bug.php?id=58382
Try to ANALIZE table and watch cardinality again
I have two timestamps in different formats.
One is a UTC string that looks like this: "2014-09-19T20:55:51Z"
The other is a unix timestamp that looks like this: 1411159809
These two timestamps happen to be five minutes apart. As I verified with this Epoch Converter tool.
I want to put both of these time stamps (along with the data that they describe) into my database, but when I do they don't agree.
The table I'm using looks like this:
+--------------+--------------+------+-----+---------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------------------+-----------------------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| property_key | int(11) | YES | | NULL | |
| updated_at | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| created_at | timestamp | NO | | 0000-00-00 00:00:00 | |
| value | varchar(128) | YES | | NULL | |
| name | varchar(128) | YES | | NULL | |
+--------------+--------------+------+-----+---------------------+-----------------------------+
As queries, I have:
insert into events (value,property_key,created_at) values ('datapoints/56892923',273827,'2014-09-19T20:55:51Z')
and
insert into events (value,name,created_at) values ('240','HeartRate',FROM_UNIXTIME(1411159809))
When I pull these entries back out of the database, this is what I get:
*************************** 1. row ***************************
id: 1
property_key: 273827
updated_at: 2014-09-19 17:42:44
created_at: 2014-09-19 20:55:51
value: datapoints/56892923
name: NULL
*************************** 2. row ***************************
id: 2
property_key: NULL
updated_at: 2014-09-19 17:43:09
created_at: 2014-09-19 14:50:09
value: 240
name: HeartRate
2 rows in set (0.00 sec)
updated_at becomes the timestamp for the creation of the row, as expected. What bothers me is that created_at is six hours different. I'm guessing MYSQL is interpreting the UTC as local time, but that's not what I want. How do I get these timestamps to behave the way I want?
Reading the manual for FROM_UNIXTIME():
Returns a representation of the unix_timestamp argument as a value in 'YYYY-MM-DD HH:MM:SS' or YYYYMMDDHHMMSS.uuuuuu format, depending on whether the function is used in a string or numeric context. The value is expressed in the current time zone.
Also for the definition of the TIMESTAMP data type:
MySQL converts TIMESTAMP values from the current time zone to UTC for storage, and back from UTC to the current time zone for retrieval.
So, if you're using TIMESTAMP columns you should be OK, all you have to do is use TZ_CONVERT to ensure the timestamp gets formatted as UTC.
Is there any way to convert the warning that MySQL is issuing about an invalid datetime into a hard error? I've tried using SET sql_mode='TRADITIONAL'; which apparently is supposed to turn (some) things that are warnings into errors, but it does not have any effect here. This is MySQL 5.1.56. Something that works on a session-level would be ideal, but I'll take what I can get.
mysql> describe test_table2;
+----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+-------------+------+-----+---------+-------+
| value | int(11) | YES | | NULL | |
| name | varchar(16) | YES | | NULL | |
| sometime | datetime | YES | | NULL | |
+----------+-------------+------+-----+---------+-------+
3 rows in set (0.00 sec)
mysql> select * from test_table2;
+-------+-------+---------------------+
| value | name | sometime |
+-------+-------+---------------------+
| 1 | one | 2002-09-01 10:00:00 |
| 2 | two | 2002-09-02 11:00:00 |
| 3 | three | 2002-09-03 12:00:00 |
| 4 | four | 2002-01-04 13:00:00 |
| 5 | five | 2002-01-05 14:00:00 |
+-------+-------+---------------------+
5 rows in set (0.00 sec)
mysql> select * from test_table2 where sometime = 'foo';
Empty set, 2 warnings (0.00 sec)
Warning (Code 1292): Incorrect datetime value: 'foo' for column 'sometime' at row 1
Warning (Code 1292): Incorrect datetime value: 'foo' for column 'sometime' at row 1
With SET sql_mode='TRADITIONAL', doing an INSERT with an invalid date causes an error, but doing a SELECT with an invalid date still causes a warning. You can trigger the error by passing the (possibly invalid) date value to this query first:
CREATE TEMPORARY TABLE IF NOT EXISTS date_guard (date DATE) SELECT 'foo' AS date;
where 'foo' is the date value you want to validate.
Who is supposed to see the error?
If this is a fixed string 'foo' just try converting 'foo' to a date and see if you can a valid result (i.e. not 00-00-000). Do a pre-query to check the validity of the date, and then continue after.
I have not been able to make MySQL give an error in this case (or even convert the invalid date to a NULL - it insists on making it 00-00-0000).