I have a VARCHAR field completion_date that contains a timestamp date (ex 1319193919). Is it possible to run a query on such a field that compares it with NOW() ? I'm trying:
SELECT * FROM (`surveys`) WHERE `active` = '1' AND `archived` = '0' AND `completion_date` > 'NOW()'
But the results are not really what I'm expecting, is this cause of the VARCHAR? If so, what kind of date field am I better off using? The data must remain a Linux timestamp.
Convert NOW() to a timestamp using UNIX_TIMESTAMP()
SELECT *
FROM (`surveys`)
WHERE `active` = '1' AND `archived` = '0' AND `completion_date` > UNIX_TIMESTAMP(NOW())
Also, remove the quotes you had around 'NOW()'
mysql> SELECT UNIX_TIMESTAMP(NOW());
+-----------------------+
| UNIX_TIMESTAMP(NOW()) |
+-----------------------+
| 1319288622 |
+-----------------------+
N.B. In case you need it, the inverse of this function is FROM_UNIXTIME() to convert a timestamp into the default MySQL DATETIME format.
As mentioned in comments below, if you have the ability to make the change, it is recommended to use a real DATETIME type instead of VARCHAR() for this data.
A Linux timestamp can easily be stored in a BIGINT (or an UNSIGNED INT), which would make the type of comparisons you're trying to do possible. A VARCHAR is going to do a lexical, not numeric, comparison and which is NOT what you want. Using a BIGINT in conjunction with converting NOW() with UNIX_TIMESTAMP() should get you what you want.
It might even be better to store it using a DATETIME data type and do the conversion when you select the data. Storing it as a DATETIME future proofs your application in the event that you move to or add a different platform where a Linux timestamp isn't appropriate. Then you only need to modify your code, not convert your data to have it continue to work. I'm working on a project now where dates were stored as character data and it's been no end of problems getting the old data into shape to use with the new application, though you might experience fewer problems than us because you're storing a time stamp, not a formatted date, with its attendant typos.
Related
I have recently upgraded a MySQL data store from some ungodly many-years-out-of-date version to 8.0.26.
In one particular table I store dates associated with each record, but occasionally there are as-yet-unknown future dates. These have always been stored in the format YYYY-MM-??, so the field type is VARCHAR(10) rather than DATE, as would be expected if it was possible to always be exact. The field data is otherwise reliably YYYY-MM-DD.
However, queries to order this data have recently stopped working as expected, with MySQL reckoning that such an unknown date should be ordered BEFORE an exact date.
A query boils down to something like this: SELECT * FROM table WHERE date_field <= CURDATE()
(Today is 3rd December, so CURDATE is evaluating as 2021-12-03. The same occurs when using the literal string value 2021-12-03 rather than the CURDATE function, so it's definitely a sorting issue rather than clash between data types.)
In those old MySQL versions previously running, 2021-12-?? would evaluate higher/greater than an exact date like 03, and thus not be returned. This would also be expected in line with ASCII sort ordering. Now, however, any such ?? records are also returned, the question mark character apparently being sorted as before/less than a digit.
For the moment I can force the correct and expected behaviour by utilising REPLACE in my query, but this is process-heavy, ugly and inconvenient: SELECT * FROM table WHERE REPLACE(date_field , '??', '99') <= CURDATE()
Can anyone shed some light on why this is occurring and how I might correct it? It is presumably a MySQL bug, given the standard ASCII ordering and the previous experience (of many years standing) of it working correctly?
EDIT: Thanks to the initial replies pointing me to collation. The database uses almost entirely plain English with only occasional accents (etc), so I've rarely had to touch the default settings in the past.
As per ProGu and Álvaro González's responses, I've begun digging around and test queries without the real table/database involved do indeed return as suggested. However, as soon as I attempt to run anything on the real table, it's still not behaving as expected.
The table is on InnoDB, and all tables and (textual) fields across the database are utf8mb4/utf8mb4_0900_ai_ci. I have tried forcing the collation both at query level and by changing the actual table and field collation, yet that pesky 2021-12-?? is always returned, no matter which I choose. I have attempted various query formats to no avail:
SELECT * FROM table WHERE date_field <= CURDATE() ORDER BY date_field COLLATE utf8mb4_0900_ai_ci DESC
SELECT * FROM table WHERE date_field COLLATE utf8mb4_0900_ai_ci <= CURDATE() COLLATE utf8mb4_0900_ai_ci ORDER BY date_field COLLATE utf8mb4_0900_ai_ci DESC
Test based on Álvaro's code, correctly returning 2021-12-03:
with sample_data (sample_value) as (
select '2021-12-??'
union all select '2021-12-03'
)
select *
from sample_data
where sample_value <= CURDATE()
order by sample_value COLLATE utf8mb4_0900_ai_ci DESC LIMIT 1;
Is my collation inexperience showing; have I missed something really obvious?
EDIT 2
All tables and all text columns (plus connection) are already set to utf8mb4 and utf8mb4_0900_ai_ci.
See this DB Fiddle, which also incorrectly returns 2021-12-?? as apparently less than the comparison value (current date, 2021-12-08). I can find no collation that returns the real smaller value (2021-10-31 in the sample data).
Going back to Rick's initial reply:
SELECT "2021-12-??" < "2021-12-03"
This returns 1, i.e. that 03 IS greater than ??. Why? ASCII ordering is clear that digit characters come before - less than - the question mark character.
As in the original version of my post, it seems to me that MySQL is getting character ordering wrong when it is using digits as string rather than int.
Compare:
SELECT "?" < "0"; = 1
SELECT "?" < 0; = 0
This is a collation issue. You're probably relying on the default collation and that has changed.
You can change the collation at query level to figure out which ones suits your need and then adjust the table or column collation accordingly:
with sample_data (sample_value) as (
select '?'
union all select '0'
)
select *
from sample_data
order by sample_value COLLATE utf8mb4_bin;
Result
0
?
with sample_data (sample_value) as (
select '?'
union all select '0'
)
select *
from sample_data
order by sample_value COLLATE utf8mb4_0900_as_cs;
Result
?
0
Demo
Please note I mean collation and not encoding. You should be able to keep your current encoding if it isn't UTF-8.
Edit #1: these snippets are only a tool to decide which collation to choose. The fix to the problem is not to add a random ORDER BY clause at the end of your query, the fix is to change the table (or column) collation:
Edit #2:
where sample_value <= CURDATE() seemingly ignores table collation, but that's probably due to automatic casting from date type. If you force a cast to text things change:
where sample_value <= cast(curdate() as char(10))
Demo
My advice is that you first set a good known default collation everywhere (tables, connection...). It's possible that that will fix all issues.
I think you will need to change the code a little. It is unclear which of the cases below apply to your code.
mysql> SELECT "2021-12-??" < "2021-12-03";
+-----------------------------+
| "2021-12-??" < "2021-12-03" |
+-----------------------------+
| 1 |
+-----------------------------+
mysql> SELECT "2021-12-??" < CURDATE();
ERROR 1525 (HY000): Incorrect DATE value: '2021-12-??'
mysql> SELECT "2021-12-??" < CAST(CURDATE() AS CHAR);
+----------------------------------------+
| "2021-12-??" < CAST(CURDATE() AS CHAR) |
+----------------------------------------+
| 1 |
+----------------------------------------+
mysql> SELECT DATE(NOW()) <= CAST(CURDATE() AS CHAR);
+----------------------------------------+
| DATE(NOW()) <= CAST(CURDATE() AS CHAR) |
+----------------------------------------+
| 1 |
+----------------------------------------+
Look at the following sql.
CREATE SCHEMA IF NOT EXISTS `scheduler`;
USE `scheduler` ;
CREATE TABLE IF NOT EXISTS `scheduler`.`JobHistory` (
`Id` INT NOT NULL AUTO_INCREMENT,
`Job` INT NOT NULL,
`StartTime` TIMESTAMP NOT NULL,
`FinishTime` TIMESTAMP NOT NULL,
PRIMARY KEY (`Id`),
INDEX `fk_JobHistory_Job_idx` (`Job` ASC));
It is throwing ErrorCode: 1067. Invalid default value for 'Finish Time'
But I'm not giving any default value for finish time, also there is another time stamp StartTime which is exactly same and I'm not getting any exception for that one.
Although #jsnplank is right that timestamps are treated differently and you should consider using datetime datatype for these 2 particular columns, however, he fails to explain the error message.
The error message is most likely the result of a combination of how mysql treats timestamp fields when no default value is provided and your sql mode settings.
You define both timestamp columns as not null, without any specific default value set. This means that the 1st timestamp column's default value will be current_timestamp() and will also be updated to current_timestamp() whenever the record changes. This is why the 1st timestamp field does not generate an error message, no matter which of the 2 is the 1st one.
However, the 2nd not null timestamp column's default value will be '0000-00-00 00:00:00' if you do not explicitly define a default value.
See this blog post for more details.
Probably no_zero_date sql mode is also enabled on your server either explicitly or as part of strict sql mode. This sql mode generates an error if you want set '0000-00-00 00:00:00' as a default value or would like to insert this value into any date field.
So, you can use timestamp data type in your table, but make the 2nd one either nullable or provide 0 or any valid date (such as the epoch) as an explicit default value.
Since you are marking start and end dates with these fields, uding datetime instead of timestamp as datatype may be a good idea.
You should be using DATETIME data types for your StartTime and FinishTime columns. TIMESTAMPS have a very specific usage. See http://www.sqlteam.com/article/timestamps-vs-datetime-data-types
Referencing an article from the year 2000 is not really helpful as a lot already changed since then and might not be true any longer. As others already mentioned in other related questions, TIMESTAMP values reference a specific point in time relative to January 1st, 1970 at UTC. DATETIME values on the contrary just store some date and time without a reference to any point in time. They are more like display values, a clock on the wall that shows you some value. A datetime of 2018-01-12 14:00:00 might be a different time in different timezones if you want to keep track of when something happened.
TIMESTAMP on the other hand is always stored as UTC and when read or used in datetime function, automatically is converted back to the connection's or database default timezone. So when your connection is set to +02:00, the actual value that will be stored in the TIMESTAMP column, will be 2018-01-12 12:00:00 instead of 2018-01-12 14:00:00. When you then read the column with a +05:00 connection, you will see 2018-01-12 17:00:00. A DATETIME value would always stay at 2018-01-12 14:00:00 no matter what timezone is set for the database or connection.
So for tracking when something has happened or when something will/should happen, TIMESTAMP is the way to go. When you just want to store a fixed date time independent of the timezone, then use DATETIME (e.g. users shall receive an email at 2 in the morning, as 2 in the morning is the same for everyone).
If I use a QSqlTableModel to access a MySQL database I can convert a TIMESTAMP field using the following:
QDateTime dateTime = index(section, column).data().toDateTime();
QString str = dateTime.toString("yyyy-MM-dd hh:mm:ss.zzz");
So str shows, i.e. 2014-06-22 22:11:44.221.
But I want to access the database using QSqlQuerry, so I do:
QDateTime dateTime = query.value(column).toDateTime();
str = dateTime.toString("yyyy-MM-dd hh:mm:ss.zzz");
But now I'm missing the milliseconds, str shows 2014-06-22 22:11:44.000. What's the proper way to see the milliseconds?
If I do str = query.value(column).toString(); then I get 2014-06-22T22:11:44.
From this page:
https://dev.mysql.com/doc/refman/5.6/en/datetime.html
A DATETIME or TIMESTAMP value can include a trailing fractional
seconds part in up to microseconds (6 digits) precision. In
particular, as of MySQL 5.6.4, any fractional part in a value inserted
into a DATETIME or TIMESTAMP column is stored rather than discarded.
So, the millisecond is there in MySQL! But the query.value() does not get it - at this point in the Qt history as pointed by #peppe here.
Relating back to the original question: There is no proper way to see the millisecond since the query does not have it. One alternative could be to modify the query, from:
SELECT timestamp FROM table;
to
SELECT DATE_FORMAT(timestamp, '%Y-%c-%e %H:%i:%s.%f') as timestamp FROM table;
And then finish the job with:
QString str = query.value(column).toString();
QDateTime dateTime = QDateTime::fromString(str, "yyyy-MM-dd hh:mm:ss.zzz000");
I got the insight from here.
From MySQL 5.1 documentation:
A DATETIME or TIMESTAMP value can include a trailing fractional seconds part in up to microseconds (6 digits) precision. Although this fractional part is recognized, it is discarded from values stored into DATETIME or TIMESTAMP columns.
It seems like seconds is the best you can do with timestamp.
I have a column called "Time" this column is currently set to Varchar since i have some corrupted Time in some updates.Some Time has non ASCII characters etc. So how can i sort out all these corrupted and non properly formated time fields and set to NULL ? So that i can safely convert the Time column back to DateTime. The normal time fields in the updates are usually in the format of 2013-07-24 14:37:56
I was thinking of sorting it out by doing something like :
SELECT * FROM updates WHERE TIME not LIKE '....-..-.. ..:..:..'
But i don't know if that is the right regex approach and most efficient.
You could try using MySQL STR_TO_DATE on your varchar column:
update updates set `time` = null
where str_to_date(`time`,'%Y-%m-%d %H:%i:%s') is null;
So how can i sort out all these corrupted and non properly formated time fields and set to NULL ?
If CAST of such TIME column returns a NULL, then you can set it to NULL or a desired date time string value.
Example with Select:
mysql> select #ts:=cast( '2013-07╞ƒ~¥14:37:56' as datetime ) ts, #ts is null;
+------+-------------+
| ts | #ts is null |
+------+-------------+
| NULL | 1 |
+------+-------------+
Example for update:
update table_name
set time_coumn = null -- or a valid date time string value
where cast( time_column as datetime ) is null
Once you have set the column value to a NULL, you can then set valid datetime values for those records which have NULL values.
SELECT * FROM updates WHERE DATE_FORMAT(`TIME`, '%Y-%m-%d %H:%i:%s') IS NULL
Function DATE_FORMAT will try to format TIME in 'Y-m-d H:i:s' format, if function is not able to format TIME it will return NULL, so that's how you'll know which one is not in appropriate format and then you can handle them manually or set them to null with UPDATE query
Not a very good title, so my apologies.
For some reason, (I wasn't the person who did it, i digress) we have a table structure where the field type for a date is varchar. (odd).
We have some dates, such as:
1932-04-01 00:00:00 and 1929-07-04 00:00:00
I need to do a query which will convert these date strings into a unix time stamp, however, in mySQL if you convert a date which is before 1970 it will return 0.
Any ideas?
Thanks so much!
EDIT: Wrong date format. ooops.
Aha! We've found a solution!
The SQL to do it:
SELECT DATEDIFF( STR_TO_DATE('04-07-1988','%d-%m-%Y'),FROM_UNIXTIME(0))*24*3600 -> 583977600
SELECT DATEDIFF( STR_TO_DATE('04-07-1968','%d-%m-%Y'),FROM_UNIXTIME(0))*24*3600 -> -47174400
This could be useful for future reference.
You can test it here: http://www.onlineconversion.com/unix_time.htm
I've adapted the DATEDIFF workaround to also include time not just days. I've wrapped it up into a stored function, but you can just extract the SELECT part out if you don't want to use functions.
DELIMITER |
CREATE FUNCTION SIGNED_UNIX_TIMESTAMP (d DATETIME)
RETURNS BIGINT
DETERMINISTIC
BEGIN
DECLARE tz VARCHAR(100);
DECLARE ts BIGINT;
SET tz = ##time_zone;
SET time_zone = '+00:00';
SELECT DATEDIFF(d, FROM_UNIXTIME(0)) * 86400 +
TIME_TO_SEC(
TIMEDIFF(
d,
DATE_ADD(MAKEDATE(YEAR(d), DAYOFYEAR(d)), INTERVAL 0 HOUR)
)
) INTO ts;
SET time_zone = tz;
return ts;
END|
DELIMITER ;
-- SELECT UNIX_TIMESTAMP('1900-01-02 03:45:00');
-- will return 0
-- SELECT SIGNED_UNIX_TIMESTAMP('1900-01-02 03:45:00');
-- will return -2208888900
convert these date strings into a
unix time stamp
Traditional Unix timestamps are an unsigned integer count of seconds since 1-Jan-1970 therefore can't represent any date before that.
At best you will have mixed results depending on the system you are using to represent the timestamp.
From wikipedia
There was originally some controversy
over whether the Unix time_t should
be signed or unsigned. If unsigned,
its range in the future would be
doubled, postponing the 32-bit
overflow (by 68 years). However, it
would then be incapable of
representing times prior to 1970.
Dennis Ritchie, when asked about this
issue, said that he hadn't thought
very deeply about it, but was of the
opinion that the ability to represent
all times within his lifetime would be
nice. (Ritchie's birth, in 1941, is
around Unix time −893 400 000.) The
consensus is for time_t to be signed,
and this is the usual practice. The
software development platform for
version 6 of the QNX operating system
has an unsigned 32-bit time_t, though
older releases used a signed type.
It appears that MySQL treats timestamps as an unsigned integer, meaning that times before the Epoc will all resolve to 0.
This being the case, you always have the option to implement your own unsigned timestamp type and use that for your calculations.
If its feasible for your problem, you could shift all your mysql times by, say 100 years, and then work with those adjusted timestamps or re calculate the negative timestamp value.
As some have said, make sure your system is using 64bits to represent the timestamp otherwise you'll hit the year 2038 problem.
I have not tried the above solutions but this might in case you are not able to retrieve the date value from the MySQL database in the form of timestamp, then this operation can also be tried
SELECT TIMESTAMPDIFF(second,FROM_UNIXTIME(0),'1960-01-01 00:00:00');
To get the max. Range +/- wise use this query on your birthday field, in my case "yyyy-mm-dd" but you can change it to your needs
select name, (#bday:=STR_TO_DATE(birthday,"%Y-%m-%d")),if(year(#bday)<1970,UNIX_TIMESTAMP(adddate(#bday, interval 68 year))-2145916800,UNIX_TIMESTAMP(#bday)) from people
I feel like we're making this much too difficult...
Use my functions below so you can convert anything to and from unix timestamps, much like you do in a browser.
Call functions like this:
select to_unix_time('1776-07-04 10:02:00'),
from_unix_time(-6106024680000);
By compiling these:
delimiter $$
create function to_unix_time (
p_datetime datetime
) returns bigint
deterministic
begin
declare v_ret bigint;
select round(timestampdiff(
microsecond,
'1970-01-01 00:00:00',
p_datetime
) / 1000, 0)
into v_ret;
return v_ret;
end$$
create function from_unix_time (
p_time bigint
) returns datetime(6)
deterministic
begin
declare v_ret datetime(6);
select '1970-01-01 00:00:00' +
interval (p_time * 1000) microsecond
into v_ret;
return v_ret;
end$$
Use Date instead of timestamps. Date will solve your Problems.
check this link