MySQL: Incorrect Date Value - mysql

I am parsing an xml file that has following data structure:
<row Id="253858" UserId="40883" Name="Scholar" Date="2009-03-08T01:52:32.570" />
<row Id="253860" UserId="19483" Name="Supporter" Date="2009-03-08T01:57:31.733" />
<row Id="253861" UserId="74951" Name="Autobiographer" Date="2009-03-08T02:02:32.390" />
I used a ruby script to parse this data and insert them into a mysql database. Here is how my data table looks like:
+---------+-------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+---------+-------------+------+-----+-------------------+-----------------------------+
| id | int(11) | NO | PRI | NULL | |
| user_id | int(11) | NO | | NULL | |
| name | varchar(40) | YES | | NULL | |
| created | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
+---------+-------------+------+-----+-------------------+-----------------------------+
This xml file has large number of records and until the parser gets to 3rd record I don't get any error and all the data get parse correctly and insert into the table:
| 253857 | 23658 | Critic | 2009-03-08 01:52:32 |
| 253858 | 40883 | Scholar | 2009-03-08 01:52:33 |
| 253860 | 19483 | Supporter | 2009-03-08 01:57:32 |
+--------+---------+--------------------+---------------------+
But when we get to the record with row Id="253861" I get the following mysql error:
load.rb:21:in `execute': Incorrect datetime value: '2009-03-08T02:02:32.390' for column 'created' at row 1 (Mysql::Error)
from load.rb:21:in `on_start_element'
from load.rb:133:in `parse'
from load.rb:133:in `<main>'
incase if you need the ruby method that insert records:
def on_start_element(element, attributes)
if element == 'row'
#st.execute(attributes['Id'], attributes['UserId'], attributes['Name'], attributes['Date'])
end
end
end
I don't think this is related to script, because I extracted the record from xml file and tried to insert directly into my mysql table and I got the following error:
mysql> insert into badge values(253861,74951,'Autobiographer','2009-03-08T02:02:32.390');
ERROR 1292 (22007): Incorrect datetime value: '2009-03-08T02:02:32.390' for column 'created' at row 1
I also tried to insert the record that is after the above record in the xml file and I got the same results:
mysql> insert into badge values(253862,49628,'Teacher','2009-03-08T02:12:30.807');
ERROR 1292 (22007): Incorrect datetime value: '2009-03-08T02:12:30.807' for column 'created' at row 1
So something in that date string makes mysql unhappy. What I couldn't figure out is that date and previous records that has the same date structure didn't have any problem. Hope I have explain the problem clear with information.

Since you said you insert dates in a couple of places I would suggest you write and helper method to do date conversions that you can reuse everywhere you need to insert dates
require 'date'
def iso8601_to_mysql_datetime(date)
DateTime.parse(date).to_time.strftime("%F %T")
end
iso8601_to_mysql_datetime('2009-03-08T02:02:32.390')
=> "2009-03-08 02:02:32"
NOTE: The above converts a ISO8601 into a string that MySQL understands
MySQL date and time literal documentation can be found here:
http://dev.mysql.com/doc/refman/5.6/en/date-and-time-literals.html

Related

Error while updating file into mysql

i am trying to load csv data file into mysql , which later on i need to pass to HDFS but while updating the file
mysql > load data infile '/var/lib/mysql-files/online_retail.csv' into table retail fields terminated by ',' ignore 1 lines;
ERROR : "Incorrect date value: '12/1/2010 8:26' for column
'invoicedate' at row 1"
mysql> DESCRIBE retail;
+-------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+--------------+------+-----+---------+-------+
| InvoiceNo | int(11) | YES | | NULL | |
| stockcode | varchar(20) | YES | | NULL | |
| Description | varchar(500) | YES | | NULL | |
| Quantity | int(11) | YES | | NULL | |
| invoicedate | date | YES | | NULL | |
| UnitPrice | double | YES | | NULL | |
| CustomerId | int(11) | YES | | NULL | |
| Country | varchar(30) | YES | | NULL | |
+-------------+--------------+------+-----+---------+-------+
If you give the data type as "DATE", Date format should be in the form of "YYYY-MM-DD" or "YY-MM-DD".
Either make the data type as STRING, or transform the data to the YYYY-MM-DD format...
I.e from 12/1/2010 8:26 to 2010/12/01
Hope this helps
As said eggyal and as documented under Date and Time Literals:
MySQL recognizes DATE values in these formats:
As a string in either 'YYYY-MM-DD' or 'YY-MM-DD' format. A “relaxed” syntax is permitted: Any punctuation character may be used as the delimiter between date parts. For example, '2012-12-31', '2012/12/31', '2012^12^31', and '2012#12#31' are equivalent.
As a string with no delimiters in either 'YYYYMMDD' or 'YYMMDD' format, provided that the string makes sense as a date. For example, '20070523' and '070523' are interpreted as '2007-05-23', but '071332' is illegal (it has nonsensical month and day parts) and becomes '0000-00-00'.
As a number in either YYYYMMDD or YYMMDD format, provided that the number makes sense as a date. For example, 19830905 and 830905 are interpreted as '1983-09-05'.
And for your specific case, a few lines below :
MySQL recognizes DATETIME and TIMESTAMP values in these formats:
As a string in either 'YYYY-MM-DD HH:MM:SS' or 'YY-MM-DD HH:MM:SS' format. A “relaxed” syntax is permitted here, too: Any punctuation
character may be used as the delimiter between date parts or time
parts. For example, '2012-12-31 11:30:45', '2012^12^31 11+30+45',
'2012/12/31 11*30*45', and '2012#12#31 11^30^45' are equivalent.
The only delimiter recognized between a date and time part and a fractional seconds part is the decimal point.
The date and time parts can be separated by T rather than a space. For example, '2012-12-31 11:30:45' '2012-12-31T11:30:45' are
equivalent.
As a string with no delimiters in either 'YYYYMMDDHHMMSS' or 'YYMMDDHHMMSS' format, provided that the string makes sense as a date.
For example, '20070523091528' and '070523091528' are interpreted as
'2007-05-23 09:15:28', but '071122129015' is illegal (it has a
nonsensical minute part) and becomes '0000-00-00 00:00:00'.
As a number in either YYYYMMDDHHMMSS or YYMMDDHHMMSS format, provided that the number makes sense as a date. For example,
19830905132800 and 830905132800 are interpreted as '1983-09-05 13:28:00'
.
Therefore your expression is not a valid MySQL neither a date nor a timestamp literal. In order to comply with one of the literal formats detailed above, you must either create a script that makes your time data comply with the DATETIME or TIMESTAMP format and probably update your database to change the invoicedate type.

How to select a line from a MySQL text field

Say I have a mysql table which contains two columns, one is a job number, and the other contains the stderr of the respective job:
+----------+---------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+---------+------+-----+---------+-------+
| job | int(11) | YES | UNI | NULL | |
| jerrors | text | YES | | NULL | |
+----------+---------+------+-----+---------+-------+
The text contains newline characters.
I would like to select from jerrors only the lines containing a given word, i.e. something like
(\n.*word*.*\n)
How would I construct such query? If I do
SELECT job, jerrors FROM mytable WHERE jerrors LIKE "%No such file%";
then I get the whole text in the column.
Here's a sample text
Error in <TCling::RegisterModule>: cannot find dictionary module A2Dict_rdict.pcm
Error in <TCling::RegisterModule>: cannot find dictionary module A1Dict_rdict.pcm
Error in <TCling::RegisterModule>: cannot find dictionary module SpectraDict_rdict.pcm
PedestalT0 File : conf/A2-roT0.job1685_0-run1685_2.dat: No such file or directory
[A1DataDecoder] WARNING: 12 Slots for 20 pedestals! Using '16' for last 19 pedestals ...
Error in <TInterpreter::TCling::AutoLoad>: failure loading library libMathMore.so for ROOT::Math::GSMIntegrator

mysql issue when reading in Date type from .txt file with str_to_date

I have been given the task of reading in a .txt file into a mySQL table.
Here is what I did.
CREATE TABLE PERSON(PIDM INT, FNAME VARCHAR(20), LNAME VARCHAR(20), SSN INT(8), DOB DATE, GENDER CHAR(2));
LOAD DATA LOCAL INFILE '/pathtoTextFile'
INTO TABLE PERSON
FIELDS TERMINATED BY ', '
LINES TERMINATED BY '\n'
IGNORE 1 ROWS
(PIDM, FNAME, LNAME, SSN, #DATE, GENDER)
SET DOB = STR_TO_DATE(#DATE, '%m-%d-%Y');
But I get several warnings from doing this.
including:
+---------+------+-----------------------------------------------------------------+
| Level | Code | Message |
+---------+------+-----------------------------------------------------------------+
| Warning | 1411 | Incorrect datetime value: '12/14/1957' for function str_to_date |
| Warning | 1411 | Incorrect datetime value: '01/13/1969' for function str_to_date |
| Warning | 1264 | Out of range value for column 'SSN' at row 3 |
| Warning | 1411 | Incorrect datetime value: '02/14/1976' for function str_to_date |
| Warning | 1366 | Incorrect integer value: '' for column 'SSN' at row 4 |
| Warning | 1411 | Incorrect datetime value: '03/12/1989' for function str_to_date
I haven't been able to figure out why I get an invalid date from the str_to_date function/method.
This is supposed to be a learning excercise but I have been at it for like 2 hours and haven't gotten anywhere. Does anyone have any tips or any idea of what I'm doing wrong?
Thank you
P.S. The purpose is not to change the .txt file for it to work, but to be able to handle incorrect inputs.
It depends on the source data but if there is a mixture of MM/DD/YYYY and MM-DD-YYYY you could replace the unwanted delimiter with the wanted delimiter, then convert to date.
Note: because Month First and Day First formats are "ambiguous" (e.g. 7/8/2014 could be a date in July or August) they should be avoided whenever possible.

Mysql date warning data truncated

I'm having an interesting issue with Mysql DATE format.
I have this table :
| id | int(11) | NO | PRI | NULL | auto_increment |
| file_path | varchar(255) | YES | | NULL | |
| date_export | date | YES | | NULL | |
When i'm updating a row using the date function : NOW(), the date is updated with this format :
'2014-01-23'
But when i'm using another date format, like hand-written one like :
update backup_conf_allied set date_export='2014-23-01' where file_path='IDF-952584-SW1' ;
The date_export column transforms into :
'0000-00-00'
Warning table tells me that :
| Warning | 1265 | Data truncated for column 'date_export' at row 3628 |
Why? The date format is the same as NOW() function.
Thanks.
Posted query
update backup_conf_allied set `date_export='2014-23-01'` where file_path='IDF-952584-SW1' ;
What it should be
update backup_conf_allied set `date_export='2014-01-23'` where file_path='IDF-952584-SW1' ;
MySQL Support DATE format as 'YYYY-MM-DD' , Year then Month then Date, So you are updating a Date column with wrong value "2014-23-01" , There are only 12 months in year, you are using month 23 which is invalid that's MySQL is converting it to ZERO DATE (0000-00-00)
I had a similar problem recently, my issue was that I was trying to upload a datetime object as a date object. Although that's not the same issue that Gui O was having, if you run into this, make sure that you're actually trying to upload a date object.
I'm having this same problem but for a different reason. My MySQL column has a datetime type, but my datetime values are from Python, and they look like this: `'2021-04-01 03:58:50.088087'.
So the result is to truncate before the period:
t = t[0:19]
e.g.:
>>> datetime.datetime.now().isoformat()
'2021-04-03T08:28:41.602373'
>>> datetime.datetime.now().isoformat()[0:19]
'2021-04-03T08:28:44'
>>>

View hook only displays 1969 dates

I have a table that looks like:
+--------------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+-------------+------+-----+---------+-------+
| ProductsDownloadId | int(11) | NO | PRI | 0 | |
| RCContactID | int(11) | NO | MUL | NULL | |
| product_name | varchar(50) | YES | MUL | NULL | |
| download_date | timestamp | YES | | NULL | |
+--------------------+-------------+------+-----+---------+-------+
I'm writing a module for this table to be viewable in a Drupal 6 View.
I followed the example I found here:
http://drupalcontrib.org/api/drupal/contributions--views--docs--docs.php/function/hook_views_data/6
So I exposed the download_date as thus:
$data['products_downloaded']['download_date']=array(
'title'=>t("Download Date"),
'help'=>t("When Product was downloaded by the user"),
'field' => array(
'handler' => 'views_handler_field_date',
'click sortable' => TRUE,
),
'sort' => array(
'handler' => 'views_handler_sort',
),
'filter' => array(
'handler' => 'views_handler_filter_date',
),
);
But when I add it to a view, all the dates are displayed as "12/31/1969 - 19:33". And none of the dates in my table are:
EDIT: Corrected query:
mysql> select count(1) from products_downloaded where download_date <'2000-12-31 23:59:59.999999';
+----------+
| count(1) |
+----------+
| 0 |
+----------+
1 row in set (0.04 sec)
I also did a custom date format with the format 'r' in the View and I got
Wed, 31 Dec 1969 19:33:31 -0500 for all the dates.
So what did I do wrong on my module?
Your query:
select count(1) from products_downloaded where download_date <'12/31/2000 - 19:33'
Your problem here is that MySQL expects dates to be given in a different format to that.
You need to provide your dates in the following format:
'2007-12-31 23:59:59.999999'
(you can drop the microseconds, seconds, etc to get the precision you need as required)
So in your case, your query should look like this:
select count(1) from products_downloaded where download_date <'2000-12-31 19:33'
This should query the field correctly.
By the way -- If you have dates showing up unexpectedly as 1969, it implies that perhaps you've been using the wrong format in other queries as well. You may want to check that too.
See the MySQL manual page for date times: http://dev.mysql.com/doc/refman/5.1/en/date-and-time-functions.html
What I was able to determine was that PHP or Drupal wasn't able to understand whatever was being returned to the processor. I messed around with a custom hook and got the value to be accepted by the DateTime constructor. From there . . . it was easy to get the date formats back.