I'm new to SQL, currently I'm doing a task about join two datasets, one of the dataset was created by myself, here's the query I used:
USE `abcde`;
CREATE TABLE `test_01`(
`ID` varchar(50) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`NUMBER01` bigint(20) NOT NULL DEFAULT '0',
`NUMBER02` bigint(20) NOT NULL,
`date01` date DEFAULT NULL,
PRIMARY KEY (`ID`, `date01`))
Then I load the data from a csv file to this table, the csv file looks like this:
ID NUMBER01 NUMBER02 DATE01
aaa=ee 12345678 235896578 **2009-01-01T00:00:00**
If I query this newly-created table, it looks like this(the format of the 'DATE01' changes):
ID NUMBER01 NUMBER02 DATE01
aaa=ee 12345678 235896578 **2009-01-01**
Another dataset, I queried and exported to a csv file, the format of the date01 column is like 01/12/1979 and in SQL the format looks like 1979-12-01.
I also usedselect * from information_schema.columns to check the datatype of the columns I need to join, for the newly-created dataset:
The date column for another dataset is:
The differences are:
1. The format of the date column in csv appears different
2. The COLUMN_DEFAULT are different, one is 0000-00-00, another one is NULL.
I wonder the reason why I got empty output is probably because the difference in the 'date' format, but I'm not sure how to make them the same so that I can get something in the output, can someone gave me some hint? Thank you.
the format of the 'DATE01' changes
Of course, DATE datatype does not contain timezone info/component.
I wonder the reason why I got empty output is probably because the difference in the 'date' format
If input value have some disadvantage (like wrong data format) than according value is truncated or is set to NULL. See - you must obtain a bunch of warnings during the importing similar to "truncate incorrect value".
If the date field in CSV have wrong format then you must use intermediate user-defined variable for accepting raw value, and apply proper converting expression to it in SET clause. Like
LOAD DATA INFILE ...
INTO TABLE tablename (field1, ..., #date01)
SET date01 = STR_TO_DATE(#date01, '%d/%m/%Y');
Related
I have a column of decimals, but MySQL is reading them as TEXT. I tried this to alter the column by:
ALTER TABLE `engine_type_project`.`weo_data_eu_test`
CHANGE COLUMN `Mean_GDP_all_time` `Mean_GDP_all_time` DECIMAL(6,2) NULL DEFAULT NULL;
An original value is: 3,282.772
But my code returns it as: 3.00
Prior to this, I attempted:
SELECT CAST('Mean_GDP_all_time' AS DECIMAL(6,2))
FROM weo_data_eu_test;
But the entire column returned as 0.00
In casting non-numeric values to numeric, mysql does not expect commas. So it gives up looking for additional parts of the number after the "3".
Before changing the type, remove the commas with:
update weo_data_eu_test set Mean_GDP_all_time=replace(Mean_GDP_all_time,',','')
fiddle
i have a column with name 'expdate' with type 'varchar' with values like this:
2021-02-27
28-02-2023
29/02/2024
.
.
.
now i want change all date values like format %d-%m-%Y to the %Y-%m-%d and save to new column with type date and use:
UPDATE `Users`
SET `fixed_expdate` = STR_TO_DATE(REPLACE(`expdate`,"/",'-'), "%d-%m-%Y")
where `expdate` != '0000-00-00'
but sql command show error like:
SQL query:
UPDATE `Users` SET `fixed_expdate` = STR_TO_DATE(REPLACE(`expdate`,"/",'-'), "%d-%m-%Y") where `expdate` != '0000-00-00'
MySQL said: Documentation
#1411 - Incorrect datetime value: '2021-02-27' for function str_to_date
i need to sql command,What is your solution?
You must use different converting expressions for each separate source data format. And you must check that only the values which matches the pattern used in current query are included into the convertion.
This can be performed using a lot of separate queries, each converts one of existing source formats.
-- convert the dates which have correct format
UPDATE `Users`
SET `fixed_expdate` = `expdate`
WHERE `expdate` REGEXP '\d\d\d\d-\d\d-\d\d';
-- convert dated which looks like 28-02-2023
UPDATE `Users`
SET `fixed_expdate` = STR_TO_DATE(`expdate`, "%d-%m-%Y")
WHERE `expdate` REGEXP '\d\d-\d\d-\d\d\d\d';
-- convert dated which looks like 29/02/2024
UPDATE `Users`
SET `fixed_expdate` = STR_TO_DATE(`expdate`, "%d\/%m\/%Y")
WHERE `expdate` REGEXP '\d\d\/\d\d\/\d\d\d\d';
-- and so on
After each separate convertion (or after a lot of convertions) you may look at the rows which were not converted
SELECT `expdate`
FROM `Users`
WHERE `fixed_expdate` IS NULL
LIMIT XXX
and build the next converting expression for a pattern which was not used yet - until all values are successfully converted.
MySQL said: Documentation
#1411 - Incorrect datetime value: '2021-02-27' for function str_to_date
This is due to 2 separate formats which differs in day and month posession (which are swapped) - they cannot be distinguished easily. You must use additional checking if both formats are present in the data. And you cannot distinguish what format must be applied if both formats are possible (for example, when the value is '01/02/2021').
UPDATE (copied from the comment)
Your regular expressions should should probably have anchors for the beginning and end of the string ('^' and '$'). – Gordon Linoff
I'm trying to import a XML file into a MySQL Table. In the XML file there is a timestamp in <CurrentTime> in the following format:
2016-01-26T09:52:19.3420655+01:00
This timstamp should go into the corresponding DATETIME CurrentTime column in my Table. So I did the following
LOAD XML INFILE 'xxx.xml'
INTO TABLE test.events
ROWS IDENTIFIED BY '<Event>'
SET CurrentTime = str_to_date(CurrentTime, '%Y-%m-%dT%H:%i:%s.%f');
But it quits with the error
Error Code: 1292. Incorrect datetime value: '2016-01-25T16:22:24.1840792+01:00' for column 'CurrentTime' at row 1
So it seems it doesn't convert the string at all. Why?
I think that error is thrown when the string value from the file is loaded directly to the column. The error is thrown before you get to the SET clause.
Here's an abbreviated example of how to use user-defined variables to pass the value of a field down to the SET, bypassing the assignment to the column.
Note that the columns _row and account_number are populated directly from the first two fields in the file. The later fields in the file are assigned to user-defined variables (identifiers beginning with #.
The SET clause evaluates the user-defined variables, and assigns the result of the expression to the actual column in the table.
In this example, the "dates" were formatted YYYYMMDD. I used the STR_TO_DATE() function to have that string converted to a proper DATE.
I abbreviated this sample somewhat, but it demonstrates the approach of reading field values into user-defined variables.
CREATE TABLE _import_water
(`_row` INT
,`account_number` VARCHAR(255)
,`total_due` DECIMAL(18,2)
,`end_date` DATE
,`start_date` DATE
,`ccf` DECIMAL(18,4)
)
LOAD DATA LOCAL INFILE '//server/share$/users/me/mydir/myfile.csv'
INTO TABLE _import_water
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(_row
,account_number
,#total_due
,#end_date
,#start_date
,#ccf
)
SET `total_due` = NULLIF(#total_due,'')
, `end_date` = STR_TO_DATE(#end_date,'%Y%m%d')
, `start_date` = STR_TO_DATE(#start_date,'%Y%m%d')
, `ccf` = NULLIF(#ccf,'')
Also, it doesn't look like there's any problem with your STR_TO_DATE, it seems to evaluate just fine.
testing...
SELECT STR_TO_DATE('2016-01-25T16:22:24.1840792+01:00','%Y-%m-%dT%H:%i:%s.%f') AS mydatetime
returns:
mydatetime
--------------------------
2016-01-25 16:22:24.184079
Good Morning All;
I currently have a MySQL table where there are 3 date fields (Columns) that were loaded as strings in this format 20140101 YYYYmmdd. I would like to convert this to a date format 2014/01/01 YYYY/mm/dd. Can someone please provide a simple sql syntax that would alter the table to a date format from a string and change the column to display the dates like this 2014/01/01 and not like 20140101. Thanks to all
Try this:
date_format(str_to_date(datecolumn, '%Y%m%d'),'%Y/%m/%d')
If you just want to reformat the values in the VARCHAR column, assuming that the column with sufficient length e.g. VARCHAR(10), and all the values are eight characters in length...
You could do something like this:
UPDATE mytable t
SET t.mycol = CONCAT( LEFT( t.mycol ,4)
, '/'
, SUBSTR( t.mycol ,5,2)
,'/'
, SUBSTR( t.mycol ,7,2)
)
WHERE CHAR_LENGTH(t.mycol) = 8
We want something in the statement that will prevent the statement from "working" a second time, if it's inadvertently re-run. It doesn't have to be CHAR_LENGTH. We might want to include a check that the value doesn't already contain a slash character AND t.mycol NOT LIKE '%/%'.
But why on earth are "date" values being stored in character columns, rather than in DATE datatype, which is custom designed for storing and working with date values?
ALTER TABLE mytable MODIFY mycol DATE ... ;
(If the column is defined as NOT NULL, has a default value, has a comment, those attributes can be retained, they need to be included in the new column specification, e.g.
ALTER TABLE mytable MODIFY mycol DATE NOT NULL COMMENT 'creation date';
Note that DATE columns do not have a "format" per se. When converting to string, MySQL uses date format '%Y-%m-%d'. And MySQL expects string literals representing date values to be in that same format. To get a value from a DATE column converted to string in format 'yyyy/mm/dd'.
SELECT DATE_FORMAT(date_col,'%Y/%m/%d') AS date_col
To get a string value in that format converted to DATE datatype
SELECT STR_TO_DATE('2015/06/01','%Y/%m/%d')
When I import a csv file into MySQL (phpmyadmin), for all integer values that are not specified in the file but have a default of null there is an error message: #1366 - Incorrect integer value: '' for column 'id' at row 1.. I have these questions:
a. How do I import a csv file that does not have the row-id specified if the DB table has that id defined as auto-increment?
b. What do I need in the csv file or in the table column specification in phpmyadmin for integer column that have a default of null?
Here is are sample rows from the csv file.
id,year,month,date,day,description,ranking
,,3,1,,,
,,3,2,,,
,,3,3,,,
,,3,4,,,
,,3,5,,,
,,3,6,,,
,,3,7,,,
,,3,7,,"Saints Perpetua and Felicity, Martyrs",
,,3,8,,,
,,3,8,,"Saint John of God, Religious",
,,3,9,,,
,,3,9,,"Saint Frances of Rome, Religious",
,,3,10,,,
The columns that cause the error are id, year, ranking. They are all integer columns. The column id is auto increment. The other columns are INT(11) with a default of NULL. Thanks.
CSV has no concept of "Nulls". It's impossible to differentiate between a field that is null, and a field that has a legitimately empty value (e.g. empty string). You'll have to massage the rows as you load them prior to query insertion, to replace any 'empty strings' with appropriate NULLs
e.g.
$row = fgetcsv(...);
$row[0] = 'NULL';