When I import a csv file into MySQL (phpmyadmin), for all integer values that are not specified in the file but have a default of null there is an error message: #1366 - Incorrect integer value: '' for column 'id' at row 1.. I have these questions:
a. How do I import a csv file that does not have the row-id specified if the DB table has that id defined as auto-increment?
b. What do I need in the csv file or in the table column specification in phpmyadmin for integer column that have a default of null?
Here is are sample rows from the csv file.
id,year,month,date,day,description,ranking
,,3,1,,,
,,3,2,,,
,,3,3,,,
,,3,4,,,
,,3,5,,,
,,3,6,,,
,,3,7,,,
,,3,7,,"Saints Perpetua and Felicity, Martyrs",
,,3,8,,,
,,3,8,,"Saint John of God, Religious",
,,3,9,,,
,,3,9,,"Saint Frances of Rome, Religious",
,,3,10,,,
The columns that cause the error are id, year, ranking. They are all integer columns. The column id is auto increment. The other columns are INT(11) with a default of NULL. Thanks.
CSV has no concept of "Nulls". It's impossible to differentiate between a field that is null, and a field that has a legitimately empty value (e.g. empty string). You'll have to massage the rows as you load them prior to query insertion, to replace any 'empty strings' with appropriate NULLs
e.g.
$row = fgetcsv(...);
$row[0] = 'NULL';
Related
I am trying to insert data from an UTF-8 encoded CSV file into Redshift database but I get the error when attempting to insert timestamp into a column which has timestamp data type.
Here's a sample CSV:
employeeId,employeeDept,employeeName,shiftStartTime,shiftEndTime,onPremises
KL214691,John Smith,operations,2023-01-17 09:01:34,2023-01-17 16:52:41,1
KL214692,Samantha Kennedy,operations,2023-01-17 08:31:54,2023-01-17 16:09:10,1
Here's a sample table DDL:
create table historical_metrics_agent_status_time_on_status
(
employeeid varchar(10),
employeename varchar(100),
employeedept varchar(50),
shiftstarttime timestamp encode az64,
shiftendtime timestamp encode az64,
onpremises boolean,
importdatetime timestamp encode az64
)
sortkey (employeeid);
The error message shows that there's an invalid digit - on position 4 in column shiftstarttime which has raw field value 2023-01-17 09:01:34. It looks like it's not reading timestamp from CSV file properly. Is there something I'm missing in CSV?
Check stl_load_errors for the exact row that is failing. My guess is that one of the VARCHAR columns has a comma (,) in it and is throwing off the alignment of the CSV to table columns. Like if one of the names is entered as “Smith, Joe”.
UPDATE kpi.data
SET MetricValue = ''
WHERE (MetricValue IS NULL )
and PeriodDate = '2020-01-02'
and ReportID = 4
I got this error
Msg 8114, Level 16, State 5, Line 4 Error converting data type varchar
to numeric.
What you are trying to do does not make any sense, so it cannot work, and it will not work.
If the data type of a column is numeric then the column can only contain numbers.
If the data type of a column is numeric and nullable the column can contain either a number or null.
There is no other value that a numeric field can receive, either in MySQL or in any other relational database that I have ever heard of.
Perhaps what you want to do is to convert null to blank when selecting, (not when inserting/updating,) in which case you should look at some other Q&A like this one: MySql Query Replace NULL with Empty String in Select
I'm new to SQL, currently I'm doing a task about join two datasets, one of the dataset was created by myself, here's the query I used:
USE `abcde`;
CREATE TABLE `test_01`(
`ID` varchar(50) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`NUMBER01` bigint(20) NOT NULL DEFAULT '0',
`NUMBER02` bigint(20) NOT NULL,
`date01` date DEFAULT NULL,
PRIMARY KEY (`ID`, `date01`))
Then I load the data from a csv file to this table, the csv file looks like this:
ID NUMBER01 NUMBER02 DATE01
aaa=ee 12345678 235896578 **2009-01-01T00:00:00**
If I query this newly-created table, it looks like this(the format of the 'DATE01' changes):
ID NUMBER01 NUMBER02 DATE01
aaa=ee 12345678 235896578 **2009-01-01**
Another dataset, I queried and exported to a csv file, the format of the date01 column is like 01/12/1979 and in SQL the format looks like 1979-12-01.
I also usedselect * from information_schema.columns to check the datatype of the columns I need to join, for the newly-created dataset:
The date column for another dataset is:
The differences are:
1. The format of the date column in csv appears different
2. The COLUMN_DEFAULT are different, one is 0000-00-00, another one is NULL.
I wonder the reason why I got empty output is probably because the difference in the 'date' format, but I'm not sure how to make them the same so that I can get something in the output, can someone gave me some hint? Thank you.
the format of the 'DATE01' changes
Of course, DATE datatype does not contain timezone info/component.
I wonder the reason why I got empty output is probably because the difference in the 'date' format
If input value have some disadvantage (like wrong data format) than according value is truncated or is set to NULL. See - you must obtain a bunch of warnings during the importing similar to "truncate incorrect value".
If the date field in CSV have wrong format then you must use intermediate user-defined variable for accepting raw value, and apply proper converting expression to it in SET clause. Like
LOAD DATA INFILE ...
INTO TABLE tablename (field1, ..., #date01)
SET date01 = STR_TO_DATE(#date01, '%d/%m/%Y');
I'm trying to import a XML file into a MySQL Table. In the XML file there is a timestamp in <CurrentTime> in the following format:
2016-01-26T09:52:19.3420655+01:00
This timstamp should go into the corresponding DATETIME CurrentTime column in my Table. So I did the following
LOAD XML INFILE 'xxx.xml'
INTO TABLE test.events
ROWS IDENTIFIED BY '<Event>'
SET CurrentTime = str_to_date(CurrentTime, '%Y-%m-%dT%H:%i:%s.%f');
But it quits with the error
Error Code: 1292. Incorrect datetime value: '2016-01-25T16:22:24.1840792+01:00' for column 'CurrentTime' at row 1
So it seems it doesn't convert the string at all. Why?
I think that error is thrown when the string value from the file is loaded directly to the column. The error is thrown before you get to the SET clause.
Here's an abbreviated example of how to use user-defined variables to pass the value of a field down to the SET, bypassing the assignment to the column.
Note that the columns _row and account_number are populated directly from the first two fields in the file. The later fields in the file are assigned to user-defined variables (identifiers beginning with #.
The SET clause evaluates the user-defined variables, and assigns the result of the expression to the actual column in the table.
In this example, the "dates" were formatted YYYYMMDD. I used the STR_TO_DATE() function to have that string converted to a proper DATE.
I abbreviated this sample somewhat, but it demonstrates the approach of reading field values into user-defined variables.
CREATE TABLE _import_water
(`_row` INT
,`account_number` VARCHAR(255)
,`total_due` DECIMAL(18,2)
,`end_date` DATE
,`start_date` DATE
,`ccf` DECIMAL(18,4)
)
LOAD DATA LOCAL INFILE '//server/share$/users/me/mydir/myfile.csv'
INTO TABLE _import_water
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(_row
,account_number
,#total_due
,#end_date
,#start_date
,#ccf
)
SET `total_due` = NULLIF(#total_due,'')
, `end_date` = STR_TO_DATE(#end_date,'%Y%m%d')
, `start_date` = STR_TO_DATE(#start_date,'%Y%m%d')
, `ccf` = NULLIF(#ccf,'')
Also, it doesn't look like there's any problem with your STR_TO_DATE, it seems to evaluate just fine.
testing...
SELECT STR_TO_DATE('2016-01-25T16:22:24.1840792+01:00','%Y-%m-%dT%H:%i:%s.%f') AS mydatetime
returns:
mydatetime
--------------------------
2016-01-25 16:22:24.184079
I just don't seem to get a solution for my problem! I need to import this into SQL Server.
The 2nd column (and a few more) from a .CSV MySQL export contains the tstamp field, which I need converted.
I created the table, but the bulk import did not work. Got the following error message
Msg 4864, Level 16, State 1, Line 4
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 2, column 1 (tstamp).
Msg 4864, Level 16, State 1, Line 4
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 3, column 1 (tstamp).
Herewith the code.
-- Recreate the table
CREATE TABLE [Majestic].[dbo].hdiyouth
(tstamp datetime NOT NULL
)
GO
-- Bulk insert the data from csv file
-- Ensure the file(s) is/are closed!
BULK
INSERT [Majestic].[dbo].hdiyouth
FROM 'C:\Path\CSV\hdiyouth.csv'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = ';',
ROWTERMINATOR = '\n'
)
GO
Try checking out LOAD DATA: http://dev.mysql.com/doc/refman/5.1/en/load-data.html
A bit down there is an example of how to convert a column before inserting:
mysql> LOAD DATA INFILE '/tmp/bit_test.txt'
-> INTO TABLE bit_test (#var1) SET b= CAST(#var1 AS UNSIGNED);
how about creating a SSIS package to do that?
This link may help you
The problem is, that you have the value "0" in the tstamp and tstamp_updated columns, which are of datatype timestamp, right?
MySQL supports NULL values in timestamp columns, also represented by '0000-00-00 00:00:00'. SQL Server does not support this. Don't get me wrong, it supports NULL value in timestamp columns, but not the '0' of MySQL. The easiest way to solve this may be using SSIS like Diego suggested. I personally solved this issue by converting MySQL NULL values to '1970-01-01', which is the minimum value for timestamp columns.