Load data from CSV inside bit field in mysql - mysql

What's the right syntax to insert a value inside a column of type bit(1) in `MySQL'?
My column definition is:
payed bit(1) NOT NULL
I'm loading the data from a csv where the data is saved as 0 or 1.
I've tried to do the insert using:
b'value' or 0bvalue (example b'1' or 0b1)
As indicated from the manual.
But I keep getting this error:
Warning | 1264 | Out of range value for column 'payed' at row 1
What's the right way to insert a bit value?
I'm not doing the insert manually but I'm loading the data from a csv (using load data infile) in which the data for the column is 0 or 1.
This is my load query, I've renamed the fields for privacy questions, there's no error in that definition:
load data local infile 'input_data.csv' into table table
fields terminated by ',' lines terminated by '\n'
(id, year, field1, #date2, #date1, field2, field3, field4, field5, field6, payed, field8, field9, field10, field11, project_id)
set
date1 = str_to_date(#date1, '%a %b %d %x:%x:%x UTC %Y'),
date2 = str_to_date(#date2, '%a %b %d %x:%x:%x UTC %Y');
show warnings;
This is an example row of my CSV:
200014,2013,0.0,Wed Feb 09 00:00:00 UTC 2014,Thu Feb 28 00:00:00 UTC 2013,2500.0,21,Business,0,,0,40.0,0,PROSPECT,1,200013
Update:
I didn't find a solution with the bit, so I've changed the column data type from bit to tinyint to make it work.

I've finally found the solution and I'm posting it here for future reference. I've found help in the mysql load data manual page.
So for test purpose my table structure is:
+--------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------+-------------+------+-----+---------+-------+
| id | int(11) | NO | PRI | NULL | |
| nome | varchar(45) | YES | | NULL | |
| valore | bit(1) | YES | | NULL | |
+--------+-------------+------+-----+---------+-------+
My csv test file is:
1,primo_valore,1
2,secondo_valore,0
3,terzo_valore,1
The query to load the csv into the table is:
load data infile 'test.csv' into table test
fields terminated by ',' lines terminated by '\n'
(id, nome, #valore) set
valore=cast(#valore as signed);
show warnings;
As you can see do load the csv you need to do a cast cast(#valore as signed) and in your csv you can use the integer notation 1 or 0 to indicate the bit value. This is because BIT values cannot be loaded using binary notation (for example, b'011010').

Replace the "0" values in the csv by no value at all. That worked for me.

You can use BIN() function like this :
INSERT INTO `table` VALUES (`column` = BIN(1)), (`column` = BIN(0));

Let me guess, but I think you should ignore 1st line of your CSV file in LOAD query.
See "IGNORE number LINES"

Related

mysql change HEX number to decimal in LOAD DATA LOCAL INFILE

all,
I have a test.csv file with Id in HEX numbers as below:
Id, DateTime,...
66031851, ...
2E337E4E, ...
The table_test is created using MYSQL as below:
CREATE TABLE table_test(
Id BIGINT NOT NULL,
DateTime DATETIME NOT NULL,
OtherId BIGINT NOT NULL,
...,
PRIMARY KEY (Id, DateTime, OtherId)
)ENGINE=InnoDB DEFAULT CHARSET=utf8;
The created table_test is as below:
+---------------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------------+--------------+------+-----+---------+-------+
| Id | bigint(20) | NO | PRI | NULL | |
| DateTime | datetime | NO | PRI | NULL | |
I am using MYSQL as below to load the data in a table:
load data local infile 'test.csv' replace into table table_test character set utf8mb4 fields terminated by ',' ENCLOSED BY '\"' lines terminated by '\n' ignore 1 lines SET Id=CONV(Id, 16, 10);
Also tried:
SET Id=cast(CONV(Id, 16, 10) AS UNSIGNED)
and
SET Id=cast(CONV(CONVERT(Id,CHAR), 16, 10) AS UNSIGNED)
But the HEX numbers with letters like "2E337E4E" do not work. They become some very big number which is bigger than a BIGINT. But when I try MYSQL below:
select CONV('2E337E4E', 16, 10);
It works as expected with the correct result "775126606". So I think I miss a step in "LOAD DATA" to make the Id as string for the CONV(). Searched for some time, but did not find a solution.
Anyone has some idea or hint?
Thanks very much
Zhihong
The typical solution for this type of problem is to load the value into a user-defined-variable, then do the conversion in a SET statement.
Something like this should work for you:
load data local infile 'test.csv'
replace into table table_test
character set utf8mb4
fields terminated by ','
ENCLOSED BY '\"'
lines terminated by '\n'
ignore 1 lines
(#Id, `DateTime`, <explicitly list all other columns>)
SET Id=CONV(#Id, 16, 10);

LOAD FILE,wrong output in the last row and column

This is sql file
# mo.sql
DROP TABLE IF EXISTS mo;
## _CREATE_TABLE_
CREATE TABLE mo
(
name CHAR(30),
age INT,
salary INT
);
## _CREATE_TABLE_
LOAD DATA LOCAL INFILE 'moja-2001.txt' INTO TABLE mo;
I run it from Linux terminal
mysql -p cookbook < mo.sql
No warning,no errors.
SELECT * FROM mo;
+--------------------------+------+--------+
| name | age | salary |
+--------------------------+------+--------+
| jova jovic | 24 | NULL |
| ceda prashak | 25 | NULL |
| toma grobar 28 20001 | NULL | NULL |
+--------------------------+------+--------+
I have created txt file with geany text editor
jova jovic 24 999
ceda prashak 25 1000
toma grobar 28 20001
Why is salary column wrong?Why is third row wrong also?
The last row does not use the correct separators for values. I suspect the other rows separate the values using tabs and the last one uses spaces. The same for column salary.
Make sure you use the same separator for values. It's better to use comma (it is visible and less error prone) and use the FIELDS TERMINATED BY clause of the LOAD DATA statement to inform MySQL about it.
Change the file to look like this:
jova jovic,24,999
ceda prashak,25,1000
toma grobar,28,20001
and import it like this:
LOAD DATA LOCAL INFILE 'moja-2001.txt' INTO TABLE mo FIELDS TERMINATED BY ',';
Read more about the LOAD DATA statement.

load data infile Date time in mysql results in zeros

can someone help me with this, my resulting table is showing only zeros for the timestamp. I tried changing the field type to datatime and timestamp but not luck
MYSQL
LOAD DATA LOCAL INFILE 'vals.csv'
INTO TABLE vals
FIELDS TERMINATED BY ","
LINES TERMINATED BY "\n"
IGNORE 1 LINES
(#varTimeSt,NOMINAL,NAME,ID,VAL) SET DATETIME = STR_TO_DATE(#varTimeSt,'%d/%m/%Y %h:%i:%s');
CSV File
DATETIME,NAME,ID,VAL
"25/08/2016 02:00:00",tom,cNum6,12
"25/08/2016 02:00:00",Charles,cNum7,10.58
"25/08/2016 02:00:00",Donal,cNum8,10.18
"25/08/2016 02:00:00",Duncan,cNum7,10.31
Resulting table
DATETIME,NAME,ID,VAL
0000-00-00 00:00:00,tom,cNum6,12
0000-00-00 00:00:00,Charles,cNum7,10.58
0000-00-00 00:00:00,Donal,cNum8,10.18
0000-00-00 00:00:00,Duncan,cNum7,10.31
Your problem is that your dates are not in native mysql DATETIME format (%Y-%m-%d %H:%i:%s), this can be shown as such:
mysql> SELECT CAST('25/08/2016 02:00:00' AS DATETIME);
+-----------------------------------------+
| CAST('25/08/2016 02:00:00' AS DATETIME) |
+-----------------------------------------+
| NULL |
+-----------------------------------------+
1 row in set, 1 warning (0,00 sec)
mysql> SHOW WARNINGS;
+---------+------+-------------------------------------------------+
| Level | Code | Message |
+---------+------+-------------------------------------------------+
| Warning | 1292 | Incorrect datetime value: '25/08/2016 02:00:00' |
+---------+------+-------------------------------------------------+
1 row in set (0,00 sec)
The solution is to get them into MySQL format, this could be done by manipulating your DATETIME column on disk or at source. This may be more difficult than the 'self sufficient' solution I'll outline below.
There exists the string to date (STR_TO_DATE) function to solve this kind of issue once data is inside the database, here is a demonstration with your format. Otherwise see the manual here. (See the DATE_FORMAT section for a table with the codes you can use for different date formats.):
mysql> SELECT CAST(STR_TO_DATE('25/08/2016 02:00:00', '%d/%m/%Y %H:%i:%s') AS DATETIME);
+---------------------------------------------------------------------------+
| CAST(STR_TO_DATE('25/08/2016 02:00:00', '%d/%m/%Y %H:%i:%s') AS DATETIME) |
+---------------------------------------------------------------------------+
| 2016-08-25 02:00:00 |
+---------------------------------------------------------------------------+
1 row in set (0,00 sec)
Personally in this situation I'd load the data into a temporary table that accepted the date fields as a string and then use an INSERT with the STR_TO_DATEfunction to get the data into the final table. A final solution (untested, guideline only) would look like this:
CREATE TEMPORARY TABLE valsLoad (DATETIME TEXT,NAME TEXT,ID TEXT,VAL INT);
LOAD DATA LOCAL INFILE 'vals.csv'
INTO TABLE valsLoad
FIELDS TERMINATED BY ","
LINES TERMINATED BY "\n"
IGNORE 1 LINES
(#varTimeSt,NOMINAL,NAME,ID,VAL) SET DATETIME = STR_TO_DATE(#varTimeSt,'%d/%m/%Y %h:%i:%s');
INSERT INTO vals
SELECT STR_TO_DATE(DATETIME, '%d/%m/%Y %H:%i:%m'), NAME, ID, VAL FROM valsLoad;
DROP TEMPORARY TABLE valsLoad;
Please let me know if you have any questions.
Regards,
James

MySQL CSV import: datetime value

I have been banging my head against a wall trying to import datetime values from a .csv file.
Here's the import statement.
LOAD DATA LOCAL INFILE 'myData.csv'
INTO TABLE equity_last_import
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
(equity,last,#last_date)
SET last_date = STR_TO_DATE( #last_date, '%Y-%m-%d %H:%i:%s')
Here's a sample of the data:
4108,48.74,"2013-09-16 16:15:04"
4249,8.1,"2013-09-16 16:15:04"
4197,3.81,"2013-09-16 17:20:00"
4139,26.81,"2013-09-16 16:15:04"
4218,24.83,"2013-09-16 17:20:00"
4260,79.72,"2013-09-16 16:15:04"
4270,450.12,"2013-09-16 17:20:00"
4242,30.38,"2013-09-16 16:15:04"
4193,1.42,"2013-09-16 16:15:04"
4134,3.77,"2013-09-16 16:15:04"
I am able to import date values using STR_TO_DATE() but I not able to get datetime values to import. I have tried several different date formats other than '%Y-%m-%d %H:%i:%s' and I always get a null datetime [0000-00-00 00:00:00]. I have also tried not using STR_TO_DATE(), since the string is in the default MySQL datetime format.
Any help will be appreciated.
I ran into the same problem. I fixed it by changing the format for the date column in my CSV file to match the MySQL datetime format.
Open CSV in Excel.
Highlight the column.
Right-click on the column.
Click on Format Cells.
Pick Custom.
Use yyyy/mm/dd hh:mm:ss in the Type field.
Click ok
My CSV successfully imported after I changed the datetime format as above.
The date in your data file is already in a format MySQL should natively understand. It's just enclosed in double quotes. You need to tell LOAD DATA INFILE how to deal with the quotes. Try something like this:
LOAD DATA LOCAL INFILE 'myData.csv'
INTO TABLE equity_last_import
FIELDS OPTIONALLY ENCLOSED BY '"' TERMINATED BY ','
LINES TERMINATED BY '\n'
(equity,last,last_date)
Update:
Since you've said it doesn't work, I created a test table and verified that it does work. Here's the proof:
I've highlighted your csv data from the question and pasted into a new file called myData.csv in my system's /tmp folder. Then I connected to the mysql console, switched to the test database and ran the following:
mysql> create table equity_last_import (equity int, last decimal(10,2), last_date datetime) engine=innodb;
Query OK, 0 rows affected (0.02 sec)
mysql> LOAD DATA LOCAL INFILE '/tmp/myData.csv'
-> INTO TABLE equity_last_import
-> FIELDS OPTIONALLY ENCLOSED BY '"' TERMINATED BY ','
-> LINES TERMINATED BY '\n'
-> (equity,last,last_date);
Query OK, 10 rows affected (0.00 sec)
Records: 10 Deleted: 0 Skipped: 0 Warnings: 0
mysql> select * from equity_last_import;
+--------+--------+---------------------+
| equity | last | last_date |
+--------+--------+---------------------+
| 4108 | 48.74 | 2013-09-16 16:15:04 |
| 4249 | 8.10 | 2013-09-16 16:15:04 |
| 4197 | 3.81 | 2013-09-16 17:20:00 |
| 4139 | 26.81 | 2013-09-16 16:15:04 |
| 4218 | 24.83 | 2013-09-16 17:20:00 |
| 4260 | 79.72 | 2013-09-16 16:15:04 |
| 4270 | 450.12 | 2013-09-16 17:20:00 |
| 4242 | 30.38 | 2013-09-16 16:15:04 |
| 4193 | 1.42 | 2013-09-16 16:15:04 |
| 4134 | 3.77 | 2013-09-16 16:15:04 |
+--------+--------+---------------------+
10 rows in set (0.00 sec)
See? It works perfectly.
Another Update:
You've specified that you're getting the following error now:
Out of range value for column 'last_date' at row 1
Does your CSV file have a header? If so, you may want to add IGNORE 1 LINES to your LOAD DATA INFILE command to tell MySQL to skip over the header.
Pulled my hair out over this also because I'm not importing in the above suggested way.
Workaround: Created a temporary field temp_date of type "VARCHAR" on your import table and have no problems loading the data. I then was able to perform an update on my date column which is of type date.
update table
set date = temp_date
I was having the same trouble and here's what I discovered, I think it might help you
The problem regards a GMT conflict:
The database I was extracting the .csv file was on GMT 00:00, so there wasn't daylight saving time.
My local server (which was the one I was trying to insert the .csv file) was running on my computer (system's) GMT, by default. In my case it was GMT -3, which is affected by daylight saving time.
SQL has a special way to deal with daylight saving time, it ignores the exact date and time when the daylight saving time starts to happen. In my case, it was october 20 and all the records between 00 and 1 AM of that day simply weren't recognized by the server, even if the format was correct, because they simply 'didn't exist' (see more here). This was affecting all timestamp records, not only the specifics of daylight saving time.
My solution was to set the local server to GMT +00, before creating the new table and import the .csv, using
SET time_zone='+00:00';
And then when I imported the .csv file all the records were read properly. I think if you set the time zone equal to the one that generated the .csv file should work!
Open CSV in Excel.
Highlight the column.
Right-click on the column.
Click on Format Cells.
Pick Custom.
Use same format as SQL date format yyyy-mm-dd
Click ok and save
it's working fine for me

Set timestamp on insert when loading CSV [duplicate]

This question already has answers here:
How can i add date as auto update when import data from csv file?
(2 answers)
Closed 9 years ago.
I have a Timestamp field that is defined to be automatically updated with the CURRENT_TIMESTAMP value.
It works fine when I fire a query, but when I import a csv (which I'm forced to do since one of the fields is longtext) , the update does not work.
I have tried to:
Give timestamp column as now() function in csv
Manually enter timestamp like 2013-08-08 in the csv
Both the approaches do not work
From what I gather, after updating your question, is that you're actually updating rows using a CSV, and expect the ON UPDATE clause to set the value of your timestamp field to be updated.
Sadly, when loading a CSV into a database you're not updating, but inserting data, and overwriting existing records. At least, when using a LOCAL INFILE, if the INFILE isn't local, the query will produce an error, if it's a local file, these errors (duplicates) will produce warnings and the operation will continue.
If this isn't the case for you, perhaps consider following one of the examples on the doc pages:
LOAD DATA INFILE 'your.csv'
INTO TABLE tbl
(field_name1, field_name2, field_name3)
SET updated = NOW()
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY ('\n');
Just in case you can't/won't/forget to add additional information, loading a csv int a MySQL table is quite easy:
LOAD DATA
LOCAL INFILE '/path/to/file/filename1.csv'
INTO TABLE db.tbl
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(`field_name1`,`field_name2`,`field_name3`)
If you create a table along the lines of:
CREATE TABLE tbl(
id INT AUTO_INCREMENT PRIMARY KEY, -- since your previous question mentioned auto-increment
field_name1 VARCHAR(255) NOT NULL PRIMARY KEY, -- normal fields
field_name2 INTEGER(11) NOT NULL PRIMARY KEY,
field_name3 VARCHAR(255) NOT NULL DEFAULT '',
-- when not specified, this field will receive current_timestamp as value:
inserted TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-- if row is updated, this field will hold the timestamp of update-time
updated TIMESTAMP NOT NULL DEFAULT 0
ON UPDATE CURRENT_TIMESTAMP
)ENGINE = INNODB
CHARACTER SET utf8 COLLATE utf8_general_ci;
This query is untested, so please be careful with it, it's just to give a general idea of what you need to do to get the insert timestamp in there.
This example table will work like so:
> INSERT INTO tbl (field_name1, field_name2) VALUES ('foobar', 123);
> SELECT FROM tbl WHERE field_name1 = 'foobar' AND field_name2 = 123;
This will show:
+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+
| id | field_name1 | field_name2 | field_name3 | inserted | updated |
+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+
| 1 | foobar | 123 | | 2013-08-07 00:00:00 | 0000-00-00 00:00:00 |
+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+
As you can see, because we didn't explicitly insert a value into the last three fields, MySQL used their DEFAULT values. For field_name3, an empty string was used, for inserted, the default was CURRENT_TIMESTAMP, for updated the default value was 0 which, because the field-type is TIMESTAMP is represented by the value 0000-00-00 00:00:00. If you were to run the following query next:
UPDATE tbl
SET field_name3 = 'an update'
WHERE field_name1 = 'foobar'
AND field_name2 = 123
AND id = 1;
The row would look like this:
+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+
| id | field_name1 | field_name2 | field_name3 | inserted | updated |
+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+
| 1 | foobar | 123 | an update | 2013-08-07 00:00:00 | 2013-08-07 00:00:20 |
+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+
that's all. Some basics can be found here, on mysqltutorial.org, but best keep the official manual ready. It's not bad once you get used to it.
Perhaps this question might be worth a quick peek, too.