Working here on bulk insert, which skips 2 records. Explanation below:
My table (works fine, Auto-incrementing Job_Id):
create table avjobs ( Job_Id int NOT NULL AUTO_INCREMENT PRIMARY KEY,
Job_Name varchar(255), Job_Seq varchar(255), Job_Date varchar(255),
Start_Time time, End_Time time, Runtime time, Status varchar(255) );
Here is my csv file:
JOB1A|0029|20140506|14:01:05|15:00:01|0:59:45|FINISHED
JOB2B|0030|20140506|15:01:05|16:00:01|0:59:55|INITIATED
Here is the BULK INSERT that I am using:
LOAD DATA LOCAL INFILE '/tmp/jobs.csv' INTO TABLE avjobs FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
Query OK, 3 rows affected, 9 warnings (0.00 sec)
Here is output for select:
mysql> select * from avjobs;
+--------+----------+----------+----------+------------+----------+----------+--------+
| Job_Id | Job_Name | Job_Seq | Job_Date | Start_Time | End_Time | Runtime | Status |
+--------+----------+----------+----------+------------+----------+----------+--------+
| 1 | 0029 | 20140506 | 14:01:05 | 15:00:01 | 00:23:55 | 00:00:00 | NULL |
| 2 | 0030 | 20140506 | 15:01:05 | 16:00:01 | 00:59:55 | 00:00:00 | NULL |
+--------+----------+----------+----------+------------+----------+----------+--------+
The Bulk insert skips, somehow, the job name as well as status.
Can you please advise what is wrong in syntax?
You must specify your columns:
LOAD DATA LOCAL INFILE '/tmp/jobs.csv'
INTO TABLE avjobs (Job_Name, Job_Seq, Job_Date, Start_Time, End_Time, Runtime, Status)
FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
because your import file doesn't contain your Job_ID, see manual
LOAD DATA INFILE 'persondata.txt' INTO TABLE persondata; By default,
when no column list is provided at the end of the LOAD DATA INFILE
statement, input lines are expected to contain a field for each table
column. If you want to load only some of a table's columns, specify a
column list:
Related
When I'm trying to import csv into MySQL table, I'm getting an error
Data too long for column 'incident' at row 1
I'm sure the values are not higher than varchar(12). But, still I'm getting the error.
MariaDB [pagerduty]>
LOAD DATA INFILE '/var/lib/mysql/pagerduty/script_output.csv'
REPLACE INTO TABLE incidents
ignore 1 lines;
ERROR 1406 (22001): Data too long for column 'incident' at row 1
MariaDB [pagerduty]>
LOAD DATA INFILE '/var/lib/mysql/pagerduty/script_output.csv'
INTO TABLE incidents
ignore 1 lines;
ERROR 1406 (22001): Data too long for column 'incident' at row 1
While trying with REPLACE, the data is uploading only one column(which set on primary key)
MariaDB [pagerduty]>
LOAD DATA INFILE '/var/lib/mysql/pagerduty/script_output.csv'
IGNORE INTO TABLE incidents
ignore 1 lines;
Query OK, 246 rows affected, 1968 warnings (0.015 sec)
Records: 246 Deleted: 0 Skipped: 0 Warnings: 1968
**Columns:**
+----------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+-------+
| incident | varchar(12) | NO | PRI | NULL | |
| description | varchar(300) | YES | | NULL | |
| status | varchar(12) | YES | | NULL | |
| urgency | varchar(7) | YES | | NULL | |
| service | varchar(27) | YES | | NULL | |
| trigger | varchar(25) | YES | | NULL | |
| team | varchar(20) | YES | | NULL | |
| incident_start | datetime(6) | YES | | NULL | |
| incident_end | datetime(6) | YES | | NULL | |
| resolved_by | varchar(20) | YES | | NULL | |
+----------------+--------------+------+-----+---------+-------+
10 rows in set (0.003 sec)
By default, MySQL looks for a TAB character to separate values. Your file is using a comma, so MySQL reads the entire line and assumes it is the value for the first column only.
You need to tell MySQL that the column terminator is a comma, and while you're at it, tell it about the enclosing double quotes.
Try this:
LOAD DATA INFILE '/var/lib/mysql/pagerduty/script_output.csv' REPLACE INTO TABLE incidents
columns terminated by ','
optionally enclosed by '"'
ignore 1 lines;
Reference
If you THINK your data appears ok, and its still nagging about the too long data, how about creating a new temporary table structure and set your first column incident to a varchar( 100 ) just for grins... maybe even a few others if they too might be causing a problem.
Import the data to THAT table to see if same error or not.
If no error, then check the maximum trimmed length of the data in respective columns and analyze the data itself... bad format, longer than expected, etc.
Once resolved, then you can pull into production once you have figured it out. You could also always PRE-LOAD the data into this larger table structure, truncate it before each load so no dups via primary key on a fresh load.
I have had to do that in the past, also was efficient for pre-qualifying lookup table IDs for new incoming data. Additionally could apply data cleansing in the temp table before pulling into production.
I have a table name :: users with id and name in mysql , the id is primary key with auto increment and name is varchar(50) and now i have a bunch of unique names almost 70 , how should i insert it in bulk , i know basic statements but i didn't found something best related to this problem
mysql> desc users;
+-------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(100) | YES | | NULL | |
+-------+--------------+------+-----+---------+----------------+
The MySQL docs says:
INSERT statements that use VALUES syntax can insert multiple rows. To do this, include multiple lists of comma-separated column values, with lists enclosed within parentheses and separated by commas. Example:
INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);
Also you can use the Insert ... Select ... syntax:
INSERT INTO tbl_name (a,b,c)
SELECT a,b,c
FROM ...
Sorry man for me it was simple only insert into users (name) values ("name1"),("name2");
If you have all the information you want to insert inside file you might use LOAD DATA INFILE
LOAD DATA INFILE 'InfoToBeInsertedInsideDB.csv'
INTO TABLE tbl_name
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS;
The csv looks like:
Here's the content of a very simple text file to show my problem. Somehow, I end up using "|" as the separator. But that's not the problem...
10|Antonio
11|Carolina
12|Diana
13|Alejandro
Here is the code I use to create that very simple table and to load the file into it.
CREATE TABLE IF NOT EXISTS names
(
id INT NOT NULL PRIMARY KEY AUTO_INCREMENT,
name VARCHAR(100)
);
LOAD DATA LOCAL INFILE
'C:\\Users\\Carolina\\Desktop\\Tmp\\TablasCSV\\names.csv'
INTO TABLE names
CHARACTER SET 'utf8'
FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\r\n';
Here is the result of a simple select:
mysql> SELECT * FROM names;
+----+-----------+
| id | name |
+----+-----------+
| 1 | Antonio |
| 11 | Carolina |
| 12 | Diana |
| 13 | Alejandro |
+----+-----------+
4 rows in set (0.00 sec)
It has worked fine for me until I encounter the case, in which the first record value for the "id" was not "1".
I always get "1" after the load.
Any one has noticed this problem?
Is this a bug?
So far, I am fixing the record using an UPDATE command after the load, but I don't like it!!!!
I have a table of the following format:
mysql> describe tweet_info;
+-----------+--------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+--------------+------+-----+-------------------+-----------------------------+
| tweet_id | bigint(20) | NO | PRI | NULL | |
| user_id | bigint(20) | YES | | NULL | |
| tweet | varchar(140) | YES | | NULL | |
| timestamp | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| fav_count | int(11) | YES | | NULL | |
| lat | float | YES | | NULL | |
| longi | float | YES | | NULL | |
| hashtags | varchar(140) | YES | | NULL | |
+-----------+--------------+------+-----+-------------------+-----------------------------+
8 rows in set (0.00 sec)
and a file called mini.txt of the following schema:
<tweet_id> <user_id> <tweet_text> <timestamp> <favourite_count> <latitude> <longitude> <hashtags>
244435656850411520 522575984 #SGodoyAlmirall #hongostibetanos Sat Sep 08 14:02:56 +0000 2012 0 -70.29044372 -18.48140825 hongostibetanos
When I used the following query:
load data infile 'mini.txt'into table tweet_info fields terminated by '\t' lines terminated by '\n';
The query works fine and all lines in the file are inserted into my database. Just that the timestamp is not well handled and all of them stay null. Upon searching the internet a bit, I found that we can set the format of the timestamp as follows:
load data infile 'mini.txt' into table tweet_info fields terminated by '\t' lines terminated by '\n' (#var4) SET timestamp=STR_TO_DATE(#var4,'%a %b %d %H:%i:%s +0000 %Y');
However, this generates the following error:
ERROR 1062 (23000): Duplicate entry '0' for key 'PRIMARY'
This seems weird since
There are no duplicates in my file (I have manually checked on the small file on which I am presently running my command).
The first command didn't say anything about the duplication of entries and was OK.
I would be really grateful if someoen could help me out.
You need to list all the columns in the column list:
load data infile 'mini.txt'
into table tweet_info
fields terminated by '\t'
lines terminated by '\n'
(tweet_id, user_id, tweet_text, #var4, favourite_count, latitude, longitude, hashtags>)
SET timestamp=STR_TO_DATE(#var4,'%a %b %d %H:%i:%s +0000 %Y');
Your code was assigning the first column in the input file to #var4, convering that to a date, and then inserting a row with only the timestamp column specified. So it was defaulting all the other columns, and creating duplicate tweet_id = 0 rows.
I want to import data from excel to a mysql database using the command line client.
This is an example of how my csv-file is built:
Name 1 | 1 | 2 | 3 |
Name 2 | 1 | 2 | 3 |
Name 3 | 1 | 2 | 3 |
I'm using the code:
LOAD DATA LOCAL INFILE 'path to file.csv'
INTO TABLE table_name
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n';
I get the "Query OK" and with this code the formatting on the table should be almost exactly as the csv-file but I get this result:
| NULL | NULL | NULL |
| NULL | NULL | NULL |
| NULL | NULL | NULL |
What is wrong?
seems u have used '|' in your csv file as delimiters instead of
comma, try the code as
`LOAD DATA LOCAL INFILE 'path to file.csv' INTO TABLE table_name FIELDS TERMINATED BY '|'LINES TERMINATED BY '\n';`
Assume the following table structure:
CREATE TABLE `table_name` (
`name` VARCHAR(20) CHARACTER SET utf8 DEFAULT NULL,
`value1` INT(11) DEFAULT NULL,
`value2` INT(11) DEFAULT NULL,
`value3` INT(11) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
and the file, csv-file.csv:
Name 1,1,2,3
Name 2,1,2,3
Name 3,1,2,3
when I run the statement:
mysql> LOAD DATA INFILE '/path/csv-file.csv'
-> INTO TABLE `table_name`
-> FIELDS TERMINATED BY ','
-> LINES TERMINATED BY '\n';
mysql> SELECT `name`, `value1`, `value2`, `value3`
FROM `table_name`;
get the following result:
+--------+--------+--------+--------+
| name | value1 | value2 | value3 |
+--------+--------+--------+--------+
| Name 1 | 1 | 2 | 3 |
| Name 2 | 1 | 2 | 3 |
| Name 3 | 1 | 2 | 3 |
+--------+--------+--------+--------+
3 rows in set (0.00 sec)