Import CSV using LOAD DATA getting wrong values - mysql

I have a big csv (near 100mb) that I would like to import in a table with the following structure:
+-------------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+------------------+------+-----+---------+----------------+
| id | int(11) unsigned | NO | PRI | NULL | auto_increment |
| cep | varchar(255) | YES | MUL | NULL | |
| site | text | YES | | NULL | |
| cidade | text | YES | | NULL | |
| uf | text | YES | | NULL | |
| cepbase | text | YES | | NULL | |
| segmentacao | text | YES | | NULL | |
| area | text | YES | | NULL | |
| cepstatus | int(1) | YES | | NULL | |
| score | int(11) | NO | | NULL | |
| fila | int(11) | NO | | NULL | |
+-------------+------------------+------+-----+---------+----------------+
I was about to write some code to import but I've found a MySQL command that does the job to me. So I've write the following:
LOAD DATA LOCAL INFILE '/Users/user/Downloads/base.csv'
INTO TABLE cep_status_new
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\r\n'
IGNORE 1 ROWS
(#id,#cep,#site,#cidade,#uf,#cepbase,#segmentacao,#area,#cepstatus,#score,#fila)
SET id=NULL, cep=#col1, site='GOD', cidade=#col6, uf=#col7, cepbase='-', segmentacao=#col9, cepstatus=#col2, area='BING', score=99999, fila=5;
To try this code, I've removed thousand lines from my CSV and let only 2 lines: header and an input example:
cep,status,gang,bang,random,mock,awesome,qwert,hero
01019000,0,00387,00388,3550308,SAO PAULO,SP,011,B2
The code runs without problem but my insert is pretty strange:
mysql> select * from cep_status_new;
+----+------+------+--------+---------+---------+-------------+------+-----------+-------+------+
| id | cep | site | cidade | uf | cepbase | segmentacao | area | cepstatus | score | fila |
+----+------+------+--------+---------+---------+-------------+------+-----------+-------+------+
| 1 | 1 | GOD | 24655 | 3554805 | - | SP | BING | 0 | 99999 | 5 |
+----+------+------+--------+---------+---------+-------------+------+-----------+-------+------+
1 row in set (0.01 sec)
Why values from CSV are not being filled correctly?

According to this specification the column list after IGNORE 1 ROWS decides how the columns of the CSV file are mapped to columns of the table. It can either list the table columns in the order of the file or it can load the file columns into variables. With the column list
(#id,#cep,#site,#cidade,#uf,#cepbase,#segmentacao,#area,#cepstatus,#score,#fila)
you are loading 11 columns of the CSV file into variables named "id", "cep", etc. In the SET statement you then need to declare how the columns of the table are constructed from the variables. With the given statement you are refering to variables #col1 etc. that are not defined anywhere and consequently have undefined values.
The corrected statement (that I sadly can't test myself right now) should be:
INTO TABLE cep_status_new
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\r\n'
IGNORE 1 ROWS
(#col1,#col2,#col3,#col4,#col5,#col6,#col7,#col8,#col9)
SET id=NULL, cep=#col1, site='GOD', cidade=#col6, uf=#col7, cepbase='-', segmentacao=#col9, cepstatus=#col2, area='BING', score=99999, fila=5;

Related

Import CSV table and skip first column from mysql table

I got one table like this:
| Field | Type | Null | Key | Default | Extra |
+------------+-------------+------+-----+---------+----------------+
| my_id | int(5) | NO | PRI | NULL | auto_increment |
| col1 | varchar(20) | YES | | NULL | |
| col2 | varchar(20) | YES | | NULL | |
| col3 | varchar(20) | YES | | NULL | |
and I want to import a CSV file into this table. Problem is that in CSV I don't have my_id column, so I need to insert data beginning from 2nd column (col1) as the first must be and kind of count of each row.
Edit: I just did the basic import and mysql "removed" my first column from CSV:
LOAD DATA LOCAL INFILE "/home/bruno/myfield.csv"
INTO TABLE teste
FIELDS TERMINATED BY ','
ESCAPED BY ''
LINES TERMINATED BY '\n';
When I import I got:
| my_id | col2 | col3 | col4 |
+------------+----------------------+------------------+-----------------+
| 1 | lorem ipsum | SER1 | testingmyfield4 |
| 2 | dolor emet | SER1 | testingmyfield4 |
You can use a sql query to get the columns and save it in file like the query below
SELECT col1,col2,.. FROM tablename INTO OUTFILE
'location where u want to save the file'
or you can the try the tools which will be available in internet like mysqlyog,mysqlworkbench
Hope you find this answer helpfull

Why is LOAD DATA from text file into table not working properly in mysql?

I have a table event as:
+--------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------+--------------+------+-----+---------+-------+
| name | varchar(20) | YES | | NULL | |
| date | date | YES | | NULL | |
| type | varchar(15) | YES | | NULL | |
| remark | varchar(255) | YES | | NULL | |
+--------+--------------+------+-----+---------+-------+
Now I wish to insert some data into this table from a text file event.txt. The text file is as:
Tommy 2000-01-02 litter 4 kittens, 3 females, 1 male
Bowser 2001-04-26 vet needed break straightened
puffball 2002-05-13 birthday gave him a new chew toy
The whitespaces after values are tabs. The last values are sentences with actual whitespaces. When I load this txt file into the table eventit doesn't load the values properly. I run the query:
load data local infile 'D:/Softwear/mysql/install/data/event.txt' into table event;
The table generated is as following:
+----------+------------+----------+-------------------------------+
| name | date | type | remark |
+----------+------------+----------+-------------------------------+
|Tommy | 2000-01-02 | litter | 4 kittens, 3 females, 1 male
|ser | 2001-04-26 | vet | needed break straightened
| puffball | 2002-05-13 | birthday | gave him a new chew toy |
+----------+------------+----------+-------------------------------+
The value bowser is truncated. Why is it so? When I uploaded a similar pet.txt file into table pet then the table generated correctly. I used notepad to write the txt file, used tab after values and newline after rows.

MySQL CSV Import Fails - "Data too long for column 'air_date' at row 1"

I'm trying to import a CSV file into a MySQL table and I'm having all kinds of trouble getting it to work. Here's what I'm trying to do:
I am working on a video database and have an existing table with data already in it called episodes. Here's how it's set up:
+--------------+-----------------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+-----------------------+------+-----+-------------------+-----------------------------+
| title | varchar(40) | NO | MUL | NULL | |
| media_id | varchar(11) | NO | | NULL | |
| ep_info | varchar(75) | YES | | NULL | |
| air_date | varchar(20) | NO | | NULL | |
| trt | varchar(8) | NO | | NULL | |
| times_played | mediumint(9) unsigned | NO | | 0 | |
| last_played | timestamp | YES | | NULL | |
| entered | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| ep_desc | varchar(300) | NO | | NULL | |
+--------------+-----------------------+------+-----+-------------------+-----------------------------+
The primary key is the id field, with the title field set as a foreign key to the shows table. The shows table looks like this:
+-------------+-------------+------+-----+------------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+-------------+------+-----+------------+-------+
| title | varchar(50) | NO | PRI | NULL | |
| title_image | varchar(50) | NO | | NULL | |
| gif_image | varchar(50) | NO | | NULL | |
| info_url | varchar(30) | NO | | shows.html | |
+-------------+-------------+------+-----+------------+-------+
My CSV file is in the following format:
"Big Wolf On Campus","BWOC0102","Season 1 Episode 2: The Bookmobile","April 9, 1999";"21:57",NULL,NULL,NULL,NULL,"Once every 70 years, a window of transference opens that offers Tommy a chance to pass his curse to another person. Merton volunteers but that same day a bookmobile shows up in Pleasantville and people start disappearing."
"Big Wolf On Campus","BWOC0103","Season 1 Episode 3: Butch Comes To Shove","April 16, 1999","21:06",NULL,NULL,NULL,NULL,"When a character from a 1950s educational film gets sick of the rules he decides to leave the movie for Pleasantville. While there Butch decides to find someone to bring back to his black-and-white world - and Stacey is at the top of his list."
During the import, I want the data in the CSV added to the existing data in the table. I also want the last_played field set to NULL (only updated when the show plays), the entered field set with a current timestamp, and the id field auto_incremented with the next value for the table.
Here is my import statement:
LOAD DATA INFILE 'ytv.csv' INTO TABLE episodes
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n';
The resulting error message:
ERROR 1406 (22001): Data too long for column 'air_date' at row 1
What am I doing wrong here? It seems like the data is getting shifted over one column when it's importing (such that ep_info from the CSV is going into the air_date column) but I can't figure out why. Any insight would be much appreciated for this MySQL novice.
It seems you have some new episodes with no mapping entry in shows table. You can create a new table like episodes, remove any constraints, load the data to the new table, insert all missing show titles to your show table, then insert episodes from the new table to the episodes table.
Or you can delete the foreign key, load the data, amend you shows table, then add the foreign key back.

LOAD DATA LOCAL INFILe: loading data form text file auto incerment syntax

I'm trying to load data from a .txt file into mysql. The table looks like this:
+-------------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+------------------+------+-----+---------+----------------+
| user_name | varchar(10) | NO | MUL | NULL | |
| user_id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| listings | int(10) unsigned | NO | | 0 | |
| connections | int(10) unsigned | NO | | 0 | |
| flags | int(10) unsigned | NO | | 0 | |
| days | int(10) unsigned | NO | | 0 | |
| email | varchar(30) | NO | | NULL | |
| suspend | tinyint(1) | YES | | 0 | |
+-------------+------------------+------+-----+---------+----------------+
when I load the data using LOAD DATA LOCAL INFILE how do I handle the user_id column? All the other items have data already. I'd like the database to assign the user_id's. Is that possible?
What should column under user_id look like in the text file?
I used the following:
name \N 2 4 2 100 emal#mail.com o
Each entry was automatically given the correct id in place of the \N.
If there's a better way, please let me know.
"You can also discard an input value by assigning it to a user variable and not assigning the variable to a table column" read here
Example:
LOAD DATA LOCAL INFILE 'file.txt' INTO TABLE tablename (user_name, #skip, listings, connections, flags, days, email, suspend);

retrieving date in SELECT statement from mysql

I wrote the following mysql code:
select trade_dt,
ticker_id,
settle_price,
volume
from hist
where volume > 0 and trade_dt between '06/22/2011' and '06/30/2011';
but unfortunately, it returns dates outside the time window as follows:
+------------+------------+--------------+--------+
| trade_dt | ticker_id | settle_price | volume |
+------------+------------+--------------+--------+
| 06/23/2006 | N (Jul 06) | 156.900000 | 90 |
| 06/26/2006 | N (Jul 06) | 155.600000 | 63 |
| 06/27/2006 | N (Jul 06) | 159.300000 | 79 |
| 06/28/2006 | N (Jul 06) | 159.600000 | 57 |
| 06/29/2006 | N (Jul 06) | 143.400000 | 511 |
| 06/30/2006 | N (Jul 06) | 140.200000 | 342 |
| 06/23/2005 | V (Oct 05) | 151.200000 | 61 |
| 06/23/2011 | U (Sep 11) | 22.500000 | 6284 |
| 06/24/2011 | U (Sep 11) | 23.100000 | 4505 |
| 06/27/2011 | U (Sep 11) | 22.650000 | 3118 |
| 06/28/2011 | U (Sep 11) | 22.100000 | 3707 |
| 06/29/2011 | U (Sep 11) | 21.500000 | 5830 |
| 06/30/2011 | U (Sep 11) | 20.750000 | 9207 |
| 06/23/2008 | F (Jan 09) | 23.260000 | 2 |
and I wonder if that is because my trade_dt is defined as a string in hist table.EDITED table to replace char(10) with date
desc hist;
+-----------------+---------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+---------------+------+-----+---------+----------------+
| futures_id | int(11) | NO | PRI | NULL | auto_increment |
| trade_dt | date | NO | | NULL | |
| ticker_id | varchar(46) | NO | MUL | NULL | |
| settle_price | decimal(10,6) | NO | | NULL | |
| change_in_price | decimal(10,6) | NO | | NULL | |
| volume | bigint(11) | NO | | NULL | |
| open_int | bigint(11) | NO | | NULL | |
+-----------------+---------------+------+-----+---------+----------------+
how do I fix my date problem?
ok, I changed the trade_dt field to date instead of char(10) and now when I run the below statement to load the data into the DB, it inserts blanks for dates, just because how the date is formatted in the data file.
LOAD DATA LOCAL INFILE '$fn' INTO TABLE $tn FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n' (trade_dt,ticker_id,settle_price,change_in_price, volume, open_int);
this is a sample of raw data:
03/30/2012,Z (Dec 12),25.81,25.81,25.50,25.70,25.60,-0.45,24,0,318
As you note, it's because the data type is strings. You could get the correct result by casting the column to a date type inside the query, but that's incredibly inefficient. The only sound way to fix is to change the type of the trade_dt column, inside the table definition.
select trade_dt, ticker_id, settle_price, volume from
hist where volume > 0 and trade_dt between '2011-06-22' and '2011-06-30';
MySQL retrieves and displays DATE values in 'YYYY-MM-DD' format.
The problem is really with string field type.
You have two options:
Change field to date (preferred)
Store string dates in 'YYYY/MM/DD' format
While saving data you get date as dd/mm/yyyy just explode the date and save as in correct format sql accepts (YYYY-MM-DD);
$date = '26/07/2001';
$data= explode("/",$date);
$dateField = $data[2]."-".$data[1]."-".$data[0];
save this $dateField as date in sql table...