I wrote this SQL source:
LOAD DATA LOCAL INFILE '"+exportDataFile+"' INTO TABLE ex_patients_variations FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'
This source should load to the database a huge file (right now it contain over than 1 million lines, and in the future it could contain billions lines).
The problem is its not load all the lines in the file.
I try to run it some times, and everytime I use this SQL source, it upload different number of lines (between 550,000 to 660,000 lines, when I have more than 1 million lines).
How can I solve it and upload all the file lines in one time to the database?
when it is only for one time you can split the file in some pieces
with
split --lines=100000 FILENAME
So it will split it in 10 file with 100000 lines each.
also you can test to set the variable max-allowed-packet= in my.cnf to a higher value. you must restart the server
Related
I'm trying to import an csv file to my database table, the origin of this file was a previous database with the same structure. My issue is that it imports only 1000 rows instead of the whole 62k+ file. The script i'm using is:
LOAD DATA INFILE 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/covid19.csv'
INTO TABLE covid19.covid19
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(id,date,iso2,iso3,country,continent,cases,deaths,population);
Some clients have a option, where they reduce the number of returned Rows with a LIMIT 1000.
You should check, how many rows you actually have with
SELECT COUNT(*) FROM covid19.covid19;
You should see the actual number of inserted rows, as the command didn't show any warnungs or errors.
I ran into a problem that I didn't understand. I have a .csv file that is about 10 GB in size. I'm transferring it into the database. When I use only the innodb engine, 1.863.941 lines are added. But when I use the Aria engine, 1.765.972 lines are added.
I can't see how many rows there are in this one because it is so big .csv file.
The database I use: Mysql 5.7.24 - Mariadb 10.3.12
The SQL Command I use:
LOAD DATA LOCAL INFILE "'.$file.'"
INTO TABLE '.$table.'
CHARACTER SET UTF8
FIELDS TERMINATED by \',\'
ENCLOSED BY \'"\'
LINES TERMINATED BY \'\r\n\''
I am not getting any error code when transferring SQL.
I have a large CSV file I am trying to import into MySQL (around 45GB, around 150mil rows, most columns small but one with variable length text, can get up to KBs of size). I am using LOAD DATA LOCAL INFILE to try and import it but the server always times out my connection before it finishes. I have tried modifying the global connection timeout variables to fix this, but it already has some hours before it times out. Is there another way to import a database this large, or am I doing something wrong with this approach?
LOAD DATA LOCAL INFILE 'filename.csv' INTO TABLE table CHARACTER SET latin1
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\'
LINES TERMINATED BY '\r\n';
I am executing this command on Windows 10, with the MySQL command line. My MySQL version is 8.0.
The way I have handled this in the past is by writing a php script that reads the file and outputs the top 50% into a new file, then deletes those rows. Then perform two load data infiles, one for original and one for the new.
For smaller csv files, the following has always worked for me...
LOAD DATA LOCAL INFILE 'C:/store_emails2.csv' INTO TABLE
mytable FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n';
Now, I am trying to do this with a file that has 100K rows. Mysql only takes about 50K rows, though. The odd part is that it takes random rows, not just the first 50K. I have no idea what the problem could be, as the file is formatted correctly and I've imported data this way many times. Any ideas as to what may be the problem?
EDIT: including what part of the csv looks like...
CUST_NUM,LAST_NAME,FIRST_NAME,MIDDLE_INIT,PREFIX,SUFFIX,ADDRESS_1,ADDRESS_2,CITY,STATE,ZIP,HOME_TEL,BUS,MAILING,MOD_DATE,SIGN_UP_DATE,BIRTHDAY_DATE,CASHIER,STR,EMAIL
1234556767,name,name,,,,,,,,,9999999999,,Y,6007,607,0101,341,71,email
I use LOAD DATA LOCAL INFILE to upload csv files to MySQL.
If the csv was created by a mac, I include LINES TERMINATED BY '\r' in the query.
If the csv was created by a MS Office, I include LINES TERMINATED BY '\n' in the query.
If the csv was created in Open Office, I omit LINES TERMINATED BY altogether.
Is there a way I can formulate my query so the csv will be uploaded regardless of how it was created?
Look up line-termination stripping procs. There should be a few resources out there which look at \r\n issues and how to fix them in different coding sets.