I am uploading a csv file via the LOAD DATA INFILE command in a bash script.
I need to find out how many lines of data was uploaded. I need to know this to compare with the number of lines in the csv.
This is to verify that all the data mentioned in the CSV file was uploaded.
I am using a bash script to do this upload as this will excuted on bi-hourly everyday.
How can I get the number of rows uploaded.
mysql -u $_db_user -p$_db_password -D $_db --local-infile=1 <<EOF
LOAD DATA LOCAL INFILE '$_csv_directory/TempImport.csv'
into table TempImport
fields terminated by ','
enclosed by '"'
lines terminated by '\r\n';
EOF
Related
I have a set of raw data files named using a pattern like a-1.txt, a-2.txt, etc. I am using the LOAD DATA LOCAL INFILE command in a MySQL script to load the raw data files into the database. That command cannot be run in a stored procedure. I'd like to avoid doing a copy/paste for loading 20 raw data files, like I described, in the MySQL script and would much rather use a LOOP to load the raw data files, but LOOP cannot be used outside of a stored procedure.
What's the best way to handle this? How do I get the MySQL script to do this?
Assuming you are using Bash, you can run the following to generate the SQL file.
rm testfile.sql
ls *.txt | xargs -I inputfile echo "LOAD DATA LOCAL INFILE 'inputfile' INTO TABLE mytable FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\r\n';" >> testfile.sql
Then to run it, you can add another line.
mysql -h localhost -u root -pXXXXXXX mydatabase < testfile.sql
I'm trying to bypass the -secure-file-priv option that MySQL has enabled to export a table into a .csv file.
I've been running SELECT * FROM final_table INTO OUTFILE 'test.csv' FIELDS ENCLOSED BY '"' TERMINATED BY ';' ESCAPED BY '"' LINES TERMINATED BY '\r\n'; into the terminal, but it has been returning The MySQL server is running with the --secure-file-priv option so it cannot execute this statement
Previously, I bypassed this error when importing files by running LOAD DATA LOCAL INFILE instead of LOAD DATA INFILE which had the same error and was wondering if there was a way to do it with exporting as well.
Thanks!
Answering my own question since I found a solution that might help out other people.
mysql -u root -p [database] -e '[SELECT * FROM TABLE_NAME]' > data_export.csv
got me exactly what I needed without disabling secure-file-priv!
I am trying to build an automated CSV to MYSQL dump which occurs everytime the CSV file in a certain directory is updated.
Once the file is updated, the following line of code is executed in a bash script:
mysql -u root -p$MASTER_DB_PASSW < /usr/local/scripts/order.sql
The contents of order.sql are as follows:
use test;
truncate test.ORDER;
load data infile '/home/test/ORDER.csv' into table test.ORDER fields terminated by ','
enclosed by '"'
lines terminated by '\r\n'
IGNORE 1 LINES
(order_date);
For some reason, when I run order.sql manually in Workbench, it works perfectly... but when I run it on the server, the contents of the csv file do not get dumped. Any tips/advice? Thanks in advance.
I solved it.
I had an error in my bash script. I found this by running:
bash -x scriptname
So i'm trying to create a script that I can run that will do a batch import of csv files into a table.
I'm having trouble getting the script to work.
Here is the script i'm running:
#!/bin/bash
for f in *.csv
do
"/opt/lampp/bin/mysql -e use test -e LOAD DATA LOCAL INFILE ("$f") INTO TABLE temp_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES (DATE, TIME, SITE_NAME, SITE_IP, TOTAL_TALKTIME, EDGE_UL_BYTES, EDGE_DL_BYTES);"
done
When I run the script I receive the following error message:
./script.sh: line 5: unexpected EOF while looking for matching `''
./script.sh: line 7: syntax error: unexpected end of file
The load data local infile command works fine directly in mysql.
When you want to use literal double quotes in double quoted strings, escape them with \". Since mysql doesn't care about line feeds, you can also break the line to make it more readable:
#!/bin/bash
for f in *.csv
do
/opt/lampp/bin/mysql -e "use test" -e "
LOAD DATA LOCAL INFILE '$f'
INTO TABLE temp_table
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(DATE, TIME, SITE_NAME, SITE_IP, TOTAL_TALKTIME,
EDGE_UL_BYTES, EDGE_DL_BYTES);"
done
mysql -u<username> -p<password> -h<hostname> <db_name> --local_infile=1 -e "use <db_name>" -e"LOAD DATA LOCAL INFILE '<path/file_name>'
IGNORE INTO TABLE <table_name>
FIELDS TERMINATED BY '\t'
OPTIONALLY ENCLOSED BY '\"'"
LOAD DATA
LOCAL INFILE "file.txt"
REPLACE INTO TABLE file
FIELDS TERMINATED BY '|'
(attribute1, attribute2)
LOAD DATA
LOCAL INFILE "file2.txt"
REPLACE INTO TABLE file2
FIELDS TERMINATED BY '|'
(attribute3, attribute4)
I have a single text file composed of several of these "LOAD DATA" commands. I receive an error message saying line 6, or the start of the 2nd command, is not proper syntax. And if I try to introduce a "lines terminated by '\n'" code, it says it is not allowed with my mysql version.
You should add a ';' at the end of each load statement.
LOAD DATA
LOCAL INFILE "file.txt"
REPLACE INTO TABLE file
FIELDS TERMINATED BY '|'
(attribute1, attribute2);
LOAD DATA
LOCAL INFILE "file2.txt"
REPLACE INTO TABLE file2
FIELDS TERMINATED BY '|'
(attribute3, attribute4);
See also ERROR 1148: The used command is not allowed with this MySQL version
You can specify that as an additional option when setting up your client connection:
mysql -u myuser -p --local-infile somedatabase