I am trying to build an automated CSV to MYSQL dump which occurs everytime the CSV file in a certain directory is updated.
Once the file is updated, the following line of code is executed in a bash script:
mysql -u root -p$MASTER_DB_PASSW < /usr/local/scripts/order.sql
The contents of order.sql are as follows:
use test;
truncate test.ORDER;
load data infile '/home/test/ORDER.csv' into table test.ORDER fields terminated by ','
enclosed by '"'
lines terminated by '\r\n'
IGNORE 1 LINES
(order_date);
For some reason, when I run order.sql manually in Workbench, it works perfectly... but when I run it on the server, the contents of the csv file do not get dumped. Any tips/advice? Thanks in advance.
I solved it.
I had an error in my bash script. I found this by running:
bash -x scriptname
Related
I have a set of raw data files named using a pattern like a-1.txt, a-2.txt, etc. I am using the LOAD DATA LOCAL INFILE command in a MySQL script to load the raw data files into the database. That command cannot be run in a stored procedure. I'd like to avoid doing a copy/paste for loading 20 raw data files, like I described, in the MySQL script and would much rather use a LOOP to load the raw data files, but LOOP cannot be used outside of a stored procedure.
What's the best way to handle this? How do I get the MySQL script to do this?
Assuming you are using Bash, you can run the following to generate the SQL file.
rm testfile.sql
ls *.txt | xargs -I inputfile echo "LOAD DATA LOCAL INFILE 'inputfile' INTO TABLE mytable FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\r\n';" >> testfile.sql
Then to run it, you can add another line.
mysql -h localhost -u root -pXXXXXXX mydatabase < testfile.sql
I'm trying to bypass the -secure-file-priv option that MySQL has enabled to export a table into a .csv file.
I've been running SELECT * FROM final_table INTO OUTFILE 'test.csv' FIELDS ENCLOSED BY '"' TERMINATED BY ';' ESCAPED BY '"' LINES TERMINATED BY '\r\n'; into the terminal, but it has been returning The MySQL server is running with the --secure-file-priv option so it cannot execute this statement
Previously, I bypassed this error when importing files by running LOAD DATA LOCAL INFILE instead of LOAD DATA INFILE which had the same error and was wondering if there was a way to do it with exporting as well.
Thanks!
Answering my own question since I found a solution that might help out other people.
mysql -u root -p [database] -e '[SELECT * FROM TABLE_NAME]' > data_export.csv
got me exactly what I needed without disabling secure-file-priv!
I am uploading a csv file via the LOAD DATA INFILE command in a bash script.
I need to find out how many lines of data was uploaded. I need to know this to compare with the number of lines in the csv.
This is to verify that all the data mentioned in the CSV file was uploaded.
I am using a bash script to do this upload as this will excuted on bi-hourly everyday.
How can I get the number of rows uploaded.
mysql -u $_db_user -p$_db_password -D $_db --local-infile=1 <<EOF
LOAD DATA LOCAL INFILE '$_csv_directory/TempImport.csv'
into table TempImport
fields terminated by ','
enclosed by '"'
lines terminated by '\r\n';
EOF
LOAD DATA
LOCAL INFILE "file.txt"
REPLACE INTO TABLE file
FIELDS TERMINATED BY '|'
(attribute1, attribute2)
LOAD DATA
LOCAL INFILE "file2.txt"
REPLACE INTO TABLE file2
FIELDS TERMINATED BY '|'
(attribute3, attribute4)
I have a single text file composed of several of these "LOAD DATA" commands. I receive an error message saying line 6, or the start of the 2nd command, is not proper syntax. And if I try to introduce a "lines terminated by '\n'" code, it says it is not allowed with my mysql version.
You should add a ';' at the end of each load statement.
LOAD DATA
LOCAL INFILE "file.txt"
REPLACE INTO TABLE file
FIELDS TERMINATED BY '|'
(attribute1, attribute2);
LOAD DATA
LOCAL INFILE "file2.txt"
REPLACE INTO TABLE file2
FIELDS TERMINATED BY '|'
(attribute3, attribute4);
See also ERROR 1148: The used command is not allowed with this MySQL version
You can specify that as an additional option when setting up your client connection:
mysql -u myuser -p --local-infile somedatabase
I'm using a bash script to pull data from online sources. Right now I just have it writing to a text file, but it would be better if the script could automatically put this data into mysql tables. How can this be done? Examples would be helpful.
Suppose you download a .csv file. which has header and have a database test in mysql.
Download the file first.
wget http://domain.com/data.csv -O data.csv
Dump the data to mysql table tbl
cat <<FINISH | mysql -uUSERNAME -pPASSWORD test
LOAD DATA INFILE 'data.csv' INTO TABLE `tbl`
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES;
FINISH
Here USERNAME must have FILE privilege.
You can use bash like this:
#!/bin/bash
params="dbname -uuser -ppassswd"
echo "SELECT * FROM table;" | mysql $params
# or
mysql $params <<DELIMITER
SELECT * FROM table;
DELIMITER