I have a batch process that looks like this:
for /F %%a in (Files_april.txt) do (BlaBla_filedecryptionv3.exe -i FullRefresh_%%a.afi -k keys.txt -l customer_layout.txt -o customer_%%a.txt -t d -d \124)
mysql -q -h beast --port=3310 -u jxxxx --password="xxxx" di < load_april.sql
mysql -q -h beast --port=3310 -u xxxxx --password="xxxx" di < exportapril.sql > april.txt
I was wondering if it is possible to take the same idea of the %%a that I used to decrypt the files in a mysql qry.
The load_april.sql looks like:
load data local infile 'W:\\New DLP\\customer_20140331_0.txt' into table di.dlp_monthly_201404
fields terminated by '|'
lines terminated by '\r\n';
load data local infile 'W:\\New DLP\\customer_20140331_1.txt' into table di.dlp_monthly_201404
fields terminated by '|'
lines terminated by '\r\n';
Where the "20140331_0" and "20140331_1" are what I want replaced so it would be something like this:
load data local infile 'W:\\New DLP\\customer_%%a.txt' into table di.dlp_monthly_201404
fields terminated by '|'
lines terminated by '\r\n';
Is this possible? Any help would be great.
You can include an sql query directly on the mysql command line, rather than loading it from a file. The resulting file would look something like:
for /F %%a [...]
mysql [...] -e "load data local infile 'path\\%%a.txt' into [...]"
[...]
Where [...] needs to be expanded according to your needs.
Related
can someone please assist, I am new to mysql, and i have noticed that "LOAD DATA LOCAL INFILE" does not work in mysql event scheduler to update my databases from a normal .csv
So I'm trying to setup a "cron job" in linux to run a shell script to do the LOAD DATA INFILE to my databases, but im getting errors on the following shell script, please help correct it, see my script layout below...
#!/bin/bash
mysql -u root -p xxxxxxx testdb --local_infile=1 -e"LOAD DATA LOCAL INFILE '/mnt/mysqldb/mysqldb-new/mysql/CK-BATCH-FTP/Acelity/activity.csv'
INTO TABLE acelity_activity
FIELDS TERMINATED BY ';'
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS;
Thank you for helping me
You must enclose the Double quotes in ENCLOSED BY and i think that you have a finalizing Double quote in you original code
#!/bin/bash
mysql -u root -p xxxxxxx testdb --local_infile=1 -e"LOAD DATA LOCAL INFILE '/mnt/mysqldb/mysqldb-new/mysql/CK-BATCH-FTP/Acelity/activity.csv'
INTO TABLE acelity_activity
FIELDS TERMINATED BY ';'
ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS;"
And you should test the command, in wokbench or phpmyadmin
Thanks for the help #nbk
So i modified the script above a little, and the following worked for me: (I removed the db name in the mysql syntax and add it in the query)
mysql -u root -pxxxxxxx -e"LOAD DATA LOCAL INFILE '/mnt/mysqldb/mysqldb-new/mysql/CK-BATCH-FTP/File/XXXX.csv'
INTO TABLE <DB NAME>.<TABLE NAME>
FIELDS TERMINATED BY ';'
ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS;"
If you are using Mysql8 then this will help you.
mysql -udbuser -pXXXXX -h host-pc test1 --local_infile=1 -e "LOAD DATA LOCAL INFILE '/home/abc/data.csv' INTO TABLE tempTable FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n' (col1, col2);";
Make sure there is no space in username and password i.e. it should be like this as mentioned in the above command.
Also ENCLOSED BY needs to write in the mentioned format only.
Since I'm using workbench and want to load the text file data.
And I would like to add file name in column.
Let me giving you my example.
My text file: 20170205077.txt
Data in "20170205077.txt":
1|Something|Something_2, 1|Something|Something_3
Also the query is
LOAD DATA LOCAL INFILE "20170205077.txt" INTO "DB.TABLE"
FIELDS TERMINATED BY '|' Lines terminated by ', '
When executed I hope the table like this:
| FileName |userId| data | data |
|20170205077| 1 |Something|Something_2|
|20170205077| 1 |Something|Something_3|
How to enhance the query?
Create a bat file: (ex. upload.bat)
Put it and all your text files in C:\import (name of folder and files must not have SPACE). Then run the bat file to import
#echo off
setlocal enabledelayedexpansion
FOR %%f IN ("*.txt") DO (
set old=%%~dpnxf
set new=!old:\=\\!
"C:\Program Files\MySQL\MySQL Server 5.7\bin\mysql" -e "load data local infile '"!new!"' IGNORE into table DB.TABLE (userID, data1, data2) FIELDS TERMINATED BY '|' OPTIONALLY ENCLOSED BY '"""' LINES TERMINATED BY ',' SET Filename = %%nf" -h yourhost -u youruser -pYourpassword YourDB
echo UPLOAD %%~nxf DONE
)
pause
I'm trying to simplify a load data local infile, by putting it into a .sh file, and bash to run it.
Here is my count_portal.sh file
mysql -h host -u root -p password
load data local infile
"/workplace/user/dump/count_portal.txt"
replace into table test.icqa_count_portal
fields terminated by '\t' lines terminated by '\n' ignore 1 lines;
And here is my bash script
bash /home/user/Desktop/count_portal.sh I get an output that doesn't do what it is designed to do. When I simply make the count_portal.sh contain mysql -h host it longs in when running the script.
I figured it out. Here is my file.
#!/bin/bash
/usr/bin/mysql --host=host --user=root --password=password --database=test<<EOFMYSQL
load data local infile '/workplace/user/ETLdump/count_portal.txt' replace INTO TABLE count_portal fields terminated by '\t' LINES
TERMINATED BY '\n' ignore 1 lines;
EOFMYSQL
Works flawlessly!
Try removing the space between the user and password switches. So it would be something like:
mysql -h host -uroot -ppassword etc....
The spaces seem to cause problems for me when doing similar things inside bash scripts.
missed -e ?
mysql -h host -u root -ppassword dbname -e "load data local infile '/workplace/user/dump/count_portal.txt' replace into table test.icqa_count_portal fields terminated by '\t' lines terminated by '\n' ignore 1 lines;"
I agree with #PasteBT, his answer should work well. I think there are escape problem or shell variable is empty.
Have you tried this?:
echo "LOAD DATA LOCAL INFILE '/workplace/user/dump/count_portal.txt' REPLACE INTO TABLE test.icqa_count_portal FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' IGNORE 1 LINES;" | mysql -h host -u root -ppassword
and you original shell script is wrong. could you post again what did you do?
mysql -u<username> -p<password> -h<hostname> <db_name> --local_infile=1 -e "use <db_name>" -e"LOAD DATA LOCAL INFILE '<path/file_name>'
IGNORE INTO TABLE <table_name>
FIELDS TERMINATED BY '\t'
OPTIONALLY ENCLOSED BY '\"'"
I'm using a bash script to pull data from online sources. Right now I just have it writing to a text file, but it would be better if the script could automatically put this data into mysql tables. How can this be done? Examples would be helpful.
Suppose you download a .csv file. which has header and have a database test in mysql.
Download the file first.
wget http://domain.com/data.csv -O data.csv
Dump the data to mysql table tbl
cat <<FINISH | mysql -uUSERNAME -pPASSWORD test
LOAD DATA INFILE 'data.csv' INTO TABLE `tbl`
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES;
FINISH
Here USERNAME must have FILE privilege.
You can use bash like this:
#!/bin/bash
params="dbname -uuser -ppassswd"
echo "SELECT * FROM table;" | mysql $params
# or
mysql $params <<DELIMITER
SELECT * FROM table;
DELIMITER
I have a mysql datbase with a tables that have the same column names as csv files that i recieve. I used to use a windows batch file to upload then into mysql.
#echo off
echo TRUNCATE TABLE `alarms`; > importalarm.sql
echo Creating list of MySQL import commands
for %%s in (*.csv) do
echo LOAD DATA LOCAL INFILE '%%s' INTO TABLE `alarms`
FIELDS TERMINATED BY ',' ENCLOSED BY ' '
ESCAPED BY '/'LINES TERMINATED BY '\r\n' IGNORE 1 LINES; >> importcsv.sql;
echo mysql -u root -p uninetscan < importalarm.sql
I need to turn this into a linux script.
Any ideas.
Here's how you'd do something similar on linux:
#!/bin/bash
echo 'TRUNCATE TABLE `alarms`;' > importalarm.sql
echo "Creating list of MySQL import commands"
for f in *.csv; do
echo "LOAD DATA LOCAL INFILE '$f' INTO TABLE \`alarms\` FILEDS TERMINATED BY ',' ENCLOSED BY ' ' ESCAPED BY '/' LINES TERMINATED BY '\r\n' IGNORE 1 LINES;" >> importcsv.sql;
done
echo 'mysql -u root -p uninetscan < importalarms.sql'
I noticed that you are sending the output to 2 files (importalarm.sql and importcsv.sql), not sure if it's what you want, but it's really easy to change.
Also, the last line is just echoing the command, not actually executing it. If you want to execute it, just remove the echo and quotes.