After trying for a full day, I'm hoping someone here can help me make below script work. I've combined information from multiple threads (example) and websites, but can't get it to work.
What I'm trying to do:
I'm trying to get a MariaDB10 database called 'stock_db' on my Synology NAS to load all *.csv files from a specific folder (where I save downloaded historical prices of stocks) and add these to a table called 'prices'. The files are all equally named "price_history_'isin'.csv".
Below SQL statement works when running it individually from HeidiSQL on my Windows machine:
Working SQL
LOAD DATA LOW_PRIORITY LOCAL INFILE 'D:\\Downloads\\price_history_NL0010366407.csv'
IGNORE INTO TABLE `stock_db`.`prices`
CHARACTER SET utf8
FIELDS TERMINATED BY ';'
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 2 LINES
(#vdate, #vprice)
SET
isin = 'NL0010366407',
date = STR_TO_DATE(#vdate, '%d-%m-%Y'),
price = #vprice
;
The issue
Unfortunately, when I try to batch loading all csv's from a folder on my NAS through below script, I keep getting the same error.
#!/bin/bash
for filename in ./price_history/*.csv; do
echo $filename
isin=${filename:30:12}
echo $isin
/volume1/#appstore/MariaDB10/usr/local/mariadb10/bin/mysql -u root -p \
"LOAD DATA LOW_PRIORITY LOCAL INFILE '$filename'\
IGNORE INTO TABLE 'stock_db.prices'\
CHARACTER SET utf8\
FIELDS TERMINATED BY ';'\
OPTIONALLY ENCLOSED BY '"'"'"'\
ESCAPED BY '"'"'"'\
LINES TERMINATED BY '\r\n'\
IGNORE 2 LINES (#vdate, #vprice)\
SET\
isin = '$isin',\
date = STR_TO_DATE(#vdate, '%d-%m-%Y'),\
price = #vprice;"
done
ERROR 1102 (42000): Incorrect database name
What I've tried
Took the database name out of stock_db.prices and mentioned it separately as [database] outside of the quoted SQL statement - Doesn't work
Changed quotes around 'stock_db.prices' in many different ways - Doesn't work
Separated the SQL into a separate file and referenced it '< stmt.sql' - Complicates things even further and couldn't get it to work at all (although probably preferred)
Considered (or even preferred) using a PREPARE statement, but seems I can't use this in combination with LOAD DATA (reference)
Bonus Question
If someone can help me do this without having to re-enter the user's password or putting the password in the script, this would be really nice bonus!
Update
Got the 'Incorrect Database Error' resolved by adding '-e' option
Now I have a new error on the csv files:
ERROR 13 "Permission Denied"
While the folder and files are full access for everyone.
Anyone any thoughts to this?
Thanks a lot!
Try to set database using -D option: change the first line to
/volume1/#appstore/MariaDB10/usr/local/mariadb10/bin/mysql -D stock_db -u root -p \ ...
You may have an error in this line IGNORE INTO TABLE 'stock_db.prices'\ - try to remove the single quotes.
Create file .my.cnf in your user's home directory and put the following information into it:
[client]
password="my password"
Info about option files.
'stock_db.prices'
Incorrect quoting. This will work since neither are keywords:
stock_db.prices
This will also work:
`stock_db`.`prices`
Note that the db name and the table name are quoted separately, using backtics.
I can't predict what will happen with this nightmare:
'"'"'"'
Related
Inside a directory /mylog/ I have a bunch of CSV files. Each CSV file only has one line of data. I need this data to be inputted into a MySQL Database.
An example of a line would be:
2015-08-14 00:00:00,HOSTNAME,10271kB,17182kB,92874kB,10%,/dev/disk1,/
I need to remove the 'kB' from each file size and remove the % from the percentage field. I also need to make sure that the date time and hostname are always unique and no duplicate entries are ever put in.
This is what I started to write so far. But I'm obviously missing the database name to use and removing the kB and %. If there's anything else wrong or missing, let me know. There's also the fact mysql is called each time, is there a way to do multiple load data?
Shell script:
#!/bin/bash
for f in /var/log/mylog/*.csv
do
mysql -e "load data local infile '"$f"' into table myTable fields TERMINATED BY ',' LINES TERMINATED BY '\n'" -u myUser --password=myPassword --local-infile
done
I'm using this syntax to insert 50 records into a mySQL table from within a php script:
$sql = "LOAD DATA INFILE '/home/myusername/timeshts/hmo/tiertwo/myBuilders/BuilderTwo/sql-cron-files/overtime-item.sql' INTO TABLE `ot-item` CHARACTER SET 'utf8' FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'";
$result = mysql_query($sql,$conn);
if($result) { echo "Load data Done!!!"; }
I'm mainly used to Windows paths and locations, this is Linux and i think its a path issue - i'm using an absolute server path since im not sure where the 'data directory or database directory is as advised in the mySQL documentation. I suppose i haven't got access to either of these directories on a shared host - so absolute is my best bet. Is there something i'm missing in the syntax that i just cant see?? I wrote the script so i know that lines are terminated by a new line character and columns are separated with a single commma char.
Just a guess, but maybe you are missing a / at the beginning of the path.
You need file privilege .
Contact your DBA to execute below command
grant file on . to 'usr'#'%';
I have read many postings and done numerous google searches with different key words to find the answer to this question. While there are many postings on the topic, none of the postings gives an answer that works on my machine, so I am creating a new posting.
I have MySQL 5.6 installed on a windows development machine that is not connected to a network. I am trying to export data from a table into an outfile which I can then use for other purposes. the query runs successfully. In fact, when I try to run the same query a second time, I get a message saying that the outfile already exists. But when I go hunting for the file by its name, or using windows explorer, I cannot find it.
WHERE IS THE OUTFILE IN WINDOWS 7, USING MYSQL 5.6?
Here is code to create the outfile:
SELECT * INTO OUTFILE 'table.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM table
Use the following syntax :
- you tell him where to put it !
SELECT * INTO OUTFILE 'c:/table.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM table
It normally stored in the DATA_DIR parameter location.
From the reference manual for the SELECT statement:
SELECT ... INTO OUTFILE is the complement of LOAD DATA INFILE.
From the reference manual for the LOAD DATA INFILE:
The server uses the following rules to locate the file:
If the file name is an absolute path name, the server uses it as given.
If the file name is a relative path name with one or more leading components, the server searches for the file relative to the server's data directory.
If a file name with no leading components is given, the server looks for the file in the database directory of the default database.
So I would look in the default database directory. To find out which it is:
show variables like 'datadir';
You can, of course, define the location of the file specifying in your select... into outfile... sentence.
Also, if you have the MySQL installed in your cliente (or you are working on localhost), you can write something like this in the command line:
mysql [connection parameters] -e"select ..." > yourFile.txt
This will dump the result of your select statement into yourFile.txt in the current directory.
Hope this helps
So I am trying to access my servers databse reomtely and have it run commands to export several tables all to individual csv files. So what I have is a commmand line command parameters that look like this:
mysql -h 198.xxx.xxx.xxx -u user-p < file.txt
The contents of file.txt looks like this:
SELECT * FROM log
INTO OUTFILE 'C:\USERS\username\Desktop\log.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
SELECT * FROM permission_types
INTO OUTFILE 'C:\USERS\username\Desktop\permission_types.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
SELECT * FROM personal_info_options
INTO OUTFILE 'C:\USERS\username\Desktop\personal_info_options.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
I am not sure that I have the syntax for this right or if this is even possible I have been doing a bunch of research trying to get examples. People usually tell you concept but never seem to give you the code you need to test them its always something like:
mysql -h localhost -u user-p < somefile and they don't show you contents
of a file for example
I am running windows 7, I installed WAMPServer and it has MYSQL version 5.5.24, which I am access via commandline. I am not sure about the FILEDS TERMINATED BY or the ENCLOSED BY or LINES TERMINATED BY... do I need those at all? Will that actually save to my local machine? I am nervous about running this script I don't want to make a mistake and mess up the database. Also is .txt ok for the script file?
Any help you can give would be great.
I am not sure that I have the syntax for this right or if this is even possible I have been doing a bunch of research trying to get examples.
Your syntax is correct, except that each SELECT statement should be terminated with a semicolon. Note that you will also need to specify the database in which your tables reside—it's easiest to do this as an argument to mysql:
mysql -h 198.xxx.xxx.xxx -u user-p mydb < file.txt
I am not sure about the FILEDS TERMINATED BY or the ENCLOSED BY or LINES TERMINATED BY... do I need those at all?
As documented under SELECT ... INTO Syntax:
Here is an example that produces a file in the comma-separated values (CSV) format used by many programs:
SELECT a,b,a+b INTO OUTFILE '/tmp/result.txt'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM test_table;
As explained under LOAD DATA Syntax:
If you specify no FIELDS or LINES clause, the defaults are the same as if you had written this:
FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\'
LINES TERMINATED BY '\n' STARTING BY ''
It goes on to explain:
Conversely, the defaults cause SELECT ... INTO OUTFILE to act as follows when writing output:
Write tabs between fields.
Do not enclose fields within any quoting characters.
Use “\” to escape instances of tab, newline, or “\” that occur within field values.
Write newlines at the ends of lines.
Note that because LINES TERMINATED BY '\n' is the default, you could omit that clause; but the FIELDS clauses are necessary for CSV output.
Will that actually save to my local machine?
No. As documented under SELECT ... INTO Syntax:
The SELECT ... INTO OUTFILE 'file_name' form of SELECT writes the selected rows to a file. The file is created on the server host, so you must have the FILE privilege to use this syntax. file_name cannot be an existing file, which among other things prevents files such as /etc/passwd and database tables from being destroyed. The character_set_filesystem system variable controls the interpretation of the file name.
The SELECT ... INTO OUTFILE statement is intended primarily to let you very quickly dump a table to a text file on the server machine. If you want to create the resulting file on some other host than the server host, you normally cannot use SELECT ... INTO OUTFILE since there is no way to write a path to the file relative to the server host's file system.
However, if the MySQL client software is installed on the remote machine, you can instead use a client command such as mysql -e "SELECT ..." > file_name to generate the file on the client host.
It is also possible to create the resulting file on a different host other than the server host, if the location of the file on the remote host can be accessed using a network-mapped path on the server's file system. In this case, the presence of mysql (or some other MySQL client program) is not required on the target host.
I am nervous about running this script I don't want to make a mistake and mess up the database.
SELECT statements only read data from the database and do not make any changes to its content: thus they cannot "mess up the database".
Also is .txt ok for the script file?
Any extension will work: neither the MySQL client nor server software see it (your operating system reads the file and sends its content to the client program, which in turn sends that content to the server; the operating system is ambivalent to the file extension here).
On Windows, a .txt extension will associate the file with a text editor (e.g. Notepad) so that it can be readily opened for editing. Personally I would prefer .sql as it more accurately describes the file's content, and I would then associate that extension with a suitable editor—but none of that is necessary.
I am working with a large database 1.5 gig in size and hundreds of tables / fields. I need to convert all tables into CSV files. PhpMyAdmin does not do this easily / times out.
I would rather use a shell / mysql command or a script to get the data out and into CSV.
Note:
I am looking to export ALL tables of the database - in 1 shot. I can not produce an export command for every single table individually.
You can use mysqldump:
The mysqldump command can also generate output in CSV, other delimited text, or XML format.
In particular, look at the following arguments:
--tab=path
--fields-[optionally-]enclosed-by
--fields-escaped-by
--fields-terminated-by
--lines-terminated-by
--no-create-info
You will need to do this table by table, see below.
SELECT *
INTO OUTFILE '/tmp/products.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
ESCAPED BY '\\'
LINES TERMINATED BY '\n'
FROM products
Note that the directory must be writable by the MySQL database server. If it's not, you'll get an error message like this:
#1 - Can't create/write to file '/tmp/products.csv' (Errcode: 13)
Also note that it will not overwrite the file if it already exists, instead showing this error message:
#1086 - File '/tmp/products.csv' already exists
Source: http://www.electrictoolbox.com/mysql-export-data-csv/
Information about the software : sql2csv
Download link exe : http://www.convert-in.com/demos/sql2csv.exe
This is best option I found around for windows. With the software we can connect to local and remote DB server and select schema. In one shot we can extract all tables data into Valid CSV files.
Features :