SQL Remote Connection with file read - mysql

So I am trying to access my servers databse reomtely and have it run commands to export several tables all to individual csv files. So what I have is a commmand line command parameters that look like this:
mysql -h 198.xxx.xxx.xxx -u user-p < file.txt
The contents of file.txt looks like this:
SELECT * FROM log
INTO OUTFILE 'C:\USERS\username\Desktop\log.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
SELECT * FROM permission_types
INTO OUTFILE 'C:\USERS\username\Desktop\permission_types.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
SELECT * FROM personal_info_options
INTO OUTFILE 'C:\USERS\username\Desktop\personal_info_options.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
I am not sure that I have the syntax for this right or if this is even possible I have been doing a bunch of research trying to get examples. People usually tell you concept but never seem to give you the code you need to test them its always something like:
mysql -h localhost -u user-p < somefile and they don't show you contents
of a file for example
I am running windows 7, I installed WAMPServer and it has MYSQL version 5.5.24, which I am access via commandline. I am not sure about the FILEDS TERMINATED BY or the ENCLOSED BY or LINES TERMINATED BY... do I need those at all? Will that actually save to my local machine? I am nervous about running this script I don't want to make a mistake and mess up the database. Also is .txt ok for the script file?
Any help you can give would be great.

I am not sure that I have the syntax for this right or if this is even possible I have been doing a bunch of research trying to get examples.
Your syntax is correct, except that each SELECT statement should be terminated with a semicolon. Note that you will also need to specify the database in which your tables reside—it's easiest to do this as an argument to mysql:
mysql -h 198.xxx.xxx.xxx -u user-p mydb < file.txt
I am not sure about the FILEDS TERMINATED BY or the ENCLOSED BY or LINES TERMINATED BY... do I need those at all?
As documented under SELECT ... INTO Syntax:
Here is an example that produces a file in the comma-separated values (CSV) format used by many programs:
SELECT a,b,a+b INTO OUTFILE '/tmp/result.txt'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM test_table;
As explained under LOAD DATA Syntax:
If you specify no FIELDS or LINES clause, the defaults are the same as if you had written this:
FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\'
LINES TERMINATED BY '\n' STARTING BY ''
It goes on to explain:
Conversely, the defaults cause SELECT ... INTO OUTFILE to act as follows when writing output:
Write tabs between fields.
Do not enclose fields within any quoting characters.
Use “\” to escape instances of tab, newline, or “\” that occur within field values.
Write newlines at the ends of lines.
Note that because LINES TERMINATED BY '\n' is the default, you could omit that clause; but the FIELDS clauses are necessary for CSV output.
Will that actually save to my local machine?
No. As documented under SELECT ... INTO Syntax:
The SELECT ... INTO OUTFILE 'file_name' form of SELECT writes the selected rows to a file. The file is created on the server host, so you must have the FILE privilege to use this syntax. file_name cannot be an existing file, which among other things prevents files such as /etc/passwd and database tables from being destroyed. The character_set_filesystem system variable controls the interpretation of the file name.
The SELECT ... INTO OUTFILE statement is intended primarily to let you very quickly dump a table to a text file on the server machine. If you want to create the resulting file on some other host than the server host, you normally cannot use SELECT ... INTO OUTFILE since there is no way to write a path to the file relative to the server host's file system.
However, if the MySQL client software is installed on the remote machine, you can instead use a client command such as mysql -e "SELECT ..." > file_name to generate the file on the client host.
It is also possible to create the resulting file on a different host other than the server host, if the location of the file on the remote host can be accessed using a network-mapped path on the server's file system. In this case, the presence of mysql (or some other MySQL client program) is not required on the target host.
I am nervous about running this script I don't want to make a mistake and mess up the database.
SELECT statements only read data from the database and do not make any changes to its content: thus they cannot "mess up the database".
Also is .txt ok for the script file?
Any extension will work: neither the MySQL client nor server software see it (your operating system reads the file and sends its content to the client program, which in turn sends that content to the server; the operating system is ambivalent to the file extension here).
On Windows, a .txt extension will associate the file with a text editor (e.g. Notepad) so that it can be readily opened for editing. Personally I would prefer .sql as it more accurately describes the file's content, and I would then associate that extension with a suitable editor—but none of that is necessary.

Related

PlanetScale : Import / Export to CSV in MySQL Shell requires Access to Servers Filesystem

If I want to do an export to csv or an import from csv, I will need access to the filesystem from within my MySQL Database Shell.
e.g.
SELECT id, filename
FROM attachments
INTO OUTFILE '/tmp/results.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n';
I am using PlanetScale right now, and i don't know how or where I can get access to the servers filesystem in order to import or export data from within the mysql shell.
See https://vitess.io/docs/14.0/reference/compatibility/mysql-compatibility/
The SELECT ... INTO form of SELECT in MySQL enables a query result to
be stored in variables or written to a file. Vitess supports SELECT
... INTO DUMPFILE and SELECT ... INTO OUTFILE constructs for unsharded
keyspaces but does not support storing results in variable. Moreover,
the position of INTO must be towards the end of the query and not in
the middle. An example of a correct query is as follows:
SELECT * FROM <tableName> INTO OUTFILE 'x.txt' FIELDS TERMINATED BY ';' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\t' LINES TERMINATED BY '\n'
For sharded keyspaces this statement can still be used but only after
specifying the exact shard with a USE Statement.
(emphasis mine)
So you must know which server to use, because it's required that you specify the shard.

inserting csv files into mysql db

I have csv file with delimiter ';'. It looks following (third column should be empty):
id;theme_name;description
5cbde2fe-b70a-5245-bbde-c2504a4bced1;DevTools;allow web developers to test and debug their code. They are different from website builders and integrated development environments (IDEs) in that they do not assist in the direct creation of a webpage, rather they are tools used for testing the user interface of a website or web application.
c8bfc406-aaa6-5cf9-94c3-09fc54b934e7;AI;
Here is my script for inserting data from csv into db:
mysql -u ${MYSQL_USER} --password=${MYSQL_PASSWORD} ${MYSQL_DATABASE} --local-infile=1 -h ${MYSQL_HOST} -e"LOAD DATA LOCAL INFILE '/tmp/init_data/$file' INTO TABLE $table_name FIELDS TERMINATED BY ';' OPTIONALLY ENCLOSED BY '\"' IGNORE 1 LINES";
When I'm making SELECT statement, Im getting carriage return (\r) in last column in response:
Here is response from mysql
{
"themeName": "DevTools",
"description": "allow web developers to test and debug their code. They are different from website builders and integrated development environments (IDEs) in that they do not assist in the direct creation of a webpage, rather they are tools used for testing the user interface of a website or web application.\r"
}, {
"themeName": "AI",
"description": "\r"
}
When I add delimiter ';' after last column in csv file, carriage return disappeared from response.
for example: c8bfc406-aaa6-5cf9-94c3-09fc54b934e7;AI;;
Why mysql add \r into third column ?
Is there any possible way how to solve it ? (except replace in select statement)
Thanks
I bet your CSV file comes from Windows. Those files have \r\n at the end of every line.
Add this to your command ...
LINES TERMINATED BY '\\r\\n'
and things should work. If they don't try this
LINES TERMINATED BY '\r\n'
On UNIX-derived systems (like Linux) the default LINES TERMINATED BY '\n' works.
On Mac systems you need LINES TERMINATED BY '\r'.
If you add a trailing ; column separator you create a fourth column in your CSV. But the table you're loading only has three columns, so LOAD DATA INFILE ignores that fourth column, which has a \r in it.
Why this difference, you ask? Old-timey teletype devices (the original terminals for MS-DOS and UNIX) needed a Return code to move the print head back to the first column, and a Newline code to move the paper up one line. The UNIX Bell Labs team decided their tty driver would add the extra Return code so lines ended with a particular single character. MS-DOS's team (umm, Bill Gates) left them both in.
Why Mac with just Return? Maybe somebody knows.
Answer:
According to #O.Jones answer I needed to add LINES TERMINATED BY '\r' but alsoI need to add \n
mysql -u ${MYSQL_USER} --password=${MYSQL_PASSWORD} ${MYSQL_DATABASE} --local-infile=1 -h ${MYSQL_HOST} -e"LOAD DATA LOCAL INFILE '/tmp/init_data/$file' INTO TABLE $table_name FIELDS TERMINATED BY ';' ENCLOSED BY '\"' LINES TERMINATED BY '\r\n' IGNORE 1 LINES";

MySQL 's 'secure-file-priv' not allowing load data infile, even from the allocated safe folder

I am trying to upload a large .csv file into a database (not on a server, just on my local computer only for my use) on MySQL. I have been having the Error Code: 1290. The MySQL server is running with the --secure-file-priv option so it cannot execute this statement error thrown.
I put my .csv file into the specified 'safe' folder (C:/ProgramData/MySQL/MySQL Server 5.7/Uploads) specified in the 'my' file, but it kept giving me the same error. Every solution I've seen has been using that folder destination and since that doesn't work can anyone help me get around or turn off the secure-file-priv option?
Here is my code in case wanted:
LOAD DATA INFILE 'C:\ProgramData\MySQL\MySQL Server 5.7\Uploads\my-file.csv'
INTO TABLE my-table
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS;
Thank you for any help
From the MySQL documentation for LOAD DATA INFILE:
The file name must be given as a literal string. On Windows, specify backslashes in path names as forward slashes or doubled backslashes. The character_set_filesystem system variable controls the interpretation of the file name.
LOAD DATA INFILE "C:/ProgramData/MySQL/MySQL Server 5.7/Uploads/my-file.csv"
INTO TABLE my-table
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS;
I had the same issue. Using forward slashes instead of backslashes (even on Windows machine) fixed it for me.

where is the outfile created by mysql 5.6

I have read many postings and done numerous google searches with different key words to find the answer to this question. While there are many postings on the topic, none of the postings gives an answer that works on my machine, so I am creating a new posting.
I have MySQL 5.6 installed on a windows development machine that is not connected to a network. I am trying to export data from a table into an outfile which I can then use for other purposes. the query runs successfully. In fact, when I try to run the same query a second time, I get a message saying that the outfile already exists. But when I go hunting for the file by its name, or using windows explorer, I cannot find it.
WHERE IS THE OUTFILE IN WINDOWS 7, USING MYSQL 5.6?
Here is code to create the outfile:
SELECT * INTO OUTFILE 'table.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM table
Use the following syntax :
- you tell him where to put it !
SELECT * INTO OUTFILE 'c:/table.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM table
It normally stored in the DATA_DIR parameter location.
From the reference manual for the SELECT statement:
SELECT ... INTO OUTFILE is the complement of LOAD DATA INFILE.
From the reference manual for the LOAD DATA INFILE:
The server uses the following rules to locate the file:
If the file name is an absolute path name, the server uses it as given.
If the file name is a relative path name with one or more leading components, the server searches for the file relative to the server's data directory.
If a file name with no leading components is given, the server looks for the file in the database directory of the default database.
So I would look in the default database directory. To find out which it is:
show variables like 'datadir';
You can, of course, define the location of the file specifying in your select... into outfile... sentence.
Also, if you have the MySQL installed in your cliente (or you are working on localhost), you can write something like this in the command line:
mysql [connection parameters] -e"select ..." > yourFile.txt
This will dump the result of your select statement into yourFile.txt in the current directory.
Hope this helps

MySQL Results to a File

How do I write the results from a mysql query to file? I just need something quick. Output can be CSV, XML, HTML, etc.
SELECT a,b,a+b
FROM test_table
INTO OUTFILE '/tmp/result.txt'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(the docs show INTO OUTFILE up in the SELECT .. portion which may work as well, but I've never tried it that way)
http://dev.mysql.com/doc/refman/5.0/en/select.html
INTO OUTFILE creates a file on the server; if you are on a client and want it there, do:
mysql -u you -p -e "SELECT ..." > file_name
if you have phpMyAdmin installed, it is a nobrainer: Run the query (haven't got a copy loaded, so I can't tell you the details, but it really is easy) and check neer bottom for export options. CSV will be listed, but I think you can also have SQL if you like :)
phpMyAdmin will give CSV in Excels dialect, which is probably what you want...
You can use MySQL Query Browser to run the query and then just go to File -> Export Resultset and choose the output format. The options are CSV, HTML, XML, Excel and PLIST.