How can I load a file with "MySQL LOAD FILE" features from remote server?
As stated in the manual for LOAD DATA, you can use the LOCAL keyword:
The LOCAL keyword, if specified, is interpreted with respect to the client end of the connection:
If LOCAL is specified, the file is read by the client program on the client host and sent to the server. The file can be given as a full path name to specify its exact location. If given as a relative path name, the name is interpreted relative to the directory in which the client program was started.
When using LOCAL with LOAD DATA, a copy of the file is created in the server's temporary directory. This is not the directory determined by the value of tmpdir or slave_load_tmpdir, but rather the operating system's temporary directory, and is not configurable in the MySQL Server. (Typically the system temporary directory is /tmp on Linux systems and C:\WINDOWS\TEMP on Windows.) Lack of sufficient space for the copy in this directory can cause the LOAD DATA LOCAL statement to fail.
If LOCAL is not specified, the file must be located on the server host and is read directly by the server. The server uses the following rules to locate the file:
If the file name is an absolute path name, the server uses it as given.
If the file name is a relative path name with one or more leading components, the server searches for the file relative to the server's data directory.
If a file name with no leading components is given, the server looks for the file in the database directory of the default database.
Note that, in the non-LOCAL case, these rules mean that a file named as ./myfile.txt is read from the server's data directory, whereas the file named as myfile.txt is read from the database directory of the default database. For example, if db1 is the default database, the following LOAD DATA statement reads the file data.txt from the database directory for db1, even though the statement explicitly loads the file into a table in the db2 database:
LOAD DATA INFILE 'data.txt' INTO TABLE db2.my_table;
Windows path names are specified using forward slashes rather than backslashes. If you do use backslashes, you must double them.
For security reasons, when reading text files located on the server, the files must either reside in the database directory or be readable by all. Also, to use LOAD DATA INFILE on server files, you must have the FILE privilege. See Section 6.2.1, “Privileges Provided by MySQL”. For non-LOCAL load operations, if the secure_file_priv system variable is set to a nonempty directory name, the file to be loaded must be located in that directory.
Using LOCAL is a bit slower than letting the server access the files directly, because the contents of the file must be sent over the connection by the client to the server. On the other hand, you do not need the FILE privilege to load local files.
With LOCAL, the default duplicate-key handling behavior is the same as if IGNORE is specified; this is because the server has no way to stop transmission of the file in the middle of the operation. IGNORE is explained further later in this section.
LOCAL works only if your server and your client both have been configured to permit it. For example, if mysqld was started with --local-infile=0, LOCAL does not work. See Section 6.1.6, “Security Issues with LOAD DATA LOCAL”.
Related
I'm running mysql v8.0.24 on ubuntu server. I have created a table called uranium in the database stocks. I'm now trying to run the command below in mysql as root, to load records from a csv on the server into the table. I'm getting the error below, does anyone see what the issue might be and can you suggest how to fix it?
code:
LOAD DATA INFILE '/home/user/uranium.csv' INTO TABLE stocks.uranium FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 ROWS;
error:
ERROR 1290 (HY000): The MySQL server is running with the --secure-file-priv option so it cannot execute this statement
By default MySQL has secure_file_priv set, which restricts the directories that it will access for INFILE and OUTFILE operations. You can set the MySQL configuration to allow a nominated folder by editing the [mysqld] section of mysql.cnf*.
Ubuntu also has AppArmor running, that will prevent MySQL from seeing the file even if it has the correct permissions. You will need to update that too.
For this reason I recommend you create a dedicated directory for this sort of import, rather than using a home directory. For the purposes of this answer I have use /sqlfiles.
For Ubuntu and other Debian derived versions of Linux, go to /etc/mysql/mysql.conf.d
Using your favourite editor, edit the file mysql.cnf that you find there. (There are other files with the same name elsewhere in the /etc/mysql directory. Make sure you're editing the correct one)
At the end of the [mysqld] section add the line
secure-file-priv=/sqlfiles
Save the file, then create the folder and set the ownership to mysql
sudo mkdir /sqlfiles
sudo chown mysql:mysql /sqlfiles
Now update the AppArmor profile. Go to /etc/apparmor.d/local and edit the file usr.sbin.mysqld. Add these lines to the end of that file:
/sqlfiles/ r,
/sqlfiles/** rwk,
Finally, restart AppArmor
sudo service apparmor restart
This is working on my development server running Ubuntu 20.04, MySQL 8.0.24
Good luck!
* If you create the entry in mysql.cnf but leave the filename blank you will disable the effect of the variable, and MySQL will read or write to any folder. AppArmor will still prevent access unless you update that too.
Look at this answer
In spite the fact that your csv file is located on the server's file system, I think you should include LOCAL keyword:
LOAD DATA LOCAL INFILE ...
Or
The file has to be located in the database directory or have world
read permissions, and the client username must have the FILE
privilege.
I think your csv file is not
...located in the database directory or have world read permissions...
Note that
...client username must have the FILE privilege...
MySql8 documentation: doc and doc
LOAD DATA INFILE gets the file from the database server's local
filesystem. The file has to be located in the database directory or
have world read permissions, and the client username must have the
FILE privilege.
LOAD DATA LOCAL INFILE reads the file on the client, and sends the
contents to the server.
You can find more details in the documentation.
And
If LOCAL is specified, the file is read by the client program on the client host and sent to the server.
If LOCAL is not specified, the file must be located on the server host and is read directly by the server.
Credit goes to barmar and tim-biegeleisen
I have found the solution to load xml file data to mysql db table Here
My question is how I can load remote location xml file and dump it to mysql database table.
If you have any idea or have done something like this please help to find the solution.
Thank you
LOAD XML statement does not support retrieving files from a remote location. The file has to be either on the server, or on the client (LOCAL - if enabled in configuration). You need to write a script that saves the file either to the server or to the client from the remote location.
If you manage to map a remote shared directory to the local fileszstem, then you can import data that way.
I'm loading a csv file into a mysql instance running on local host. I can do this using the LOAD DATA LOCAL INFILE syntax, but what I'm wondering is why I need the LOCAL if the server and file are on the same machine. Without the local, I get an error saying:
ERROR 13 (HY000): Can't get stat of '/path/to/myfile.csv' (Errcode: 13)
That is because the system account under which MySQL is working have no rights to read this file.
When you don't specify LOCAL the file is being read directly by the MySQL server process and have to have rights to read this file.
When you add LOCAL the file is being read by mysql client program not the server process and in your case apparently has access to your csv file, which is I presume is located somewhere in the user directory of an account under which you're working.
If you put this file to a directory where MySQL process has rights to read from (e.g. /tmp/myfile.csv) LOAD DATA INFILE will work just fine.
LOAD DATA INFILE
The LOCAL keyword affects where the file is
expected to be found:
If LOCAL is specified, the file is read by the client program on the client host and sent to the server. The file can be given as a full
path name to specify its exact location. If given as a relative path
name, the name is interpreted relative to the directory in which the
client program was started.
If LOCAL is not specified, the file must be located on the server host and is read directly by the server.
The best place to look these up are in MySQL docs. Check out http://dev.mysql.com/doc/refman/5.1/en/load-data.html
The --local option causes mysqlimport to read data files from the
client host.
I need to load a CSV file from client machine to MySQL server database.
I am trying LOAD DATA INFILE.
My confusion is regarding ACCESS PERMISSIONs required to use
- LOAD DATA INFILE
- LOAD DATA LOCAL INFILE
Earlier I believed that I need FILE privilege to use both of them.
I came across this line in mysql documentation :
when reading text files located on the server, the files must either reside in the database directory or be readable by all. Also, to use LOAD DATA INFILE on server files, you must have the FILE privilege. See Section 6.2.1, “Privileges Provided by MySQL”. For non-LOCAL load operations, if the secure_file_priv system variable is set to a nonempty directory name, the file to be loaded must be located in that directory.
Looking at this, I got confused.
Do I need FILE privilege to load FILE from client machine using LOCAL option?
We do not need FILE privilege to LOAD data-file from a remote machine to MySQL Server. We need --local-infile option on Client machine enabled for that.
We need FILE privilege when we are trying to LOAD a data-file which is present on MySQL server. Additionally, mysql demon should also have access to READ from directory where data-file is placed.
This is probably too simplistic a question but here goes.
I have a client that will drop xls files into a folder on our FTP site. I need to check if a file exists, I need to move it from the FTP folder to a folder on the server. Once the processing is done, I need to send another (but different) xls file back to a folder on the same FTP server.
I can see that there is an FTP task and I can connect to the FTP site, but I am unsure on how to specify where to send the file and also how to only select a file at a time.
I think if I just concentrate on the first part, I can work on getting the file back as a second step.
So the end result is to check the folder on the FTP site, if a file exists move it to the server.
The SSIS FTP task wraps the basic FTP syntax you would use if you were connecting to the FTP site interactively. Here's a review of basic FTP syntax.
So here's what you should be looking for when you're editing the FTP task. 1) The task needs to log into the FTP server, 2) it needs to know that it is performing a GET operation, 3) it needs to know the path and filename of the file it is supposed to retrieve from the FTP server, and 4) it needs to know where to drop the file on the local server.
So, in the FTP Task Editor, you want to go to the General tab and create an FTP connection. Then go to the File Transfer tab, and then set the "Operation" -> "Receive files", and fill in values for the Local Path and the Remote Path. (Or you can keep those paths in SSIS variables and have the task get them from there.)
The IsTransferAscii setting is False by default. This means it will assume it is transferring a binary file. Alternatively, if you tell it to treat it like an Ascii file, it will try to fix the line endings to account for the different combinations of carriage return and line feed characters used by various operating systems. You don't want that if you want to transfer the file verbatim, but you might want it if you're going back and forth between Windows and Linux or something.
You should also learn a little interactive FTP syntax. I often use this to figure out why SSIS is having a problem transferring files. Go to the command prompt and type "ftp". You can then type "?" to see a list of commands. Or just type "ftp yourservername", log in, and use cd and ls to walk around the directory structure and see what's there.