I'm kinda new to SQL. I am trying to add data to an alredy created table, the csv looks like these, it is 132645 lines long:
iso_code;continent;location;date;population;total_cases;new_cases;new_cases_smoothed;total_deaths;new_deaths;new_deaths_smoothed;total_cases_per_million;new_cases_per_million;new_cases_smoothed_per_million;total_deaths_per_million;new_deaths_per_million;new_deaths_smoothed_per_million;reproduction_rate;icu_patients;icu_patients_per_million;hosp_patients;hosp_patients_per_million;weekly_icu_admissions;weekly_icu_admissions_per_million;weekly_hosp_admissions;weekly_hosp_admissions_per_million
AFG;Asia;Afghanistan;24/02/2020;398354280;50;50;NULL;NULL;NULL;NULL;126;126;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;
AFG;Asia;Afghanistan;25/02/2020;398354280;50;0;NULL;NULL;NULL;NULL;126;0;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;
AFG;Asia;Afghanistan;26/02/2020;398354280;50;0;NULL;NULL;NULL;NULL;126;0;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;
AFG;Asia;Afghanistan;27/02/2020;398354280;50;0;NULL;NULL;NULL;NULL;126;0;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;
AFG;Asia;Afghanistan;28/02/2020;398354280;50;0;NULL;NULL;NULL;NULL;126;0;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;
And my sql query is this:
LOAD DATA LOCAL INFILE 'C:\\Users\\Usuario\\Desktop\\CovidDeaths.csv' INTO TABLE coviddeaths
columns terminated by ';'
LINES TERMINATED BY '\r\n';
I have some alredy uploaded data to the coviddeaths table using the import wizards but the upload was really slow, so tried to finish importing using the LOAD DATA INFILE query.
To see how many lines have been alredy uploaded I used
select count(*) from coviddeaths;
And threw:
count(*)
11312
When I try to run the LOAD DATA LOCAL INFILE query, nothing happens. The mouse pointer spins for a second like it tries to run it but it doesnt add any additional row. It doesnt throw any errors like the ones I have read about Error Code: 1148. The used command is not allowed with this MySQL version. I also tried to place the csv file in the Uploads folder, where my Secure File Priv option pointed to in my.ini:
# Secure File Priv.
secure-file-priv="C:/ProgramData/MySQL/MySQL Server 8.0/Uploads"
It just does nothing. Any ideas?
Thanks in advance!
EDIT 2: Database Schema:
The error comes from the fact that both client and server must enable an option to allow local datafiles to be imported. If you use the command-line client, you would need to open the client like this:
mysql --local-infile -u root -p
In addition, the MySQL Server would need to be configured with the local-infile option.
The secure-file-priv is not needed, because that controls importing or exporting files on the server, not from the client.
See also ERROR 1148: The used command is not allowed with this MySQL version or my answer to "mysql 8.0" local infile fails I tried and display the settings. Could it be permission issues?
The error message is unusually obscure. MySQL error messages are usually more accurate and informative. I logged a bug in 2019 to request an improved error message, and they did fix it in MySQL 8.0.
I am trying to load data localy from pc to MySQL database ('texnologialogismikou') but I get a lot of errors. I searched a lot in forums and in MySQL site but I couldn't find a solution.
I tried to change variables, move the file in the Workbench directory and some other things but nothing worked.
load data local infile 'C:\Users\Mattheos\Desktop\ActualTotalLoad-10days.csv'
Into table texnologialogismikou.actualtotalload
fields terminated by ';' enclosed by '\"' lines terminated by '\n'
Some of the errors I got:
Error Code: 2. File 'C:ProgramDataMySQLMySQL Server 8.0Uploads' not found (OS errno 2 - No such file or directory)
Error Code: 1148. The used command is not allowed with this MySQL version
Error Code: 1290. The MySQL server is running with the --secure-file-priv option so it cannot execute this statement
P.S. I know that there are posts like this but this is the last thing I could do. I tried everything I could find and nothing worked.
On mySQL worckbench at the Manage Server Connections window -> Select your connection -> Go to the Advance tab and insert OPT_LOCAL_INFILE=1 in the others textinput.
The last error message is the relevant one:
Error Code: 1290. The MySQL server is running with the --secure-file-priv option so it cannot execute this statement
The --secure-file-priv option restricts from which directory files can be loaded, as explained in the documentation:
If set to the name of a directory, the server limits import and export operations to work only with files in that directory.
You can display the configured directory with:
show variables like "secure_file_priv";
Just move the file to import to that directory, and run the load statement again.
try mysqlimport
https://dev.mysql.com/doc/refman/8.0/en/mysqlimport.html
or Try community edition of tools like Pentaho ETL which will load data really fast.
I am running mysql version 5.7.27 for Linux on a remote server.
I have a php Script that should load some big csv files into a database via following code
$query = "LOAD DATA INFILE
'C:/some/big/file/file.csv'
REPLACE INTO TABLE csv_table
CHARACTER SET UTF8
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES";
The script is running normally but mysql sends out the following error:
Errormessage: The MySQL server is running with the --secure-file-priv option so it cannot execute this statement
I already did a lot of googling about this error and eventually found some helpful tips about how to set the variable and what to set:
https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_secure_file_priv
I also learned mysql will look at 3 places for the configuration:
/etc/my.cnf
/etc/mysql/my.cnf
/usr/local/etc/my.cnf ~/.my.cnf
in respective order.
The my.cnf includes etc/mysql/mysql.conf.d file so I put the following line there
[mysqld]
secure_file_priv = '/home/my/folder/'
When I check the setting via
SHOW VARIABLES LIKE 'secure-f%';
I get the correct value!
But when I try to execute the script, I still receive the same old error message. Since the setting is correctly printed, I am not sure what to make of this.
Please help me figure out how to get rid of this error. Thanks
EDIT:
Thanks for the hint! It's embarassing but it was really like you said. I tested locally (worked) and copy pasted the code onto the server, where there is no C:/ .. of course. Thank you.
After this I got a different error which could be solved with the help of this:
LOAD DATA INFILE Error Code : 13
I have a kind of huge amount of data on a .txt file (something like 90.000 lines and 10 columns, each 'item' separed by comma) and I need to find a way to import it from my txt file to a mysql table.
I did some research and found the following code:
LOAD DATA INFILE "/my_path/file.txt" INTO TABLE "table" CHARACTER SET "utf8" FIELDS TERMINATED BY "," LINES TERMINATED BY "\r\n";
but that gives me the following error:
Error Code: 29. File '/my_path/file.txt' not found (OS errno 13 - Permission denied) 0.000 sec
So after searching a little bit more, I found that a lot of people had this issue but solved just by putting the word 'LOCAL' (LOAD DATA LOCAL INFILE), but that gives me this error:
Error Code: 1148. The used command is not allowed with this MySQL version 0.000 sec
(I'm using MySQL Workbench 8.0)
So I tried to set some global variables like secure-file-priv="", loose-local-infile=1 and local-infile=1 in my my.ini file, but nothing seems to work.
Can anyone help me please...
Ps 1: I can't use the wizard import because even though it would work, it's too slow
Ps 2: I tried to convert my txt file to csv but it didn't work either.
Without LOCAL, LOAD DATA INFILE tells the MySQL server to find the file on the file system of its own machine and read it.
So, the path and file have to exist on the server machine. And, the credentials (the username) under which the MySQL server runs must have read access to the file.
Your permission denied error hints to me that the MySQL server can't read the file.
The OP had 2 problems:
a) mysql did not have access to the input file from a permissions perspective
b) the input file was not correctly encoded.
Once the access issue was resolved and the input file encoding had been corrected to utf8, the load worked correctly.
I use MySQL queries all the time in PHP, but when I try
LOAD DATA INFILE
I get the following error
#1045 - Access denied for user 'user'#'localhost' (using password: YES)
Does anyone know what this means?
I just ran into this issue as well. I had to add LOCAL to my SQL statement.
For example, this gives the permission problem:
LOAD DATA INFILE '{$file}' INTO TABLE {$table}
Add LOCAL to your statement and the permissions issue should go away. Like so:
LOAD DATA LOCAL INFILE '{$file}' INTO TABLE {$table}
I had this problem. I searched around and did not find a satisfactory answer. I summarise below the results of my searches.
The access denied error could mean that:
'user'#'localhost' does not have the FILE privilege (GRANT FILE on *.* to user#'localhost'); or,
the file you are trying to load does not exist on the machine running mysql server (if using LOAD DATA INFILE); or,
the file you are trying to load does not exist on your local machine (if using LOAD DATA LOCAL INFILE); or,
the file you are trying to load is not world readable (you need the file and all parent directories to be world-readable: chmod 755 directory; and, chmod 744 file.dat)
Try using this command:
load data local infile 'home/data.txt' into table customer;
This should work. It worked in my case.
Ensure your MySQL user has the FILE privilege granted.
If you are on shared web hosting, there is a chance this is blocked by your hosting provider.
I found easy one if you are using command line
Login asmysql -u[username] -p[password] --local-infile
then SET GLOBAL local_infile = 1;
select your database by use [db_name]
and finally LOAD DATA LOCAL INFILE 'C:\\Users\\shant\\Downloads\\data-1573708892247.csv' INTO TABLE visitors_final_test FIELDS TERMINATED BY ','LINES TERMINATED BY '\r \n' IGNORE 1 LINES;
The string from Lyon gave me a very good tip: On Windows, we need to use slahes and not backslashes. This code works for me:
File tempFile = File.createTempFile(tableName, ".csv");
FileUtils.copyInputStreamToFile(data, tempFile);
JdbcTemplate template = new JdbcTemplate(dataSource);
String path = tempFile.getAbsolutePath().replace('\\', '/');
int rows = template.update(MessageFormat
.format("LOAD DATA LOCAL INFILE ''{0}'' INTO TABLE {1} FIELDS TERMINATED BY '',''",
path, tableName));
logger.info("imported {} rows into {}", rows, tableName);
tempFile.delete();
If you are trying this on MySQL Workbench,
Go to connections -> edit connection -> select advanced tab
and add OPT_LOCAL_INFILE=1 in the 'Others' text field.
Now restart the connection and try.
I ran into the same issue, and solve it by folowing those steps :
activate load_infile variable
grand file permission to my custom mysql user
deactivate secure_file_priv variable (my file was uploaded by the webserver to the /tmp folder which is of course not the secured directory of myslq /var/lib/mysql-file)
For this 3rd point, you can refer to : https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_secure_file_priv
BR,
AD
This happened to me as well and despite having followed all the steps described by Yamir in his post I couldn't make it work.
The file was in /tmp/test.csv with 777 permissions. The MySQL user had file permissions, LOCAL option was not allowed by my MySQL version, so I was stuck.
Finally I was able to solve the problem by running:
sudo chown mysql:mysql /tmp/test.csv
I discovered loading MySQL tables can be fast and painless (I was using python / Django model manager scripts):
1) create table with all columns VARCHAR(n) NULL e.g.:
mysql> CREATE TABLE cw_well2( api VARCHAR(10) NULL,api_county VARCHAR(3) NULL);
2) remove headers (first line) from csv, then load (if you forget the LOCAL, you’ll get “#1045 - Access denied for user 'user'#'localhost' (using password: YES)”):
mysql> LOAD DATA LOCAL INFILE "/home/magula6/cogswatch2/well2.csv" INTO TABLE cw_well2 FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'
-> ;
Query OK, 119426 rows affected, 19962 warnings (3.41 sec)
3) alter columns:
mysql> ALTER TABLE cw_well2 CHANGE spud_date spud_date DATE;
mysql> ALTER TABLE cw_well2 CHANGE latitude latitude FLOAT;
voilà!
I was trying to insert data from CSV to MYSQL DB using python.
You can try the below method to load data from CSV to Database.
Make a connection with the Database using pymysql or MySQL.connector any library you want in python.
Make Sure you are able to use the in-line while connecting for that while providing host, user, and password try to add local_inline=True.
Skipping to load data part.
sql = f'''LOAD DATA LOCAL infile "filename.csv" INTO TABLE schema.tablename FILED TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\r\n'''' Note: If you have column names in CSV, use IGNORE ROW 1 LINES. The execute the sql by: cursor.execute(sql) conn.commit() conn.close()
It probably means that the password you supplied for 'user'#'localhost' is incorrect.