I am scanning 5000 csv files into a database. Regrettably the files have '', for 0. Thus when ever I run my script it crashes. I heard that this error is possible to avoid by simply disabling strict mode. So I attempted to disable strict mode to allow me to read in an empty string as a 0 for my numeric fields. However the error persisted.
So does disabling strict mode allow '' to be read into a int field? (the '' is two qoutes i.e. empty string)
If so why did setting
sql_mode=''
in the my.ini config file not fix the problem.
Thank you!
I guess you import the CSV file using LOAD DATA INFILE command. Before you execute this command, type:
SET sql_mode = '';
More information about various SQL modes can be found in the documentation.
I need do it this, and work correctly:
To disable strict SQL mode, SSH in to your server as root
Create this file: /etc/mysql/conf.d/disable_strict_mode.cnf
Open the file and enter these two lines:
[mysqld]
sql_mode=IGNORE_SPACE,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
Restart MySQL with this command: sudo service mysql restart
You can modify you LOAD DATA INFILE Statement to correct the values. Something along these lines should work.
LOAD DATA INFILE 'filepath.csv'
INTO TABLE MyTable(Column1,#Col2,#Col3)
SET Column2=CASE WHEN #Col2 = '' THEN 0 ELSE #Col2 END
,Column3=CASE WHEN #Col3 = '' THEN 0 ELSE #Col3 END;
This query imports the value as is into column1, and the fixes the values for columns 2 and 3. Using this, you don't have to disable strict mode, and you are actually in control of what data goes into your database, are are able to fix it in a reliable way. You can also use this feature to change date formats, or import hex encoded blob values. Very useful feature.
I'm putting this in an answer as I am yet to have the rep to comment on your follow up question about the settings not "holding". It has been my experience with MySQL that the settings in the configuration file are only read on starting the server so that you would have to restart the server for new settings to take effect.
The server is the daemon that the command-line client communicates with when you run commands so that restarting the command line client does not actually restart the server itself.
A more elaborate explanation is provided in this answer: https://serverfault.com/a/79051.
Related
I have a node.js script which is connecting to a local MariaDB server, creating some tables, and then attempting to load some data into them by executing a SQL query like this:
LOAD XML LOCAL INFILE 'C:\\path\\to\\some_data.xml'
INTO TABLE some_data
ROWS IDENTIFIED BY '<data>';
The script fails with an error like:
SqlError: (conn=60, no: 45034, SQLState: 45034) LOCAL INFILE wrong filename. 'C:\path\to\some_data.xml' doesn't correspond to query LOAD XML LOCAL INFILE 'C:\\path\\to\\some_data.xml'
INTO TABLE some_data
ROWS IDENTIFIED BY '<data>';. Query cancelled. Check for malicious server / proxy
The node.js script code is like:
const pathToXmlFile = path.resolve(__dirname, '../some_data.xml')
.replace(/\\/g, '\\\\');
// .replace(/\\/g, '/');
// none of the above work, and neither does no escaping at all
query = `LOAD XML LOCAL INFILE '${pathToXmlFile}'
INTO TABLE some_data
ROWS IDENTIFIED BY '<data>';`;
I tried escaping the path with forward slashes (I'm running the script through Git Bash), the error remains the same, only the way the path is displayed changes. I have set permitLocalInfile: 'true' on the MariaDB client connection and local_infile=1 on both the [mysqld] and [client] sections of my.ini. The node.js connection user has the FILE privilege.
Node version is 16.13.0, MariaDB npm module version is 2.5.5, all running on the same machine with Windows 10 (MariaDB server too, on localhost).
The above query works perfectly fine when executed from HeidiSQL.
Any ideas on what I'm doing wrong?
I've create a mariadb bug for that : https://jira.mariadb.org/browse/CONJS-181
As a workaround, you might use "/" in place of "\": using "C:/path/to/some_data.xml" won't be escaped avoiding this bug
It seems that it's a bug with the mariadb node connector, I have opened an issue here:
https://github.com/mariadb-corporation/mariadb-connector-nodejs/issues/183
In the meantime, I have tried the mysql2 and mysql connectors, the first doesn't seem to support the permitLocalInfile: 'true' connection option, so I switch to the standard mysql connector and everything works fine (with a few changes in the code because of slight differences in method calling).
If the bug in the mariadb connector is fixed, I will update the answer here.
I'm kinda new to SQL. I am trying to add data to an alredy created table, the csv looks like these, it is 132645 lines long:
iso_code;continent;location;date;population;total_cases;new_cases;new_cases_smoothed;total_deaths;new_deaths;new_deaths_smoothed;total_cases_per_million;new_cases_per_million;new_cases_smoothed_per_million;total_deaths_per_million;new_deaths_per_million;new_deaths_smoothed_per_million;reproduction_rate;icu_patients;icu_patients_per_million;hosp_patients;hosp_patients_per_million;weekly_icu_admissions;weekly_icu_admissions_per_million;weekly_hosp_admissions;weekly_hosp_admissions_per_million
AFG;Asia;Afghanistan;24/02/2020;398354280;50;50;NULL;NULL;NULL;NULL;126;126;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;
AFG;Asia;Afghanistan;25/02/2020;398354280;50;0;NULL;NULL;NULL;NULL;126;0;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;
AFG;Asia;Afghanistan;26/02/2020;398354280;50;0;NULL;NULL;NULL;NULL;126;0;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;
AFG;Asia;Afghanistan;27/02/2020;398354280;50;0;NULL;NULL;NULL;NULL;126;0;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;
AFG;Asia;Afghanistan;28/02/2020;398354280;50;0;NULL;NULL;NULL;NULL;126;0;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;NULL;
And my sql query is this:
LOAD DATA LOCAL INFILE 'C:\\Users\\Usuario\\Desktop\\CovidDeaths.csv' INTO TABLE coviddeaths
columns terminated by ';'
LINES TERMINATED BY '\r\n';
I have some alredy uploaded data to the coviddeaths table using the import wizards but the upload was really slow, so tried to finish importing using the LOAD DATA INFILE query.
To see how many lines have been alredy uploaded I used
select count(*) from coviddeaths;
And threw:
count(*)
11312
When I try to run the LOAD DATA LOCAL INFILE query, nothing happens. The mouse pointer spins for a second like it tries to run it but it doesnt add any additional row. It doesnt throw any errors like the ones I have read about Error Code: 1148. The used command is not allowed with this MySQL version. I also tried to place the csv file in the Uploads folder, where my Secure File Priv option pointed to in my.ini:
# Secure File Priv.
secure-file-priv="C:/ProgramData/MySQL/MySQL Server 8.0/Uploads"
It just does nothing. Any ideas?
Thanks in advance!
EDIT 2: Database Schema:
The error comes from the fact that both client and server must enable an option to allow local datafiles to be imported. If you use the command-line client, you would need to open the client like this:
mysql --local-infile -u root -p
In addition, the MySQL Server would need to be configured with the local-infile option.
The secure-file-priv is not needed, because that controls importing or exporting files on the server, not from the client.
See also ERROR 1148: The used command is not allowed with this MySQL version or my answer to "mysql 8.0" local infile fails I tried and display the settings. Could it be permission issues?
The error message is unusually obscure. MySQL error messages are usually more accurate and informative. I logged a bug in 2019 to request an improved error message, and they did fix it in MySQL 8.0.
I'm trying to upload a csv file to a local database using MySQL with mybatis. I created a mysql container to act as the server and use datagrip to access the data. I want to use:
<insert id="batchInsertCsv" parameterType="string">
LOAD DATA LOCAL INFILE #{filePath}
INTO TABLE DATA
COLUMNS TERMINATED BY ','
IGNORE 1 LINES
</insert>
to upload the data but my current error is: Loading local data is disabled; this must be enabled on both the client and server sides. Using:
SET GLOBAL local_infile = true;
I have set local_infile on the server side and checked it with:
SHOW GLOBAL VARIABLES LIKE 'local_infile';
Can anyone offer help or advice? Thanks.
LOAD DATA LOCAL INFILE must be allowed both in server and client and you have already done the server part.
As MyBatis is built on top of JDBC API, you should add a connection property to the JDBC URL to enable the feature in client. e.g.
jdbc:mysql://127.0.0.1:3309/test?allowLoadLocalInfile=true
Note that there are several properties related to this feature.
allowLoadLocalInfile
allowLoadLocalInfileInPath
allowUrlInLocalInfile
The first one is a simple TRUE/FALSE switch.
The latter two allow you to set some restriction on the file being uploaded.
Please see the documentation for the details.
https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-connp-props-security.html
I am running mysql version 5.7.27 for Linux on a remote server.
I have a php Script that should load some big csv files into a database via following code
$query = "LOAD DATA INFILE
'C:/some/big/file/file.csv'
REPLACE INTO TABLE csv_table
CHARACTER SET UTF8
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES";
The script is running normally but mysql sends out the following error:
Errormessage: The MySQL server is running with the --secure-file-priv option so it cannot execute this statement
I already did a lot of googling about this error and eventually found some helpful tips about how to set the variable and what to set:
https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_secure_file_priv
I also learned mysql will look at 3 places for the configuration:
/etc/my.cnf
/etc/mysql/my.cnf
/usr/local/etc/my.cnf ~/.my.cnf
in respective order.
The my.cnf includes etc/mysql/mysql.conf.d file so I put the following line there
[mysqld]
secure_file_priv = '/home/my/folder/'
When I check the setting via
SHOW VARIABLES LIKE 'secure-f%';
I get the correct value!
But when I try to execute the script, I still receive the same old error message. Since the setting is correctly printed, I am not sure what to make of this.
Please help me figure out how to get rid of this error. Thanks
EDIT:
Thanks for the hint! It's embarassing but it was really like you said. I tested locally (worked) and copy pasted the code onto the server, where there is no C:/ .. of course. Thank you.
After this I got a different error which could be solved with the help of this:
LOAD DATA INFILE Error Code : 13
I'm having problems using 'LOAD DATA INFILE' via a db.executesql() command: I'm getting "InternalError: (1148, u'The used command is not allowed with this MySQL version')"
A bit of digging, and I find I need to set --local-infile as a mysql option (MySQL: Enable LOAD DATA LOCAL INFILE). I've modified my.cnf, but that doesn't seem to be being picked up by web2py. How can I pass options to the mysql client which is used by web2py?