I use parametric queries for normal insert/updates for security.
How do I do that for queries like this:
LOAD DATA INFILE '/filepath' INTO TABLE mytable
In my case, the path to the file would be different everytime (for different requests). Is it fine to proceed like this (since I am not getting any data from outside, the file is from the server itself):
path = /filepath
"LOAD DATA INFILE" + path + "INTO TABLE mytable"
Since LOAD DATA is not listed in SQL Syntax Allowed in Prepared Statements you can't prepare something like
LOAD DATA INFILE ? INTO TABLE mytable
But SET is listed. So a workaround could be to prepare and execute
SET #filepath = ?
And then execute
LOAD DATA INFILE #filepath INTO TABLE mytable
Update
In Python with MySQLdb the following query should work
LOAD DATA INFILE %s INTO TABLE mytable
since no prepared statement is used.
To answer your "is it fine to proceed like this" question, your example code will fail because the resulting query will be missing quotes around the filename. If you changed it to the following it could run, but is still a bad idea IMO:
path = "/filepath"
sql = "LOAD DATA INFILE '" + path + "' INTO TABLE mytable" # note the single quotes
While you may not be accepting outside input today, code has a way of sticking around and getting reused/copied, so you should use the API in a way that will escape your parameters:
sql = "LOAD DATA INFILE %s INTO TABLE mytable"
cursor.execute(sql, (path,))
And don't forget to commit if autocommit is not enabled.
Related
I have a LOAD DATA mysql query im trying to run and need help to fix one thing.
Here is the query im running
$query = "LOAD DATA LOCAL INFILE '$file_app'
INTO TABLE tbl_user_tmp
LINES STARTING BY '{'
TERMINATED BY '/>'
(#name)
set
name=SUBSTRING_INDEX(#name,'\"',-2),
activity=SUBSTRING_INDEX(SUBSTRING_INDEX(#name,'}',1),'/',1),
class=SUBSTRING_INDEX(SUBSTRING_INDEX(#name,'}',1),'/',-1),
user = '$user'" ;
Here is the example of data from file im trying to load.
<item component="ComponentInfo{com.apps.aaa.roadside/com.apps.aaa.roadside.Splash}" drawable="aaa_roadside1" />
With the above query i get the following
name=aaa_roadside1"
activity=com.apps.aaa.roadside
class=com.apps.aaa.roadside.Splash
Everything is correct but name. I need to removed that last "
I thought the below query would work but it does not. any ideas?
This was working before when i had TERMINATED BY set to this '\" />'
However this will not account for entires that might not have that space at end like this
<item component="ComponentInfo{com.apps.aaa.roadside/com.apps.aaa.roadside.Splash}" drawable="aaa_roadside1"/>
I need to account for both
$query = "LOAD DATA LOCAL INFILE '$file_app'
INTO TABLE tbl_user_tmp
LINES STARTING BY '{'
TERMINATED BY '/>'
(#name)
set
name=SUBSTRING(SUBSTRING_INDEX(#name,'\"',-2),-1),
activity=SUBSTRING_INDEX(SUBSTRING_INDEX(#name,'}',1),'/',1),
class=SUBSTRING_INDEX(SUBSTRING_INDEX(#name,'}',1),'/',-1),
user = '$user'" ;
I try to insert the data from a csv file into a mysql database using a powershell script. When using a (dummy) variable in the LOAD DATA query I run into troubles.
Reproducible example:
Create a Mysql database and table with
CREATE DATABASE loadfiletest;
USE loadfiletest;
CREATE TABLE testtable (field1 INT, field2 INT DEFAULT 0);
Create a csv file named loadfiletestdata.csv containing
1,3
2,4
Create the powershell script (don't forget to change the db password and possibly the username)
[system.reflection.assembly]::LoadWithPartialName("MySql.Data")
$mysqlConn = New-Object -TypeName MySql.Data.MySqlClient.MySqlConnection
$mysqlConn.ConnectionString = "SERVER=localhost;DATABASE=loadfiletest;UID=root;PWD=pwd"
$mysqlConn.Open()
$MysqlQuery = New-Object -TypeName MySql.Data.MySqlClient.MySqlCommand
$MysqlQuery.Connection = $mysqlConn
$MysqlQuery.CommandText = "LOAD DATA LOCAL INFILE 'C:/path/to/files/loadfiletestdata.csv' INTO TABLE loadfiletest.testtable FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '""' LINES TERMINATED BY '\r\n' (field1, field2)"
$MysqlQuery.ExecuteNonQuery()
Put everything in the folder C:/path/to/files/ (should also be your path in the powershell script) and run the script. This populates the table testtable with
field1 field2
1 3
2 4
as one would expect. This implies that quotes and such are like they should be. Each time the script is executed, those values are inserted in the table. Now, when I replace in the one but last line of the powershell script (field1, field2) by (field1, #dummy), I would expect that the values
field1 field2
1 0
2 0
are inserted into the table. However, I receive the error
Exception calling "ExecuteNonQuery" with "0" argument(s): "Fatal error encountered during command execution."
At C:\path\to\files\loadfiletest.ps1:8 char:1
+ $queryOutput = $MysqlQuery.ExecuteNonQuery()
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : MySqlException
When running the query with #dummy from a mysql client it works. Also the syntax looks the same to me as what can be found in the mysql manual (somewhere in the middle of the page, look for #dummy).
A few further experiment that I did, suggest that any LOAD DATA query containing a variable #whatever gives the error.
So the questions:
Why doesn't it work?
Is there a way to execute a LOAD DATA query with (dummy) variables from powershell?
If not, is there an elegant workaround?
Obvious workarounds are creating an intermediate csv file according to the layout of the table or creating an intermediate table matching the layout of the csv file. However that seems ugly and cumbersome for something that imho should "just work".
Note: The present question is a follow up and generalization of this question. I chose to start a new one since replacing the old content would make the answers already given obsolete and adding the content of this question would make the old question veeeeery long and full of useless sidetracks.
I know this is old, but I had the same problem and I found the solution here:
http://blog.tjitjing.com/index.php/2009/05/mysqldatamysqlclientmysqlexception-parameter-id-must-be-defined.html
Quoting from the above blog:
"Starting from version 5.2.2 of the Connector you should add the Allow User Variables=True Connection String Setting in order to use User Defined Variables in your SQL statements.
Example of Connection String:
Database=testdb;Data Source=localhost;User Id=root;Password=hello;Allow User Variables=True"
Thank you for down-voting my answer.
For my database class the teacher assigned us to use Oracle SQL to design an application. Because I have more experience with mySQL he said I could use it instead.
I want to make my assignment look as simliar to his example as possible. What his example consists of of is one file run.sql that looks like this:
#start //this runs start.sql which creates the tables
DESC table_name; //do this for all tables
#insert //this runs insert.sql that creates dummy data
SELECT * FROM table_name; //do this for all tables
#query //this runs query.sql that runs our sample queries
#drop //this kills all the data
Is there a way to do something simliar?
Namely a way to write a query that calls external queries and outputs all data to an output.txt file?
Use 'source' to input the *.sql files
use 'create procedure' to generate the 'drop' function
use "into outfile '/file/path';" on your select to write out.
double redirect to append: "into outfile '>>/file/path';"
The source command for the mysql command-line client could do the job here:
source start.sql;
DESC table_name;
You can get more commands with help.
I'm trying to run this query from a .Net application
LOAD DATA LOCAL INFILE 'testsFile.txt'
INTO TABLE Test
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(idTest, SampleID, Analyst, #Analysed, Device, Comments, #TotalRUL, #RULOne, #RULTwo, #RULThree, #RULFour, Uploaded)
SET
Analysed = nullif(#Analysed,''),
TotalRUL = nullif(#TotalRUL,''),
RULOne = nullif(#RULOne,''),
RULTwo = nullif(#RULTwo,''),
RULThree = nullif(#RULThree,''),
RULFour = nullif(#RULFour,'')
When I run this query from MySQL Workbench everything works fine, but when I use my .net application to run the query I get the following exception:
Parameter '#Analysed' must be defined.
I don't think I can use a declare statement outside of a stored procedure and I cant use a stored procedure due to my use of the LOAD DATA statement
What to do? Is this checkmate?
Sure you can't. If your query works with Workbench, this sounds like a .net bug.
I suggest you try "stupid" solutions like using backticks (after the # and after Analyzed... sorry, Stack Overflows autoformatting doesnt allow me to show you what I mean) or changing the variable's name.
How to use MySQL user-variables with ADO.NET
seems to have the answer to this
I needed to add "allow user variables" to the connection string
I have this slot:
void Managment::dbExportTriggered()
{
save = QFileDialog::getSaveFileName(this, trUtf8("Export db"),
QDir::currentPath() + "Backup/",
trUtf8("Dumped database (*.sql)"));
sqlQuery = "SELECT * INTO OUTFILE '" + save + ".sql' FROM Users, Data";
//QMessageBox::critical(0, trUtf8("query dump"), QString::number(query.exec(sqlQuery)));
query.exec(sqlQuery);
}
And I have this query:
sqlQuery = "SELECT * INTO OUTFILE " + save + " FROM Users, Data";
I execute normally but no dumped file appear, the backup directory has the right permission, the dumped database must be in client.
UPDATE:
After a search I found that the INTO OUTFILE query will dump database in the server not in the client as I was thought, so my question now how can I dump database in remote MySQL server, any quick methods with out any external tools like mysqldump client.
SELECT ... INTO OUTFILE creates a file on the MySQL server machine, with permissions matching whoever the MySQL server runs as. Unless you have root access on the MySQL server to retrieve the file that you're exporting, SELECT ... INTO OUTFILE is unlikely to do what you want.
In fact, I think I'd go so far as to say that if you're trying to use SELECT ... INTO OUTFILE from a GUI client, you're probably taking the wrong approach to your problem.
Just an idea: Another approach is to call mysqldump with QProcess. With some google-fu this seems to be an example:
..
if (allDatabases->isChecked()) {
arguments << "--all-databases";
} else {
arguments << "--databases";
foreach(QListWidgetItem *item, databasesList->selectedItems())
arguments << item->text();
}
proc->setReadChannel(QProcess::StandardOutput);
QApplication::setOverrideCursor(Qt::WaitCursor);
proc->start("mysqldump", arguments);
..
Thus, you can also add some parameters to dump only a specific table.
Edit:
Just note from the mysql doc on the SELECT ... INTO OUTFILE statement:
If you want to create the resulting
file on some other host than the
server host, you normally cannot use
SELECT ... INTO OUTFILE since there is
no way to write a path to the file
relative to the server host's file
system.
Thus you must roll your own, or you can use mysql -e as suggested by the above documentation.
Did you dump/print save to check it's valid? Does currentPath() return a trailung "/"?
Could there be difference between the path as seen by your client program and as (to be) seen by the server?
Does the user have the necessary privileges (file privilege for sure, maybe more)
Can't you get an error message from the log?
Are you getting any errors running the sql statement?
I notice that you're concatenating the filename into the SQL query without surrounding it by quotation marks. Your code will yield something like
SELECT * INTO OUTFILE /path/to/somewhere FROM Users, Data
but the MySQL documentation says it wants something like
SELECT * INTO OUTFILE '/path/to/somewhere' FROM Users, Data
Also keep the following in mind:
The file is created on the server host, so you must have the FILE privilege to use this syntax. file_name cannot be an existing file, which among other things prevents files such as /etc/passwd and database tables from being destroyed.
If you're looking on your client, you won't see the file there, even if the operation succeeds.