I have a LOAD DATA mysql query im trying to run and need help to fix one thing.
Here is the query im running
$query = "LOAD DATA LOCAL INFILE '$file_app'
INTO TABLE tbl_user_tmp
LINES STARTING BY '{'
TERMINATED BY '/>'
(#name)
set
name=SUBSTRING_INDEX(#name,'\"',-2),
activity=SUBSTRING_INDEX(SUBSTRING_INDEX(#name,'}',1),'/',1),
class=SUBSTRING_INDEX(SUBSTRING_INDEX(#name,'}',1),'/',-1),
user = '$user'" ;
Here is the example of data from file im trying to load.
<item component="ComponentInfo{com.apps.aaa.roadside/com.apps.aaa.roadside.Splash}" drawable="aaa_roadside1" />
With the above query i get the following
name=aaa_roadside1"
activity=com.apps.aaa.roadside
class=com.apps.aaa.roadside.Splash
Everything is correct but name. I need to removed that last "
I thought the below query would work but it does not. any ideas?
This was working before when i had TERMINATED BY set to this '\" />'
However this will not account for entires that might not have that space at end like this
<item component="ComponentInfo{com.apps.aaa.roadside/com.apps.aaa.roadside.Splash}" drawable="aaa_roadside1"/>
I need to account for both
$query = "LOAD DATA LOCAL INFILE '$file_app'
INTO TABLE tbl_user_tmp
LINES STARTING BY '{'
TERMINATED BY '/>'
(#name)
set
name=SUBSTRING(SUBSTRING_INDEX(#name,'\"',-2),-1),
activity=SUBSTRING_INDEX(SUBSTRING_INDEX(#name,'}',1),'/',1),
class=SUBSTRING_INDEX(SUBSTRING_INDEX(#name,'}',1),'/',-1),
user = '$user'" ;
Related
I use parametric queries for normal insert/updates for security.
How do I do that for queries like this:
LOAD DATA INFILE '/filepath' INTO TABLE mytable
In my case, the path to the file would be different everytime (for different requests). Is it fine to proceed like this (since I am not getting any data from outside, the file is from the server itself):
path = /filepath
"LOAD DATA INFILE" + path + "INTO TABLE mytable"
Since LOAD DATA is not listed in SQL Syntax Allowed in Prepared Statements you can't prepare something like
LOAD DATA INFILE ? INTO TABLE mytable
But SET is listed. So a workaround could be to prepare and execute
SET #filepath = ?
And then execute
LOAD DATA INFILE #filepath INTO TABLE mytable
Update
In Python with MySQLdb the following query should work
LOAD DATA INFILE %s INTO TABLE mytable
since no prepared statement is used.
To answer your "is it fine to proceed like this" question, your example code will fail because the resulting query will be missing quotes around the filename. If you changed it to the following it could run, but is still a bad idea IMO:
path = "/filepath"
sql = "LOAD DATA INFILE '" + path + "' INTO TABLE mytable" # note the single quotes
While you may not be accepting outside input today, code has a way of sticking around and getting reused/copied, so you should use the API in a way that will escape your parameters:
sql = "LOAD DATA INFILE %s INTO TABLE mytable"
cursor.execute(sql, (path,))
And don't forget to commit if autocommit is not enabled.
I try to insert the data from a csv file into a mysql database using a powershell script. When using a (dummy) variable in the LOAD DATA query I run into troubles.
Reproducible example:
Create a Mysql database and table with
CREATE DATABASE loadfiletest;
USE loadfiletest;
CREATE TABLE testtable (field1 INT, field2 INT DEFAULT 0);
Create a csv file named loadfiletestdata.csv containing
1,3
2,4
Create the powershell script (don't forget to change the db password and possibly the username)
[system.reflection.assembly]::LoadWithPartialName("MySql.Data")
$mysqlConn = New-Object -TypeName MySql.Data.MySqlClient.MySqlConnection
$mysqlConn.ConnectionString = "SERVER=localhost;DATABASE=loadfiletest;UID=root;PWD=pwd"
$mysqlConn.Open()
$MysqlQuery = New-Object -TypeName MySql.Data.MySqlClient.MySqlCommand
$MysqlQuery.Connection = $mysqlConn
$MysqlQuery.CommandText = "LOAD DATA LOCAL INFILE 'C:/path/to/files/loadfiletestdata.csv' INTO TABLE loadfiletest.testtable FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '""' LINES TERMINATED BY '\r\n' (field1, field2)"
$MysqlQuery.ExecuteNonQuery()
Put everything in the folder C:/path/to/files/ (should also be your path in the powershell script) and run the script. This populates the table testtable with
field1 field2
1 3
2 4
as one would expect. This implies that quotes and such are like they should be. Each time the script is executed, those values are inserted in the table. Now, when I replace in the one but last line of the powershell script (field1, field2) by (field1, #dummy), I would expect that the values
field1 field2
1 0
2 0
are inserted into the table. However, I receive the error
Exception calling "ExecuteNonQuery" with "0" argument(s): "Fatal error encountered during command execution."
At C:\path\to\files\loadfiletest.ps1:8 char:1
+ $queryOutput = $MysqlQuery.ExecuteNonQuery()
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : MySqlException
When running the query with #dummy from a mysql client it works. Also the syntax looks the same to me as what can be found in the mysql manual (somewhere in the middle of the page, look for #dummy).
A few further experiment that I did, suggest that any LOAD DATA query containing a variable #whatever gives the error.
So the questions:
Why doesn't it work?
Is there a way to execute a LOAD DATA query with (dummy) variables from powershell?
If not, is there an elegant workaround?
Obvious workarounds are creating an intermediate csv file according to the layout of the table or creating an intermediate table matching the layout of the csv file. However that seems ugly and cumbersome for something that imho should "just work".
Note: The present question is a follow up and generalization of this question. I chose to start a new one since replacing the old content would make the answers already given obsolete and adding the content of this question would make the old question veeeeery long and full of useless sidetracks.
I know this is old, but I had the same problem and I found the solution here:
http://blog.tjitjing.com/index.php/2009/05/mysqldatamysqlclientmysqlexception-parameter-id-must-be-defined.html
Quoting from the above blog:
"Starting from version 5.2.2 of the Connector you should add the Allow User Variables=True Connection String Setting in order to use User Defined Variables in your SQL statements.
Example of Connection String:
Database=testdb;Data Source=localhost;User Id=root;Password=hello;Allow User Variables=True"
Thank you for down-voting my answer.
I'm trying to run this query from a .Net application
LOAD DATA LOCAL INFILE 'testsFile.txt'
INTO TABLE Test
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(idTest, SampleID, Analyst, #Analysed, Device, Comments, #TotalRUL, #RULOne, #RULTwo, #RULThree, #RULFour, Uploaded)
SET
Analysed = nullif(#Analysed,''),
TotalRUL = nullif(#TotalRUL,''),
RULOne = nullif(#RULOne,''),
RULTwo = nullif(#RULTwo,''),
RULThree = nullif(#RULThree,''),
RULFour = nullif(#RULFour,'')
When I run this query from MySQL Workbench everything works fine, but when I use my .net application to run the query I get the following exception:
Parameter '#Analysed' must be defined.
I don't think I can use a declare statement outside of a stored procedure and I cant use a stored procedure due to my use of the LOAD DATA statement
What to do? Is this checkmate?
Sure you can't. If your query works with Workbench, this sounds like a .net bug.
I suggest you try "stupid" solutions like using backticks (after the # and after Analyzed... sorry, Stack Overflows autoformatting doesnt allow me to show you what I mean) or changing the variable's name.
How to use MySQL user-variables with ADO.NET
seems to have the answer to this
I needed to add "allow user variables" to the connection string
I'm having an issue with an app I'm working on.
The app allows a user to upload a CSV file which gets processed by the app and in turn creates records into a number of tables. In order to improve performance, for one of the tables, it produces a new CSV file in order to use the mysql LOAD DATA INFILE functionality.
Instead, it seems to be increasing the time it takes to process.
I'm pushing all processing into the background using sidekiq. It seems to be creating the CSV without any problems, however when I execute the load data query it just sits there and I have no idea what it's doing.
My processing function does the following :
CSV.open(output_path, 'w+', { force_quotes: true }) do |writer|
writer << headers
while rows.count > 0
....
data_sets.each do |ds|
writer << [UUIDTools::UUID.random_create, resp, row[set], ds.id, now, now]
set += 1
end
resp += 1
end
end
sql = "LOAD DATA LOCAL INFILE '#{output_path}'
INTO TABLE data_set_responses
FIELDS TERMINATED BY ',' ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
(id, response_number, response, data_set_id, created_at, updated_at)"
con = ActiveRecord::Base.connection
con.execute("SET autocommit = 0;")
con.execute("SET unique_checks = 0;")
con.execute("SET foreign_key_checks = 0;")
con.execute("LOCK TABLES data_set_responses WRITE;")
con.execute(sql)
con.execute("UNLOCK TABLES;")
con.execute("COMMIT;")
con.execute("SET autocommit = 1;")
con.execute("SET unique_checks = 1;")
con.execute("SET foreign_key_checks = 1;")
As of right now, my sidekiq process has been running for 22 minutes and still hasn't finished. It should be inserting around 700k rows which shouldn't be taking anywhere near this long!
The table I'm inserting into has a binary field for it's primary key (uuid) so I don't know if that's slowing it down?
Any ideas?
I ended up changing my data structure to one that didn't require the vast number of rows that this structure did. I've got it down to a matter of seconds :)
I am relatively new to Perl.I have been trying to insert data to database from a text file using a CGI script.I have written code for it and it's working properly.but when i try to impose a limit on the data that is inserted using LIMIT keyword there is a problem.Please check where i am going wrong and what needs to be amended.Thanks for your advice.
here is the code
#!/usr/bin/perl
use warnings;
use strict;
use CGI ':standard';
use DBI;
if(param())
{
my #params=param();
my $limit=param('limit')||'';
my $dbh =DBI->connect("DBI:mysql:sample","root","");
my $var="LOAD DATA LOCAL INFILE 'C:/test.txt' INTO TABLE sample2 FIELDS TERMINATED BY '\n' WHERE LIMIT 0,$limit";
my $sth = $dbh->do($var) or die "prepare failed: " . $dbh->errstr();
print $sth ."Records inserted";
$sth->finish();
$dbh->disconnect();
print
header(),
start_html(
-title=>'Welcome',
-text=>'#520063'
),
#h1("Records have been displayed"),
end_html();
}
else
{
print
header(),
start_html('A Simple Form'),
h1('Please enter the limit '),
start_form(),
'Limit: ',
textfield(-name=>'limit'),
br(),
#'Phone Number: ',
#textfield(-name => 'number'),
#br(),
submit(),
end_form(),
end_html();
}
Without an error message I can't be totally sure, but the problem is likely in your SQL.
LOAD DATA LOCAL INFILE 'C:/test.txt'
INTO TABLE sample2
FIELDS TERMINATED BY '\n'
WHERE LIMIT 0,$limit
There's problems.
There WHERE clause is empty (LIMIT is not part of the WHERE clause)
LOAD DATA INFILE does not take a WHERE clause
LOAD DATA INFILE does not take a LIMIT clause
Basically that whole last line won't work. In general, if a SQL command doesn't work in a program try firing up the MySQL command line and debug the command there.
LOAD DATA INFILE does not have a way to limit how many lines it will pull in, this is an oft requested feature. To work around it I would suggest copying and truncating the file before feeding it to MySQL.