I'm trying to run this query from a .Net application
LOAD DATA LOCAL INFILE 'testsFile.txt'
INTO TABLE Test
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(idTest, SampleID, Analyst, #Analysed, Device, Comments, #TotalRUL, #RULOne, #RULTwo, #RULThree, #RULFour, Uploaded)
SET
Analysed = nullif(#Analysed,''),
TotalRUL = nullif(#TotalRUL,''),
RULOne = nullif(#RULOne,''),
RULTwo = nullif(#RULTwo,''),
RULThree = nullif(#RULThree,''),
RULFour = nullif(#RULFour,'')
When I run this query from MySQL Workbench everything works fine, but when I use my .net application to run the query I get the following exception:
Parameter '#Analysed' must be defined.
I don't think I can use a declare statement outside of a stored procedure and I cant use a stored procedure due to my use of the LOAD DATA statement
What to do? Is this checkmate?
Sure you can't. If your query works with Workbench, this sounds like a .net bug.
I suggest you try "stupid" solutions like using backticks (after the # and after Analyzed... sorry, Stack Overflows autoformatting doesnt allow me to show you what I mean) or changing the variable's name.
How to use MySQL user-variables with ADO.NET
seems to have the answer to this
I needed to add "allow user variables" to the connection string
Related
I use parametric queries for normal insert/updates for security.
How do I do that for queries like this:
LOAD DATA INFILE '/filepath' INTO TABLE mytable
In my case, the path to the file would be different everytime (for different requests). Is it fine to proceed like this (since I am not getting any data from outside, the file is from the server itself):
path = /filepath
"LOAD DATA INFILE" + path + "INTO TABLE mytable"
Since LOAD DATA is not listed in SQL Syntax Allowed in Prepared Statements you can't prepare something like
LOAD DATA INFILE ? INTO TABLE mytable
But SET is listed. So a workaround could be to prepare and execute
SET #filepath = ?
And then execute
LOAD DATA INFILE #filepath INTO TABLE mytable
Update
In Python with MySQLdb the following query should work
LOAD DATA INFILE %s INTO TABLE mytable
since no prepared statement is used.
To answer your "is it fine to proceed like this" question, your example code will fail because the resulting query will be missing quotes around the filename. If you changed it to the following it could run, but is still a bad idea IMO:
path = "/filepath"
sql = "LOAD DATA INFILE '" + path + "' INTO TABLE mytable" # note the single quotes
While you may not be accepting outside input today, code has a way of sticking around and getting reused/copied, so you should use the API in a way that will escape your parameters:
sql = "LOAD DATA INFILE %s INTO TABLE mytable"
cursor.execute(sql, (path,))
And don't forget to commit if autocommit is not enabled.
I'm trying to build a Data Integration job uses pass through to extract data from a view in a MySQL database.
Wev'e been using pass through a lot in the project, mostly extracting data from Redshift,
however with MySQL I was not able to do make it work properly.
It keeps complaining a table is missing even though when pass through is off, view is found and data is extracted...
tried every trick I know, starting from enabling case-sensitive DBMS object names, to manually remove single/double quotes from the statement just in case MySQL confuses confuses it with something else...
No luck.
ODBC driver is [MySQL][ODBC 5.3(a) Driver][mysqld-5.5.53].
Ran on a Windows environment.
Any idea how to solve this?
Thank you in advance.
EDIT
So, first of all, one correction (even though not that important - I extract from a view, not a table).
This is the code generated by SAS Create Table transformation, pass through enabled. I only put an asterisk instead of the full list of columns:
proc sql;
connect to ODBC
(
READBUFF=10000 DATASRC="cmp.web_api" AUTHDOMAIN="MYSQL_CMP_Auth"
);
create table work."W7ZZZKOC"n as
select
*
from connection to ODBC
(
select
V_BI_ACCOUNT.ACCOUNT_NAME,
V_BI_ACCOUNT.ACQUISITION_SOURCE__C,
V_BI_ACCOUNT.ZUORA__ACTIVE__C,
V_BI_ACCOUNT.ADDRESS_LINE_1__C,
V_BI_ACCOUNT.ADDRESS_LINE_2__C,
V_BI_ACCOUNT.ADDRESS_LINE_3__C,
V_BI_ACCOUNT.AGREEMENT_DATE,
V_BI_ACCOUNT.AGREEMENT_LEGAL_CLAUSE_1__C,
V_BI_ACCOUNT.AGREEMENT_LEGAL_CLAUSE_2__C,
V_BI_ACCOUNT.PERSONBIRTHDATE,
V_BI_ACCOUNT.BLOCKED_REASON__C,
V_BI_ACCOUNT.BRAND__C,
V_BI_ACCOUNT.CPN__C,
V_BI_ACCOUNT.ACCCREATEDBYID,
V_BI_ACCOUNT.ACCCREATEDDATE,
V_BI_ACCOUNT.CURRENCY_PREFERENCE__C,
V_BI_ACCOUNT.CUSTOMER_FULL_NAME__PC,
V_BI_ACCOUNT.ACCOUNTID,
V_BI_ACCOUNT.ZUORA__CUSTOMERPRIORITY__C,
V_BI_ACCOUNT.DELIVERY_SALUTATION__C,
V_BI_ACCOUNT.DISPLAY_NAME,
V_BI_ACCOUNT.PERSONEMAIL,
V_BI_ACCOUNT.EMAILKEY__C,
V_BI_ACCOUNT.FACEBOOKKEY,
V_BI_ACCOUNT.FIRSTNAME,
V_BI_ACCOUNT.GENDER__C,
V_BI_ACCOUNT.PHONE,
V_BI_ACCOUNT.ACCLASTACTIVITYDATE,
V_BI_ACCOUNT.ACCLASTMODIFIEDDATE,
V_BI_ACCOUNT.LASTNAME,
V_BI_ACCOUNT.OTHER_EMAIL__C,
V_BI_ACCOUNT.PI_TYPE__C,
V_BI_ACCOUNT.ACCPARENTID,
V_BI_ACCOUNT.POSTCODE__C,
V_BI_ACCOUNT.PRIMARY_ACCOUNT_OF_THIS_CUSTOMER,
V_BI_ACCOUNT.ACCPRIMARY__C,
V_BI_ACCOUNT.ACCREASON_FOR_STATUS__C,
V_BI_ACCOUNT.ZUORA__SLA__C,
V_BI_ACCOUNT.ZUORA__SLASERIALNUMBER__C,
V_BI_ACCOUNT.SALUTATION,
V_BI_ACCOUNT.ACCSYSTEMMODSTAMP,
V_BI_ACCOUNT.PERSONTITLE,
V_BI_ACCOUNT.ZUORA__UPSELLOPPORTUNITY__C,
V_BI_ACCOUNT.X_CODE__C,
V_BI_ACCOUNT.ZUORA__ACCOUNT_ID__C,
V_BI_ACCOUNT.ZUORA__PAYMENTMETHODID__C,
V_BI_ACCOUNT.CITY,
V_BI_ACCOUNT.ORIGINAL_CREATED_DATE,
V_BI_ACCOUNT.SOURCE_SYSTEM_ID,
V_BI_ACCOUNT.STATUS,
V_BI_ACCOUNT.ZUORA__CONTACT_ID,
V_BI_ACCOUNT.ACCISDELETED,
V_BI_ACCOUNT.BILLING_ACCOUNT_NAME,
V_BI_ACCOUNT.ACZCREATEDDATE,
V_BI_ACCOUNT.ACZSYSTEMMODSTAMP,
V_BI_ACCOUNT.ACZLASTACTIVITYDATE,
V_BI_ACCOUNT.ZUORA__ACCOUNT__C,
V_BI_ACCOUNT.ZUORA__ACCOUNTNUMBER__C,
V_BI_ACCOUNT.ZUORA__AUTOPAY__C,
V_BI_ACCOUNT.ZUORA__BALANCE__C,
V_BI_ACCOUNT.ZUORA__CREDITCARDEXPIRATION__C,
V_BI_ACCOUNT.ZUORA__CURRENCY__C,
V_BI_ACCOUNT.ZUORA__MRR__C,
V_BI_ACCOUNT.ZUORA__PAYMENTTERM__C,
V_BI_ACCOUNT.ZUORA__PURCHASEORDERNUMBER__C,
V_BI_ACCOUNT.ZUORA__LASTINVOICEDATE__C,
V_BI_ACCOUNT.COUNTRY_NAME,
V_BI_ACCOUNT.COUNTRY_CODE,
V_BI_ACCOUNT.FAVOURITE_FOOTBALL_CLUB,
V_BI_ACCOUNT.COUNTY
from
web_api.V_BI_ACCOUNT as V_BI_ACCOUNT
);
%rcSet(&sqlrc);
disconnect from ODBC;
quit;
And again, when I extract data without pass through - works successfully,
I found out the problem was a column name exceeds 32 positions.
As SAS supports up column names up to 32,
the query fails to find PRIMARY_ACCOUNT_OF_THIS_CUSTOMER as the original column name is PRIMARY_ACCOUNT_OF_THIS_CUSTOMER__C.
EDIT
One more thing I found out is, MySQL doesn't like specifying schema name nor aliases.
Therefore,
From clause to only specify table name i.e : 'from v_bi_account' rather than 'web_api.v_bi_account'
and do not use aliases i.e use 'from v_bi_account' rather than 'from v_bi_account as v_bi_account'
Thank you guys so much for your help.
I have a LOAD DATA mysql query im trying to run and need help to fix one thing.
Here is the query im running
$query = "LOAD DATA LOCAL INFILE '$file_app'
INTO TABLE tbl_user_tmp
LINES STARTING BY '{'
TERMINATED BY '/>'
(#name)
set
name=SUBSTRING_INDEX(#name,'\"',-2),
activity=SUBSTRING_INDEX(SUBSTRING_INDEX(#name,'}',1),'/',1),
class=SUBSTRING_INDEX(SUBSTRING_INDEX(#name,'}',1),'/',-1),
user = '$user'" ;
Here is the example of data from file im trying to load.
<item component="ComponentInfo{com.apps.aaa.roadside/com.apps.aaa.roadside.Splash}" drawable="aaa_roadside1" />
With the above query i get the following
name=aaa_roadside1"
activity=com.apps.aaa.roadside
class=com.apps.aaa.roadside.Splash
Everything is correct but name. I need to removed that last "
I thought the below query would work but it does not. any ideas?
This was working before when i had TERMINATED BY set to this '\" />'
However this will not account for entires that might not have that space at end like this
<item component="ComponentInfo{com.apps.aaa.roadside/com.apps.aaa.roadside.Splash}" drawable="aaa_roadside1"/>
I need to account for both
$query = "LOAD DATA LOCAL INFILE '$file_app'
INTO TABLE tbl_user_tmp
LINES STARTING BY '{'
TERMINATED BY '/>'
(#name)
set
name=SUBSTRING(SUBSTRING_INDEX(#name,'\"',-2),-1),
activity=SUBSTRING_INDEX(SUBSTRING_INDEX(#name,'}',1),'/',1),
class=SUBSTRING_INDEX(SUBSTRING_INDEX(#name,'}',1),'/',-1),
user = '$user'" ;
I try to insert the data from a csv file into a mysql database using a powershell script. When using a (dummy) variable in the LOAD DATA query I run into troubles.
Reproducible example:
Create a Mysql database and table with
CREATE DATABASE loadfiletest;
USE loadfiletest;
CREATE TABLE testtable (field1 INT, field2 INT DEFAULT 0);
Create a csv file named loadfiletestdata.csv containing
1,3
2,4
Create the powershell script (don't forget to change the db password and possibly the username)
[system.reflection.assembly]::LoadWithPartialName("MySql.Data")
$mysqlConn = New-Object -TypeName MySql.Data.MySqlClient.MySqlConnection
$mysqlConn.ConnectionString = "SERVER=localhost;DATABASE=loadfiletest;UID=root;PWD=pwd"
$mysqlConn.Open()
$MysqlQuery = New-Object -TypeName MySql.Data.MySqlClient.MySqlCommand
$MysqlQuery.Connection = $mysqlConn
$MysqlQuery.CommandText = "LOAD DATA LOCAL INFILE 'C:/path/to/files/loadfiletestdata.csv' INTO TABLE loadfiletest.testtable FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '""' LINES TERMINATED BY '\r\n' (field1, field2)"
$MysqlQuery.ExecuteNonQuery()
Put everything in the folder C:/path/to/files/ (should also be your path in the powershell script) and run the script. This populates the table testtable with
field1 field2
1 3
2 4
as one would expect. This implies that quotes and such are like they should be. Each time the script is executed, those values are inserted in the table. Now, when I replace in the one but last line of the powershell script (field1, field2) by (field1, #dummy), I would expect that the values
field1 field2
1 0
2 0
are inserted into the table. However, I receive the error
Exception calling "ExecuteNonQuery" with "0" argument(s): "Fatal error encountered during command execution."
At C:\path\to\files\loadfiletest.ps1:8 char:1
+ $queryOutput = $MysqlQuery.ExecuteNonQuery()
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : MySqlException
When running the query with #dummy from a mysql client it works. Also the syntax looks the same to me as what can be found in the mysql manual (somewhere in the middle of the page, look for #dummy).
A few further experiment that I did, suggest that any LOAD DATA query containing a variable #whatever gives the error.
So the questions:
Why doesn't it work?
Is there a way to execute a LOAD DATA query with (dummy) variables from powershell?
If not, is there an elegant workaround?
Obvious workarounds are creating an intermediate csv file according to the layout of the table or creating an intermediate table matching the layout of the csv file. However that seems ugly and cumbersome for something that imho should "just work".
Note: The present question is a follow up and generalization of this question. I chose to start a new one since replacing the old content would make the answers already given obsolete and adding the content of this question would make the old question veeeeery long and full of useless sidetracks.
I know this is old, but I had the same problem and I found the solution here:
http://blog.tjitjing.com/index.php/2009/05/mysqldatamysqlclientmysqlexception-parameter-id-must-be-defined.html
Quoting from the above blog:
"Starting from version 5.2.2 of the Connector you should add the Allow User Variables=True Connection String Setting in order to use User Defined Variables in your SQL statements.
Example of Connection String:
Database=testdb;Data Source=localhost;User Id=root;Password=hello;Allow User Variables=True"
Thank you for down-voting my answer.
I am using the following query to load into mysql:
LOAD DATA INFILE '/var/www/vhosts/httpdocs/xml/insert_feeds.csv'
INTO TABLE `deals_items`
FIELDS TERMINATED BY '###!##'
OPTIONALLY ENCLOSED BY ''
ESCAPED BY ''
LINES TERMINATED BY '###%##'
IGNORE 1 LINES
(#id,dealID,shopid,categoryid,title,permalink,url,startDate,endDate,description,extract,price,previous_price,discount,purchases,image,#location,#lat,#lng,locationText,type,settings,active)
SET id = '', lat=#lat, lng=#lng, locationText=#location, location = GeomFromText(CONCAT(POINT(#lat,#lng)))
Now this used to work just fine, but ever since an mysql was upgraded from 5.0 to 5.1 it stopped working. It now works only if I add the LOCAL to the statement.
The error I'm getting is Can't get stat of /var/www/vhosts/httpdocs/xml/insert_feeds.csv
The user was granted with full permissions to test it and the file was given 0777.
It won't work from any client (mysql,mysqli,pdo and console). Adding the local solves all problems but there is a lot of queries and I cannot go on and change them all. Further more, as I understand from the manual there are security issues with the LOCAL command.
The exact mysql version is 5.1.58-1~dotdeb.0 and the client version 5.0.51a