My file has data as below
abc|def|I completed by degreeSymbol®|210708
My load/import statement is below which is run in linux shell script. The Lang environment variable value is= en_US.UTF-8
load client from filename of del MODIFIED BY CHARDEL timestampformat="YYYYMMDD" coldel| usedefaults fastparse messages logfilename insert into tablename nonrecoverable;
In table the data is getting loaded as
abc def (null) 210708
And also when I run a select query the get the below error in the db2
com.ibm.db2.jcc.am.SqlException: Caught java.io.CharConversionException
Related
Environment: Windows 64bit, python 2.7.8 (32 bit), pyodbc v. 3.0.7
For what I need to do, I cannot use linkedservers, per internal policy.
Using python, I'm attempting to:
Export a table's data (*.dat) and its structure (format - *.fmt)
from one Sql Server (either 2008 or 2012) using bcp. The exported
files sit on my local machine. I make 2 BCP calls: one to get the
format (.fmt) file, another to get the data (.dat) (could not
find a way to do both in one step)
Import data of the given table into a database - MyDatabase - where I have full permission (per DBA's claim), and on either the same SQL server but different database, or on another server altogether. Here, my primary goal is to automate both the creation of the table to be imported (based on the exported fmt file), and the actual importing of its data.
I've got point 1 working, where I can dynamically specify the server, catalog, schema, and table to export, and python automagically creates the table.dat and table.fmt files on my local machine under a dedicated folder - DedicatedShareFolder
DedicatedShareFolder is a shared folder on my local machine that stores the exported tables and their fmt files. It is accessible to the SQL Servers I'm trying to import those tables into.
In point 2, I used python to build a SQL statement as follows:
sql = "select * into %s from
openrowset(BULK '\\\\%s\\sqltemp\\%s.dat',
FORMATFILE = '\\\\%s\\sqltemp\%s.fmt') as A" %(newTableName,os.environ['COMPUTERNAME'],
table,os.environ['COMPUTERNAME'],table)
Which ends up looking like:
select A.* into MyDatabase.dbo.blah48 from
openrowset(BULK '\\MyMachineName\DedicatedShareFolder\table.dat',
FORMATFILE = '\\MyMachineName\DedicatedShareFolder\table.fmt') as A;
I create a connection to the SQL server that has MyDatabase, and execute:
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=%s;DATABASE=%s;UID=%s;PWD=%s' % (server, catalog, login, pw))
cursor = cnxn.cursor()
rows = cursor.execute(sql).rowcount
print "Done importing %s rows" %rows
I get:
Done importing 606597 rows
Alas, the table was not created
I ran a trace on my local machine using 'ODBC Data Source Administration''s 'Tracing' tab. I opened the log file, and could not find any error pertaining to the creation of the table. I do see entries like this:
test_DB 5200-5a28 EXIT SQLDriverConnectW with return code 1 (SQL_SUCCESS_WITH_INFO)
HDBC 0x03C59190
HWND 0x00000000
WCHAR * 0x74C28B34 [ -3] "******\ 0"
SWORD -3
WCHAR * 0x74C28B34 <Invalid buffer length!> [-3]
SWORD -3
SWORD * 0x00000000
UWORD 0 <SQL_DRIVER_NOPROMPT>
DIAG [01000] [Microsoft][ODBC SQL Server Driver][SQL Server]Changed database context to 'mydatabase'. (5701)
DIAG [01000] [Microsoft][ODBC SQL Server Driver][SQL Server]Changed language setting to us_english. (5703)
DIAG [01S00] [Microsoft][ODBC SQL Server Driver]Invalid connection string attribute (0)
I suspect 'Invalid buffer length' has no effect here (couldn't find info on it in my search), and I do see driver changing context and language successfully.
I also see few entries in trace log like this:
test_DB 50c0-4fec EXIT SQLGetTypeInfo with return code -1 (SQL_ERROR)
HSTMT 0x03AC8E80
SWORD 12 <SQL_VARCHAR>
DIAG [24000] [Microsoft][ODBC Driver Manager] Invalid cursor state (0)
Nothing significant turned out in my search that would indicate 'invalid cursor state' has any effect here. Above entry could be due to other parts of code I'm still working on.
further down the trace log, I see:
test_DB 50c0-4fec ENTER SQLExecDirect
HSTMT 0x03ACB5B0
UCHAR * 0x025BBB94 [ -3] "select A.* into MyDatabase.dbo.blah48 from
openrowset(BULK '\\MyMachineName\DedicatedShareFolder\table.dat',
FORMATFILE = '\\MyMachineName\DedicatedShareFolder\table.fmt') as A\ 0"
SDWORD -3
test_DB 50c0-4fec EXIT SQLExecDirect with return code 0 (SQL_SUCCESS)
HSTMT 0x03ACB5B0
UCHAR * 0x025BBB94 [ -3] "select A.* into MyDatabase.dbo.blah48 from
openrowset(BULK '\\MyMachineName\DedicatedShareFolder\table.dat',
FORMATFILE = '\\MyMachineName\DedicatedShareFolder\table.fmt') as A\ 0"
SDWORD -3
test_DB 50c0-4fec ENTER SQLRowCount
HSTMT 0x03ACB5B0
SQLLEN * 0x0027EFD4
test_DB 50c0-4fec EXIT SQLRowCount with return code 0 (SQL_SUCCESS)
HSTMT 0x03ACB5B0
SQLLEN * 0x0027EFD4 (606597)
trace file indicates table was created. Alas it wasn't.
Out of desperation, I looked under every database on the server I'm testing on. No blah tables anywhere, even though MyDatabase is the only one I have with write permission
If I execute the same "select * into ... openrowset ... bulk ... " statement in Microsoft SQL Server Management Studio, it succeeds (logged in as the same user used in python script)
I use functions in the same script to perform many other SQL-related tasks, successfully. The import is the only thing not working.
I've also run every negative unit test I could think of to make sure no variable is getting changed midway. Nothing.
I'm a beginner in python. I'm either doing something gravely wrong in my code, or ?
if "select * into .. openrowset etc" type of statement cannot be used to achieve my goal, what other SLQ solution can I use to create a table and load its data, based on its BCP dat and fmt files?
Thanks.
Automatic commit of transactions is disabled in the Connection object by default. All of your requested work is actually being done, the transaction is just being rolled back when the connection is closed.
Couple options:
Commit your changes with either the Connection.commit() or Cursor.commit(). They are functionally the same, Cursor.commit() was added so it's not necessary to keep up with the Connection object.
Set the autocommit variable to True when the connection is created. Note that it is an argument for the pyodbc.connect() function, it's not part of the connection string. Setting autocommit with your coding style would be:
....
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=%s;DATABASE=%s;UID=%s;PWD=%s' % (server, catalog, login, pw),
autocommit=True)
....
I have a csv comma separated file containing hundreds of thousands of records in the following format:
3212790556,1,0.000000,,0
3212790557,2,0.000000,,0
Now using the SQL Server Import Flat file method works just dandy. I can edit the sql so that the table name and column names are something meaningful. Plus I also edit the data type from the default varchar(50) to int or decimal. This all works fine and sql import is able to import successfully.
However I am unable to do this same task using the Bulk Insert Query which is as follows:
BULK
INSERT temp1
FROM 'c:\filename.csv'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
GO
This query returns the following 3 errors which I have no idea how to resolve:
Msg 4866, Level 16, State 1, Line 1
The bulk load failed. The column is too long in the data file for row 1, column 5. Verify that the field terminator and row terminator are specified correctly.
Msg 7399, Level 16, State 1, Line 1
The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error.
Msg 7330, Level 16, State 2, Line 1
Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".
The purpose of my application is that there are multiple csv files in a folder that all need to go up in a single table so that I can query for sum of values. At the moment I was thinking of writing a program in C# that will execute the BULK insert in a loop (according for the number of files) and then return back with my results. I am guessing I dont need to write a code and that I can just write a script that does all of this - any one can guide me to the right path :)
Many thanks.
Edit: just added
ERRORFILE = 'C:\error.log'
to the query and I am getting 5221 rows inserted. Some times its 5222 some times its 5222 but it just fails beyond this point. Dont know whats the issue??? The CSV is perfectly fine.
SOB. WTF!!!
I cant believe that replacing \n with "0x0A" in the ROWTERMINATOR worked!!! I mean seriously. I just tried it and it worked. WTF moment!! Totally.
However what is a bit interesting is that the SQL Import wizard too only about 10 something seconds to import. The import query took well over a minute. Any guesses??
I've created an SSIS package that executes inline SQL queries from our database and is supposed to output the contents to a text file. I originally had the text file comma delimited, but changed to pipe delimted after researching the error further. I also did a substring of the FirstName field and ensure that the SSIS placeholder fields matched in length. The error message is as follows:
[Customers Flat File [196]] Error: Data conversion failed. The data conversion for
column "FirstName" returned status value 4 and status text "Text was truncated or one or more
characters had no match in the target code page.".
The SQL statement I'm using in my OLE DB Source is as follows:
SELECT
dbo.Customer.Email, SUBSTRING(dbo.Customer.FirstName, 1, 100) AS FirstName,
dbo.Customer.LastName, dbo.Customer.Gender,
dbo.Customer.DateOfBirth, dbo.Address.Zip, dbo.Customer.CustomerID, dbo.Customer.IsRegistered
FROM
dbo.Customer INNER JOIN
dbo.Address ON dbo.Customer.CustomerID = dbo.Address.CustomerID
What other fixes should I put in place to ensure the package runs without error?
Have you tried to run this query in SSMS? If so, did you get a successful result?
If you havent tried it yet, paste this query in a new SSMS window and wait for it to complete.
If the Query completes, then we dont have a problem with the query. Something could be off inside the package.
But if the query does not finish up and fails, you know where to look.
EDIT
On second thoughts, is your Customer source a flat file or something? It looks like there is a value in the Customer table/file which does not match with the output metadata of the source. Check your source again.
I am trying to insert a record into a table. I dont need to access a dat file, because all values are either constants or generated in the control file itself.
An example would be -
OPTIONS(LOAD=1)
LOAD DATA
APPEND
INTO TABLE table_name
(P_ID CONSTANT 202, NAME CONSTANT "ABC", NUM CONSTANT 1, CREATED_BY CONSTANT "DEF",
CREATION_DATE EXPRESSION "current_timestamp(6)")
However, when I execute the sqlldr command for the above ctl file, I am getting the following error -
SQL*Loader-500: Unable to open file (<file_name>.dat)
SQL*Loader-553: file not found
SQL*Loader-509: System error: No such file or directory
I tink you need to correct yor controlfile:
LOAD DATA
INFILE *
I want to store images and .docx/.doc, .pptx/.ppt, .pdf files using the front end of my software. I don't understand how to implement this and how to insert the BLOB and CLOB files into the table. Please help.
I am using Kubuntu 11.04, MySQL5, Qt 4.7.3.
Two ways:
1 - Use a LOAD_FILE function -
INSERT INTO table1 VALUES(1, LOAD_FILE('data.png'));
2 - Insert file as hex string, e.g. -
INSERT INTO table1 VALUES
(1, x'89504E470D0A1A0A0000000D494844520000001000000010080200000090916836000000017352474200AECE1CE90000000467414D410000B18F0BFC6105000000097048597300000EC300000EC301C76FA8640000001E49444154384F6350DAE843126220493550F1A80662426C349406472801006AC91F1040F796BD0000000049454E44AE426082');
INSERT INTO MY_TABLE(id, blob_col) VALUES(1, LOAD_FILE('/full/path/to/file/myfile.png')
LOAD_FILE has many conditions attached to it. From the MySQL documentation:
LOAD_FILE(file_name)
Reads the file and returns the file contents as a string. To use this
function, the file must be located on the server host, you must
specify the full path name to the file, and you must have the FILE
privilege. The file must be readable by all and its size less than
max_allowed_packet bytes. If the secure_file_priv system variable is
set to a nonempty directory name, the file to be loaded must be
located in that directory.
If the file does not exist or cannot be read because one of the
preceding conditions is not satisfied, the function returns NULL.
Also, there there are bugs with LOAD_FILE in Linux. See http://bugs.mysql.com/bug.php?id=38403 for the bug, and MySQL LOAD_FILE returning NULL for workarounds. On Ubuntu 12.04, MySQL 5.5.32, this works for me:
Copy file to /tmp
Change ownership to mysql user chown mysql:mysql /tmp/yourfile
Log into mysql as mysql root user so you are sure you have FILE privilege
Run your insert statement
Or you could merely use the MySQL Workbench, select the rows, last rows, insert a row without the blob, then just right click and select "Load Value From File".
INSERT INTO table1 VALUES(1, LOAD_FILE(data.png));
won't work but
INSERT INTO table1 VALUES(1, LOAD_FILE('data.png'));
should (assuming data.png exists in the local directory)
for those People who are getting "Column 'image' cannot be null" error while saving Blob through query :-
Open your MySql Command Line Client and login with root user and type
mysql> SHOW VARIABLES LIKE "secure_file_priv";
this will show you the secure path used by MySql to access the files. something like
+------------------+-----------------------+
| Variable_name | Value |
+------------------+-----------------------+
| secure_file_priv | /var/lib/mysql-files/ |
+------------------+-----------------------+
you can either paste files inside this folder or change the "secure_file_priv" variable value to "empty string" so that it can read file from anywhere.
If you are using mysql workbench, just right click on the field (cell) and select 'load value from file' option and then browse to the file and click open and then click on apply. It will automatically generate query like this
UPDATE `dbname`.`tablename` SET `columnname` = ? WHERE (`row` = '1');