BULK INSERT large CSV file and attach an additional column - sql-server-2008

I was able to use BULK INSERT on an SQL Server 2008 R2 database to import a CSV file (Tab delimited) with more than 2 million rows. This command is planned to run every week.
I added an additional column named "lastupdateddate" to the generated table to store the datestamp a row is updated via a INSERT trigger. But when I ran the BULK INSERT again, it failed due to mismatch in columns as there is no such a field in a raw CSV file.
Is there any possibility to configure BULK INSERT to ignore the "lastupdateddate" column when it runs?
Thanks.
-- EDIT:
I tried using a format file but still unable to solve the problem.
The table looks as below.
USE AdventureWorks2008R2;
GO
CREATE TABLE AAA_Test_Table
(
Col1 smallint,
Col2 nvarchar(50) ,
Col3 nvarchar(50) ,
LastUpdatedDate datetime
);
GO
The csv "data.txt" file is:
1,DataField2,DataField3
2,DataField2,DataField3
3,DataField2,DataField3
The format file is like:
10.0
3
1 SQLCHAR 0 7 "," 1 Col1 ""
2 SQLCHAR 0 100 "," 2 Col2 SQL_Latin1_General_CP1_CI_AS
3 SQLCHAR 0 100 "," 3 Col3 SQL_Latin1_General_CP1_CI_AS
The SQL command I ran is:
DELETE AAA_Test_Table
BULK INSERT AAA_Test_Table
FROM 'C:\Windows\Temp\TestFormatFile\data.txt'
WITH (formatfile='C:\Windows\Temp\TestFormatFile\formatfile.fmt');
GO
The error received is:
Msg 4864, Level 16, State 1, Line 2
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 2, column 1 (Col1).
Msg 4832, Level 16, State 1, Line 2
Bulk load: An unexpected end of file was encountered in the data file.
Msg 7399, Level 16, State 1, Line 2
The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error.
Msg 7330, Level 16, State 2, Line 2
Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".

Yes you can using a format file as documented Here and use that format with bcp command with -f option like -f format_file_name.fmt.
Well another option would be; import all the data (I mean all fields) and then drop the non wanted column lastupdateddate using SQL like
ALTER TABLE your_bulk_insert_table DROP COLUMN lastupdateddate

Related

sqlldr generating ORA-00984: column not allowed here - while trying to add constant text

I need to load several similar csv files into work tables with the same format for onward processing but for some of the data I get 'ORA-00984: column not allowed here' errors.
I can't change the layout of the csv but the ordering of the columns in the work table and the format of the sqlldr control file are in my control.
What do I need to change to get sqlldr to load this data?
EDIT: Solution: The following change to the .ctl file:
col6_fixedchar constant "abc", fixes the issue, interestingly sqlldr is quite happy with interpreting "3600" as a number.
Below is a sample:
table:
create table test_sqlldr
(
col1_date date,
col2_char varchar2(15),
col3_int number(5),
col4_int number(5),
col5_int number(5),
-- fixed and dummy fields
col6_fixedchar varchar2(15),
col7_nullchar varchar2(20),
col8_fixedint number(5)
);
csv:
cat /tmp/test_sqlldr.csv
2019-08-27 09:00:00,abcdefghi,3600,0,0
2019-08-27 09:00:00,jklmnopqr,3600,0,0
2019-08-27 09:00:00,stuvwxyza,3600,3598,3598
2019-08-27 09:00:00,bcdefghij,3600,0,0
ctl:
cat /tmp/test_sqlldr.ctl
load data infile '/tmp/test_sqlldr.csv'
insert into table test_sqlldr
fields terminated by ',' optionally enclosed by '"' TRAILING NULLCOLS
(
col1_date timestamp 'yyyy-mm-dd hh24:mi:ss',
col2_char,
col3_int,
col4_int,
col5_int,
col6_fixedchar "abc",
col8_fixedint "3600"
)
This generates the following output:
/opt/oracle/product/112020_cl_64/cl/bin/sqlldr <db credentials> control='/tmp/test_sqlldr.ctl' ; cat test_sqlldr.log
SQL*Loader: Release 12.2.0.1.0 - Production on Wed Aug 28 10:26:00 2019
Copyright (c) 1982, 2017, Oracle and/or its affiliates. All rights reserved.
Path used: Conventional
Commit point reached - logical record count 4
Table TEST_SQLLDR:
0 Rows successfully loaded.
Check the log file:
test_sqlldr.log
for more information about the load.
SQL*Loader: Release 12.2.0.1.0 - Production on Wed Aug 28 10:26:00 2019
Copyright (c) 1982, 2017, Oracle and/or its affiliates. All rights reserved.
Control File: /tmp/test_sqlldr.ctl
Data File: /tmp/test_sqlldr.csv
Bad File: /tmp/test_sqlldr.bad
Discard File: none specified
(Allow all discards)
Number to load: ALL
Number to skip: 0
Errors allowed: 50
Bind array: 64 rows, maximum of 256000 bytes
Continuation: none specified
Path used: Conventional
Table TEST_SQLLDR, loaded from every logical record.
Insert option in effect for this table: INSERT
TRAILING NULLCOLS option in effect
Column Name Position Len Term Encl Datatype
------------------------------ ---------- ----- ---- ---- ---------------------
COL1_DATE FIRST * , O(") DATETIME yyyy-mm-dd hh24:mi:ss
COL2_CHAR NEXT * , O(") CHARACTER
COL3_INT NEXT * , O(") CHARACTER
COL4_INT NEXT * , O(") CHARACTER
COL5_INT NEXT * , O(") CHARACTER
COL6_FIXEDCHAR NEXT * , O(") CHARACTER
SQL string for column : "abc"
COL8_FIXEDINT NEXT * , O(") CHARACTER
SQL string for column : "3600"
Record 1: Rejected - Error on table TEST_SQLLDR, column COL4_INT.
ORA-00984: column not allowed here
Record 2: Rejected - Error on table TEST_SQLLDR, column COL4_INT.
ORA-00984: column not allowed here
Record 3: Rejected - Error on table TEST_SQLLDR, column COL4_INT.
ORA-00984: column not allowed here
Record 4: Rejected - Error on table TEST_SQLLDR, column COL4_INT.
ORA-00984: column not allowed here
Table TEST_SQLLDR:
0 Rows successfully loaded.
4 Rows not loaded due to data errors.
0 Rows not loaded because all WHEN clauses were failed.
0 Rows not loaded because all fields were null.
Space allocated for bind array: 115584 bytes(64 rows)
Read buffer bytes: 1048576
Total logical records skipped: 0
Total logical records read: 4
Total logical records rejected: 4
Total logical records discarded: 0
Run began on Wed Aug 28 10:26:00 2019
Run ended on Wed Aug 28 10:26:00 2019
Elapsed time was: 00:00:00.14
CPU time was: 00:00:00.03
Try: col6_fixedchar CONSTANT "abc"

insert and fetch strings and matrices to/from MySQL with Matlab

I need to store data in a database. I have installed and configured a MySQL database (and an SQLite database) in Matlab. However I cannot store and retrieve anything other than scalar numeric values.
% create an empty database called test_data base with MySQL workbench.
% connect to it in Matlab
conn=database('test_database','root','XXXXXX','Vendor','MySQL');
% create a table to store values
create_test_table=['CREATE TABLE test_table (testID NUMERIC PRIMARY KEY, test_string VARCHAR(255), test_vector BLOB, test_scalar NUMERIC)'];
curs=exec(conn,create_test_table)
Result is good so far (curs.Message is an empty string)
% create a new record
datainsert(conn,'test_table',{'testID','test_string','test_vector','test_scalar'},{1,'string1',[1,2],1})
% try to read out the new record
sqlquery='SELECT * FROM test_table8';
data_to_view=fetch(conn,sqlquery)
Result is bad:
data_to_view =
1 NaN NaN 1
From the documentation for "fetch" I would expect:
data_to_view =
1×4 table
testID test_string test_vector test_scalar
_____________ ___________ ______________ ________
1 'string1' 1x2 double 1
Until I learn how to read blobs I'd even be willing to accept:
data_to_view =
1×4 table
testID test_string test_vector test_scalar
_____________ ___________ ______________ ________
1 'string1' NaN 1
I get the same thing with an sqlite database. How can I store and then read out strings and blobs and why isn't the data returned in table format?
Matlab does not document that the default options for SQLite and MySQL database retrieval are to attempt to return everything as a numeric array. One only needs this line:
setdbprefs('DataReturnFormat','cellarray')
or
setdbprefs('DataReturnFormat','table')
in order to get results with differing datatypes. However! now my result is:
data_to_view =
1×4 cell array
{[2]} {'string1'} {11×1 int8} {[1]}
If instead I input:
datainsert(conn,'test_table',{'testID','test_string','test_vector','test_scalar'},{1,'string1',typecast([1,2],'int8'),1})
Then I get:
data_to_view =
1×4 cell array
{[2]} {'string1'} {16×1 int8} {[1]}
which I can convert like so:
typecast(data_to_view{3},'double')
ans =
1 2
Unfortunately this does not work for SQLite. I get:
data_to_view =
1×4 cell array
{[2]} {'string1'} {' �? #'} {[1]}
and I can't convert the third part correctly:
typecast(unicode2native(data_to_view{1,3}),'double')
ans =
0.0001 2.0000
So I still need to learn how to read an SQLite blob in Matlab but that is a different question.

Oracle 12C: Error getting while create BLOB column table with JSON type

While trying to create a table with column as BLOB data type holding JSON value, I am putting one constraint to check whether the BLOB column is JSON or not. I am getting below error in Oracle 12C database.
CREATE TABLE colorTab (
id NUMBER,
color BLOB ) ;
ALTER TABLE colorTab
ADD CONSTRAINT ensure_json CHECK (color IS JSON STRICT);
Error:
Table COLORTAB created.
Error starting at line : 7 in command -
ALTER TABLE colorTab
ADD CONSTRAINT ensure_json CHECK (color IS JSON STRICT)
Error report -
ORA-40499: no format defined for binary data type
40499. 00000 - "no format defined for binary data type"
*Cause: The provided JSON input was of binary type but a format was
not defined.
*Action: Define a format for binary input.
I am referring to the below blog in oracle:
https://blogs.oracle.com/jsondb/entry/storing_json_in_blob_columns
JSON support is available starting with 12.1.0.2.0. It is not available in 12.1.0.1.0.
That said it does not appear that your error is from running with 12.1.0.1.0.
Running with 12.1.0.1.0 I get
SQL*Plus: Release 12.2.0.1.0 Production on Thu Apr 27 14:16:35 2017
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Last Successful login time: Thu Apr 27 2017 14:16:27 -07:00
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
SQL> CREATE TABLE colorTab (
2 id NUMBER,
3 color BLOB ) ;
Table created.
SQL>
SQL> ALTER TABLE colorTab
2 ADD CONSTRAINT ensure_json CHECK (color IS JSON STRICT);
ADD CONSTRAINT ensure_json CHECK (color IS JSON STRICT)
*
ERROR at line 2:
ORA-00908: missing NULL keyword
SQL>
And with 12.1.0.2.0 (BP13)..
C:\Users\Mark D Drake>sqlplus scott/oracle#PDB12102
SQL*Plus: Release 12.2.0.1.0 Production on Thu Apr 27 14:20:17 2017
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Last Successful login time: Thu Apr 27 2017 14:18:43 -07:00
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
SQL> CREATE TABLE colorTab (
2 id NUMBER,
3 color BLOB ) ;
Table created.
SQL>
SQL> ALTER TABLE colorTab
2 ADD CONSTRAINT ensure_json CHECK (color IS JSON STRICT);
Table altered.
SQL>
What is your exact environment (include bundle patches etc...)
If you don't have bundle patch 13 or later can you try
SQL> ALTER TABLE colorTab
2 ADD CONSTRAINT ensure_json CHECK (color IS JSON FORMAT JSON STRICT);
Table altered.
SQL>
If this works I think you have a version without the mandatory JSON patches applied. I would STRONGLY recommend updating to the latest database bundled patches if that is the case.
SQL> CREATE TABLE colorTab (
2 id NUMBER,
3 color BLOB
4 )
5 /
Table created.
SQL> INSERT INTO colorTab VALUES(1, utl_raw.cast_to_raw (' { "color": "black "dummy"", "rgb": [0,0,0], "hex": "#000000"
} '))
2 /
1 row created.
SQL> Select json_value(utl_raw.cast_to_varchar2(color) format json,'$.color' ERROR on error)
2 from colorTab
3 /
from colorTab
*
ERROR at line 2:
ORA-40441: JSON syntax error
SQL>
With valid JSON
SQL> DROP TABle COLORTAB
2 /
Table dropped.
SQL> CREATE TABLE colorTab (
2 id NUMBER,
3 color BLOB
4 )
5 /
Table created.
SQL> INSERT INTO colorTab VALUES(1, utl_raw.cast_to_raw (' { "color": "black dummy", "rgb": [0,0,0], "hex": "#000000" }
'))
2 /
1 row created.
SQL> Select json_value(utl_raw.cast_to_varchar2(color) format json,'$.color' ERROR on error)
2 from colorTab
3 /
JSON_VALUE(UTL_RAW.CAST_TO_VARCHAR2(COLOR)FORMATJSON,'$.COLOR'ERRORONERROR)
--------------------------------------------------------------------------------
black dummy
SQL>
I got the same error in the version 12.1.0.2.0 while running the Unit test
Caused by: java.sql.SQLException:
ORA-40499: no format defined for binary data type
DEFINE _O_VERSION = "Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options" (CHAR)
DEFINE _O_RELEASE = "1201000000" (CHAR)
DEFINE _PWD = "C:\DDrive\Softwares\sqldeveloper-18\sqldeveloper\sqldeveloper\bin" (CHAR)
But the same is working fine in the version
DEFINE _O_VERSION = "Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production" (CHAR)
DEFINE _O_RELEASE = "1202000000" (CHAR)
DEFINE _PWD = "C:\DDrive\Softwares\sqldeveloper-18\sqldeveloper\sqldeveloper\bin" (CHAR)
Try with the Oracle DB version 12.2.0.1.0

BulkInsert into table with Identity column (T-SQL)

1) Can I do a BulkInsert from a CSV file, into a table, such that the table has an identity column that is not in the CSV, and gets automatically assigned?
2) Is there any rule that says the table that I'm BulkInsert'ing into, has to have the same columns in the same order as the flat file being read?
This is what I'm trying to do. Too many fields to include everything...
BULK INSERT ServicerStageACS
FROM 'C:\POC\DataFiles\ACSDemo1.csv'
WITH (FORMATFILE = 'C:\POC\DataFiles\ACSDemo1.Fmt');
GO
SELECT * FROM ServicerStageACS;
Error:
Msg 4864, Level 16, State 1, Line 3 Bulk load data conversion error
(type mismatch or invalid character for the specified codepage) for
row 1, column 1 (rowID).
I'm pretty sure the error is because I have an identity.
FMT starts like this:
9.0
4
1 SQLCHAR 0 7 "," 1 Month SQL_Latin1_General_CP1_CI_AS
2 SQLCHAR 0 100 "," 2 Client SQL_Latin1_General_CP1_CI_AS
A co-worker recommended that it was easier to do the bulk insert into a view. The view does not contain the identity field, or any other field not to be loaded.
truncate table ServicerStageACS
go
BULK INSERT VW_BulkInsert_ServicerStageACS
FROM 'C:\POC\DataFiles\ACSDemo1.csv'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
GO
SELECT * FROM ServicerStageACS;

SQL SERVER 2008 R2 DBCC PAGE PUZZLED

i did a test about DBCC IND and DBCC page .
Engine: SQL Server 2008
Script:
CREATE TABLE dbo.t2
(
ID int,
Col1 varchar(10),
Col2 varchar(20)
);
INSERT INTO dbo.t2 SELECT 1,REPLICATE('a',10),REPLICATE('b',10)
DECLARE #DBID int,#TableID int;
SELECT #DBID=DB_ID(),#TableID=OBJECT_ID('dbo.t2');
DBCC IND(#DBID,#TableID,-1)
--this shows :
1 274 1 278 1307151702 0 1 72057594039107584 In-row data 1
DBCC TRACEON (3604)
--DBCC PAGE (#DBID, 1,22,3)
--then dump the page info
DBCC PAGE(#DBID, 1,274,1)
shows:
DATA:
Slot 0, Offset 0x60, Length 51, DumpStyle BYTE
Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP VARIABLE_COLUMNS VERSIONING_INFO
Record Size = 51
Memory Dump #0x000000000F7EA060
0000000000000000: 70000800 01000000 03000002 001b0025 †p..............%
0000000000000010: 00616161 61616161 61616162 62626262 †.aaaaaaaaaabbbbb
0000000000000020: 62626262 62000000 00000000 00310000 †bbbbb........1..
0000000000000030: 00000†††††††††††††††††††††††††††††††...
OFFSET TABLE:
Row - Offset
0 (0x0) - 96 (0x60)
i try to explain:
byte one
0x70 01110000
BIT 0: SQL SERVER 2005/2008
BIT 1-3 :PRIMARY DATA
BIT 4 : HAS NULL
BIT 5:HAS VAR
BIT 6:NOT USED
BIT 7: NOT GHOST
Second Byte
0x00 : NOT USED
Thrid Byte
0x0008 :offset 8
the fix length int ,i got value 1
then next byte
0300 : 3 columns
00 : not null
0002 :2 columns
var column endoffset 0x001B (i got value aaa....)
var column end offset 0x0025 (i got value bbbb...)
but ,i don's know the meaning about last bytes....
please help me ,thks a lot
It's the 14-byte versioning tag [timestamep, pointer into tempdb].
From the master, Paul Randal
Inside the Storage Engine: Anatomy of a page. And Extent. And Record
It's part of the Record, as per comments to last link.
Each database row may use up to 14 bytes at the end of the row for row versioning information. The row versioning information contains the transaction sequence number of the transaction that committed the version and the pointer to the versioned row. These 14 bytes are added the first time the row is modified, or when a new row is inserted, under any of these conditions:
READ_COMMITTED_SNAPSHOT or ALLOW_SNAPSHOT_ISOLATION options are ON.
The table has a trigger.
Multiple Active Results Sets (MARS) is being used.
Online index build operations are currently running on the table.