I was trying to insert a JSON file into a table which has only one column varchar2(4000) using sql loader. After I load I see the file text is loaded in multiple rows instead of one row , but I want them in one row , the whole file in one column and one row. Not sure why this is happening , is there a option to tell in the control file? here is my control file:
LOAD DATA
INFILE 'c:\json\sample-order.json'
INTO TABLE at_jsondocs
FIELDS
( jsontext CHAR(4000) )
See Alex Poole's explanation here, but the column in your table should be a CLOB, and you need to structure your control file like this:
LOAD DATA
INFILE *
INTO TABLE at_jsondocs
(
x FILLER CHAR(1),
jsontext LOBFILE(CONSTANT "c:\json\sample-order.json") TERMINATED BY EOF
)
BEGINDATA
0
Related
I am trying to use the bigquery export functionality to push data out to GCS in json format.
At the end of the process inorder to validate the count of exported records in the GCS file, I am creating an external table with auto schema detection just to take a count of records in the GCS files exported.
This works for single exported files. But for tables greater than 1gb in size, i use the wild card inorder to split into multiple files. This results in multiple files with some empty files as well created.
The empty files are causing an error while querying the external table : "400 Schema has no fields".
Please suggest any ideas to:
Either make sure that empty files do not get created in the export operation for multiple files scenario
To ignore empty files in the external table creation.
Any other way to take count of records in GCS after the export operation
I had the same problem but I found a workaround: it seems a TEMP TABLE does the trick.
(EDIT: reading the doc I noticed "export data" has always been described for BigQuery tables, non for custom selects. And since I never experienced empty files when exporting real tables, I gave temp tables the same chance)
Imagine we have the following query:
EXPORT DATA OPTIONS(
uri='gs://mybucket/extract-here/*.csv.gz'
, format='CSV'
, compression='GZIP'
, overwrite=true
, header=true
, field_delimiter=","
) AS (
WITH mytable AS (
SELECT col FROM UNNEST([1,2,3,4,5,6,7,8]) AS col
)
SELECT * FROM mytable
);
You can rewrite it as following:
BEGIN
CREATE TEMP TABLE _SESSION.tmpExportTable AS (
WITH mytable AS (
SELECT col FROM UNNEST([1,2,3,4,5,6,7,8]) AS col
)
SELECT * FROM mytable
);
EXPORT DATA OPTIONS(
uri='gs://mybucket/extract-here/*.csv.gz'
, format='CSV'
, compression='GZIP'
, overwrite=true
, header=true
, field_delimiter=","
) AS
SELECT * FROM _SESSION.tmpExportTable;
END;
I've got a process that creates a csv file that contains ONE set of values that I need to import into a field in a MySQL database table. This process creates a specific file name that identifies the values of the other fields in that table. For instance, the file name T001U020C075.csv would be broken down as follows:
T001 = Test 001
U020 = User 020
C075 = Channel 075
The file contains a single row of data separated by commas for all of the test results for that user on a specific channel and it might look something like:
12.555, 15.275, 18.333, 25.000 ... (there are hundreds, maybe thousands, of results per user, per channel).
What I'm looking to do is to import directly from the CSV file adding the field information from the file name so that it looks something like:
insert into results (test_no, user_id, channel_id, result) values (1, 20, 75, 12.555)
I've tried to use "Bulk Insert" but that seems to want to import all of the fields where each ROW is a record. Sure, I could go into each file and convert the row to a column and add the data from the file name into the columns preceding the results but that would be a very time consuming task as there are hundreds of files that have been created and need to be imported.
I've found several "import CSV" solutions but they all assume all of the data is in the file. Obviously, it's not...
The process that generated these files is unable to be modified (yes, I asked). Even if it could be modified, it would only provide the proper format going forward and what is needed is analysis of the historical data. And, the new format would take significantly more space.
I'm limited to using either MATLAB or MySQL Workbench to import the data.
Any help is appreciated.
Bob
A possible SQL approach to getting the data loaded into the table would be to run a statement like this:
LOAD DATA LOCAL INFILE '/dir/T001U020C075.csv'
INTO TABLE results
FIELDS TERMINATED BY '|'
LINES TERMINATED BY ','
( result )
SET test_no = '001'
, user_id = '020'
, channel_id = '075'
;
We need the comma to be the line separator. We can specify some character that we are guaranteed not to tppear to be the field separator. So we get LOAD DATA to see a single "field" on each "line".
(If there isn't trailing comma at the end of the file, after the last value, we need to test to make sure we are getting the last value (the last "line" as we're telling LOAD DATA to look at the file.)
We could use user-defined variables in place of the literals, but that leaves the part about parsing the filename. That's really ugly in SQL, but it could be done, assuming a consistent filename format...
-- parse filename components into user-defined variables
SELECT SUBSTRING_INDEX(SUBSTRING_INDEX(f.n,'T',-1),'U',1) AS t
, SUBSTRING_INDEX(SUBSTRING_INDEX(f.n,'U',-1),'C',1) AS u
, SUBSTRING_INDEX(f.n,'C',-1) AS c
, f.n AS n
FROM ( SELECT SUBSTRING_INDEX(SUBSTRING_INDEX( i.filename ,'/',-1),'.csv',1) AS n
FROM ( SELECT '/tmp/T001U020C075.csv' AS filename ) i
) f
INTO #ls_u
, #ls_t
, #ls_c
, #ls_n
;
while we're testing, we probably want to see the result of the parsing.
-- for debugging/testing
SELECT #ls_t
, #ls_u
, #ls_c
, #ls_n
;
And then the part about running of the actual LOAD DATA statement. We've got to specify the filename again. We need to make sure we're using the same filename ...
LOAD DATA LOCAL INFILE '/tmp/T001U020C075.csv'
INTO TABLE results
FIELDS TERMINATED BY '|'
LINES TERMINATED BY ','
( result )
SET test_no = #ls_t
, user_id = #ls_u
, channel_id = #ls_c
;
(The client will need read permission the .csv file)
Unfortunately, we can't wrap this in a procedure because running LOAD DATA
statement is not allowed from a stored program.
Some would correctly point out that as a workaround, we could compile/build a user-defined function (UDF) to execute an external program, and a procedure could call that. Personally, I wouldn't do it. But it is an alternative we should mention, given the constraints.
I need to enter a text value that to represent the year (will be the same for every row) in a set of data being imported from a csv file. I am getting a syntax error each time. How do I specify the text value so that it will populate the column properly?
Load data local infile 'C:/Users/Candace.....csv'
into table estimate(State, '2010', Population)
fields terminated by ',';
Not tested, though according to the documentation it should work:
LOAD DATA INFILE 'file.csv'
INTO TABLE estimate
(State, Population)
SET Year = 2010;
Relevant part from the doc:
The SET clause can be used to supply values not derived from the input file.
First time using sqlldr, simply trying to load a file with single column, and cannot find a proper way to do it :(
What do I put as a delimiter?
Here's my .ctl file :
load data
infile 'myfile.dat'
into table mytable
fields terminated by ''
(mycolumn)
I keep getting errors in the .log like :
Record 4: Rejected - Error on table ..., column ....
ORA-12899: value too large for column "... (actual: 80, maximum: 24)
Even though the values in the file are max 8 chars each or smth :
string1
string2
string3
Any help will be much appreciated.
Many thanks,
G
You don't need the fields terminated by line in this case but you should have a TRUNCATE or APPEND depending on if you want to keep existing data or not.
load data
infile 'myfile.dat'
truncate
into table mytable
(mycolumn)
Why not just change the size of your column to 80
Alter table modify varchar2(80)
I have around 1000 html files in my local computer and I have to import them into database table in a single column . I mean one full file into one cell and 1000 HTML files into 1000 rows .
Table structure
CREATE TABLE `content` (
`ID` smallint(5) unsigned NOT NULL AUTO_INCREMENT,
`data` text ,
PRIMARY KEY (`ID`)
)
one row contains ID and one html file contents
I tried
load DATA LOCAL INFILE 'C:\\Users\\userD\\Desktop\\data\\alter-data.html' INTO TABLE content LINES TERMINATED BY '</html>';
since each file end line is </html> but that didn't help to load any data ..
Can any one help me on this?
Thank You
Regards
Kiran
I needed to do the same thing and spent ages working it out but it turned out to be simple.
Note the test table, test_html field was defined as a "text" field
truncate table test;
load data local infile 'data.htm'
into table test
fields terminated by '</html>'
lines terminated by '</html>'
(test_html);
I used the below Java program and able to upload the contents
http://www.roseindia.net/tutorial/java/core/files/javafilesavetodatabase.html
Thanks All
Regards
Kiran