I am very confused about LOAD DATA INFILE
after searching SO and google I have found no help on what I am attempting to do.
I want to create a new table, and load the contents of a csv file. The csv files first row is the column names I want.
Or if that cannot be done, how can I load the file without knowing how many columns exist?
LOAD DATA INFILE 'file.txt'
INTO TABLE t1
(... unknown number of columns ...);
You're going to need a tool other than mysql to load csvs without a predetermined schema.
Related
I have created an empty table A in MySQL and inserted data from a local csv file, A.csv, using LOAD DATA LOCAL INFILE. Now I want to create more tables just like A and insert data from other different local csv files (they all have the same fields).
So for table B it will be something like:
CREATE TABLE B LIKE A;
LOAD DATA LOCAL INFILE 'mypath/B.csv'
...
I need to repeat this process for about 20 tables. How can I write a loop procedure to automate the process?
Any help on this would be much appreciated!
I have two CSV-files that I uploaded to the Azure Blob Storage within HDInsight. I can upload these two files to the cluster without problems. I then create two Hive-tables with...
CREATE EXTERNAL TABLE IF NOT EXISTS hive_table1(id int, age string, date string...)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\;' STORED AS TEXTFILE LOCATION '/user/hive/warehouse'
Similar syntax goes for the other table.
Now I want to load the first CSV-file into the first table and the second CSV-file into the second table (resulting in non-corresponding columns).
I use...
LOAD DATA INPATH '/file/file1.csv' OVERWRITE INTO TABLE hive_table1;
...and am able to load the CSV-file data into the first table. But..., not only is the first data set loaded into the first Hive table, it also loads the exact same file's data into the second Hive table.
Obviously, I only want to have the first data set loaded into one table and the second distinct data set only into the other table.
Can anyone help pointing out errors or contribute with a possible solution?
Thanks in advance.
It looks like you just need to specify a different 'LOCATION' for the second table. When you do the 'LOAD DATA', Hive is actually copying data into that path. If both tables have the same 'LOCATION', they will share the same data.
Your location is what creating problem. You have given same location for both the tables. As the tables are external the file will be created directly under your path.
Also LOAD DATA INPATH '/file/file1.csv' OVERWRITE INTO TABLE hive_table1; will overwrites the already existing file. This is what happening with your tables. As Farooque mentioned for different tables the location should be unique to get the desired results.
I see you are creating external table and creating 2 tables having single files each.
You have to follow the simple steps as below:
Create table
CREATE EXTERNAL TABLE IF NOT EXISTS hive_table1(id int, age string, date string...)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ';' STORED AS TEXTFILE LOCATION '/user/hive/warehouse/table1_dir/'
Copy file to HDFS location
hdfs dfs -put '/file/file1.csv' '/user/hive/warehouse/table1_dir/'
Similary for second table
Create table
CREATE EXTERNAL TABLE IF NOT EXISTS hive_table2(id int, age string, date string...)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ';' STORED AS TEXTFILE LOCATION '/user/hive/warehouse/table2_dir/'
Copy file to HDFS location
hdfs dfs -put '/file/file2.csv' '/user/hive/warehouse/table2_dir/'
Note: If you are using more than one table, then their location should be unique.
I am trying to load a file using the command below:
'LOAD DATA LOCAL INFILE 'H://test-Project//SampleFiles//Employee.csv'
IGNORE INTO Table Employee_Tbl.
(emp_code,emp_name,emp_pi_id);
One of the columns in the table is the file name. THe file content of 'Employee.csv' does not consist of the 'file name' explicitly. And I need is the value 'Employee.csv' to be populated into this table along with insert of the data rows from the file. Any thoughts as to how I can do this? Would greatly appreciate your help.
thanks
I have written the below query to insert data from xml file to a mysql table:
load XML local infile 'D:\\a.xml' into table asdb.abc rows identified by '<ApplicantName>';
It inserts the values in the table. I have a column named AckNo in the abc table. I need to insert the xml values where ackNo is equal to a particular value. e.g. i tried writing the below query but its not working:
load XML local infile 'D:\\a.xml' into table asdb.abc rows identified by '<ApplicantName>' where ackNo='1';
Would really appreciate someones help.
I don't think that you can use a WHERE clause with LOAD INFILE.
Check the documentation.
I think you have to parse your XML file and keep only values you want, then generates another XML file, before using LOAD DATA.
Or easier and quicker :
Load everything
DELETE FROM your_table WHERE your_id != "the_value_you_want"
Initially, I created a database called "sample" and updated the data from massive size CSV file.
Whenever I have small changes in .csv file (some data are added/deleted/modified), I have to update this in database too. Always updating the entire .csv file (large) is not efficient.
Is there any efficient way to update the modified data from .csv file to database?
Assuming that you are using LOAD DATE INFILE for importing from CSV, try using this syntax:
LOAD DATA INFILE 'file_name'
IGNORE
INTO TABLE `tbl_name`
...
...
IGNORE keyword will skip any rows in the CSV that duplicate any existing row in the table causing a conflict with a unique key. Read more here.
This will be more quicker and efficient than importing the complete CSV again.