Is it possible to open and read from a text file in a MYSQL stored procedure? I have a text file with a list of about 50k telephone numbers, and want to write a stored procedure that will open the file, read the 50k lines and store it as rows in a table. I cannot load the file directly using LOAD IN FILE as the table has additional columns that I have to set.
Thanks!
In the end I used LOAD IN FILE. Apparently you can set which columns get populated by using the SET keyword:
LOAD DATA LOCAL INFILE '/temp/input_file.txt' IGNORE INTO TABLE TEST.TEST_INSERT (INSERT_FIELD) SET VERSION=1, LIST_ID=ID_GEN(), CREATE_DATE=NOW(), CREATE_BY='TESTUSER';
Related
How should I specify the columns names when I want to load a MySQL table from a local .csv file using the LOAD DATA LOCAL FILE command in MySQL?
You specify a comma seperated list of columns, in parentheses, at the end (but before the optional SET parameters).
See https://dev.mysql.com/doc/refman/8.0/en/load-data.html#load-data-column-list
I have two CSV-files that I uploaded to the Azure Blob Storage within HDInsight. I can upload these two files to the cluster without problems. I then create two Hive-tables with...
CREATE EXTERNAL TABLE IF NOT EXISTS hive_table1(id int, age string, date string...)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\;' STORED AS TEXTFILE LOCATION '/user/hive/warehouse'
Similar syntax goes for the other table.
Now I want to load the first CSV-file into the first table and the second CSV-file into the second table (resulting in non-corresponding columns).
I use...
LOAD DATA INPATH '/file/file1.csv' OVERWRITE INTO TABLE hive_table1;
...and am able to load the CSV-file data into the first table. But..., not only is the first data set loaded into the first Hive table, it also loads the exact same file's data into the second Hive table.
Obviously, I only want to have the first data set loaded into one table and the second distinct data set only into the other table.
Can anyone help pointing out errors or contribute with a possible solution?
Thanks in advance.
It looks like you just need to specify a different 'LOCATION' for the second table. When you do the 'LOAD DATA', Hive is actually copying data into that path. If both tables have the same 'LOCATION', they will share the same data.
Your location is what creating problem. You have given same location for both the tables. As the tables are external the file will be created directly under your path.
Also LOAD DATA INPATH '/file/file1.csv' OVERWRITE INTO TABLE hive_table1; will overwrites the already existing file. This is what happening with your tables. As Farooque mentioned for different tables the location should be unique to get the desired results.
I see you are creating external table and creating 2 tables having single files each.
You have to follow the simple steps as below:
Create table
CREATE EXTERNAL TABLE IF NOT EXISTS hive_table1(id int, age string, date string...)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ';' STORED AS TEXTFILE LOCATION '/user/hive/warehouse/table1_dir/'
Copy file to HDFS location
hdfs dfs -put '/file/file1.csv' '/user/hive/warehouse/table1_dir/'
Similary for second table
Create table
CREATE EXTERNAL TABLE IF NOT EXISTS hive_table2(id int, age string, date string...)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ';' STORED AS TEXTFILE LOCATION '/user/hive/warehouse/table2_dir/'
Copy file to HDFS location
hdfs dfs -put '/file/file2.csv' '/user/hive/warehouse/table2_dir/'
Note: If you are using more than one table, then their location should be unique.
I have a database of entries consisting of a 'name', 'id' and a 'description', but currently the 'description' field is set to the file path of a .txt file that actually contains the description content. Each .txt file's name is each row's 'id', plus the .txt extension and they all reside in the same directory.
Can I load and replace each 'description' field with the content from the relevant text file (using MySQL)?
You can't write a MySQL query directly that will read the description values from your file system. That would require the MySQL server to be able to read raw text from files in your file system. You Can't Do That™.
You certainly can write a program in your favorite host language (php, java, PERL, you name it) to read the rows from your database, and update your description rows.
You could maybe contrive to issue a LOAD DATA INFILE command to read each text file. But the text files would have to be very carefully formatted to resemble CSV or TSV files.
Purely using mysql this would be a difficult, if not impossible exercise because mysql does not really offer any means to open files.
The only way to open an external text file from mysql is to use LOAD DATA INFILE command, which imports the text file into a mysql table. What you can do is to write a stored procedure in mysql that:
Create a temporary table with a description field large enough to accommodate all descriptions.
Reads all id and description field contents into a cursor from your base table.
Loop through the cursor and use load data infile to load the given text file's data into your temporary table. This is where things can go wrong. The account under which mysql daemon / service runs needs to have access to the directories and fiels where the description files are stored. You also need to be able to parametrise the load data infile command to read the full contents of the text file into a single field, so you need to set the field and line terminated by parameters to such values that cannot be found in any of the description files. But, even for this you need to use a native UDF (user defined function) that can execute command line programs because running load data infile directly from stored procedures is not allowed.
See Using LOAD DATA INFILE with Stored Procedure Workaround-MySQL for full description how to this.
Issue an update statement using the id from the cursor to update the description field in your base table from the temporary table.
Delete the record from your temp table.
Go to 3.
It may be a lot easier to achieve this from an external programming language, that has better file manipulation functions and can update each record in your base table accordingly.
Suppose we have table with a DECIMAL column with values, for example: 128.98, 283.98, 21.20.
I want to import some CSV Files to this table. However, in the columns of these files, I have values like 235,69, 23,23, with comma instead of points.
I know I can REPLACE that column, but is there some way of doing that before LOAD INFILE?
I do not believe you can simultaneously replace that column and load the data. Looks like you will have to do multiple steps to get the results you want.
Load the data first into a raw table using the LOAD INFILE command. This table can be identical to the main table. You can use the Create Table like command to create the table.
Process the data (i.e. change the comma to a . where applicable) in the raw table.
select the data from the raw table and insert into main table either with row by row processing or bulk insert.
This can all be done in a stored procedure (SP) or by a 3rd party script written in python, php, etc...
If you want to know more about SP's in Mysql, Here is a useful link.
I have a text file to be imported in a MySQL table. The columns of the files are comma delimited. I set up an appropriate table and I used the command:
load data LOCAL INFILE 'myfile.txt' into table mytable FIELDS TERMINATED BY ‘,’;
The problem is, there are several spaces in the text file, before and after the data on each column, and it seems that the spaces are all imported in the tables (and that is not what I want). Is there a way to load the file without the empty spaces (other than processing each row of the text file before importing in MySQL)?
As far as I understand, there's no way to do this during the actual load of the data file dynamically (I've looked, as well).
It seems the best way to handle this is to either use the SET clause with the TRIM
function
("SET column2 = TRIM(column2)")
or run an update on the string columns after loading, using the TRIM() function.
You can also create a stored procedure using prepared statements to run the TRIM function on all columns in a specified table, immediately after loading it.
You would essentially pass in the table name as a variable, and the sp would use the information_schema database to determine which columns to upload.
If you can use .NET, CSVReader is a great option(http://www.codeproject.com/KB/database/CsvReader.aspx). You can read data from a CSV and specify delimiter, trimming options, etc. In your case, you could choose to trim left and right spaces from each value. You can then either save the result to a new text file and import it into the database, or loop through the CsvReader object and insert each row into the database directly. The performance of CsvReader is impressive. Hope this helps.