I have a query that loads data from a text file into a table, finds a row with a max value in the "date_time" column and deletes all rows that are less than the max value. But this file will be updated several times a day and each time only one row with the max value will remain in the table.
LOAD DATA INFILE 'C:/users/user/desktop/download.txt' IGNORE INTO TABLE download1
FIELDS TERMINATED BY ';'
LINES TERMINATED BY '\r\n'
IGNORE 3 LINES;
DELETE FROM download1 WHERE date_time < (SELECT * FROM (SELECT MAX(date_time) AS MaxDatetime FROM download1) AS t)
How can I make the past max value also remain in the table when executing a query with an updated file?
text file:
table:
Updated based on question edit and comments.
Since the id field in the table is auto_increment, it provides a continuously increading value. Get the max value of the id field before uploading your new file and use that to limit your delete to newer records only:
SET #OLDMAXID = IFNULL((SELECT MAX(id) FROM download1), 0);
LOAD DATA INFILE 'C:/users/user/desktop/download.txt' IGNORE INTO TABLE download1
FIELDS TERMINATED BY ';'
LINES TERMINATED BY '\r\n'
IGNORE 3 LINES;
DELETE FROM download1 WHERE date_time < (SELECT * FROM (SELECT MAX(date_time) AS MaxDatetime FROM download1) AS t) and id > #OLDMAXID;
Related
Need help with the below SQL*Loader script. I have created the below test script to load the data into the 3 tables based on the location and the work_type columns.But data is not getting inserted correctly into the 3 tables. I am not able to understand how to do multiple comparisons on the data in when clause. Please help.
LOAD DATA
INFILE *
REPLACE
INTO
TABLE
xx_dumy_test_emp_table
WHEN (WORK_TYPE='EMP') AND (Work_Location = 'IND')
FIELDS TERMINATED BY ';'
TRAILING NULLCOLS
(
Work_Type
,Employee_Role
,First_name
,Last_Name
,Work_Location
)
INTO
TABLE
xx_dumy_test_oversea_employess
WHEN (WORK_TYPE='EMP') AND (Work_Location <> 'IND')
FIELDS TERMINATED BY ';'
TRAILING NULLCOLS
(
Work_Type POSITION(1:3)
,Employee_Role
,First_name
,Last_Name
,Work_Location
)
INTO
TABLE
xx_dumy_test_mgr_table
WHEN (WORK_TYPE='MGR') AND (Work_Location = 'IND')
FIELDS TERMINATED BY ';'
TRAILING NULLCOLS
(
Work_Type POSITION(1:3)
,Employee_Role
,First_name
,Last_Name
,Work_Location
)
--DATA OF THE SCRIPT
EMP;Test Ops1;Gautam;Hoshing;IND;
MGR;Test Ops2;Steve;Tyler;IND;
EMP;Test Ops3;Hyana;Motler;JPY;
Regards,
Gautam.
I want to load data from a CSV file into a table in my database. However, the first two columns are an ID (AUTO INCREMENT, PRIMARY KEY) and CURRENT DATE (CURRENT DATE ON UPDATE) field. The source file will therefore not have these fields.
How would I get the file to load? The below code doesn't work. I have also tried leaving out id and action_date from the fields in the brackets.
LOAD DATA INFILE '/upliftments//UPLIFTMENTS_20160901.csv'
INTO TABLE database.return_movements
CHARACTER SET latin1
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
IGNORE 1 LINES
( id
, action_date
, current_code
, serial_number
, new_code
);
I am loading data using SQLldr into multiple tables using 1 control file. The sequence in the first table(seq.nextval) becomes input to the 2nd table(seq.currval). This does not work as the last number generated in the sequence becomes the currval for the remaining tables and it inserts that number to all the rows in the second and subsequent tables, unless I specify rows=1 to commit every row. Is there an alternative to specify commit after certain number still keeing the currval?
Control file:
LOAD DATA INFILE '.\DATA_FILES\USERS.csv'
APPEND
INTO TABLE TABLE1
FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY '"'
trailing NULLCOLS
( USER_ID "SEQ_USER_ID.NEXTVAL",
USER_NAME,
USER_PASSWORD "(select user_password from USERS where
USER_NAME='DEMO')",
EXPIRY_DATE "(select SYSDATE+1000 from dual)",
FAILED_ATTEMPTS CONSTANT "0"
)
INTO TABLE TABLE2
FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY '"'
trailing NULLCOLS
( USER_GROUP_ID "SEQ_USER_GROUP_ID.NEXTVAL",
USER_ID "SEQ_USER_ID.CURRVAL",
GROUP_ID CONSTANT "0"
)
Thanks,
Anan
I am populating a MySQL table with a csv file pulled from a third party source. Every day the csv is updated and I want to update rows in MySQL table if an occurrence of column a, b and c already exists, else insert the row. I used load data infile for the initial load but I want to update against a daily csv pull. I am familiar with INSERT...ON DUPLICATE, but not in the context of a csv import. Any advice on how to nest LOAD DATA LOCAL INFILE within INSERT...ON DUPLICATE a, b, c - or if that is even the best approach would be greatly appreciated.
LOAD DATA LOCAL INFILE 'C:\\Users\\nick\\Desktop\\folder\\file.csv'
INTO TABLE db.tbl
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 lines;
Since you use LOAD DATA LOCAL INFILE, it is equivalent to specifying IGNORE: i.e. duplicates would be skipped.
But
If you specify REPLACE, input rows replace existing rows. In other words, rows that have the same value for a primary key or unique index as an existing row.
So you update-import could be
LOAD DATA LOCAL INFILE 'C:\\Users\\nick\\Desktop\\folder\\file.csv'
REPLACE
INTO TABLE db.tbl
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 lines;
https://dev.mysql.com/doc/refman/5.6/en/load-data.html
If you need a more complicated merge-logic, you could import CSV to a temp table and then issue INSERT ... SELECT ... ON DUPLICATE KEY UPDATE
I found that the best way to do this is to insert the file with the standard LOAD DATA LOCAL INFILE
LOAD DATA LOCAL INFILE
INTO TABLE db.table
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 lines;
And use the following to delete duplicates. Note that the below command is comparing db.table to itself by defining it as both a and b.
delete a.* from db.table a, db.table b
where a.id > b.id
and a.field1 = b.field1
and a.field2 = b.field2
and a.field3 = b.field3;
To use this method it is essential that the id field is an auto incremental primary key.The above command then deletes rows that contain duplication on field1 AND field2 AND field3. In this case it will delete the row with the higher of the two auto incremental ids, this works just as well if we were to use < instead of >.
I have a table that looks like this:
products
--------
id, product, sku, department, quantity
There are approximately 800,000 entries in this table. I have received a new CSV file that updates all of the quantities of each product, for example:
productA, 12
productB, 71
productC, 92
So there are approximately 750,000 updates (50,000 products had no change in quantity).
My question is, how do I import this CSV to update only the quantity based off of the product (unique) but leave the sku, department, and other fields alone? I know how to do this in PHP by looping through the CSV and executing an update for each single line but this seems inefficient.
You can use LOAD DATA INFILE to bulk load the 800,000 rows of data into a temporary table, then use multiple-table UPDATE syntax to join your existing table to the temporary table and update the quantity values.
For example:
CREATE TEMPORARY TABLE your_temp_table LIKE your_table;
LOAD DATA INFILE '/tmp/your_file.csv'
INTO TABLE your_temp_table
FIELDS TERMINATED BY ','
(id, product, sku, department, quantity);
UPDATE your_table
INNER JOIN your_temp_table on your_temp_table.id = your_table.id
SET your_table.quantity = your_temp_table.quantity;
DROP TEMPORARY TABLE your_temp_table;
I would load the update data into a seperate table UPDATE_TABLE and perform an update within MySQL using:
UPDATE PRODUCTS P SET P.QUANTITY=(
SELECT UPDATE_QUANTITY
FROM UPDATE_TABLE
WHERE UPDATE_PRODUCT=P.PRODUCT
)
I dont have a MySQL at hand right now, so I can check the syntax perfectly, it might be you need to add a LIMIT 0,1 to the inner SELECT.
Answer from #ike-walker is indeed correct but also remember to double check how your CSV data if formatted. Many times for example CSV files can have string fields enclosed in double quotes ", and lines ending with \r\n if working on Windows.
By default is assumed that no enclosing character is used and line ending is \n.
More info and examples here https://mariadb.com/kb/en/importing-data-into-mariadb/
This can be fixed by using additional options for FIELDS and LINES
CREATE TEMPORARY TABLE your_temp_table LIKE your_table;
LOAD DATA INFILE '/tmp/your_file.csv'
INTO TABLE your_temp_table
FIELDS
TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"' -- new option
LINES TERMINATED BY '\r\n' -- new option
(id, product, sku, department, quantity);
UPDATE your_table
INNER JOIN your_temp_table on your_temp_table.id = your_table.id
SET your_table.quantity = your_temp_table.quantity;
DROP TEMPORARY TABLE your_temp_table;