I have an ssis package source ODBC, after aggregate using vlookup inserting records into sql table using OLEDB. Using vlookup inserting new records in destination table(sql) and matching records in temp table(sql). And the table is having Primary key on date field. Inserting records, some records are dupliating violating PK. So here base on date field, delete records of 3 days and insert again.
In ssis First taken Execute task, and written delete query, then Data Flow to move data form source to destination. Here i am not getting Correct records.
Task:
Before inserting Delete records and insert them.
Please help me out how to achieve this.
Related
I need to give my customer some sample data of each table in the dbase. I have duplicated my dbase, and now in the duplicate dbase I want to delete all records in all tables, except for 1 record. It doesn't matter which record remains. Then I will do an export for them without a million records.
What is the query I need to run to do this? Each table can have different primary keys.
I am using mysql 5.7.34 command line.
The following is an example to better explain my scenario. My database table has following columns
Column -1: Operating_ID (which is the primary key)
Column -2: Name
Column -3: Phone
Column -4: Address
Column -5: Start Date
Column -6: End Date
The values for the columns 1,2,3,4 come from an extract and this extract is pushed to the database daily using SSIS data flow task
The values for the columns 5 and 6 are user inputted from a web applicxation and saved to the database.
Now in SSIS process instead of throwing violation of primary key error, i need to update Columns 2,3,4 if the primary key i.e column 1 already exists.
First i considered replace but that deletes the user inputted data columns 4,5.
I would like to keep the data in columns 4,5 and update columns 2,3,4 when column 1 already exists.
Do a LOOKUP for Operating_ID. Change that lookup from "FAIL ON NOT FOUND" to "REDIRECT ROWS TO NO MATCH"
If match not found, go to INSERT
If match found, go to UPDATE. You can run OLAP commands to update, but if it is a large set of data, you are better off putting into a table, and doing an UPDATE with a JOIN
This is what I would do. I would put all the data in a staging table. Then I woudl use a data flow to insert the new records and the source of that dataflow would be the staging table with a not exists clause referencing the prod table.
Then I would use an Execute SQL task in the control flow to update the data for existing rows.
I am creating a SSIS package in which I have 15 Data flow tasks each for different table.In each data flow task I am inserting the records from source to destination and I am checking if record is available in destination I am updating that record if not inserting as new record.Now I want to store updated records count,new records count and table name in audit table
How can I do that easily?
I'm attempting to insert, update, and delete all in one MySQL query. I have a DB with about 100 records with a primary key. I'm updating the DB from a CSV file. What I would like to happen is if a record is in the csv and not in the db, then add it. If it's in the db and the csv, update it. If it's in the db and not in the csv, delete it. I have the insert and update part working, but I'm hung up on the delete part.
Here is my query so far:
INSERT INTO mydb
(tourID,agent) VALUES (:tourID,:agent)
ON DUPLICATE KEY UPDATE
tourID=:tourID
Is there anything like 'on non duplicate key delete'?
Have an extra column named "toDelete". At the beginning of your transaction, set it to true for all rows. When you update a row, set it to false. When you are done, delete every row where toDelete is still true.
Here's some pseudocode:
For each record in the CSV file, run your INSERT INTO query.
Run a DELETE FROM mydb WHERE tourID NOT IN (comma-separated list of tourIDs from CSV)
How you come up with that comma-separated list of tourIDs from CSV depends on how you're processing your CSV file.
Does any one how can I only sync the new records from table A to table B in SSIS?
I have 2 tables which table A and table B. Table A will be update time to time from user. However, records from table B is sync from table A every 30 mins. I have created a SSIS job to do the sync, see below for more details:
I have an issue here, every times when I'm ruining the job, it will copy all data from table A and insert into table B (this causing duplicate records had been insert). Any way that I can set the job so that it will only sync the new records into table B?
I would use a MERGE statement into an execute SQL Task.
Please review some examples about how to use the merge command:
Using MERGE in SQL Server to insert, update and delete at the same time
The MERGE Statement in SQL Server 2008
Kind Regards,
Paul