I have a problem with my current data flow, where I try to delete entries, that already exist in my SQL DB. Based on the following thread, I am using a full outer join and an Alter Row step to get rid of duplicate entries: ADF copy data activity - check for duplicate records before inserting into SQL db
The only difference is, that the attribute _id is of the data type varchar.
I followed all the necessary steps:
And that is the preference of my AlterRow step:
Moreover, I am facing a strange problem, that I regularly have to set up the mapping of FixNames#{_id} as Azure detects it as an error. Setting it up again in the mapping part, the error is gone.
An altenative could be a stored procedure, but I don't understand, why the duplicates with my current version are still inserted.
Related
I'm using a data flow task and 2 Ole Db sources. The 2 sources bring in data from tables on 2 different databases on the same server. The 2 tables can be mapped by ids. All of the ids from the second table (closedstops) exist in the first table (stops). I need to remove all the the closed stops by id from the first table. Afterwards I need to export the first table out of the database into a text file.
Do I need to use a merge join before deleting or do I need to use a ole db command to delete records (see attached screenshot). I have looked at many questions and answers on stackoverflow as well as tutorials and none of them quite answer my question. Any help is greatly appreciated. Thank you.
Closed stops is the driver table. Leave it be.
Instead of an OLE DB Source table for "stops" change that to a Lookup Component. You are only interested in rows that match.
And then you can use your OLE DB Command to fire off single delete statements.
My preference for performance and traceability would be to insert all the "to be deleted" ids into a table on the Stops database. When the Data Flow has completed, an Execute SQL Task would then fire up to perform the deletes in a set based operation(s).
I have a Java Application that uses the JDBC:ODBC bridge for a connection with DBF Files linked to a Microsoft Access Database (thus, I'm using the driver for a connection to a Microsoft Access Database)
There is a table named SALFAC, that contains the following fields: NRO_FAC, COD_ITE, CAN_ITE, PRC_ITE and DSC_ITE among other columns. When I perform the following query: SELECT NRO_FAC, COD_ITE, CAN_ITE, PRC_ITE, DSC_ITE FROM SALFAC, without a WHERE clause it works fine. But when I execute the following: SELECT NRO_FAC, COD_ITE, CAN_ITE, PRC_ITE and DSC_ITE WHERE NRO_FAC=151407, my program throws a SQLException with the message The search key was not found in any record.
The NRO_FAC column is an integer type column, so using quotes results in a syntax error.
I compacted and repaired the entire database with no avail. Also, I tested the query directly on Microsoft Access 2010 and it gave me the same error. Yesterday I tested with another JDBC:ODBC bridge to the DBF files directly and also gave me the same error with the same query.
There is no blank spaces on the tables names and the columns names.
¿Is there any additional step to do in order to make queries like these work? I need execute the query with the WHERE clause. Also, for each DBF file is a NTX file. Must I do something with these files as well?
Thanks in advance
EDIT: I found something yesterday that might help. I changed the way I search the rows by inserting the entire DBF table content in an MS-Access temporary table, row by row, and then execute the query in the temprary table. It inserted the first 9 rows correctly, but the 10th row was next to a row that is marked as deleted and then the query crashed. Does the "marked-as-deleted" rows affect a query in MS-Access and/or dBase? If it does, is it possible ignore the "marked-as-deleted" rows using the JDBC:ODBC bridge? Also, must I install the Clipper commands (like DBU or PACK) in the server (it doesn't have them)?
I know there's information on this already but in my particular case the available resources I can find on it aren't working.
I'm trying to copy the data from a table in one database over to another table (also pre-existing) in another database. The table I'm trying to copy from contains null rows at the beginning which are preventing me from copying the data over because the second table will not allow null rows. I've tried deleting them, excluding them, including only the relevant rows, etc. Most of the stuff I've tried won't work with my particular version of mysql (syntax error). The specific error I'm getting when I try to move the data between databases is ERROR 1136.
mysql version: 5.5.41-0ubuntu0.14.04.1
I have an SSIS job which is getting data from an excel file and updating a database table SampleTable".
for instance consider,I am updating the following fields from the table "SampleTable" :
field1
field2
LastUpdated
the field (LastUpdated) needs to be changed only if any change happens to the values of either of the fields(i.e. field1 and field2)
the logic for updation is written in a script component.
and fetching the record value from database and comparing it with the value from the excel will be a huge performance hit.
hence i m looking for a performance friendly solution so that only once I will fetch the data from the database table (before the script execution may be) and store it temporarily somewhere .
There is option where redirect updates to staging/temp table and only insert new records to original table. Then synchronize records between original and staging table by using SP/Task. It's just isolation of update and insert operations. But you will see gain in performance.
You can achieve same by Lookup transformation so that you can avail the facility of Cache Connection Manager.
This article will provide guideline on how to use Lookup tranformation for Update/Insert and this about MSDN cache connection manager.
Using the cache connection manager
Checking if a row exists and if it does, has it changed?
Have a look of this article as well for performance best practices.
I need help in writing queries which will update MySql tables from SQL server.
I have created linked server and select queries work fine but I'm getting errors while doing update. I'm really new to writing such type of queries so please help me understanding error message and what it means.
My update query:
UPDATE openquery(stagedb_za, 'Select acm_flag FROM aol_center WHERE nid = 6439')
Set acm_flag = 'P'
Error:
OLE DB provider "MSDASQL" for linked server "stagedb_za" returned message "Row cannot be located for updating. Some values may have been changed since it was last read.".
Msg 7343, Level 16, State 4, Line 1
The OLE DB provider "MSDASQL" for linked server could not UPDATE table "[MSDASQL]". The rowset was using optimistic concurrency and the value of a column has been changed after the containing row was last fetched or resynchronized.
This whole issue goes away with a setting on the MySQL ODBC connector.
On the SQL Server that is linking to MySQL, go into the configure screen on the MySQL ODBC connection that is used by the linked server. Open "Details". On the "cursor/results" tab. Place a check in the "Return matched rows instead of affected rows".
Upon making that configuration change, updates that set a MySQL field value to the value it already has will not return an error.
Before I discovered this on another forum, I had put a LOT of code in to filter out the offending values on a field by field basis. This is easy.
Ok, I got the answer.
It seems like when your are updating column value and if new value is same as existing one then it is treated as there is no change so no updation will happen.
In my case I was updating acm_flag to 'P' but it's value was already 'P' so no updation happened. When I tried to update it to different value it worked just fine.