column mapping trouble within ADFv2 - json

I have a source .csv with 21 columns and a destination table with 25 columns.
Not ALL columns within the source have a home in the destination table and not all columns in the destination table come from the source.
I cannot get my CopyData task to let me pick and choose how I want the mapping to be. The only way I can get it to work so far is to load the source data to a "holding" table that has a 1:1 mapping and then execute a stored procedure to insert data from that table into the final destination.
I've tried altering the schemas on both the source and destination to match but it still errors out because the ACTUAL source has more columns than the destination or vice versa.
This can't possibly be the most efficient way to accomplish this but I'm at a loss as to how to make it work.
The error code that is returned is some variation on:
"errorCode": "2200",
"message": "ErrorCode=UserErrorInvalidColumnMappingColumnCountMismatch,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Invalid column mapping provided to copy activity: '{LONG LIST OF COLUMN MAPPING HERE}', Detailed message: Different column count between target structure and column mapping. Target column count:25, Column mapping count:16. Check column mapping in table definition.,Source=Microsoft.DataTransfer.Common,'",
"failureType": "UserError",
"target": "LoadPrimaryOwner"

Have you tried mapping the columns in the graphical editor? Just click on the copy activity, then mapping and click the blue button "Import Schemas". This will import both schemas and let you pick which column from source maps with which column from sink.
Hope this helped!

In the sink dataset delete the columns that you don't want to be mapped.
delete the columns that are not required in the sink by selecting and then click delete button
[
In order for the copy to work smoothly.
1.The source dataset should have all the columns in the same sequence.
2. All the columns selected in the sink dataset have to be mapped.

It seems that you were trying to extract the 16 columns from the source table to target table. If your target is Sql Server or Azure Sql DB, you can try the following settings:
Set the source structure as 21 columns in csv file.
Set the column mapping with 16 columns mapping as your wanted data.
Set the target structure as 16 columns, which has the same name and order in the column mapping definition.

Related

SSIS package to Insert flat file data to two different tables

I want to insert flat file data to two different sql table.But some additional field coming from flat file should be inserted to other table on the basis of indicator field but the usual field coming should be inserted into the regular table.
The other issue,the additional field to be inserted cannot be inserted directly because of no column mapping.
eg:
1234 056 Y Tushar
5678 065 N
So 1234 056 should be inserted to regular table but indicator Y tells us that Tushar should be inserted to other table.
But the table in which I want to Insert Tushar cannot be done directly as it does not have 1234 column name.
For indicator N also it should get inserted normally in the base table.
So what I did was I used a conditional split and then used ole db command but it it inserting multiple records in the table.
If you put a Multicast task right after your flat file source, you can create extra copies of your data set. Then you can use one copy to insert into Regular Table, and then you can put your Conditional Split on the second copy.
Your data flow would then look like this:
In my Flat File Source I defined four columns:
The Multicast doesn't need any configuration, and I assume the Regular Table destination isn't giving you the trouble. So next, you'd create the Indicator check with a Conditional Split task. Check for a value of Y like this:
Then just map whichever available columns you want to insert into Other Table. I chose the second column (I called mine Seq) and the Name column. You may have these named differently.

SSIS parameters/variables in destination mapping

I am working on a large SSIS data migration project in which all of the output tables have fields for the creation date & user as well as the last update date and user. The values will be the same for all of the records in all of the output tables.
Is there a way to define parameters or variables that will appear in the destination mapping window, and can be used to populate the output table?
If I use a sql statement in the source, I could, of course, include extra fields for this, but then I also have to add a Data Conversion task for translating the string fields from varchar to nvarchar.
You cannot do this in the destination mapping.
As you've already considered, you could do this by including the extra fields in the source, but then you are passing all that uniform data through the entire dataflow and perhaps having to convert it as well.
A third option would be to run through your data flow without those columns at all (let them be NULL in the destination), and then follow the data flow with an UPDATE that sets those columns with a package variable value.

SSIS - Reuse Ole DB source when matching Fact against lookup table twice

I am pretty new to SSIS and BI in general, so first of all sorry if this is a newbie question.
I have my source data for the fact table in a csv, so I want to match the ids against the surrogate keys in lookup tables.
The data structure in the csv is like this
... userId, OriginStationId, DestinyStationId,..
What I am trying to accomplish is to match the data against my lookup table. So what I am doing is
Reading Lookup data using OLE DB Source
Reading my csv file
Sorting both inputs by the same field
Doing a left join by Id, in order to get the SK
This way, if there is no match (aka can't find the surrogate key) I can redirect that to a rejected csv and handle it later.
something like this:
(sorry for the spanish!)
I am doing this for each dimension, so I can handle each one with different error codes.
Since OriginStationId and DestinyStationId are two values from the same dimension (they both match against the same lookup table), I wanted to know if there's a way to avoid reading two times the data from the table (I mean, not to use two ole db sources to read twice the data from the same table).
I tried adding a second output to the sort but I am not allowed to. The same goes to adding another output from OLE DB Source.
I see there's an "cache option", is the best way to go ? (Although it would impy creating anyway another OLE DB source.. right?)
The third option I thought of was joining by the two fields, but since there is only one field in the lookup table (the same field) I am getting an error when I try to map both colums from my csv against the same column in my Lookup table
There are columns missing with the sort order 2 to 2
What is the best way to go for this ?
Or I am thinking something incorrectly ?
If something was not clear let me know and I'll update my question
Any time you wish you could have multiple outputs from a component that only allows one, all you have to do is follow that component with the Multicast component, whose sole purpose is to split a Data Flow stream into multiple outputs.
Gonzalo
I have just used this article on how to derive columns for a data warehouse building:- How to Populate a Fact Table using SSIS (part 1).
Using this I built a simple package that reads a CSV file with two columns that are used to derive separate values from the same CodeTable. The CodeTable has two fields Id and Description.
The Data Flow has two "Lookup" tasks. The first one joins the attribute Lookup1 against the Description to derive its Id. The second joins the attribute Lookup2 against the Description to derive a different Id.
Here is the Data Flow:-
Note the "Data Conversion" was required to convert the string attributes from the CSV file into "Unicode string [DT_WSTR]" so they could be joined to the nvarchar(50) description attribute in the table.
Here is the Data Conversion:-
Here is the first Lookup (the second one joins "Copy of Lookup2" to the Description):-
Here is the Data Viewer output with the to two derived Ids CodeTableFirstId and CodeTableSecondId:-
Hopefully I understand your problem and this is of use to you.
Cheers John

Importing excel file to access and set up columns field name

I'm having an access tool where I'm importing an excel file with table information. The system is creating a new table with this info with column fields (F1,F2,F3, etc.) and under it there is 10 lines with data and after that a table. I need the information from this table to be appended in another table in Access. I'm having the code and the append query, but sometimes some of the columns in excel file are change their places and this is a problem for my table 2. I would like to ask you is it possible somehow to change automatically the nameing of the column fields in the first table when I'm importing the info from the excel sheet.
Thank you in advance! - Here is a screenshot. The yellow one to go to the grey one.

Using Multi Cast transformation for destinations with different columns

What is the use of Multicast Transformation Task ? With this task, is it possible to send to two destinations from a single source, while each destination has different columns ?
I assume that you are referring to Multicast Transformation inside the Data Flow task. If so, yes it is possible. The purpose of the transformation is to channel data from a single source to n number of Transformation tasks or Destinations.
If source has following columns
Source
Column 1
Column 2
Column 3
and destinations have these columns.
Destination 1 Destination 2
Column 1 Column 2
Column 3
Both destinations will be able to see Columns 1 - 3 that are available in Source. You have to map the columns accordingly in the respective destinations. Refer below example:
Example:
Screenshot #1 shows that Source has two columns Header and Value.
Screenshot #2 shows that Destination 1 has both columns Header and Value. They are mapped accordingly.
Screenshot #3 shows that Destination 2 has only column Header. It is mapped accordingly.
Screenshot #4 shows sample package execution.
Hope that helps.
Screenshot #1:
Screenshot #2:
Screenshot #3:
Screenshot #4:
#Siva did a good job of explaining the how. I'm going to tackle the "What is the use of Multicast Transformation Task?" question.
Let me give you examples of how I have used it or seen it used. First, we like to store the data in a staging table that contains just the raw unchanged data (this makes it easier for us to research data issues to see if the data problem came from a bug in our process or bad data sent by the client.) and at the same time I want to send the same data to another staging table that will be used to transform the data.
Sometimes we use Mulitcast to take denormalized files and send them to normalized data tables. So the names go to the person table, the addresses go to the address table and the phones go to the phone table.
Multicast can be used to do several different transformations on different data fields in the same source at the same time rather than one at a time and then bring all the revised data back together in a Merge join. So one path checks the States to make sure they are valid or converts the long names to the 2 character abbreviations and another checks the zip codes and adds the leading zeros that got lost because the data came from an Excel file. Then the cleaned address data is put back together with the correct values we want for insertion to our database. This can speed up cleaning as data is being scrubbed simultaneously not one step at a time.