I have following ONE dataset with additional columns not mentioned for ease of reading,
Address1_PostCode Address2_PostCode
Address1_Line1 Address2_Line1
Address1_Line2 Address2_Line2
Address1_Country Address2_Country
Desired output is,
Address2_PostCode
Address2_Line1
Address2_Line2
Address2_Country
Here is what I am trying to do,
If any of Address1 field has data then overwrite Address2 fields, e.g. if Address1 only has postcode and Address2 has Country, the final result will have only postcode and country will be empty or null
If all fields of Address1 are empty then don't do anything
I already searched on my own and I can see there is functionalities like replace column to add new and replacenull function, but I am not able to understand them enough to achieve my goals
Add a Derived Column transformation that has a new column expression with a boolean result that will check whether all Address1 fields are empty/null or not.
Add another Derived Column transformation after that one (unless you want to repeat the validation expression on each column) that checks for this bool result and reassigns the proper field from either Address1 or Address2, on each field you need.
On this last step you can either add new columns to the flow or overwrite existing ones, just make sure that you are using the ones being checked.
I would suggest an alternate data model which allows you to keep your original data intact.
Add a multicast (this will allow the data to duplicate)
Have one flow handle the normal non-address flow
Add a new flow to handle person address data.
Add a script task (this will be used to normalize the addresses
Mark the key as input and all the address columns as input
Create a new output (call it Address) with PersonKey, AddressType, AddressLine1, AddressLine2, PostalCode, Country
Add this simple code.
AddressOutputBuffer.AddRow();
AddressOutputBuffer.PersonKey = Row.PersonKey;
AddressOutputBuffer.AddressType = "Address1";
AddressOutputBuffer.AddressLine1 = Row.Address1_Line1;
... (Add the rest in here)
AddressOutputBuffer.AddRow();
AddressOutputBuffer.PersonKey = Row.PersonKey;
AddressOutputBuffer.AddressType = "Address2";
AddressOutputBuffer.AddressLine1 = Row.Address2_Line1;
... (Add the rest in here)
Write this new Person Address info to a new table (you can build whatever logic you want to the queries you write or you can create a view to handle your specific logic.)
NOTE: You may need to null handle the code above
Ex:
AddressOutputBuffer.AddressLine1 = !Row.Address1_Line1_IsNull?Row.Address1_Line1:"";
Related
I am trying to accomplish something that is pretty easy to do in SQL, but seemingly very challenging to do in SSIS without using SQL. Basically, I need to consolidate and concatenate a field of a many-to-one relationship.
Given entities: [Contract Item] (many) to (one) [Account]
There is a field [ari_productsummary] that contains the product listed on the Contract Item entity. We want to write that value to the Account as [ari_activecontractitems]. However, an Account may have more than one Contract Item record associated to it, in which case, we want to concatenate those values. We also only want the distinct values to be concatenated (distinct rows already solved within my data flow).
This can be accomplished by writing to a temporary table, and then using a query or view to obtain the summarized results as followed. I created a SQL table called TESTTABLE that contains the [ari_productsummary] from the Contract Item entity along with the referring [accountid] to map it back to Account. I then wrote the following query as a view:
SELECT distinct accountid,
(SELECT TT2.ari_productsummary + '; '
FROM TESTTABLE TT2
WHERE TT2.accountid = TT.accountid
FOR XML PATH ('')
) AS 'ari_activecontractitems'
FROM TESTTABLE TT
Executing that Query provides me the results that I want, which I can then use for importing into the Account entity as shown below:
But how do I do this in a SSIS dataflow without writing to a SQL table as a temporary placeholder for the data?? I want to do the entire process inside one dataflow container, without using a temporary SQL table/view. The whole summarization process needs to be done on the fly:
Does anyone have a solution that doesn't require a temporary SQL table/view/query, but is contained entirely within the data flow?
I am using VS 2017 and the KingswaySoft Dynamic CRM 365 ETL toolset to develop my solution/package.
Spit balling here as I don't Dynamics nor do I have the custom components.
Data Flow 1 - Contract aggregation
The purpose of this data flow is to replicate your logic in the elegant query you provided and shove that into a Cache Connection Manager (see Notes for 2008+ at the end)
KingswaySoft Dynamics Source -> Script Task -> Cache Transform
If you want to keep the sort in there, do it before the script task. The implementation I'll take with the Script Task is that it's fully blocking - that is all the rows must arrive before it can send any on. Tasks like the Merge Join are only partially blocking because the requirement of sorted data means that once you no longer have a match for the current item, you can send it on down the pipeline.
The Script Task is going to be asynchronous transformation. You'll have two output columns, your key accountid and your new derived column of ari_activecontractitems. That column will might need to be big - you'll know your data best but if it's a blob type in Dynamics (> 4k unicode or > 8k ascii characters) then you'll have to define the data type as DT_TEXT/DT_NTEXT
As inputs, you'll select accountid and ari_productsummary from your source.
The code should be pretty easy. We're going to accumulate the inbound data into a Dictionary.
// member variable
Dictionary<string, List<string>> accumulator;
The PreProcess method, we'll tack this in there to initialize our variable
// initialize in PreProcess method
accumulator = new Dictionary<string, List<string>>();
In the OnBufferRowSent (name approx)
// simulate the inbound queue
// row_id would be something like Rows.row_id
if (!accumulator.ContainsKey(row_id))
{
// Create an empty dictionary for our list
accumulator.Add(row_id, new List<string>());
}
// add it if we don't have it
if (!accumulator[row_id].Contains(invoice))
{
accumulator[row_id].Add(invoice);
}
Once you get the signal sent of no more data available, that's when you start buffering output data. The auto generated code will have placeholders for all this.
// This is how we shove data out the pipe
foreach(var kvp in accumulator)
{
// approximately thus
OutputBuffer1.AddRow();
OutputBuffer1.row_id = kvp.Key;
OutputBuffer1.ari_productsummary = string.Join("; ", kvp.Value);
}
We have an upcoming release that comes with a component that does exactly what you are trying to achieve without the need of writing custom code. The feature is currently under preview, please reach out to us for private access to the feature. You can find our contact information on our website.
UPDATE - June 5, 2020, we have made the components available for public access at https://www.kingswaysoft.com/products/ssis-productivity-pack/ as a result of our 2020 Release Wave 1. We have two components available that serve this kind of purpose. The Composition component will take input values and transform into a composite value in a SSIS column. The Decomposition component does the opposite, it would take an input value and split it into multiple rows using either delimiter-based text splitting or XML/JSON array splitting.
I'm trying to define a LOV with Universe Designer that when used in a #Prompt shows the user the item description, but gives back the corresponding item key as result of user selection.
e.g. From a dimension table of countries with two columns as COUNTRY_ISO and COUNTRY_NAME, I would like to show COUNTRY_NAME values to user with #prompt, but get back the corresponding COUNTRY_ISO as return value of #prompt.
This is possible using Index Awareness. I'll describe the solution using the sample "Island Resorts Marketing" universe, but the concept should map to your specific need.
For this example, we'll create a prompt on the "Country" object in the "Resort" class. The prompt will display the country names, but the condition will apply to the country_id field.
First, we need to identify the keys. On the Keys tab of the Resort\Country object, add a new Primary Key using Resort_Country.country_id as the source:
(in your case, you'll do this on the COUNTRY_NAME object, using COUNTRY_ISO as the primary key)
Next, we create the predefined condition that will include the prompt. Create a new Predefined Condition, named "Select country", with the following definition:
Resort_Country.country_id = #Prompt('Choose a country','A','Resort\Country',Mono,primary_key)
What we're doing is is applying the condition to the ID field (resort_country.country_id), but using the country name (the Country object) as the source. Note the "primary_key" parameter -- this tells BO to use the PK from the referenced object.
Now to test. Create a WebI report and select the "Select country" filter along with a field to display. I chose Service Line:
I run the report and am presented with a list of country names. I choose France, and I get a list of the Service Lines associated with France. The SQL generated by this query, reflecting my prompt selection, is:
SELECT
Service_Line.service_line
FROM
Service_Line,
Country Resort_Country,
Resort
WHERE
( Resort_Country.country_id=Resort.country_id )
AND ( Resort.resort_id=Service_Line.resort_id )
AND ( Resort_Country.country_id = 2 )
Note the very last line, which shows that the condition is applied using the ID, even though I selected the country name "France".
References:
- XI3.1 Designer Guide #Prompt info, starting on page 520
- Dave's blog on Index Awareness
- Dave's blog on Index Awareness with prompts
I have been struggling with an issue for a couple of days. I am sharing a Content Provider with two different apps (app A and app B). All the stuff regarding DB creation and Content Provider management is done by app A. App B just accesses it using the corresponding Authorities and a Content Provider Client.
ContentProviderClient myCPClient = this.miContext.getContentResolver().acquireContentProviderClient(this.miUri);
The problem comes up when trying to query the database in a more complex way, i.e. using some key words like GROUP BY, HAVING, etc. I need to get unique references according to one specific column (I want to use GROUP BY), and I have found out that there is no rawQuery() method for a ContentProviderClient, but a simplified query() method (compared to the one available in the class SQLiteDatabase, which allows to formulate proper MySQL commands).
I have checked this answer, but since my ContentProvider is accessed from a different app, I do not have any class like MyContentProvider.
To sum up, is there any way to make a proper query (like rawQuery()) to a ContentProvider which was generated by a different app?
I have finally got to a solution which is rather simple and sensible. I got a very good explanation about Content Providers and Content Resolvers. The latter is used to access the former, which means that they can not control what is in the provider, but get data from them. This means that you can not make a Content Provider Client to use a rawQuery() if it is not implemented (override) in the query() method of the corresponding ContentProvider.
To work around my problem, I have used a flag in my provider client and modify my content provider to read it so I can make use of GROUP BY. I just wanted to get unique references from the database according to a particular column.
Here it is the solution, which is not a very clean one, but it works quite well.
For the ContentProviderClient,
ContentProviderClient myCPClient = this.miContext.getContentResolver().acquireContentProviderClient(this.miUri);
//I declare some variables for the query
//'selection' will get all the rows whose "_id" is greater than 0, i.e. all the rows
String selection = BaseDatosParam.Tabla._ID + ">?";
String[] selectionArgs = {"0"};
//'groupBy' is not formatted in any particular way. I just need it to contain the pattern "GROUP BY"
String groupBy = "GROUP BY" + BaseDatosParam.Tabla.REF;
//the last field of the query corresponds to 'sortOrder', but I
Cursor c = myCPClient.query(Uri.parse(miUri.toString()),
projection, selection, selectionArgs, groupBy);
In the ContentProvider,
public Cursor query(Uri uri, String[] projection, String selection, String[] selectionArgs, String sortOrder)
{
String where = selection;
String groupBy = null;
SQLiteDatabase db = this.miBDManager.getWritableDatabase();
//We just check out whether 'sortOrder' includes the pattern "GROUP BY", otherwise that field will remain null
Pattern myPat = Pattern.compile("GROUP BY");
Matcher myMat = myPat.matcher(sortOrder);
if (myMat.find())
groupBy = myMat.replaceFirst("");
Cursor c = db.query(BaseDatosParam.Tabla.NOMBRE_TABLA, projection, where, selectionArgs, groupBy, null, null);
c.setNotificationUri(this.getContext().getContentResolver(), uri);
return c;
}
Regards,
Suppose you have a complete user entry with al details like department, company name, job title, some custom fields, etc.
How should we delete a field or leave it blank; for example, I if I try to leave blank company name to delete it, I get a "Cannot call method "setCompanyName" of null, so how should deletion of this information should be managed?
With single value fields like Department, CompanyName etc you can set the value to an empty string (not null) to remove the existing value. For example:
profile.setCompanyName('');
For custom fields, use .deleteCustomeField() and fields for which there can be multiple instances use the provided delete methods like .deletePhoneField(). Use them like this:
var phones = profile.getPhones();
for (var i in phones) {
phones[i].deletePhoneField();
}
The ProfilesApp Tests script has examples of all these in unit tests.
Say I have an entity Customer which has relationship with city, order etc. Now when I am adding a customer object, should I assign customer.cityid, or customer.city? From the form I get cityid from a dropdown, so to assign the city object, I will have to make a query using id selected.
If you need the City object populated, then get the City and set .City.
If you don't need the City object and are just saving and moving on, setting .CityId without getting the city object will save you the select query fetching it.
In either case, the next time the object loads, City will be available. (Both methods save the same City column in the database unless you have something strange/non-default setup).