Within Microsoft Dynamics CRM 2015, is it possible to create A Duplicate Detection Job which runs only for a single Duplicate Detection Rule (or a subset of the available rules) for a given Entity?
I am trying to identify duplicates which exist on the basis of a specific rule only, so any alternative solution would also be welcome, for example if there is some way to filter the results of a Duplicate Detection Job on the basis of which rule(s) caused a particular record to appear in the results.
Ideally I'd like a UI-only solution to this, but would consider a code-based approach if that is the only way.
Related
Is it possible to materialize the edits made as part of a scenario into a dataset in foundry?
I want for each scenario to write out the primary keys of the objects edited as part of the scenario.
The motivation is that I need to run multiple processes to compute metrics as part of the changed values for each scenario, at a scale and runtime that is not possible to do with Functions.
Edit with details:
The thing is that I am not doing actual edits to the objects for the object type, I don't want to apply it.
I tested out the "Action Log" and it does not seem like this picks up "uncommitted" actions, meaning actions that is just run as part of a scenario. Also, it does not seem to be a link to the scenario it was a part of, even if the changes were committed.
The workflow is that I have Object Type A, and I define multiple scenarios S on a subset of the objects in A.
Each scenario might make something like 50k edits to a subset of A, through multiple Actions backed by a Function.
I save some of the scenarios. Now I am able to load these scenarios and "apply" them on A again in Workshop.
However I need to be able to get all the primary keys, and the edited values of A materialized into a dataset (for each scenario), as I need to run some transformation logic to compute a metric for the change as part of each scenario (at a scale and execution time not possible in Functions).
The Action Log did not seem to help a lot for this. How do I get the "edits" as part of a saved scenario into a dataset?
The only logic you can run BEFORE applying will be functions.
Not sure about your exact logic but Function's Custom Aggregations can be very powerful: Docs here
this might not directly let you calculate the Diff but you could use the scenario compare widgets in workshop to compare your aggregation between multiple Scenarios
e.g. you have a function that sums(total profit)
Your Workshop could show:
Current Data:
$10k
Scenario A:
$5k
Scneario B:
$13k
instead of like:
Scenario A:
-$5k
Scenario B:
+$3k
Afaik there's no first class way of doing this (yet).
"Applying" a scenario basically means you're submitting the actions queued on the scenario to the ontology. So neither the actions nor the ontology are aware that they came from a scenario.
What I've been doing to achieve what you're working on is using the "Action Log". It's still in Beta so you might need to ask for it to be enabled. It will allow you on each action to define a "log" object to be created that can track the pks of your edited objects per Action.
How I do the action log is:
My Action log has the "timestamp" of the action when they were run.
My Scenarios have the "timestamp" of when it was applied.
Since "Applying a Scenario" means -> actually running all actions on Ontology (underlying data) this gets me this structure if I sort everything by timestamp:
Action 1
Action 2
Scenario A applied
Action 3
Action 4
Scenario B applied
this allows you to do a mapping later on of Action 1/2 must come from Scenario A and Action 3+4 come from Scenario B.
EDIT: Apparently you might be able to use the Scenario RID directly in the Action Logs too (which is a recent addition I haven't adopted yet)
This won't allow you tho to compute (in transforms...) anything BEFORE applying a scenario tho
I'm trying to replicate a report from old Oracle Discoverer to OBIEE.
The issue is that when I had the "pivot table prompts" to the OBIEE analysis. I'm left with duplicates instead of unique values in the drop down field. Can anyone explain how I can keep this from occurring? This only happens when I add three or more attributes/columns to the "pivot table prompts" section. This worked in Oracle Discoverer….
Be sure, for the physical connection you have defined, that you have checked the below option:
Also, I know there was a bug around this sort of functionality occurring if the column properties were set to repeat, but that is not a common thing people do, and is probably not your scenario, but it is worth a look: Bug 13147411 : 11G PROMPT IS DISPLAYING REPEATING VALUES WHEN ITEM PROPERTIES SET TO REPEAT
As mentioned by rmoff, incorrect keys can often lead to this issue. As he said, you can check the nqquery.log to see the physical SQL, or enter SET VARIABLE LOGLEVEL=4; in the SQL prefix on the advanced tab, then run your report again. This will generate a log in the admin pane of the tool that you can check easily.
The dropdown list is built from the dimension you have defined, so if your key is incorrect you'll get duplicates. Check the physical SQL in nqquery.log if in doubt to see what SQL OBIEE is generating that returns the duplicates, and make sure you have defined your logical dimension correctly.
The issue was resolved by moving "Repeat" to "Suppress" for the individual columns that I was prompting on in the pivot!
We did this a lot throughout my department because we use the exports for further manipulation rather than using OBIEE only for high level overview and summation. Each row must hold the value in order to filter, so the "Repeat" functionality is often selected.
Thanks for the help/answer!!!
I was told about pre-filtering when writing reports in CRM 2011 and it sounded like something I should add to my knowledge base so I Googled it and found:
MSDN
PowerObjects
and a whole bunch more. These gave me a good understanding of the how for both automatic and explicit. The one thing I missed though was why. I mean in general the articles said things like context specific or reduce the size of the query but I failed to translate that into the nuts and bolts business case example. For example one of the articles stated that it will greatly enhance your reports but what does that mean to the person in sales running the report.
In other words I could easily add CRMAF_ to my queries but how it makes sales reports better I couldn't explain. Note I used better as that is the word the author of one article used so I took that to mean extra capability not better performance...?
So a quick business case understanding would be great.
Thank You
JB
It's very easy to make an example.
You need to create a simple report to display a list of accounts created starting from a selected date.
You just do a SQL Query as
SELECT * FROM FilteredAccount WHERE createdon => #selected_date
You publish your report and everyone is happy.
One month later there is a new requirement and you need change the report because is necessary to filter also by country (or city, or a custom field)
Would you prefer to edit the report, adding parameters (and change the queries too) or enable pre-filtering and allow the user to apply some filters (based on fields that maybe will be added in the future) before your report runs?
I am writing the SSIS package to import the data from *.csv files to the SQL 2008 DB. The problem is that one of the file contains the duplicate records in the csv file and I want to extract only the distinct values from that source. Please see the image below.
Unfortunately, the generated files are not under my control and it is owned by the third party and I could not change the way they generated.
I did use the LookUp Component. But it only checks the existing data against the incoming data. It does not check the duplicate records in the incoming data.
I believe the sort component gives an option to remove duplicate rows.
Depends on how serious you want to get about the duplicates. Do you need a record of what was duplicated or is it enough to just get rid of them? Sort component will get rid of dups on the sort field. However, the dups may have different data in the other fields and then you want a differnt strategy. Usually I load all to staging tables and clean up from there. I send the dupes removed to an exception table (we have to answer a lot of questions from our customers about why things don't match what they sent) and I often use a set of business rules (and use either an execute SQl or data flow tasks to enforce the rules) to determine which one to pick if there are duplicates in one area but not another (say two business addresses when we can only store 1). I also make sure the client is aware of how we determine which of the two to pick.
Use SORT tool for that from Toolbox, then click on it. You will get all available input columns.
Check the column and change sortType direction and then check "remove rows with duplicate sort value".
Bring in the data from the csv file the way it is, then dedup it after it's loaded.
It'll be easier to debug, too.
I used Aggregate Component and Group By both QualificationID and UnitID. If you want, you can also use Sort Component too. Perhaps, my information might help others.
I am doing an SSIS look up transformation, looking up in a voyages table, however some of my records don't have voyages, so I get errors. Is there any way that I can skip the look up on those records.
To expand on unclepaul84's answer, you can configure your lookup component to perform one of three actions on a failed lookup.
Fail Component (the default and the action you have now from your question. Fails the job step (and maybe the entire package) when there are no matches to the row in a lookup attempt.)
Ignore Failure (Doesn't fail your job step, leaves a null in the field you brought in from the lookup i.e. Voyage name? )
Redirect Row (Doesn't fail your job step, allows you to direct rows with no voyage to a different processing flow for handling (i.e. if you want to put a default 'No Voyages' message in your Voyage Name field))
Alternatively, as John Saunders mentioned in his comment, you could test the VoyageID column and split your data flow into two paths depending upon if the VoyageID column is null. Since the Lookup component can handle this, I prefer using the single lookup rather than a conditional split followed by a lookup on one of the paths.
You could tell the lookup of component to ignore lookup failures.