I'm using a web service as my XML datasource which returns entities including multiple hierarchical data like below (codes modified for project privacy);
public class UserData {
// some scalar properties
public string Id ...
public string Name ...
public string Surname ...
// some navigational properties
public Address address ...
public CourseInfo[] courses ...
public AwardInfo[] awards ...
}
To show all the entity information inside the same report, I can create different datasets in my report to call my web service and filter preferred parts of the returned service response.
To get only UserData properties:
<Query>
<Method Name="GetUserData" Namespace="UserApp.ReportingServices"/>
<SoapAction>UserApp.ReportingServices/IReportingService/GetUserData</SoapAction>
<ElementPath IgnoreNamespaces="true">GetUserDataResponse{}/UserData</ElementPath>
</Query>
To get Address information together with UserData properties:
<Query>
<Method Name="GetUserData" Namespace="UserApp.ReportingServices"/>
<SoapAction>UserApp.ReportingServices/IReportingService/GetUserData</SoapAction>
<ElementPath IgnoreNamespaces="true">GetUserDataResponse{}/UserData/Address</ElementPath>
</Query>
To get course information together with UserData properties:
<Query>
<Method Name="GetUserData" Namespace="UserApp.ReportingServices"/>
<SoapAction>UserApp.ReportingServices/IReportingService/GetUserData</SoapAction>
<ElementPath IgnoreNamespaces="true">GetUserDataResponse{}/UserData/Courses/CourseInfo</ElementPath>
</Query>
My question raises right here: If I use multiple dataset queries like above, my report will make a web service call for every dataset it has though my service always returns the same XML response which includes all the data needed for the three datasets above.
Is there any way to re-use returned XML response of a query for the other datasets? If it would possible, my report would call the web service only one time, and datasets would filter different parts of that XML response without calling the web service again and again.
If this is not possible, than what's the best practice about showing multiple hierarchical data inside the same report? Do I have to create different web services returning different hierarchical parts of the data?
Reporting Services datasets are limited to a simple table of rows and columns - they can't handle data with multiple hierarchies as you have described.
I would redesign your web service to reflect this, probably splitting it into the three sets of data you described. Then there will be three web service calls, but without the duplication of content. They will also be executed in parallel which will probably be more efficient overall than your current design.
Related
I am trying to accomplish something that is pretty easy to do in SQL, but seemingly very challenging to do in SSIS without using SQL. Basically, I need to consolidate and concatenate a field of a many-to-one relationship.
Given entities: [Contract Item] (many) to (one) [Account]
There is a field [ari_productsummary] that contains the product listed on the Contract Item entity. We want to write that value to the Account as [ari_activecontractitems]. However, an Account may have more than one Contract Item record associated to it, in which case, we want to concatenate those values. We also only want the distinct values to be concatenated (distinct rows already solved within my data flow).
This can be accomplished by writing to a temporary table, and then using a query or view to obtain the summarized results as followed. I created a SQL table called TESTTABLE that contains the [ari_productsummary] from the Contract Item entity along with the referring [accountid] to map it back to Account. I then wrote the following query as a view:
SELECT distinct accountid,
(SELECT TT2.ari_productsummary + '; '
FROM TESTTABLE TT2
WHERE TT2.accountid = TT.accountid
FOR XML PATH ('')
) AS 'ari_activecontractitems'
FROM TESTTABLE TT
Executing that Query provides me the results that I want, which I can then use for importing into the Account entity as shown below:
But how do I do this in a SSIS dataflow without writing to a SQL table as a temporary placeholder for the data?? I want to do the entire process inside one dataflow container, without using a temporary SQL table/view. The whole summarization process needs to be done on the fly:
Does anyone have a solution that doesn't require a temporary SQL table/view/query, but is contained entirely within the data flow?
I am using VS 2017 and the KingswaySoft Dynamic CRM 365 ETL toolset to develop my solution/package.
Spit balling here as I don't Dynamics nor do I have the custom components.
Data Flow 1 - Contract aggregation
The purpose of this data flow is to replicate your logic in the elegant query you provided and shove that into a Cache Connection Manager (see Notes for 2008+ at the end)
KingswaySoft Dynamics Source -> Script Task -> Cache Transform
If you want to keep the sort in there, do it before the script task. The implementation I'll take with the Script Task is that it's fully blocking - that is all the rows must arrive before it can send any on. Tasks like the Merge Join are only partially blocking because the requirement of sorted data means that once you no longer have a match for the current item, you can send it on down the pipeline.
The Script Task is going to be asynchronous transformation. You'll have two output columns, your key accountid and your new derived column of ari_activecontractitems. That column will might need to be big - you'll know your data best but if it's a blob type in Dynamics (> 4k unicode or > 8k ascii characters) then you'll have to define the data type as DT_TEXT/DT_NTEXT
As inputs, you'll select accountid and ari_productsummary from your source.
The code should be pretty easy. We're going to accumulate the inbound data into a Dictionary.
// member variable
Dictionary<string, List<string>> accumulator;
The PreProcess method, we'll tack this in there to initialize our variable
// initialize in PreProcess method
accumulator = new Dictionary<string, List<string>>();
In the OnBufferRowSent (name approx)
// simulate the inbound queue
// row_id would be something like Rows.row_id
if (!accumulator.ContainsKey(row_id))
{
// Create an empty dictionary for our list
accumulator.Add(row_id, new List<string>());
}
// add it if we don't have it
if (!accumulator[row_id].Contains(invoice))
{
accumulator[row_id].Add(invoice);
}
Once you get the signal sent of no more data available, that's when you start buffering output data. The auto generated code will have placeholders for all this.
// This is how we shove data out the pipe
foreach(var kvp in accumulator)
{
// approximately thus
OutputBuffer1.AddRow();
OutputBuffer1.row_id = kvp.Key;
OutputBuffer1.ari_productsummary = string.Join("; ", kvp.Value);
}
We have an upcoming release that comes with a component that does exactly what you are trying to achieve without the need of writing custom code. The feature is currently under preview, please reach out to us for private access to the feature. You can find our contact information on our website.
UPDATE - June 5, 2020, we have made the components available for public access at https://www.kingswaysoft.com/products/ssis-productivity-pack/ as a result of our 2020 Release Wave 1. We have two components available that serve this kind of purpose. The Composition component will take input values and transform into a composite value in a SSIS column. The Decomposition component does the opposite, it would take an input value and split it into multiple rows using either delimiter-based text splitting or XML/JSON array splitting.
I am making a simple REST API in front of a NoSQL database that stores records as documents similar to JSON (but not exactly the same). Each record has some fields, including id for the database, and also including some derived fields, like dateCreated.
Any time I GET anything, I want to return the objects with all the fields.
// GET /users returns an array of these in JSON
// [{id:"xxx", name:"Bobby", dateCreated:"YYYY-MM-DD"]
data User = User { id :: String, name :: String, dateCreated :: XXX }
But any time I POST or PUT anything, they client should send an object with the id field and any derived fields missing. The database is responsible to create the id when I save it, and I would create some derived fields
// POST /users would need you to post only the name.
// {name:"Henry"}
data PartialUser = PartialUser { name :: String }
If resource represents objects of type User, what should I call the thing client is sending to me? Would you make all the derived fields Maybe values? Or would you create a second object called PostedUser or something?
It can be many things:
a request body
the representation of the intended resource state of the client
a command DTO which you can send to the domain logic in order to process it by CQRS
I would make it CreateUser command, but if you don't want to use CQRS and DDD, then you would probably call it as PartialUserRepresentation, or you don't create a data structure, just use the properties to create a new User entity. Ofc. if you use entities.
So I would say it depends on the architecture of your system.
I am designing a Web Application using Django Framework. I have written the model code, urls.py and view code which can be seen Here.
I have added some data into the database table. But when I try to access the object using the code below, it just shows bookInfo objects five times. I don't think I am successful enough in pulling the data from the database. Kindly help.
View
def showbooks(request):
booklist = bookInfo.objects.order_by('Name')[:10]
output = ','.join([str(id) for id in booklist])
return HttpResponse(output)
You are iterating through the object list, you just need to reference the column/attribute you want:
output = ','.join([obj.id for obj in booklist])
Alternatively you can more more finely craft you original db call, then the iterable you use will work. In this case we'll pull out a list of the 'id' attribute.
booklist = bookInfo.objects.order_by('Name').values_list('id', flat=True)[:10]
output = ','.join([id for id in booklist])
I think you are successful in pulling the data. It is just that booklist contains objects, not numeric ids. You can add __unicode__ method to you class BookInfo that is supposed to return a string representation of the object (probably book name in this case). This method is going to be invoked when str() is applied. You can find more info about __unicode__ here.
I'm working on a C# program that retrieves data from a ServiceNow database and converts that data into C# .NET objects. I'm using the JSON Web Service to return my data in JSON format.
What I want to achieve is as follows: If there is a relational mapping between a value (for
example: I have a table called Company, where CEO is not a TEXT field but an sys_id to a Employee Table) I want to be able to output that data not with an sys_id (or just displaying the name property by using the 'displayvariable' parameter) but by an object displayed in JSON.
This means that the value of a property should be an object in JSON instead of just a single value.
A few examples:
// I don't want the JSON like this
{"Company":{"CEO":"b181e841c9212c008aeb36850331fab2"}}
// Or by displaying the name of the sys_id table
{"Company":{"CEO":"James Henderson" }}
// I want the data as follows, so I can have all the data I need inside a single JSON record.
{"Company":{"CEO":{"name":"James Henderson", "age":34, "sex":"male", "office":"SBN Left Floor 23"}}}
From reading the documentation I couldn't find anything in the JSON Web Service that allowed me to display the information like this nor
find any other alternative. It should have something to do with joining the tables and displaying it all in the right format.
I have been using SNC for almost three years and have not found you can automatically join tables in a web service. Your best option would be to use a scripted web service which possibly takes a query parameter and table parameter. Then you can json serialized your result as you see fit.
Or, another option would be to generate a new processor that will traverse the GlideRecord object. The ?JSON parameter you pass in to the URL is merely a flag to pass your request to a particular processor. Unfortunately the OOB one I believe is a Java class not a JS script, so you would need to write a script much like I mentioned earlier to traverse the object path serializing the object graph as far down as your want to go.
I have 2 classes named User.groovy and Employee.groovy and I used MYSQL to save the data. What I want is to create a new User account and save it to the User table and also save some of the data to Employee table. How can I do this? I've tried extending the user to Employee but the data only saved to User and not to Employee. But If I don't extend the User, the data is only saved to Employee. What should I do so that the data simultaneously saves to two database tables at the same time? Please help me.
Actually have this in my class user:
class User {
transient springSecurityService
String username
String password
boolean enabled
boolean accountExpired
boolean accountLocked
boolean passwordExpired
.....}
and employee:
class Employee {
String name
String email
String jobDesc
....}
So what should I do next? I'm sorry for this, I'm still starting to learn grails.
Grails paradigm (as far as scaffolding is concerned) is one form - one object. As long as you stick to this paradigm, you get all the goodies, such as input validation and error reporting for free (you may also consider using the Fields plugin here http://grails.org/plugin/fields).
However, sometimes you need to collect info and create two or more objects through single form. Usually this happens when you need to initiate new subscription and collect info for both subscription details (say, Subscription entity) and user info (User entity). This is where command objects come to rescue.
http://grails.org/doc/latest/guide/theWebLayer.html#commandObjects
So, instead of expanding/bending SubscriptionController or UserController (or UserController and EmployeeController, as per your example), you create SignUpController, which handles SignUpCommand object. The SignUpCommand object is not intended to be saved, it is used as a backing object for the SignUpController.create form. When it validates, you use the signUpCommand object data to initialize 2 domain objects (that is Subscription and User) and save these objects individually within the same transaction.
You can either delegate the save operation to a service say,
if (signUpCmd.validate()) {
SignUpService.save(signUpCmd))
}
or create and save both objects right on the spot within controller
if (signUpCmd.validate()) {
Subscription subscription = new Subscription(plan: signUpCmd.plan, ...)
subscription.save()
User user = new User(username: signUpCmd.username, ...)
user.save()
}
it is mostly matter of taste and style.
Instead of calling save() directly to your user instance, call a service class that saves both the user and the employee in one atomic operation. Like, for instance:
class UserController {
/*Injection of your service in the controller class*/
def userService
And then in the save action in this same controller:
userService.save(user) // userService.save(params)
And inside this service method you will extract the data (user or params, whatever floats your boat) you want to save in a different table as long as the usual user object.