I am new to SSIS. I want to basically create tasks in SSIS to do the following
Create a task to pull data from a table
Create a task to call Webservice. Basically I need to call the webservice couple of times to get the job done. Following are the steps.
a. Call a webservice to get a session token
b. Call a webservice to get a session service token by passing session token that you get in step 1
c. Call a webervice method by passing the service token as well as uploading the data.
d. Call a webservice to end the session
What is the best possible way to do it?
You create a data flow task for 1 to get the data. Then for 2, you can do that in a script task for a and b, you can use either c# or vb. This is pretty much just programming. Then step c, it's not very clear what you do here with the data. Potentially you use a data flow task connected to the script task you created for step a and b. In the data flow you do the uploading. Then you create another script task for step d and it connects to the data flow task you created for step c.
Related
I searched but didn't find anything specific to the issue I was having. My apologies if I overlooked it.
I have a scenario where I'm pulling data from an oData source and everything works fine. I have the oData source in a loop where I loop over each of our different companies and pull the data for that company. I'm doing this to reduce the volume of data that is being returned in each oData call.
Everything works fine as long as there is data being returned from the oData call. In the attached photo you can see that the call is being made and data is being returned.
But when the oData Service is called with parameters/filters that return no data I get the following and that is when the VS_NEEDSNEWMETADATA error is thrown. The image below shows what I get when no data is returned.
So the issue isn't that I have invalid metadata due to changes made to the service (adding/removing fields). It's that nothing is being returned so there is no Metadata. Now it's possible that this is an issue with the system that I'm pulling the oData from (SAP S4) and they way that system surfaces the oData call when there is no data??
Either way, trying to figure out a way to handle this within SSIS. I tried to set validate external metadata = false but the package still fails. I could also fix this by excluding those companies in the script but then once data does exists I'd have to remember to update the scripts and redeploy.
Ideas suggestions?
You've hit the nail on the head except your metadata is changing in the no data case because the shape is different - at least from the SSIS engine perspective.
You basically need to double invoke the OData source. First invocation will simply identify is data returned. If that evaluation is true, only then does the Data Flow start running.
How can you evaluate whether your OData will return data? You're likely looking at a Script Task and C#/VB.NET and hopefully the SAP data returns don't vary between this call and next call.
If that's the case, then you need to define an OnError event handler for the Data Flow, mark the System scoped variable Propagate to False and maybe even force a success status to be returned to get the loop to continue.
I send JSON's to my app via Postman in a list with a type of mapping(CRUD) to send it to my database.
I want my controller to put all this data, from multiple senders, in a list that will send the information to my DB. The problem is that i don't know how to store in the same list the Json and the Mapping, so when my threads do their work to know if that json must be inserted, updated, deleted and so on.
Do you guys have any ideea ?
PS: It is a spring-boot app that need to be able to send 12000 objects ( made from that jsons ) to the db.
I don't see a reason for putting all data in one list and sharing it later, each HTTP request receives own thread.
On decent server you can handle couple thousands of requests/sec which perform simple CRUD operations.
My problem is: i have a function triggered by a daily Timer that is supposed to send emails to a list of addresses stored in a database, even the text of the email is an array of datas retrieved by queries in a database. The function doesn't have inputs, nor outputs (if we don't consider the text of the email, that is directly sent by the function). How am i supposed to create a set of blackbox test cases for something like this?
There are at least two ways of writing tests for these code.
You can use an in-memory database and an in-memory IMAP- or SMTP-server. Add data to the database. Configure your function to use that database and the email server. Run your function and check the inbox of the email server.
You can mock the database and the emailserver and inject them into the function's class.
How can I retrieve the number of API invocations? I know the data has to be somewhere because wso2 BAM shows piecharts with similar data...
I would like to get that number in a mediation sequencel; is that possible? Might this might be achieved via a DB-lookup?
The way how API Usage Monitoring in WSO2 API Manager works is, there is an API handler (org.wso2.carbon.apimgt.usage.publisher.APIUsageHandler) that gets invoked for each request and response passing through the API gateway. In this handler all pertinent information with regard to API usage is published to the WSO2 BAM server. The WSO2 BAM server persists this data in Cassandra database that is shipped with it. Then there is a BAM Toolbox that has been packaged with required analytic scripts written using Apache Hive that can be installed on the BAM server. These scripts would summarize the data periodically and persist the summarized data to an sql database. So the graphs and charts shown in the API Publisher web application are created using the summarized data from the sql database.
Now, if what you require is extractable from these summarized sql tables then i suppose the process is very straight forward. You could use the DBLookup mediator for this. But if some dimension of the data which you need has been lost due to the summarizing, then you will have a little more work to do.
You have two options.
The easiest approach which involves no coding at all would be to write a custom Hive script that suits your requirement and summarize data to a sql table. Then, like before use a DBLookup mediator to read the data. You can look at the existing Hive scripts that are shipped with the product to get an idea of how it is written.
If you dont want BAM in the picture, you still can do it with minimal coding as follows. The implementation class which performs the publishing is org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataBridgeDataPublisher. This class implements the interface org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataPublisher. The interface has three instace methods as follows.
public void init()
public void publishEvent(RequestPublisherDTO requestPublisherDTO)
public void publishEvent(ResponsePublisherDTO responsePublisherDTO)
The init() method runs just once during server startup. Here is where you can add all your logic which is needed to bootstrap the class.
The publishEvent(RequestPublisherDTO) is where you publish request events and publishEvent(ResponsePublisherDTO) is where you publish response events. The DTO objects are encapsulated representations of the request and response data respectively.
What you will have to do is write a new implementation for this interface and configure it as the value for DataPublisherImpl property in api-manager.xml. To make things easier you can simply extends the exsiting org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataBridgeDataPublisher, write your necessary logic to persist usage data to an sql database within the init(), publishEvent(RequestPublisherDTO) and publishEvent(ResponsePublisherDTO) and at the end of each method just call its respective super class method.
E.g. the overriding init() will call super().init(). This way you are only adding the neccessary code for your requirement, and leaving the BAM stat collection requirement to the super class.
I have a BizTalk 2010 solution that polls a database table, retrieves unprocessed records, does a transform with a map and call a 3rd party service. The happy path is working.
Here is the workflow:
Receive Location/Port from GetUnprocessedCustomers stored procedure: Poll SQL Server 2008 with WCF-SQL adapter by calling stored proc that returns unprocessed customers (WHERE IsProcessed = 0)
SendPort to 3rd Party web service: Filtered to ReceivePortName == with an outbound map to convert message returned from the above mentioned stored proc to service schema
SendPort to UpdateIsProcessed stored proc: Filtered to MessageType == with an outbound map to convert service response to stored procedure call that will update IsProcessed = 1.
I would like to catch the following possible exceptions, create a message with exception info and call an internal exception handling service.
Database or stored procedure is not accessible.
3rd party service is not accessible.
I was able to handle number 2 above by enabling Failed Message routing. Thanks to Greg.Forsythe.
My question is:
How can I create a generic solution that will capture all exceptions and send exception info (such as exception datetime, message, stack trace etc.) to an internal service?
I was able to get an answer on the BizTalk General Forum. The short answer is "There is no way to catch all exceptions". Hit the link for some options.