Very new to mule esb. I am trying to retrive json response from http request connector and then inserting it into db. Not sure why I am not able to fetch the values when I am inserting it into DB.
Getting the following error while inserting into database:
Execution of the expression
"message.payload.Data.MissingItems.Institution" failed.
(org.mule.api.expression.ExpressionRuntimeException).
My conf file
Put a debugger before foreach and inside foreach and see how the values are being stored inside the map.Run the application in debug mode and see the structure of the payload using mule expression language.You can find out the option inside the debugger.There you can put #[message.payload] or #[message.payload.Data] or #[message.payload.Data.MissingItems] to check the structure of the message.It will also help you to understand if the issue is related to transformation or anything else.
Related
I searched but didn't find anything specific to the issue I was having. My apologies if I overlooked it.
I have a scenario where I'm pulling data from an oData source and everything works fine. I have the oData source in a loop where I loop over each of our different companies and pull the data for that company. I'm doing this to reduce the volume of data that is being returned in each oData call.
Everything works fine as long as there is data being returned from the oData call. In the attached photo you can see that the call is being made and data is being returned.
But when the oData Service is called with parameters/filters that return no data I get the following and that is when the VS_NEEDSNEWMETADATA error is thrown. The image below shows what I get when no data is returned.
So the issue isn't that I have invalid metadata due to changes made to the service (adding/removing fields). It's that nothing is being returned so there is no Metadata. Now it's possible that this is an issue with the system that I'm pulling the oData from (SAP S4) and they way that system surfaces the oData call when there is no data??
Either way, trying to figure out a way to handle this within SSIS. I tried to set validate external metadata = false but the package still fails. I could also fix this by excluding those companies in the script but then once data does exists I'd have to remember to update the scripts and redeploy.
Ideas suggestions?
You've hit the nail on the head except your metadata is changing in the no data case because the shape is different - at least from the SSIS engine perspective.
You basically need to double invoke the OData source. First invocation will simply identify is data returned. If that evaluation is true, only then does the Data Flow start running.
How can you evaluate whether your OData will return data? You're likely looking at a Script Task and C#/VB.NET and hopefully the SAP data returns don't vary between this call and next call.
If that's the case, then you need to define an OnError event handler for the Data Flow, mark the System scoped variable Propagate to False and maybe even force a success status to be returned to get the loop to continue.
I have a basic NIFI flow with one GenerateFlowFile processor and one LookupRecord processor with RestLookupService.
I need to do a rest lookup and enrich the incoming flow file with rest response.
I am getting errors that the lookup service is unable to lookup coordinates with the value I am extracting from the incoming flow file.
GenerateFlowFile is configured with simple JSON
LookupRecord is configured to extract the key from the JSON and populate it to the RestLookupService. Also, JsonReader and JsonSetWriter is configured to read the incoming flow file and to write the response back to the flow file
The RestLookupService itself exits with JsonParseException about unexpected character '<'
RestLookupService is configured with my API running in the cloud in which I am trying to use the extracted KEY from the incoming flow file.
The most interesting bug is that when I configure the URL to point for example to mocky.io everything works correctly so it means that the issue itself is tight with the API URL I am using (http://iotosk.ddns.net:3006/devices/${key}/getParsingData). I have tried also removing the $key, using the exact URL, using different URLs..
Of course the API is working OK over postman/curl anything else. I have checked the logs on the container that the API is running on and there is no requests in the logs what means that nifi is failing even before reaching the API. At least on application level...
I am absolutely out of options without any clue how to solve this. And with nifi also google is not helpful.
Does anybody see any issue in the configuration or can point me in some direction what can cause this issue?
After some additional digging. The issue was connected with authentication logic even
before the API receives it and that criples the request and and returned XML as Bryan Bende suggested in the comment.
But definitely better logging in nifi will help to solve this way faster...
I am trying to use SpringXD to stream some JSON metrics data to a Oracle database.
I am using this example from here: SpringXD Example
Http call being made: EarthquakeJsonExample
My shell cmd.
stream create earthData --definition "trigger|usgs| jdbc --columns='mag,place,time,updated,tz,url,felt,cdi,mni,alert,tsunami,status,sig,net,code,ids,souces,types,nst,dmin,rms,gap,magnitude_type' --driverClassName=driver --username=username --password --url=url --tableName=Test_Table" --deploy
I would like to capture just the properties portion of this JSON response into the given table columns. I got it to the point where it doesn't give me a error on the hashing but instead just deposits a bunch of nulls into the column.
I think my problem is the parsing of the JSON itself. Since really the properties is in the Features array. Can SpringXD distinguish this for me out of the box or will I need to write a custom processor?
Here is a look at what the database looks like after a successful cmd.
Any advice? Im new to parsing JSON in this fashion and im not really sure how to find more documentation or examples with SpringXD itself.
Here is reference to the documentation: SpringXD Doc
The transformer in the JDBC sink expects a simple document that can converted to a map of keys/values. You would need to add a transformer upstream, perhaps in your usgs processor or even a separate processor. You could use a #jsonPath expression to extract the properties key and make it the payload.
Here's my situation:
I want to do this:
I have a list of URLs in a MySQL database which I want to hit using a HTTP Request to see if the response is a HTTP Status code of 404 or not.
I have done this:
Added and configured a JDBC Config Element.
Added and configured a JDBC Request Sampler. Basically a select statement that returns a table with 8 columns. I have provided 8 comma-separated variables for the 'Variable names' field, so that the results of the JDBC request can be identified with these variable names.
Created a HTTP Request Sampler that uses one of those variables ${url} in the 'Server Name or IP' field.
Though the JDBC request works flawlessly and returns a table with a bunch of rows, the problem with this is that the HTTP Request Sampler never picks up the variable from the JDBC Request result.
The HTTP Request looks like this in the 'View Results Tree':
GET http://${url}/
I have tried these solutions:
Add 'Save Responses to a File' listener to the JDBC Request. This creates a file of type '.plain' and not a CSV. Had it been a CSV, I could have utilized that CSV file by creating a CSV Data Set Config. So this attempt failed.
I have tried forcing the file name in the above attempt to always use 'C:\JMETERTest\data.csv'. But it ends up creating a new file named 'C:\JMETERTest\data.csv1.plain'. This attempt failed too.
I tried to reference the URL column as ${url_1} in the HTTP Request's Server Name field. It worked. But the problem now is that in the results tree, all the requests are going for the the URL from only the first row of the result set. I see that this is because of the row number '_1' specified in the ${url_1} above. I can use this if someone can suggest a way to parameterize the '_1' into a variable that I can loop through (probably using a 'Counter' element). I created a Counter Config Element by the Reference Name 'loopCounter'. And used this in the Server Name field of the HTTP Request:
${url_("${loopCounter}")}
But now my HTTP Requests look lamer:
GET http://${url_("${loopCounter}")}/
This did not work too.
Solution 3 looks more doable to be only if I could resolve the parameterization of the row number.
I am open to JMeter Plugin suggestions too.
I will update anything else that I try as we go on.
P.S. Please let me know if my question is not clear in anyway.
Have you tried wrapping the HTTP sampler in a ForEach controller (parent) where the variable for the controller is the URL variable obtained from the JDBC sampler?
Also, the output variable in the ForEach will be the variable you now use in the HTTP sampler.
That way it will iterate through each variable from the beginning of the index to the end and run the sampler once each time.
In 'Save responses to a File' Listener, Select Checkboxes "Don't add Suffix and Prefix". Checking these two options will ensure, you get exact Log file name.
I'm very very new to this and need some help writing an ESB script to take an event posted via HTTPS on Port 9090 in WSO2 and transform it into a message to be apended to an XML file on the server:
The HTTPS data will contain : “ID=Servername|Severity=sevtype” (Where Servername is a device name and Sevtype can either be "WARNING" or "OK" depending on whether the server is down or up)
This then needs to be transformed and appended to an existing XML file in the following format:
<event>
<componentID>Servername</componentID>
<timestamp>2012-04-27 01:37:10</timestamp> ***(Date and time the event was received)***
<severity>NORMAL</severity> ***(If original is WARNING then severity = SEVERE else it = NORMAL)***
<eti>NodeStatus</eti><etivalue>Up</etivalue> ***(If original is WARNING then severity = Down else it = Up)***
<\event>
Please could someone assist me i'm really floundering with what seems to be a simple thing
Many Many Thanks
Simon
You can write a simple task to poll your data into the server and can do a xslt transformation to construct that particular xml format message.
Some references to write a task;
http://wso2.org/project/esb/java/4.0.3/docs/configuration_language.html#TaskConcept
http://docs.wso2.org/wiki/display/ESB403/Writing+Tasks
http://wso2.org/library/2900
How do you get the ID and Severity? are they http headers?
Generally you can use payload factory mediator[1] to build the payload messages with some input parameter data.
[1] http://wso2.org/project/esb/java/4.0.3/docs/samples/message_mediation_samples.html#Sample17