I have a basic NIFI flow with one GenerateFlowFile processor and one LookupRecord processor with RestLookupService.
I need to do a rest lookup and enrich the incoming flow file with rest response.
I am getting errors that the lookup service is unable to lookup coordinates with the value I am extracting from the incoming flow file.
GenerateFlowFile is configured with simple JSON
LookupRecord is configured to extract the key from the JSON and populate it to the RestLookupService. Also, JsonReader and JsonSetWriter is configured to read the incoming flow file and to write the response back to the flow file
The RestLookupService itself exits with JsonParseException about unexpected character '<'
RestLookupService is configured with my API running in the cloud in which I am trying to use the extracted KEY from the incoming flow file.
The most interesting bug is that when I configure the URL to point for example to mocky.io everything works correctly so it means that the issue itself is tight with the API URL I am using (http://iotosk.ddns.net:3006/devices/${key}/getParsingData). I have tried also removing the $key, using the exact URL, using different URLs..
Of course the API is working OK over postman/curl anything else. I have checked the logs on the container that the API is running on and there is no requests in the logs what means that nifi is failing even before reaching the API. At least on application level...
I am absolutely out of options without any clue how to solve this. And with nifi also google is not helpful.
Does anybody see any issue in the configuration or can point me in some direction what can cause this issue?
After some additional digging. The issue was connected with authentication logic even
before the API receives it and that criples the request and and returned XML as Bryan Bende suggested in the comment.
But definitely better logging in nifi will help to solve this way faster...
Related
Just started working on an existing project making use of the Exact Online API.
While I was debugging the project I suddenly only started receiving XML results instead of JSON results from the API. I did not change anything about the endpoints being queried I was just running the existing queries trying to figure some things out.
These are the REST API docs: https://start.exactonline.nl/docs/HlpRestAPIResources.aspx
These are the XML docs: https://support.exactonline.com/community/s/knowledge-base#All-All-DNO-Content-xmlsamplecode
Typical REST API endpoints look like this:
https://start.exactonline.be/api/v1/xxxxxx/salesinvoice/SalesInvoices
Typical XML endpoints look like this:
https://start.exactonline.be/docs/XMLDownload.aspx
I also did not change any settings. I only have access to the tokens and api. I don't have access to the account.
This is an example of an endpoint and query where I previously received JSON but am now receiving XML:
https://start.exactonline.be/api/v1/xxxxxx/salesinvoice/SalesInvoices?$filter=InvoiceID eq guid'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx'&$select=InvoiceID
I tried this manually with Postman and also using the existing code from the project.
Is there some setting I am unaware of? Am i querying the wrong way? Maybe there have been some changes to the API I am unaware of that aren't listed in the release notes?
Please provide the request header Accept in your HTTP request that specifies what content format you prefer to receive: application/json. The default of Exact Online APIs is XML (but seldom to never used).
I've started using Apache Nifi and I'm still learning it and experimenting with it. I really want to use Nifi to get JSON documents from API's and put them in my Elasticsearch database. So far using the built-in getTwitter and putElasticsearch controllers this works.
However now I want to do this with other APIs than Twitter, and I'm kinda stuck here. First off I really don't even know which controller to use? I would think getHttp or invokeHttp even with 'GET' as http verb then but it doesn't seem to work. If I use the getHttp I have to give an SSL service with keystore and truststore .. like why would I have to do that?
Apache Nifi is still quite new so hard to find decent guides / information about these kinds of things. I have read and searched the documentation but haven't gotten the wiser.
An example JSON to pick up from an API is:
https://api.ssllabs.com/api/v2/getEndpointData?host=www.bnpparibasfortis.be&s=193.58.4.82
Thanks in advance for anyone that can offer some help / insight.
What processor you use to get the JSON data is entirely dependent on the API you want to hit. The GetHttp or InvokeHttp processors should work to grab the data from a URL. If you'll notice, the SSL service is an optional property for both GetHttp and InvokeHttp so you only need to you use it when you want to communicate via HTTPS. Also, from the UI you can right click on a processor and then click "usage" to bring up the documentation for that processor.
At this link[1] you can find a NiFi template that uses GetHttp to get JSON data from randomuser.me and does various processing on it. It's primarily a template to show-case the different Avro processors but the method of grabbing the JSON should be relevant.
[1] https://github.com/hortonworks-gallery/nifi-templates/blob/master/templates/Convert_To_Avro_From_CSV_and_JSON.xml
How to call a rest service that returns JSON in a APIKit based message flow. I want to prepare the request for Rest service and want to extract the JSON message in a message flow.
Can any one help me to do this?
Thanks
The recommended way to invoke/consume REST services in mule is the use of HTTP Request
Do let us know if there is a specific you are facing
The request connector is especially handy when consuming a RESTful API
that is described in a RAML file. If you reference the API's RAML file
in the connector's configuration, it will proactively offer you the
set of available resources and operations contained in the RAML file,
as well as enforce the policies described in the file. It will also
expose the API metadata to Studio, which can then be used by other
elements such as DataWeave to autocomplete fields and make
configuration much easier
Use Rest URI path to invoke the Service and you can also get json response as output based on the type of service you invoke
This might be a weird question but I am open for all the suggestions.
The background is I want to use script to automatically deploy/remove docker container on Jelastic, but unfortunately this part is not well documented in Jelastic official API document. Jelastic provided me a piece of sample code demonstrated how to use bash to create a new environment with a new docker container but it is not enough, I still don't know how to create/remove docker container by looking at the sample code.
Since Jelastic is using standard JSON API, I am wondering is there any tool which can automatically retrieve/detect that the parameters I can use with Jelastic JSON API?
If you were me, how would you get over this if there is no document as reference?
I am keen to use Jelastic, but this issue stopped me from onboarding, many thanks.
J.
All the parameters that can be used with Jelastic JSON API are specified at http://docs.jelastic.com/api/ page.
To use JSON API without document I suggests to you check the Postman API tool https://www.getpostman.com/. This application allows you to see all the sent/received data and allows you to passes JSON values without any document or any additional actions.
Simplest scenario for beginners: Go to API docs, section Users>Authorization, using Signin method you should to obtain the session value, that is necessary almost for all further actions. Then you need to obtain information about environment, section Environment>Environment, at first you should to executes GetEnvs method, then using the application identifier of the environment that was obtained from the previous command you should to executes GetEnvInfo method. As a results of the described scenario you will get all parameters and values that can be used with Jelastic JSON API for certain type of the environment.
I have a webservice method that if I call directly via url GET returns XML without issue.
However, POST to that same url with Content-Type Json, it fails.
I think I can figure out the issue (I'm guessing it's an encoding or bad character somewhere in there) but I don't know how to debug the problem.
If I set a breakpoint in the webservice, it runs to completion. The failure appears to be happening AFTER the method returns, but BEFORE the json is returned to the caller.
How can I get in between to trace the error?
Please let me know if I can provide more context to help, but I really just need to know how to get in there.
EDIT:
The web service is configured to receive POST and return JSON and in fact DOES correctly return JSON in some cases. However, there are certain calls that are failing, so I need a way to trace this or debug it somehow and figure out why some calls are not working.
The web service is likely not configured to receive POST requests, especially if you are receiving a 405 Method Not Allowed response status.
Although I didn't find a way to debug or intercept the request to find the exact answer, it turns out the problem was the size of the content being returned by the webservice. Following this answer: ASP.NET WebMethod with jQuery json, is there a size limit?
and increasing the json limit fixed the issue!
Is there a way I could have trapped this to find the error without just guessing it was a size limit?