I've started using Apache Nifi and I'm still learning it and experimenting with it. I really want to use Nifi to get JSON documents from API's and put them in my Elasticsearch database. So far using the built-in getTwitter and putElasticsearch controllers this works.
However now I want to do this with other APIs than Twitter, and I'm kinda stuck here. First off I really don't even know which controller to use? I would think getHttp or invokeHttp even with 'GET' as http verb then but it doesn't seem to work. If I use the getHttp I have to give an SSL service with keystore and truststore .. like why would I have to do that?
Apache Nifi is still quite new so hard to find decent guides / information about these kinds of things. I have read and searched the documentation but haven't gotten the wiser.
An example JSON to pick up from an API is:
https://api.ssllabs.com/api/v2/getEndpointData?host=www.bnpparibasfortis.be&s=193.58.4.82
Thanks in advance for anyone that can offer some help / insight.
What processor you use to get the JSON data is entirely dependent on the API you want to hit. The GetHttp or InvokeHttp processors should work to grab the data from a URL. If you'll notice, the SSL service is an optional property for both GetHttp and InvokeHttp so you only need to you use it when you want to communicate via HTTPS. Also, from the UI you can right click on a processor and then click "usage" to bring up the documentation for that processor.
At this link[1] you can find a NiFi template that uses GetHttp to get JSON data from randomuser.me and does various processing on it. It's primarily a template to show-case the different Avro processors but the method of grabbing the JSON should be relevant.
[1] https://github.com/hortonworks-gallery/nifi-templates/blob/master/templates/Convert_To_Avro_From_CSV_and_JSON.xml
Related
Just started working on an existing project making use of the Exact Online API.
While I was debugging the project I suddenly only started receiving XML results instead of JSON results from the API. I did not change anything about the endpoints being queried I was just running the existing queries trying to figure some things out.
These are the REST API docs: https://start.exactonline.nl/docs/HlpRestAPIResources.aspx
These are the XML docs: https://support.exactonline.com/community/s/knowledge-base#All-All-DNO-Content-xmlsamplecode
Typical REST API endpoints look like this:
https://start.exactonline.be/api/v1/xxxxxx/salesinvoice/SalesInvoices
Typical XML endpoints look like this:
https://start.exactonline.be/docs/XMLDownload.aspx
I also did not change any settings. I only have access to the tokens and api. I don't have access to the account.
This is an example of an endpoint and query where I previously received JSON but am now receiving XML:
https://start.exactonline.be/api/v1/xxxxxx/salesinvoice/SalesInvoices?$filter=InvoiceID eq guid'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx'&$select=InvoiceID
I tried this manually with Postman and also using the existing code from the project.
Is there some setting I am unaware of? Am i querying the wrong way? Maybe there have been some changes to the API I am unaware of that aren't listed in the release notes?
Please provide the request header Accept in your HTTP request that specifies what content format you prefer to receive: application/json. The default of Exact Online APIs is XML (but seldom to never used).
what is the best way to handle Json api request in symfony?
for example I have api of customers that allowed me to add new customer, edit and delete, and read as well
my question is how to develope it in symfony, as a bundle? like i have a database? with entity class?
if you have any tutorial it will be great
As a Client:
You can use Guzzle to make HTTP calls for you and then convert the json responses using native json_decode.
Other option is to use a Rest client, you will find many of them on packagist, here is one:
https://packagist.org/packages/Mashape/unirest-php
As a Server:
FOSRestBundle, its gonna provide you with many features that you'll need. here is a tutorial:
http://williamdurand.fr/2012/08/02/rest-apis-with-symfony2-the-right-way/
Of course you should check for more sources as well, I have a project of my own which uses it and you can check some code:
https://github.com/renatomefidf/sammui
You can also check the LiipHelloBundle for other multiple usages and examples:
https://github.com/liip/LiipHelloBundle
Good luck!
I am working on a demo using MarkLogic to store emails exported from Outlook as XML, so that they stay searchable and accessible when I move away from Outlook.
I am using an AngularJS front-end calling either the native MarkLogic REST services of own REST services written in JAVA using Jersey.
MarkLogic SEARCH REST service works very well to get back a list of references to documents based on various search criteria, but I also want to display information stored inside the found documents.
I would like to avoid multiple REST calls and to get back only the needed information, so I am trying to use the EVAL REST service to run an xQuery.
It works well to get XML back (inside a multipart/mixed message) but I don't seem to be able to get JSON instead which would be much more convenient and is very easy with most other MarkLogic REST services.
I could use "json:transform-to-json()" in my xQuery or transform the XML to JSON in my JAVA code, but that does not look very elegant to me.
Is there a more efficient method to get where I am trying to go ?
First, json:transform-to-json seems plenty elegant to me. But of course it's not always the right answer.
I see three options you haven't mentioned.
server-side transforms - REST search supports server-side transforms which transform each document when you perform a bulk read by query. Those server-side transforms could generate any json you need.
search extract-document-data - this the simplest way to extract portions of documents. But it seems best if your documents are json to match your json response. Otherwise you get xml in your json response . . . unless you're ok with that.
custom search snippets - another very powerful way to customize what search returns
All of these options don't require the privileges that eval requires, which is a very good thing. Since eval allows execution of arbitrary code on your server, it requires special privileges and should be used with great care. Two other options before you use eval are (1) custom xquery installed in an http server, and (2) REST extensions.
The answers from Sam are what I would suggest. Specifically I would set a search option for search-extract-document-data (This is a search API option. If you are posting the request, then you can add the option in the XML you post back. If you are using GET, then you need to register the option ahead of time and call it. Relevant URLs to assist:
https://docs.marklogic.com/guide/rest-dev/search#id_48838
https://docs.marklogic.com/guide/search-dev/appendixa#id_44222
As for json.. ML8 will transform content. Use the accept-header or just add format=json to your results...
Example - xml which is what my content is stored as:
http://localhost:8000/v1/search?q=watermellon
...
<search:result index="1" uri="/sample/collections/1.xml" path="fn:doc("/sample/collections/1.xml")" score="34816" confidence="0.5982239" fitness="0.6966695" href="/v1/documents?uri=%2Fsample%2Fcollections%2F1.xml" mimetype="application/xml" format="xml">
<search:snippet>
<search:match path="fn:doc("/sample/collections/1.xml")/x">
<search:highlight>watermellon</search:highlight>
</search:match>
</search:snippet>
</search:result>
...
Example - json which is what my content is stored as:
http://localhost:8000/v1/search?q=watermellon&format=json
...
"index":1,
"uri":"/sample/collections/1.xml",
"path":"fn:doc(\"/sample/collections/1.xml\")",
"score":34816,
"confidence":0.5982239,
"fitness":0.6966695,
"href":"/v1/documents?uri=%2Fsample%2Fcollections%2F1.xml",
"mimetype":"application/xml",
"format":"xml",
"matches":[
{
"path":"fn:doc(\"/sample/collections/1.xml\")/x",
"match-text":[
{
"highlight":"watermellon"
}
]
}
]
}
...
For real heavy-lifting, you can use server-side transforms as in Sam's description. One note about this. Server-side transformations are not part of the search API, but part of the REST API. Just mentioning it so you have some idea of which tool you are using in each case..
I am new to wso2 API Manager, trying to set it up expose my plain HTTP POST back end calls as a REST API. I am sure this is not a new pattern that I am trying to achieve here. The requirement is to convert the JSON data coming in (has array structures) into the HTTP URL query string parameters.
After some research through the documentation and other posts on this forum, decided to go with the script mediator that would parse the JSON data and convert it to a string that can be appended to the endpoint URL. Some how I am not able to achieve this.
I have followed the following post that seems to be very straight forward. As the original poster suggested, I am also not able to use getPayloadJSON() method. Even have trouble using the work around suggested there due to JSON.parse() not working.
Link
Also, this approach of editing the service bus configuration from source view does not sound like the correct option. Is there another elegant solution to achieve this? Thanks for the help in advance.
I was able to get both methods working by using an external script instead of the inline java script. Thanks
I am trying to achieve something like which is mentioned in this link
but i don't know whereto write the parsing code. I tried to write it in new method and added the method in "my adapter.xml" but nothing happens. I even don't know how to log in IBM WorkLight. I used WL.logger(some) but its throwing error that "Logger can not be called on an object".
Any help would be appreciated. Thanks in advance.!
In most cases you don't need to manually parse responses because WL adapter framework will do this for you. Anything you retrieve via WL.Server.invokeHttp API will be parsed to JSON automatically unless you specify returnedContentType:"plain". In case you DO need to manually parse JSON you can use JSON.parse() and JSON.stringify() APIs.
Server side logging is achieved via WL.Logger.debug/error/info etc. You can get more info about it here
You don't have to parse JSON data, there are libraries in Javascript to do that. Try JSON.parse(). You should learning how to write adapters and invoking them from clients. The Getting Started modules are a good place to start, specifically Module 4 in your case.