I have a case where I have to serialize a N number of http requests, perform the requests in parallel, and describe the response in the same json. To do this in json is pretty easy obviously, I could do something like:
[
{
"request": {
"uri": "http://something",
"headers": [
{
"Content-Type": "application/json"
}
],
"body": ".. snip .. you get the idea",
"other-stuff": ""
},
"response": {
// something along the lines of the request structure to be filled in after the request completed
}
},
// second request/response here, and so on.
]
But while I could make this format up myself, I feel this problem is (very) likely solved before, and there should be some sort of json standard floating around that does this. Can't find it though. Have you encountered this somewhere?
Related
My issue is that I want to give my response file names a more meaningful name but to do this I need to be able to match on any text following the id. Here is my current mapping:
{
"priority": 10,
"request": {
"urlPattern": "/api/v1/customer\\?Id=(.*)&IdType=(.*)&AdditionalInformation=(.*)",
"method": "GET"
},
"response": {
"transformers": [
"dynamic-mapping-transformer"
],
"status": 200,
"headers": {
"Content-Type": "application/json",
"Correlation-ID": "{{request.headers.Correlation-ID}}"
},
"transformerParameters": {
"bodyFileName": "{{request.query.Id}}",
"fileDirectory": "getWorkplaceDetails",
"extension": "json",
"encodeName": false,
"caseInsensitive": true
}
}
}
And the naming convention for the response files should be something along the lines of "Id_foo_bar" but "foo" and "bar" are not available in the request so there is no way for me to know them before the response is posted. I would like to use a wild card in the bodyFileName value to match on anything after the Id and was wondering if anyone knew a way to do this?
EDIT 1
I forgot to mention that the mapping needs to accept multiple files with different names which is why I am looking to use a wild card instead of a static file name.
I'm using Wiremock server to mock responses and using JSON format to mock responses.
{
"request":
{
"url": "/token",
"method": "POST",
"bodyPatterns" : [{
"contains": "username=test_user#gmail.com&password=passwordtest_security_token"
}]
},
"response":
{
"status": 200,
"headers":
{
"Content-Type" : "application/json"
},
"jsonBody": {"message": "ok"}
}
This is not working as the '#' in the email is not encoded. I need to pass "test_user%40gmail" for the request to work.
Here the change is only at one place. But for other mocks, request bodies have many special characters**(#,%*\n\s)**. Is there any way to handle the encoding part in the Wiremock.
Also, is there way to encode the string in url?
In the above example there is an accolade missing which prevents is from loading in WireMock. The below example is complete:
{
"request": {
"url": "/token",
"method": "POST",
"bodyPatterns": [{
"contains": "username=test_user#gmail.com&password=passwordtest_security_token"
}
]
},
"response": {
"status": 200,
"headers": {
"Content-Type": "application/json"
},
"jsonBody": {
"message": "ok"
}
}
}
In your question you highlighted that you need to use %40 instead of #. I have been unable to replicate this scenario. As I'm unsure if you're posting through a form, or raw string.
In the case that you use a regular Form Data option this results in failure to match.
But sending it as a raw body the result is quite the opposite. The example behaves exactly as you would described it should. Below is a screenshot of my Postman result:
So I got the short straw at work and I have to learn some Groovy. I'm part of the infrastructure team, so I don't know a lot.
Basically what I have to do is:
- Get an event (DONE)
- Parse the event with an specific json format (WORKING ON IT)
- Make a POST request against the API (NOT DONE AT ALL)
So, I'm having some problems to test the POST request I'm making.
This is my code:
//def post = new URL("https://httpbin.org/post").openConnection();
//def message = '{"message":"this is a message"}'
def post = new URL("https://api.duckduckgo.com").openConnection();
def message = '{"q=DuckDuckGo&format=json&pretty=1"}'
post.setRequestMethod("POST")
post.setDoOutput(true)
post.setRequestProperty("Content-Type", "application/json")
post.getOutputStream().write(message.getBytes("UTF-8"));
def postRC = post.getResponseCode();
println(postRC);
if(postRC.equals(200)) {
println(post.getInputStream().getText());
}
This is the response, the first test is from httpbin.org, the second test is from duckduckgo:
root#test:/mnt/c/groovy-dev/test# groovy post.groovy
200
{
"args": {},
"data": "{\"message\":\"this is a message\"}",
"files": {},
"form": {},
"headers": {
"Accept": "text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2",
"Content-Length": "31",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "Java/1.8.0_221"
},
"json": {
"message": "this is a message"
},
"origin": "190.188.58.175, 190.188.58.175",
"url": "https://httpbin.org/post"
}
root#test:/mnt/c/groovy-dev/test# groovy post.groovy
500
I got a response from httpbin, but not from duckduckgo. I don't know if I'm doing something wrong or there is a problem with the API. Although I tested it with Postman and its working correctly.
What I'm missing?
Also, the API I will have to access in the end, uses authentication. Do you have any suggestion how to handle it?
Thanks
I have a problem of huge http response with a json slab, where only portion is point of interest.
I cannot change the response structure.
here is an example
{
"searchString": "search",
"redirectUrl": "",
"0": {
"numRecords": 123,
"refinementViewModelCollector": {},
// Lots of data here
"results": [
{
"productCode": "123",
"productShortDescription": "Desc",
"brand": "Brand",
"productReview": {
"reviewScore": 0
},
"priceView": {
"salePriceDisplayable": false,
},
"productImageUrl": "url",
"alternateImageUrls": [
"url1"
],
"largeProductImageUrl": "url4",
"videoUrl": ""
},
{
"productCode": "124",
"productShortDescription": "Desc",
"brand": "Brand",
"productReview": {
"reviewScore": 0
},
"priceView": {
"salePriceDisplayable": false,
},
"preOrder": false,
"productImageUrl": "url",
"alternateImageUrls": [
"url1"
],
"largeProductImageUrl": "url4",
"videoUrl": ""
}
]
//lots of data here
}
}
My point of interest is entries in results Jason Array, but the are sitting in the middle of json
I created a small Play WS Client like this:
val wsClient: WSClient = ???
val ret = wsClient.url("url").stream()
ret.flatMap { response =>
response.body.via(JsonFraming.objectScanner(1024))
.map(_.utf8String)
.runWith(Sink.foreach(println))
}
this will not work because it will take whole json slab as Json object. I need to skip some data until "results": entry appear in the stream, then start parsing entries and skip all the rest.
Any ideas how to do this?
Check out Alpakka's JSON module, which can stream specific parts of a nested JSON structure:
response
.body
.via(JsonReader.select("$.0.results[*]"))
.map(_.utf8String)
.runWith(Sink.foreach(println)) // or runForeach(println)
There are parsers that support parsing as a stream. For a good example check out this Circe example https://github.com/circe/circe/tree/master/examples/sf-city-lots
I'd love a better, Scala-specific answer to this question, but check out the "Mixed Reads Example" in the documentation for Google's GSON library:
https://sites.google.com/site/gson/streaming
Gson also supports mixed streaming & object model access. This lets your application have the best of both worlds: the productivity of object model access with the efficiency of streaming
...
This code reads a JSON document containing an array of messages. It steps through array elements as a stream to avoid loading the complete document into memory. It is concise because it uses Gson’s object-model to parse the individual messages
This should have great memory-performance (the code reads from a Java InputStream, so the full structure is never in memory), but may require some effort to get your results into Scala case classes.
I have been playing around with Azure Logic Apps and trying to retrieve a Pocket (ReadItLater) article so that I can create a new task in my preferred Task Manager. I have Two HTTP Connectors (one for Retrieve Operation using Pocket API and another post data to Todoist (my preferred task manager).
I can retrieve the Article and the response looks like (removed a few properties below for easy reading):
{
"statusCode": 200,
"headers": {
"pragma": "no-cache",
"status": "200 OK"
},
"body": {
"status": 1,
"complete": 1,
"list": {
"586327616": {
"item_id": "586327616",
"resolved_id": "586327616",
"given_url": "http://kenwheeler.github.io/slick/?utm_source=hackernewsletter&utm_medium=email&utm_term=design&mc_cid=58c9499fa2&mc_eid=3aaf6c4e47",
"given_title": "slick - the last carousel you'll ever need",
"time_added": "1396652224",
"time_updated": "1405156517",
"resolved_title": "slick",
"resolved_url": "http://kenwheeler.github.io/slick/?utm_source=hackernewsletter&utm_medium=email&utm_term=design&mc_cid=58c9499fa2&mc_eid=3aaf6c4e47",
"excerpt": "Add slick.js before your closing <body> tag, after jQuery (requires jQuery 1.7 +) <script type=\"text/javascript\" src=\"slick/slick.min.",
"word_count": "22"
}
}
}
}
Now I want to parse the above response to retrieve individual article properties (i.e. resolved_title). The issue here is the object under the list "586327616" is dynamic and changes for every article, and I can't seem to parse this as an expression in Logic App. My current action in Logic App looks like:
"postToTodoist": {
"conditions": [
{
"expression": "#equals(outputs('getPocketArticles')['statusCode'], 200)"
},
{
"dependsOn": "getPocketArticles"
}
],
"inputs": {
"body": "#{outputs('getPocketArticles')['body']['list'][0]['resolved_title']}",
"headers": {
"Content-Type": "application/x-www-form-urlencoded"
},
"method": "POST",
"repeat": {},
"uri": "https://todoist.com/API/v6/add_item"
},
"type": "Http"
}
For the expression I have tried converting the response to string, using coalesce and trying to access using an index, but nothing seem to work. In the error, it tells me what that the available property is i.e.:
{"code":"InvalidTemplate","message":"Unable to process template language expressions in action 'postToTodoist' inputs at line '1' and column '11': 'The template language expression 'coalesce(body('getPocketArticles')['list']).resolved_title' cannot be evaluated because property 'resolved_title' doesn't exist, available properties are '586327616'. Please see https://aka.ms/logicexpressions for usage details.'."}
I feel that it is not possible to construct an expression without knowing the name of the property, has anyone done something similar?