My json data being retrieved is as of below, notice that the format is not right in terms of object and key values why is this so, I am retrieving this json data from a dataset using the jayrock rpc service for asp.net 3.5
{
"Table":{
"columns":[
"i_member_id",
"i_group_id",
"u_name",
"u_tel",
"u_email",
"u_password",
"d_timestamp",
"b_activated"
],
"rows":[
[
1,
0,
"kevin",
"1231234",
"kevin#creaworld.com.sg",
"123",
"2011-01-05T09:51:36.8730000+08:00",
true
],
[
2,
0,
"kevin2",
"asdads",
"kevin2#creaworld.com.sg",
"123123",
"2011-01-05T10:01:46.1530000+08:00",
true
]
]
}
}
Here the link is to find the json formater
http://jsonformatter.curiousconcept.com/
Better you can use third party DLL
Json.Net
It will use full for you.
Related
I've noticed that some APIs use a format of sending a stripped down version of their data via a JSON array like the following:
[
"joe",
[
5,
2,
"yellow"
]
]
And store a set of keys like the following:
[
"name",
["some_data", [
"favorite_number",
"least_favorite_number",
"car_color"
]]
]
To turn the data from a bunch of random values to a readable set of data, like the following:
{
"name": "joe",
"some_data": {
"favorite_number": 5,
"least_favorite_number": 2,
"car_color": "yellow"
}
}
I was wondering how this could be done? I'd prefer it'd be in python, but I'm fine with programming my own libraries.
After grasping at more straws than I could fit in my mouth, I've figured it out. JSON schema is what I'm supposed to be using!
I have used REST to get data from API and the format of JSON output that contains arrays. When I am trying to copy the JSON as it is using copy activity to BLOB, I am only getting first object data and the rest is ignored.
In the documentation is says we can copy JSON as is by skipping schema section on both dataset and copy activity. I followed the same and I am the getting the output as below.
https://learn.microsoft.com/en-us/azure/data-factory/connector-rest#export-json-response-as-is
Tried copy activity without schema, using the header as first row and output files to BLOB as .json and .txt
Sample REST output:
{
"totalPages": 500,
"firstPage": true,
"lastPage": false,
"numberOfElements": 50,
"number": 0,
"totalElements": 636,
"columns": {
"dimension": {
"id": "variables/page",
"type": "string"
},
"columnIds": [
"0"
]
},
"rows": [
{
"itemId": "1234",
"value": "home",
"data": [
65
]
},
{
"itemId": "1235",
"value": "category",
"data": [
92
]
},
],
"summaryData": {
"totals": [
157
],
"col-max": [
123
],
"col-min": [
1
]
}
}
BLOB Output as the text is below: which is only first object data
totalPages,firstPage,lastPage,numberOfElements,number,totalElements
500,True,False,50,0,636
If you want to write the JSON response as is, you can use an HTTP connector. However, please note that the HTTP connector doesn't support pagination.
If you want to keep using the REST connector and to write a csv file as output, can you please specify how you want the nested objects and arrays to be written ?
In csv files, we can not write arrays. You could always use a custom activity or an azure function activity to call the REST API, parse it the way you want and write to a csv file.
Hope this helps.
I have the below JSON response to be validated. I need to validate all the "createdDate" from all the Arrays irrespective. Is there any easy way to capture them or loop through them (since it has the same object name, but in different arrays) and put them in variables to do an assertion against their corresponding values from a JDBC response?
Right now I have used JSON Assertion for each and every "createdDate" using the JSON path to validate against the database value.
{
"someobject1": 123,
"Array1":
[
{
"someobject2": 2,
"createdDate": "2019-03-26T20:29:44.631+0000",
"someobject3": "SCRIPT1"
},
{
"someobject4": 3,
"createdDate": "2019-03-27T20:29:44.631+0000",
"someobject5": "SCRIPT2"
}
],
"Array2":
[
{
"someobject6": 4,
"createdDate": "2019-03-28T20:29:44.631+0000",
"someobject7": "SCRIPT3"
},
{
"someobject8": 5,
"createdDate": "2019-03-29T20:29:44.631+0000",
"someobject9": "SCRIPT4"
}
]
}
You can use JSON Assertion configured like:
Assert JSON Path Exists: $..createdDate
Expected Value: ["2019-03-26T20:29:44.631+0000","2019-03-27T20:29:44.631+0000","2019-03-28T20:29:44.631+0000","2019-03-29T20:29:44.631+0000"]
Full configuration:
More information:
JSON Path: Deep Scan Operator
JSON Path Examples
JMeter's JSON Path Extractor Plugin - Advanced Usage Scenarios
I have a problem of huge http response with a json slab, where only portion is point of interest.
I cannot change the response structure.
here is an example
{
"searchString": "search",
"redirectUrl": "",
"0": {
"numRecords": 123,
"refinementViewModelCollector": {},
// Lots of data here
"results": [
{
"productCode": "123",
"productShortDescription": "Desc",
"brand": "Brand",
"productReview": {
"reviewScore": 0
},
"priceView": {
"salePriceDisplayable": false,
},
"productImageUrl": "url",
"alternateImageUrls": [
"url1"
],
"largeProductImageUrl": "url4",
"videoUrl": ""
},
{
"productCode": "124",
"productShortDescription": "Desc",
"brand": "Brand",
"productReview": {
"reviewScore": 0
},
"priceView": {
"salePriceDisplayable": false,
},
"preOrder": false,
"productImageUrl": "url",
"alternateImageUrls": [
"url1"
],
"largeProductImageUrl": "url4",
"videoUrl": ""
}
]
//lots of data here
}
}
My point of interest is entries in results Jason Array, but the are sitting in the middle of json
I created a small Play WS Client like this:
val wsClient: WSClient = ???
val ret = wsClient.url("url").stream()
ret.flatMap { response =>
response.body.via(JsonFraming.objectScanner(1024))
.map(_.utf8String)
.runWith(Sink.foreach(println))
}
this will not work because it will take whole json slab as Json object. I need to skip some data until "results": entry appear in the stream, then start parsing entries and skip all the rest.
Any ideas how to do this?
Check out Alpakka's JSON module, which can stream specific parts of a nested JSON structure:
response
.body
.via(JsonReader.select("$.0.results[*]"))
.map(_.utf8String)
.runWith(Sink.foreach(println)) // or runForeach(println)
There are parsers that support parsing as a stream. For a good example check out this Circe example https://github.com/circe/circe/tree/master/examples/sf-city-lots
I'd love a better, Scala-specific answer to this question, but check out the "Mixed Reads Example" in the documentation for Google's GSON library:
https://sites.google.com/site/gson/streaming
Gson also supports mixed streaming & object model access. This lets your application have the best of both worlds: the productivity of object model access with the efficiency of streaming
...
This code reads a JSON document containing an array of messages. It steps through array elements as a stream to avoid loading the complete document into memory. It is concise because it uses Gson’s object-model to parse the individual messages
This should have great memory-performance (the code reads from a Java InputStream, so the full structure is never in memory), but may require some effort to get your results into Scala case classes.
My Goal is to retrieve JSON type fields in an Solr index and also perform search queries on such fields.
I have the following documents in Solr Index and using the auto generated schema utilizing schemaless feature in Solr.
POST http://localhost:8983/solr/test1/update?commitWithin=1000
[
{"id" : "1", "type_s":"book", "title_t" : "The Way of Kings", "author_s" : "Brandon Sanderson",
"miscinfo": {"provider": "orielly", "site": "US"}
},
{"id" : "2", "type_s":"book", "title_t" : "The Game of Thrones", "author_s" : "James Sanderson",
"miscinfo": {"provider": "pacman", "site": "US"}
}
]
I see the JSON types are stored as strings in the schemaField type as seen in the output for following
GET http://localhost:8983/solr/test1/schema/fields
{
"name":"miscinfo",
"type":"strings"}
I had tried using srcField as mentioned in this post. However, a query to retrieve json type returns empty response. Below are the GET request used for the same
GET http://localhost:8983/solr/test1/select?q=1&fl=miscinfo&wt=json
GET http://localhost:8983/solr/test1/select?q=1&fl=miscinfo,source_s:[json]&wt=json
Also, the search queries for values inside JSON type fields return empty response
http://localhost:8983/solr/test1/select?q=pacman&wt=json
{
"responseHeader": {
"status": 0,
"QTime": 0,
"params": {
"q": "pacman",
"json": "",
"wt": "json"
}
},
"response": {
"numFound": 0,
"start": 0,
"docs": []
}
}
Please help in searching object types in Solr.
Have you checked this: https://cwiki.apache.org/confluence/display/solr/Response+Writers
JSON Response Writer A very commonly used Response Writer is the
JsonResponseWriter, which formats output in JavaScript Object Notation
(JSON), a lightweight data interchange format specified in specified
in RFC 4627. Setting the wt parameter to json invokes this Response
Writer. Here is a sample response for a simple query like
q=id:VS1GB400C3&wt=json: