I retrieve data using Azure Data Factory from an OnPremise database and the output I get is as follows:
{
"value":[
{
"JSON_F52E2B61-18A1-11d1-B105-XXXXXXX":"{\"productUsages\":[{\"customerId\":3552,\"productId\":120,\"productionDate\":\"2015-02-10\",\"quantity\":1,\"userName\":\"XXXXXXXX\",\"productUsageId\":XXXXXX},{\"customerId\":5098,\"productId\":120,\"productionDate\":\"2015-04-07\",\"quantity\":1,\"userName\":\"ZZZZZZZ\",\"productUsageId\":ZZZZZZ}]}"
}
]
}
The entire value array is being serialized into a JSON and I end up with:
[{
"productUsages":
[
{
"customerId": 3552,
"productId": 120,
"productionDate": "2015-02-10",
"quantity": 1,
"userName": "XXXXXXXX",
"productUsageId": XXXXXX
},
{
"customerId": 5098,
"productId": 120,
"productionDate": "2015-04-07",
"quantity": 1,
"userName": "ZZZZZZZ",
"productUsageId": ZZZZZZZ
}
]
}]
I need to have a Json Object at a root level, not Json Array ([] replaced with {}). What's the easiest way to achieve that in Azure Data Factory?
Thanks
In ADF When you read any Json file it will read as array of Objects by default :
Sample data While reading Json data:
Data preview:
But when you want to move data to sink in Json format you have option called Set of objects you need to select that:
Sample data While storing in sink in form of Json data:
Output
Related
I have a Talend Job that receives a JSON(JSON format below) from a route. I need to extract data from JSON and load it to Oracle DB table.
Job
JSON format:
{
"data": [
{
"name": "FRSC-01",
"recordnum": "01",
"Expense1": "100",
"Expense2": "7265",
"Expense3": "9000"
},
{
"name": "FRSC-02",
"recordnum": "",
"Expense1": "200",
"Expense2": "6000",
"Expense3": "9000"
},
{
"name": "FRSC-03",
"recordnum": "03",
"Expense1": "200",
"Expense2": "7000",
"Expense3": "8000"
}
]
}
You can use tExtractJsonFields component to extract data from your json.
Define a schema with the columns you want from the json (name, recordNum, Expense1, Expense2, Expense3), set loop jsonpath query to "$.data[*]", and then for each column set the jsonpath expression like so:
name => "name"
recordNum => "recordNum"
...
And then just use a tMap to map the columns to your target table in the tOracleOutput component.
I have used REST to get data from API and the format of JSON output that contains arrays. When I am trying to copy the JSON as it is using copy activity to BLOB, I am only getting first object data and the rest is ignored.
In the documentation is says we can copy JSON as is by skipping schema section on both dataset and copy activity. I followed the same and I am the getting the output as below.
https://learn.microsoft.com/en-us/azure/data-factory/connector-rest#export-json-response-as-is
Tried copy activity without schema, using the header as first row and output files to BLOB as .json and .txt
Sample REST output:
{
"totalPages": 500,
"firstPage": true,
"lastPage": false,
"numberOfElements": 50,
"number": 0,
"totalElements": 636,
"columns": {
"dimension": {
"id": "variables/page",
"type": "string"
},
"columnIds": [
"0"
]
},
"rows": [
{
"itemId": "1234",
"value": "home",
"data": [
65
]
},
{
"itemId": "1235",
"value": "category",
"data": [
92
]
},
],
"summaryData": {
"totals": [
157
],
"col-max": [
123
],
"col-min": [
1
]
}
}
BLOB Output as the text is below: which is only first object data
totalPages,firstPage,lastPage,numberOfElements,number,totalElements
500,True,False,50,0,636
If you want to write the JSON response as is, you can use an HTTP connector. However, please note that the HTTP connector doesn't support pagination.
If you want to keep using the REST connector and to write a csv file as output, can you please specify how you want the nested objects and arrays to be written ?
In csv files, we can not write arrays. You could always use a custom activity or an azure function activity to call the REST API, parse it the way you want and write to a csv file.
Hope this helps.
I have the below JSON response to be validated. I need to validate all the "createdDate" from all the Arrays irrespective. Is there any easy way to capture them or loop through them (since it has the same object name, but in different arrays) and put them in variables to do an assertion against their corresponding values from a JDBC response?
Right now I have used JSON Assertion for each and every "createdDate" using the JSON path to validate against the database value.
{
"someobject1": 123,
"Array1":
[
{
"someobject2": 2,
"createdDate": "2019-03-26T20:29:44.631+0000",
"someobject3": "SCRIPT1"
},
{
"someobject4": 3,
"createdDate": "2019-03-27T20:29:44.631+0000",
"someobject5": "SCRIPT2"
}
],
"Array2":
[
{
"someobject6": 4,
"createdDate": "2019-03-28T20:29:44.631+0000",
"someobject7": "SCRIPT3"
},
{
"someobject8": 5,
"createdDate": "2019-03-29T20:29:44.631+0000",
"someobject9": "SCRIPT4"
}
]
}
You can use JSON Assertion configured like:
Assert JSON Path Exists: $..createdDate
Expected Value: ["2019-03-26T20:29:44.631+0000","2019-03-27T20:29:44.631+0000","2019-03-28T20:29:44.631+0000","2019-03-29T20:29:44.631+0000"]
Full configuration:
More information:
JSON Path: Deep Scan Operator
JSON Path Examples
JMeter's JSON Path Extractor Plugin - Advanced Usage Scenarios
I have rows of the following JSON form:
[
{
"id": 1,
"costs": [
{
"blue": 100,
"location":"courts",
"sport": "football"
}
]
}
]
I want to upload this into a redshift table as follows:
id | blue | location | sport
--------+------+---------+------
1 | 100 | courts |football
The following JSONPaths file is not successful:
{
"jsonpaths": [
"$.id",
"$.costs[0].blue",
"$.costs[0].location",
"$.costs[0].sport"
]
}
Redshift returns the following error code:
err_code: 1216 Invalid JSONPath format: Member is not an object.
How can I change the jsonpaths file to be able to upload the json as desired?
The answer to this is provided by John Rotenstein in the comments. I am just formalizing the answer here.
As shown in the documentation, the input JSON records have to be a new line delimited sequence of JSON objects. The examples show the JSON objects as pretty printed, but typically the input stream of records would be one JSON object per line.
{ "id": 1, "costs": [ { "blue": 100, "location":"courts", "sport": "football" } ] }
{ "id": 2, "costs": [ { "blue": 200, "location":"fields", "sport": "cricket" } ] }
So, technically the input record stream is not required to be a valid JSON, but a stream of delimited valid JSON objects.
My problem is I have Json file of small json file creadted with node js
I couldn't consume my json from that link and i tried to test my json file in some website like Json formatter there is this error : Multiple JSON root elements .
when i put only one json in json formatter it become right but like this example 2 json it it wrong
this is the example of my json of 2 json ,
{"#timestamp":"2017-06-11T00:28:24.112Z","type_instance":"interrupt","plugin":"cpu","logdate":"2017-06-11T00:28:24.112Z","host":"node-2","#version":"1","collectd_type":"percent","value":0}
{"#timestamp":"2017-06-11T00:28:24.112Z","type_instance":"softirq","plugin":"cpu","logdate":"2017-06-11T00:28:24.112Z","host":"node-2","#version":"1","collectd_type":"percent","value":0}
this is not a json format json must have a root an object or an array
[
{
"#timestamp": "2017-06-11T00:28:24.112Z",
"type_instance": "interrupt",
"plugin": "cpu",
"logdate": "2017-06-11T00:28:24.112Z",
"host": "node-2",
"#version": "1",
"collectd_type": "percent",
"value": 0
},
{
"#timestamp": "2017-06-11T00:28:24.112Z",
"type_instance": "softirq",
"plugin": "cpu",
"logdate": "2017-06-11T00:28:24.112Z",
"host": "node-2",
"#version": "1",
"collectd_type": "percent",
"value": 0
}
]
If you have the file contents as a string, then split the lines and JSON.parse them one by one:
const data =`{"a": 1}
{"b": 2}`;
const lines = data.split('\n')
const objects = lines.map(JSON.parse);
console.log(objects);