How to Parse Json Object Element that contains Dynamic Attribute - json

Question

We have a Json object with the following structure.
{
"results": {
"timesheets": {
"135288482": {
"id": 135288482,
"user_id": 1242515,
"jobcode_id": 17288283,
"customfields": {
"19142": "Item 1",
"19144": "Item 2"
},
"attached_files": [
50692,
44878
],
"last_modified": "1970-01-01T00:00:00+00:00"
},
"135288514": {
"id": 135288514,
"user_id": 1242509,
"jobcode_id": 18080900,
"customfields": {
"19142": "Item 1",
"19144": "Item 2"
},
"attached_files": [
50692,
44878
],
"last_modified": "1970-01-01T00:00:00+00:00"
}}
We need to access the elements that is inside the results --> timesheets --> Dynamic id.
Example:
{
"id": 135288482,
"user_id": 1242515,
"jobcode_id": 17288283,
"customfields": {
"19142": "Item 1",
"19144": "Item 2"
},
"attached_files": [
50692,
44878
],
"last_modified": "1970-01-01T00:00:00+00:00"
}
The problem is that "135288482": { is dynamic. How do we access what is inside of it.
We are trying to create data flow to parse the data. The data is dynamic, so accessing via attribute name is not possible.

Answer 1

AFAIK, as per your JSON structure and dynamic keys it might not be possible to get the desired result using Dataflow.
I have reproduced the above and able to get it done using set variable and ForEach like below.
In your JSON, the keys and the values for the ids are same. So, I have used that to get the list of keys first. Then using that list of keys, I am able to access the inner JSON object.
These are my variable in the pipeline:
First, I have a set variable of type string and stored "id" in it.
Then I have taken lookup activity to get the above JSON file from blob.
I have stored lookup the timesheets objects as string in a set variable using the below dynamic content
#string(activity('Lookup1').output.value[0].results.timesheets)
I have used split on that string with "id" and stored the result array in an array variable.
#split(variables('jsonasstring'), variables('ids'))
This will give the array like below.
Now, I took a Foreach to this array but skipped the first element. In Each iteration, I have taken first 9 indexes of the string to append variable and that is the key.
If your id values and keys are not same, then you can skip the last from split array and take the keys from the reverse side of the string as per your requirement.
This is my dynamic content for append variable activity inside ForEach #take(item(), 9)
Then I took another ForEach, and given this keys list array to it. Inside foreach you can access the JSON with below dynamic content.
#string(activity('Lookup1').output.value[0].results.timesheets[item()])
This is my pipeline JSON:
{
"name": "pipeline1",
"properties": {
"activities": [
{
"name": "Lookup1",
"type": "Lookup",
"dependsOn": [
{
"activity": "for ids",
"dependencyConditions": [
"Succeeded"
]
}
],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "JsonSource",
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"recursive": true,
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "JsonReadSettings"
}
},
"dataset": {
"referenceName": "Json1",
"type": "DatasetReference"
},
"firstRowOnly": false
}
},
{
"name": "JSON as STRING",
"type": "SetVariable",
"dependsOn": [
{
"activity": "Lookup1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "jsonasstring",
"value": {
"value": "#string(activity('Lookup1').output.value[0].results.timesheets)",
"type": "Expression"
}
}
},
{
"name": "for ids",
"type": "SetVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "ids",
"value": {
"value": "#string('\"id\":')",
"type": "Expression"
}
}
},
{
"name": "after split",
"type": "SetVariable",
"dependsOn": [
{
"activity": "JSON as STRING",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "split_array",
"value": {
"value": "#split(variables('jsonasstring'), variables('ids'))",
"type": "Expression"
}
}
},
{
"name": "ForEach to append keys to array",
"type": "ForEach",
"dependsOn": [
{
"activity": "after split",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "#skip(variables('split_array'),1)",
"type": "Expression"
},
"isSequential": true,
"activities": [
{
"name": "Append variable1",
"type": "AppendVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "key_ids_array",
"value": {
"value": "#take(item(), 9)",
"type": "Expression"
}
}
}
]
}
},
{
"name": "ForEach to access inner object",
"type": "ForEach",
"dependsOn": [
{
"activity": "ForEach to append keys to array",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "#variables('key_ids_array')",
"type": "Expression"
},
"isSequential": true,
"activities": [
{
"name": "Each object",
"type": "SetVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "show_res",
"value": {
"value": "#string(activity('Lookup1').output.value[0].results.timesheets[item()])",
"type": "Expression"
}
}
}
]
}
}
],
"variables": {
"jsonasstring": {
"type": "String"
},
"ids": {
"type": "String"
},
"split_array": {
"type": "Array"
},
"key_ids_array": {
"type": "Array"
},
"show_res": {
"type": "String"
}
},
"annotations": []
}
}
Result:
use #json() in the dynamic content to convert the below string to an object.
If you want to store this JSONs in a file, use a ForEach and inside Foreach use an SQL script to copy each object as a row to table in each iteration using JSON_VALUE . Then outside Foreach use copy activity to copy that SQL table to your destination as per your requirement.