Related
I am trying to write a logic app to parse a Json Object and Update salesforce record. I am pretty new to both Salesforce and Azure logic apps, so I am trying to figure this out. Below is my Json File
{
"ContactId": null,
"Email": "asong#uog.com",
"IsInternalUpdate": false,
"Preferences": [
{
"PrefCode": "EmailOptIn",
"CurrentValue": "Yes",
"Locale": "en-US"
},
{
"PrefCode": "MobilePhone",
"CurrentValue": "1234567890",
"Locale": "en-US"
},
{
"PrefCode": "SMSOptIn",
"CurrentValue": "Yes",
"Locale": "en-US"
},
{
"PrefCode": "ProductTrends",
"CurrentValue": "ProductTrends,OffersPromotions",
"Locale": "en-US"
},
]
}
Based on email value, I need to update a custom object in Salesforce. From the preference array, Prefcode value maps to a field in Salesforce and Current value maps to field value. i.e below snippet translates to set the value for EmailOptIn field in Salesforce to "Yes"
{
"PrefCode": "EmailOptIn",
"CurrentValue": "Yes",
"Locale": "en-US"
}
So far, I was able to pass hardcoded values and successfully update salesforce record from logic app.
I am trying to set individual variables for each field, so that I can pass it directly to salesforce. I have two issues that I am running into
What is the best way to capture the field value mapping?
I have couple of fields that allow multi select, how do I set the multiselect values. Below is an example
{
"PrefCode": "ProductTrends",
"CurrentValue": "ProductTrends,OffersPromotions",
"Locale": "en-US"
}
Below is my logic app structure
1
2
What is the best way to capture the field value mapping?
I could able to achieve your requirement by constructing JSON with required details and used parse JSON step again. Below is the flow of the logic app that worked for me.
I have couple of fields that allow multi select, how do I set the multiselect values. Below is an example
I have converted the ProductTrends into array and retrieved each item using its index as below.
array(split(body('Parse_JSON_2')?['ProductTrends'],','))[0]
array(split(body('Parse_JSON_2')?['ProductTrends'],','))[1]
Alternatively, if it is possible to make things work from the target end for multiple values, you can send in array or string directly to salesforce.
Below is the complete JSON of my logic app.
{
"definition": {
"$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
"actions": {
"Compose": {
"inputs": {
"ContactId": null,
"Email": "asong#uog.com",
"IsInternalUpdate": false,
"Preferences": [
{
"CurrentValue": "Yes",
"Locale": "en-US",
"PrefCode": "EmailOptIn"
},
{
"CurrentValue": "1234567890",
"Locale": "en-US",
"PrefCode": "MobilePhone"
},
{
"CurrentValue": "Yes",
"Locale": "en-US",
"PrefCode": "SMSOptIn"
},
{
"CurrentValue": "ProductTrends,OffersPromotions",
"Locale": "en-US",
"PrefCode": "ProductTrends"
}
]
},
"runAfter": {},
"type": "Compose"
},
"Compose_2": {
"inputs": "#array(split(body('Parse_JSON_2')?['ProductTrends'],','))[0]",
"runAfter": {
"Parse_JSON_2": [
"Succeeded"
]
},
"type": "Compose"
},
"For_each": {
"actions": {
"Append_to_string_variable": {
"inputs": {
"name": "Preferences",
"value": "\"#{items('For_each')?['PrefCode']}\":\"#{items('For_each')?['CurrentValue']}\",\n"
},
"runAfter": {},
"type": "AppendToStringVariable"
}
},
"foreach": "#body('Parse_JSON')?['Preferences']",
"runAfter": {
"Initialize_variable": [
"Succeeded"
]
},
"type": "Foreach"
},
"Initialize_variable": {
"inputs": {
"variables": [
{
"name": "Preferences",
"type": "string"
}
]
},
"runAfter": {
"Parse_JSON": [
"Succeeded"
]
},
"type": "InitializeVariable"
},
"Parse_JSON": {
"inputs": {
"content": "#outputs('Compose')",
"schema": {
"properties": {
"ContactId": {},
"Email": {
"type": "string"
},
"IsInternalUpdate": {
"type": "boolean"
},
"Preferences": {
"items": {
"properties": {
"CurrentValue": {
"type": "string"
},
"Locale": {
"type": "string"
},
"PrefCode": {
"type": "string"
}
},
"required": [
"PrefCode",
"CurrentValue",
"Locale"
],
"type": "object"
},
"type": "array"
}
},
"type": "object"
}
},
"runAfter": {
"Compose": [
"Succeeded"
]
},
"type": "ParseJson"
},
"Parse_JSON_2": {
"inputs": {
"content": "#concat('{',slice(variables('Preferences'),0,lastIndexOf(variables('Preferences'),',')),'}')",
"schema": {
"properties": {
"EmailOptIn": {
"type": "string"
},
"MobilePhone": {
"type": "string"
},
"ProductTrends": {
"type": "string"
},
"SMSOptIn": {
"type": "string"
}
},
"type": "object"
}
},
"runAfter": {
"For_each": [
"Succeeded"
]
},
"type": "ParseJson"
}
},
"contentVersion": "1.0.0.0",
"outputs": {},
"parameters": {},
"triggers": {
"manual": {
"inputs": {
"schema": {}
},
"kind": "Http",
"type": "Request"
}
}
},
"parameters": {}
}
After following the above process, I could able to get the preferences values individually.
Wanted to pick your brains on something
So, in Azure data factory, I am running a set of activities which at the end of the run produce a json segment
{"name":"myName", "email":"email#somewhere.com", .. <more elements> }
This set of activities occurs in a loop - Loop Until activity.
My goal is to have a final JSON object like this:
"profiles":[
{"name":"myName", "email":"email#somewhere.com", .. <more elements> },
{"name":"myName", "email":"email#somewhere.com", .. <more elements> },
{"name":"myName", "email":"email#somewhere.com", .. <more elements> },
...
{"name":"myName", "email":"email#somewhere.com", .. <more elements> }
]
That is a concatenation of all the individual ones.
To put in perspective, each individual item is a paged data from a rest api - and all them constitute the final response. I have no control over how many are there.
I understand how to concatenate individual items using 2 variables
jsonTemp = #concat(finalJson, individualResponse)
finalJson = jsonTemp
But, I do not know how to make it all under the single roof "profiles" afterwards.
So this is a bit of a hacky way of doing it and happy to hear a better solution.
I'm assuming you have stored all your results in an array variable (let's call this A).
First step is to find the number of elements in this array. You can
do this using the length(..) function.
Then you go into a loop, interating a counter variable and
concatenating each of the elements of the array making sure you add
a ',' in between each element. You have to make sure you do not add
the ',' after the last element(You will need to use an IF condition
to check if your counter has reached the length of the array. At the
end of this you should have 1 string variable like this.
{"name":"myName","email":"email#somewhere.com"},{"name":"myName","email":"email#somewhere.com"},{"name":"myName","email":"email#somewhere.com"},{"name":"myName","email":"email#somewhere.com"}
Now all you need to do this this expression when you are pushing the
response anywhere.
#json(concat('{"profiles":[',<your_string_variable_here>,']}'))
I agree with #Anupam Chand and I am following the same process with a different Second step.
You mentioned that your object data comes from API pages and to end the until loop you have to give a condition to about the number of pages(number of iterations).
This is my pipeline:
First I have initilized a counter and used that counter each web page URL and in until condition to meet a certain number of pages. As ADF do not support self referencing variables, I have used another temporary variable to increment the counter.
To Store the objects of each iteration from web activity, I have created an array variable in pipeline.
Inside ForEach, use append variable activity to append each object to the array like below.
For my sample web activity the dynamic content will be #activity('Data from REST').output.data[0]. for you it will be like #activity('Web1').output. change it as per your requirement.
This is my Pipeline JSON:
{
"name": "pipeline3",
"properties": {
"activities": [
{
"name": "Until1",
"type": "Until",
"dependsOn": [
{
"activity": "Counter intialization",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"expression": {
"value": "#equals('3', variables('counter'))",
"type": "Expression"
},
"activities": [
{
"name": "Data from REST",
"type": "WebActivity",
"dependsOn": [
{
"activity": "counter in temp variable",
"dependencyConditions": [
"Succeeded"
]
}
],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"url": {
"value": "https://reqres.in/api/users?page=#{variables('counter')}",
"type": "Expression"
},
"method": "GET"
}
},
{
"name": "counter in temp variable",
"type": "SetVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "tempCounter",
"value": {
"value": "#variables('counter')",
"type": "Expression"
}
}
},
{
"name": "Counter increment using temp",
"type": "SetVariable",
"dependsOn": [
{
"activity": "Data from REST",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "counter",
"value": {
"value": "#string(add(int(variables('tempCounter')),1))",
"type": "Expression"
}
}
},
{
"name": "Append web output to array",
"type": "AppendVariable",
"dependsOn": [
{
"activity": "Counter increment using temp",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "arr",
"value": {
"value": "#activity('Data from REST').output.data[0]",
"type": "Expression"
}
}
}
],
"timeout": "0.12:00:00"
}
},
{
"name": "Counter intialization",
"type": "SetVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "counter",
"value": {
"value": "#string('1')",
"type": "Expression"
}
}
},
{
"name": "To show res array",
"type": "SetVariable",
"dependsOn": [
{
"activity": "Until1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "res_show",
"value": {
"value": "#variables('arr')",
"type": "Expression"
}
}
}
],
"variables": {
"arr": {
"type": "Array"
},
"counter": {
"type": "String"
},
"tempCounter": {
"type": "String"
},
"res_show": {
"type": "Array"
},
"arr_string": {
"type": "String"
}
},
"annotations": []
}
}
Result in an array variable:
You can access this array by the variable name. If you want the output to be like yours, you can use the below dynamic content.
#json(concat('{','"profile":',string(variables('res_show')),'}')))
However, if you want to store this in a variable, you have to wrap it in #string() as currently, ADF variables only supports int, string and array type only.
We have a Json object with the following structure.
{
"results": {
"timesheets": {
"135288482": {
"id": 135288482,
"user_id": 1242515,
"jobcode_id": 17288283,
"customfields": {
"19142": "Item 1",
"19144": "Item 2"
},
"attached_files": [
50692,
44878
],
"last_modified": "1970-01-01T00:00:00+00:00"
},
"135288514": {
"id": 135288514,
"user_id": 1242509,
"jobcode_id": 18080900,
"customfields": {
"19142": "Item 1",
"19144": "Item 2"
},
"attached_files": [
50692,
44878
],
"last_modified": "1970-01-01T00:00:00+00:00"
}}
We need to access the elements that is inside the results --> timesheets --> Dynamic id.
Example:
{
"id": 135288482,
"user_id": 1242515,
"jobcode_id": 17288283,
"customfields": {
"19142": "Item 1",
"19144": "Item 2"
},
"attached_files": [
50692,
44878
],
"last_modified": "1970-01-01T00:00:00+00:00"
}
The problem is that "135288482": { is dynamic. How do we access what is inside of it.
We are trying to create data flow to parse the data. The data is dynamic, so accessing via attribute name is not possible.
AFAIK, as per your JSON structure and dynamic keys it might not be possible to get the desired result using Dataflow.
I have reproduced the above and able to get it done using set variable and ForEach like below.
In your JSON, the keys and the values for the ids are same. So, I have used that to get the list of keys first. Then using that list of keys, I am able to access the inner JSON object.
These are my variable in the pipeline:
First, I have a set variable of type string and stored "id" in it.
Then I have taken lookup activity to get the above JSON file from blob.
I have stored lookup the timesheets objects as string in a set variable using the below dynamic content
#string(activity('Lookup1').output.value[0].results.timesheets)
I have used split on that string with "id" and stored the result array in an array variable.
#split(variables('jsonasstring'), variables('ids'))
This will give the array like below.
Now, I took a Foreach to this array but skipped the first element. In Each iteration, I have taken first 9 indexes of the string to append variable and that is the key.
If your id values and keys are not same, then you can skip the last from split array and take the keys from the reverse side of the string as per your requirement.
This is my dynamic content for append variable activity inside ForEach #take(item(), 9)
Then I took another ForEach, and given this keys list array to it. Inside foreach you can access the JSON with below dynamic content.
#string(activity('Lookup1').output.value[0].results.timesheets[item()])
This is my pipeline JSON:
{
"name": "pipeline1",
"properties": {
"activities": [
{
"name": "Lookup1",
"type": "Lookup",
"dependsOn": [
{
"activity": "for ids",
"dependencyConditions": [
"Succeeded"
]
}
],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "JsonSource",
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"recursive": true,
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "JsonReadSettings"
}
},
"dataset": {
"referenceName": "Json1",
"type": "DatasetReference"
},
"firstRowOnly": false
}
},
{
"name": "JSON as STRING",
"type": "SetVariable",
"dependsOn": [
{
"activity": "Lookup1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "jsonasstring",
"value": {
"value": "#string(activity('Lookup1').output.value[0].results.timesheets)",
"type": "Expression"
}
}
},
{
"name": "for ids",
"type": "SetVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "ids",
"value": {
"value": "#string('\"id\":')",
"type": "Expression"
}
}
},
{
"name": "after split",
"type": "SetVariable",
"dependsOn": [
{
"activity": "JSON as STRING",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "split_array",
"value": {
"value": "#split(variables('jsonasstring'), variables('ids'))",
"type": "Expression"
}
}
},
{
"name": "ForEach to append keys to array",
"type": "ForEach",
"dependsOn": [
{
"activity": "after split",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "#skip(variables('split_array'),1)",
"type": "Expression"
},
"isSequential": true,
"activities": [
{
"name": "Append variable1",
"type": "AppendVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "key_ids_array",
"value": {
"value": "#take(item(), 9)",
"type": "Expression"
}
}
}
]
}
},
{
"name": "ForEach to access inner object",
"type": "ForEach",
"dependsOn": [
{
"activity": "ForEach to append keys to array",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "#variables('key_ids_array')",
"type": "Expression"
},
"isSequential": true,
"activities": [
{
"name": "Each object",
"type": "SetVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "show_res",
"value": {
"value": "#string(activity('Lookup1').output.value[0].results.timesheets[item()])",
"type": "Expression"
}
}
}
]
}
}
],
"variables": {
"jsonasstring": {
"type": "String"
},
"ids": {
"type": "String"
},
"split_array": {
"type": "Array"
},
"key_ids_array": {
"type": "Array"
},
"show_res": {
"type": "String"
}
},
"annotations": []
}
}
Result:
use #json() in the dynamic content to convert the below string to an object.
If you want to store this JSONs in a file, use a ForEach and inside Foreach use an SQL script to copy each object as a row to table in each iteration using JSON_VALUE . Then outside Foreach use copy activity to copy that SQL table to your destination as per your requirement.
I am trying to loop over the json below and print or gather all the VpcEndPointId values.
response = {
"VpcEndpoints": [
{
"VpcEndpointId": "vpce-123",
"VpcEndpointType": "GatewayLoadBalancer",
"VpcId": "vpc-test",
"ServiceName": "com.amazonaws.com",
"State": "available",
"SubnetIds": [
"subnet-random"
],
"IpAddressType": "ipv4",
"RequesterManaged": True,
"NetworkInterfaceIds": [
"eni-123"
],
"CreationTimestamp": "2022-10-28T01:23:23.924Z",
"Tags": [
{
"Key": "AWSNetworkFirewallManaged",
"Value": "true"
},
{
"Key": "Firewall",
"Value": "arn:aws:network-firewall:us-west-2"
}
],
"OwnerId": "123"
},
{
"VpcEndpointId": "vpce-123",
"VpcEndpointType": "GatewayLoadBalancer",
"VpcId": "vpc-<value>",
"ServiceName": "com.amazonaws.vpce.us-west-2",
"State": "available",
"SubnetIds": [
"subnet-<number>"
],
"IpAddressType": "ipv4",
"RequesterManaged": True,
"NetworkInterfaceIds": [
"eni-<value>"
],
"CreationTimestamp": "2022-10-28T01:23:42.113Z",
"Tags": [
{
"Key": "AWSNetworkFirewallManaged",
"Value": "True"
},
{
"Key": "Firewall",
"Value": "arn:aws:network-firewall:%l"
}
],
"OwnerId": "random"
}
]
}
The issue I am having is the dictionary being nested inside a list. I've been able to get pass one issue where I can print the VpcEndPointId KEYS in the code below but still trying to figure out how can I print values.
I tried using .values but it appears the type is a string when I try it with the code below
for endpoint in response['VpcEndpoints']:
#for vpc_endpoint in endpoint['VpcEndpointId']:
for vpc_endpoint in endpoint:
if vpc_endpoint == 'VpcEndpointId':
type(vpc_endpoint)
I'm sure there's something I'm missing and there may be a simpler solution so any suggestions should help, thanks!
To print values of VpcEndpointId, it is enough to use one loop:
for endpoint in response['VpcEndpoints']:
print(endpoint['VpcEndpointId'])
I am trying to read a json string using Li Haoyi's ujson. This is the string:
{
"dataflows": [
{
"name": "test",
"sources": [
{
"name": "person_inputs",
"path": "/data/input/events/person/*",
"format": "JSON"
}
],
"transformations": [
{
"name": "validation",
"type": "validate_fields",
"params": {
"input": "person_inputs",
"validations": [
{
"field": "office",
"validations": [
"notEmpty"
]
},
{
"field": "age",
"validations": [
"notNull"
]
}
]
}
},
{
"name": "ok_with_date",
"type": "add_fields",
"params": {
"input": "validation_ok",
"addFields": [
{
"name": "dt",
"function": "current_timestamp"
}
]
}
}
],
"sinks": [
{
"input": "ok_with_date",
"name": "raw-ok",
"paths": [
"/data/output/events/person"
],
"format": "JSON",
"saveMode": "OVERWRITE"
},
{
"input": "validation_ko",
"name": "raw-ko",
"paths": [
"/data/output/discards/person"
],
"format": "JSON",
"saveMode": "OVERWRITE"
}
]
}
]
}
And this is how I read it:
val j = os.read(os.pwd/RelPath("src/main/scala/metadata.json"))
val jsonData = ujson.read(j)
But, the return type is ujson.Obj, and not Arr(ArrayBuffer(Obj), as expected, such that when I try to get jsonData(0), what I get is json.Value$InvalidData: Expected ujson.Arr.
I am asking this question because I have tried to use the ujson object to create a upickle object, but I cannot, and I suspect it is because of this initial error.
Any ideas of why this happens? Any help would be greatly appreciated! Thanks in advance!!
The outer element of your JSON is not an array, it is an object with a single element dataflows whose value is an array. Try jsonData("dataflows")(0).