scala ujson.read() returns ujson.Obj - json

I am trying to read a json string using Li Haoyi's ujson. This is the string:
{
"dataflows": [
{
"name": "test",
"sources": [
{
"name": "person_inputs",
"path": "/data/input/events/person/*",
"format": "JSON"
}
],
"transformations": [
{
"name": "validation",
"type": "validate_fields",
"params": {
"input": "person_inputs",
"validations": [
{
"field": "office",
"validations": [
"notEmpty"
]
},
{
"field": "age",
"validations": [
"notNull"
]
}
]
}
},
{
"name": "ok_with_date",
"type": "add_fields",
"params": {
"input": "validation_ok",
"addFields": [
{
"name": "dt",
"function": "current_timestamp"
}
]
}
}
],
"sinks": [
{
"input": "ok_with_date",
"name": "raw-ok",
"paths": [
"/data/output/events/person"
],
"format": "JSON",
"saveMode": "OVERWRITE"
},
{
"input": "validation_ko",
"name": "raw-ko",
"paths": [
"/data/output/discards/person"
],
"format": "JSON",
"saveMode": "OVERWRITE"
}
]
}
]
}
And this is how I read it:
val j = os.read(os.pwd/RelPath("src/main/scala/metadata.json"))
val jsonData = ujson.read(j)
But, the return type is ujson.Obj, and not Arr(ArrayBuffer(Obj), as expected, such that when I try to get jsonData(0), what I get is json.Value$InvalidData: Expected ujson.Arr.
I am asking this question because I have tried to use the ujson object to create a upickle object, but I cannot, and I suspect it is because of this initial error.
Any ideas of why this happens? Any help would be greatly appreciated! Thanks in advance!!

The outer element of your JSON is not an array, it is an object with a single element dataflows whose value is an array. Try jsonData("dataflows")(0).

Related

Parsing JSON array to individual variables in Azure logic apps

I am trying to write a logic app to parse a Json Object and Update salesforce record. I am pretty new to both Salesforce and Azure logic apps, so I am trying to figure this out. Below is my Json File
{
"ContactId": null,
"Email": "asong#uog.com",
"IsInternalUpdate": false,
"Preferences": [
{
"PrefCode": "EmailOptIn",
"CurrentValue": "Yes",
"Locale": "en-US"
},
{
"PrefCode": "MobilePhone",
"CurrentValue": "1234567890",
"Locale": "en-US"
},
{
"PrefCode": "SMSOptIn",
"CurrentValue": "Yes",
"Locale": "en-US"
},
{
"PrefCode": "ProductTrends",
"CurrentValue": "ProductTrends,OffersPromotions",
"Locale": "en-US"
},
]
}
Based on email value, I need to update a custom object in Salesforce. From the preference array, Prefcode value maps to a field in Salesforce and Current value maps to field value. i.e below snippet translates to set the value for EmailOptIn field in Salesforce to "Yes"
{
"PrefCode": "EmailOptIn",
"CurrentValue": "Yes",
"Locale": "en-US"
}
So far, I was able to pass hardcoded values and successfully update salesforce record from logic app.
I am trying to set individual variables for each field, so that I can pass it directly to salesforce. I have two issues that I am running into
What is the best way to capture the field value mapping?
I have couple of fields that allow multi select, how do I set the multiselect values. Below is an example
{
"PrefCode": "ProductTrends",
"CurrentValue": "ProductTrends,OffersPromotions",
"Locale": "en-US"
}
Below is my logic app structure
1
2
What is the best way to capture the field value mapping?
I could able to achieve your requirement by constructing JSON with required details and used parse JSON step again. Below is the flow of the logic app that worked for me.
I have couple of fields that allow multi select, how do I set the multiselect values. Below is an example
I have converted the ProductTrends into array and retrieved each item using its index as below.
array(split(body('Parse_JSON_2')?['ProductTrends'],','))[0]
array(split(body('Parse_JSON_2')?['ProductTrends'],','))[1]
Alternatively, if it is possible to make things work from the target end for multiple values, you can send in array or string directly to salesforce.
Below is the complete JSON of my logic app.
{
"definition": {
"$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
"actions": {
"Compose": {
"inputs": {
"ContactId": null,
"Email": "asong#uog.com",
"IsInternalUpdate": false,
"Preferences": [
{
"CurrentValue": "Yes",
"Locale": "en-US",
"PrefCode": "EmailOptIn"
},
{
"CurrentValue": "1234567890",
"Locale": "en-US",
"PrefCode": "MobilePhone"
},
{
"CurrentValue": "Yes",
"Locale": "en-US",
"PrefCode": "SMSOptIn"
},
{
"CurrentValue": "ProductTrends,OffersPromotions",
"Locale": "en-US",
"PrefCode": "ProductTrends"
}
]
},
"runAfter": {},
"type": "Compose"
},
"Compose_2": {
"inputs": "#array(split(body('Parse_JSON_2')?['ProductTrends'],','))[0]",
"runAfter": {
"Parse_JSON_2": [
"Succeeded"
]
},
"type": "Compose"
},
"For_each": {
"actions": {
"Append_to_string_variable": {
"inputs": {
"name": "Preferences",
"value": "\"#{items('For_each')?['PrefCode']}\":\"#{items('For_each')?['CurrentValue']}\",\n"
},
"runAfter": {},
"type": "AppendToStringVariable"
}
},
"foreach": "#body('Parse_JSON')?['Preferences']",
"runAfter": {
"Initialize_variable": [
"Succeeded"
]
},
"type": "Foreach"
},
"Initialize_variable": {
"inputs": {
"variables": [
{
"name": "Preferences",
"type": "string"
}
]
},
"runAfter": {
"Parse_JSON": [
"Succeeded"
]
},
"type": "InitializeVariable"
},
"Parse_JSON": {
"inputs": {
"content": "#outputs('Compose')",
"schema": {
"properties": {
"ContactId": {},
"Email": {
"type": "string"
},
"IsInternalUpdate": {
"type": "boolean"
},
"Preferences": {
"items": {
"properties": {
"CurrentValue": {
"type": "string"
},
"Locale": {
"type": "string"
},
"PrefCode": {
"type": "string"
}
},
"required": [
"PrefCode",
"CurrentValue",
"Locale"
],
"type": "object"
},
"type": "array"
}
},
"type": "object"
}
},
"runAfter": {
"Compose": [
"Succeeded"
]
},
"type": "ParseJson"
},
"Parse_JSON_2": {
"inputs": {
"content": "#concat('{',slice(variables('Preferences'),0,lastIndexOf(variables('Preferences'),',')),'}')",
"schema": {
"properties": {
"EmailOptIn": {
"type": "string"
},
"MobilePhone": {
"type": "string"
},
"ProductTrends": {
"type": "string"
},
"SMSOptIn": {
"type": "string"
}
},
"type": "object"
}
},
"runAfter": {
"For_each": [
"Succeeded"
]
},
"type": "ParseJson"
}
},
"contentVersion": "1.0.0.0",
"outputs": {},
"parameters": {},
"triggers": {
"manual": {
"inputs": {
"schema": {}
},
"kind": "Http",
"type": "Request"
}
}
},
"parameters": {}
}
After following the above process, I could able to get the preferences values individually.

jsonschema Required properties inoperative with $ref

I am going to write json schema to verify tree data.
Schema consisting of top root and block below.
There may be another block below the block.
Schema for validation.
schema = {
"$schema": "http://json-schema.org/draft-04/schema",
"$ref": "#/definitions/root",
"definitions":{
"root": {
"properties": {
"name": {
"type": "string"
},
"children": {
"type": "array",
"items": [
{"$ref":"#/definitions/block"}
]
}
},
"required": ["name", "children"]
},
"block": {
"properties": {
"name": {
"type": "string"
},
"children": {
"type": "array",
"items": [
{"$ref":"#/definitions/block"}
]
}
},
"required": ["name"]
}
}
}
Below is incorrect data for testing. The last name properties do not exist.
{
"name": "group8",
"children": [
{
"name": "group7",
"children": [
{
"name": "group6",
"children": [
{
"name": "group5",
"children": [
{ ###### wrong
"children": []
}
]
}
]
}
]
}
]
}
This data validates well, but it doesn't work on a slightly complex tree.
# Error: ValidationError: file /home/gulliver/.local/lib/python2.7/site-packages/jsonschema/validators.py line 934: 'name' is a required property #
{
"name": "group8",
"children": [
{
"name": "group7",
"children": [
{
"name": "group6",
"children": [
{
"name": "group12",
"children": [
{
"name": "group11",
"children": [
{
"name": "group10",
"children": []
}
]
}
]
},
{
"name": "group9",
"children": [
{
"name": "group5",
"children": [
{ ####### wrong
"children": []
}
]
}
]
}
]
}
]
},
{
"name": "group13",
"children": [
{
"name": "null1",
"children": []
}
]
}
]
}
It does not work when the data at the bottom of the tree is invalid.
My guess is that the branch splits and this happens, does anyone know why or how to fix it?
I tested using python and jsonschema.
When items is an array, it applies the subschema values to the same index location in the array in the instance.
For example, where you define...
"items": [
{"$ref":"#/definitions/block"}
]
only the first item in the array will be tested. It has nothing to do with deep nesting. For example, the follwing data is valid according to your schema...
{
"name": "group8",
"children": [
{
"name": "group7"
},
{
"something": "else",
"Not": "name"
}
]
}
(Live demo: https://jsonschema.dev/s/etFGE)
If you modify your use of items, then it will work like you expect:
"items": {"$ref":"#/definitions/block"}
(do this for both uses)
Live demo: https://jsonschema.dev/s/rk1OD

JSON Schema with Nested Objects with different properties

The entire JSON file is rather large so I've only taken out the subsection I've had an issue with.
{
"diagrams": {
"5f759d15cd046720c28531dd": {
"_id": "5f759d15cd046720c28531dd",
"offsetX": 320,
"offsetY": 42,
"zoom": 80,
"modified": 1604279356,
"nodes": {
"5f9f5c3ccd046720c28531e4": {
"nodeID": "5f9f5c3ccd046720c28531e4",
"type": "start",
"coords": [
360,
120
],
"data": {
"name": "Start",
"color": "standard",
"ports": [
{
"type": "",
"target": "5f9f5c3ccd046720c28531e6"
}
],
"steps": []
}
},
"5f9f5c3ccd046720c28531e5": {
"nodeID": "5f9f5c3ccd046720c28531e5",
"type": "block",
"coords": [
760,
120
],
"data": {
"name": "Help Message",
"color": "standard",
"steps": [
"5f9f5c3ccd046720c28531e6",
"5f9f5c3ccd046720c28531e7"
]
}
},
"5f9f5c3ccd046720c28531e6": {
"nodeID": "5f9f5c3ccd046720c28531e6",
"type": "speak",
"data": {
"randomize": false,
"dialogs": [
{
"voice": "Alexa",
"content": "You said help. Do you want to continue?"
}
],
"ports": [
{
"type": "",
"target": "5f9f5c3ccd046720c28531e7"
}
]
}
},
"5f9f5c3ccd046720c28531e7": {
"nodeID": "5f9f5c3ccd046720c28531e7",
"type": "interaction",
"data": {
"name": "Choice",
"else": {
"type": "path",
"randomize": false,
"reprompts": []
},
"choices": [
{
"intent": "",
"mappings": []
},
{
"intent": "",
"mappings": []
}
],
"reprompt": null,
"ports": [
{
"type": "else",
"target": null
},
{
"type": "",
"target": null
},
{
"type": "",
"target": "5f9f5c3ccd046720c28531e9"
}
]
}
},
"5f9f5c3ccd046720c28531e8": {
"nodeID": "5f9f5c3ccd046720c28531e8",
"type": "block",
"coords": [
1170,
260
],
"data": {
"name": "Exit",
"color": "standard",
"steps": [
"5f9f5c3ccd046720c28531e9"
]
}
},
"5f9f5c3ccd046720c28531e9": {
"nodeID": "5f9f5c3ccd046720c28531e9",
"type": "exit",
"data": {
"ports": []
}
}
},
"children": [],
"creatorID": 42661,
"variables": [],
"name": "Help Flow",
"versionID": "5f759d15cd046720c28531db"
}
}
}
The Current JSON Schema Definition I have is:
{
"$schema":"http://json-schema.org/schema#",
"type":"object",
"properties":{
"diagrams":{
"type":"object"
}
},
"required":[
"diagrams",
]
}
The problem I am having is that within diagrams contains multiple objects with a random string as the name e.g "5f759d15cd046720c28531dd".
Then within that object there are properties such as (_id, offsetX) which I want to express as well as a nodes object, which again contains multiple objects with arbitrary names e.g ("5f9f5c3ccd046720c28531e4", "5f9f5c3ccd046720c28531e5", ...) which have a unique node definition where some nodes have different properties to other nodes (nodeID, type, data vs nodeID, type, data, coords).
My question is with all these arbitrary things such as random names as well as different properties per each node. How do I turn it into 1 JSON schema definition which covers all the cases of how a diagram/node can be made.
You can do this with additionalProperties or patternProperties.
additionalProperties applies to any property that isn't declared in properties or patternProperties.
{
"type": "object",
"additionalProperties": {
"type": "object",
"properties": {
"_id": { ... },
"offsetX": { ... },
...
}
}
}
Your property names appear to always be hex numbers. If you want to enforce that those property names are always hex numbers, you can use patternProperties. Any property that matches the regex must conform to that schema.
{
"type": "object",
"patternProperties": {
"^[0-9a-f]{24}$": {
"type": "object",
"properties": {
"_id": { ... },
"offsetX": { ... },
...
}
}
},
"additionalProperties": false
}

AVRO schema for JSON

I have a JSON which gets generated like this. I wanted to know what would the avro schema for this would be. The number of keys values in array list is not fixed. There are related posts but they have the keys referenced and do not change. In my case the keys change. The names of the variable keys keeps on changing.
"fixedKey": [
{
"variableKey1": 2
},
{
"variableKey2": 1
},
{
"variableKey3": 3
},
.....
{
"variableKeyN" : 10
}
]
The schema should be something like this:
{
"type": "record",
"name": "test",
"fields": [
{
"name": "fixedKey",
"type": {
"type": "array",
"items": [
{"type": "map", "values": "int"},
],
},
}
],
}
Here's an example of serializing and deserializing your example data:
from io import BytesIO
from fastavro import writer, reader
schema = {
"type": "record",
"name": "test",
"fields": [
{
"name": "fixedKey",
"type": {
"type": "array",
"items": [
{"type": "map", "values": "int"},
],
},
}
],
}
records = [
{
"fixedKey": [
{
"variableKey1": 1,
},
{
"variableKey2": 2,
},
{
"variableKey3": 3,
},
]
}
]
bio = BytesIO()
writer(bio, schema, records)
bio.seek(0)
for record in reader(bio):
print(record)

Read Array Value Using Dataweave in Mule

I am trying to use dataweave in Mule to read specific data values from an incoming payload. My sample payload looks like below:
{
"source": [
{
"uri": "entities/1R6xV",
"createdBy": "API_USER",
"createdTime": 1562504739146,
"attributes": {
"label": "000000000002659654",
"value": {
"Name": [
{
}
],
"Id": [
{
}
],
"Number": [
{
"type": "config/Types/Number/attributes/Number",
"ov": true,
"value": "000000000002659654",
"uri": "entities/1R6xV/attributes/Num/1ZtyT/Number/60pvN6"
}
]
}
}
}
]
}
If I need to read the "label", I can achieve that by
label: payload.source.attributes.label
Similarly, how can I read the "value" under attributes > Number. It doesn't work by:
Value: payload.source.attributes.Number.value
I am new to Dataweave. Please advise.
The problem is that the dot selector (.) works on object and on array of objects. When it is applied to an array it will apply the dot selector to all the elements of the array that are of type object and return that result.
Lets go part by part
payload.source
Returns
[
{
"uri": "entities/1R6xV",
"createdBy": "API_USER",
"createdTime": 1562504739146,
"attributes": {
"label": "000000000002659654",
"value": {
"Name": [
{
}
],
"Id": [
{
}
],
"Number": [
{
"type": "config/Types/Number/attributes/Number",
"ov": true,
"value": "000000000002659654",
"uri": "entities/1R6xV/attributes/Num/1ZtyT/Number/60pvN6"
}
]
}
}
}
]
So far so good as payload is an Object it returns the value of source that is an array
payload.source.attributes
Returns
[
{
"label": "000000000002659654",
"value": {
"Name": [
{
}
],
"Id": [
{
}
],
"Number": [
{
"type": "config/Types/Number/attributes/Number",
"ov": true,
"value": "000000000002659654",
"uri": "entities/1R6xV/attributes/Num/1ZtyT/Number/60pvN6"
}
]
}
}
]
Works ok because the result of payload.source was ended an Array of object so it will do that selection over those objects.
Now when you execute
payload.source.attributes.value.Number
It returns
[
[
{
"type": "config/Types/Number/attributes/Number",
"ov": true,
"value": "000000000002659654",
"uri": "entities/1R6xV/attributes/Num/1ZtyT/Number/60pvN6"
}
]
]
That is an array of arrays and here is where it is broken.
My Solution
You have two alternatives here
Use flatten function
flatten(payload.source.attributes.value.Number).value
Use descendant selector
payload.source.attributes.value.Number..value
Since Number is an array, you need to specify the index you want. In this case, the zeroth element:
Value: payload.source[0].attributes.value.Number[0].value
If you have multiple numbers, it would look something like this:
%dw 1.0
%output application/json
---
values: payload.source[0].attributes.value.Number map {
value: $.value
}