I need to convert a json(got after multiple calls and then merged it) into a specific structure.
here is my input json payload
[{
"shops": [{
"shop": {
"code": "AU5",
"streetName": "a",
"city": "a",
"district": "a",
"state": "a",
"postalCode": "a",
"country": "a"
}
}, {
"shop": {
"code": "b",
"streetName": "b",
"city": "b",
"district": "b",
"state": "b",
"postalCode": "b",
"country": "b"
}
}]
},
[
[{
"salesOffice": {
"shop": {
"code": "AU5"
},
"office": "MEL",
"branch": "MEL",
"district": "SPR",
"subRegion": "SPR",
"region": "AP"
}
}],
[{
"salesOffice": {
"shop": {
"code": "b"
},
"office": "999",
"branch": "999",
"district": "999",
"subRegion": "999",
"region": "999"
}
}
]
]]
below is the expected output json
{
"shops": [
{
"shop": {
"code": "AU5",
"streetName": "a",
"city": "a",
"district": "a",
"state": "a",
"postalCode": "a",
"country": "a",
"salesOffice": {
"office": "MEL",
"branch": "MEL",
"district": "SPR",
"subRegion": "SPR",
"region": "AP"
}
}
},
{
"shop": {
"code": "b",
"streetName": "b",
"city": "b",
"district": "b",
"state": "b",
"postalCode": "b",
"country": "b",
"salesOffice": {
"office": "999",
"branch": "999",
"district": "999",
"subRegion": "999",
"region": "999"
}
}
}
]}
while transformation, 'code' inside shop should match with 'code' inside salesOffice>>shop>>'code'
below is Json Schema for the output payload(should validate against the output)
{
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "object",
"properties": {
"shops": {
"type": "array",
"items": {
"type": "object",
"properties": {
"shop": {
"type": "object",
"properties": {
"code": {
"type": "string"
},
"streetName": {
"type": "string"
},
"city": {
"type": "string"
},
"district": {
"type": "string"
},
"state": {
"type": "string"
},
"postalCode": {
"type": "string"
},
"country": {
"type": "string"
}
},
"salesOffice": {
"type": "object",
"properties": {
"office": {
"type": "string"
},
"branch": {
"type": "string"
},
"district": {
"type": "string"
},
"subRegion": {
"type": "string"
},
"region": {
"type": "string"
}
}
},
"required": [
"code",
"streetName",
"city",
"district",
"state",
"postalCode",
"country",
"salesOffice"
]
}
},
"required": [
"shop"
]
}
}
},
"required": [
"shops"
]}
Any solution or any pointer would be a great help
Refer below dataweave:
%dw 1.0
%output application/json
---
{
shops: using (salesOffice = payload[1]..salesOffice)( payload[0].shops map {
shop :
{
code: $.shop.code,
streetName:$.shop.streetName,
city:$.shop.city,
district:$.shop.district,
state:$.shop.state,
postalCode:$.shop.postalCode,
country:$.shop.country,
salesOffice: using (code= $.shop.code) ( salesOffice[?(code == $.shop.code)] map {
office:$.office,
branch:$.branch,
district:$.district,
subRegion:$.subRegion,
region:$.region
})[0]
}
})
}
You can use Validate JSON Schema mule components.
Related
Im currently working on a generated collection and got a huge metadata file. Its build up like this:
[
{
"name": "amongnfts #1",
"description": "This is one member of the AmongNFT collection. It's unique and only exists once, but may be a bit cooler than the others.",
"image": "/1.png",
"dna": "055b93fd187e52f85e3ea8b0ad660fa51aaea9e0",
"edition": 1,
"date": 1645368881553,
"attributes": [
{
"trait_type": "background",
"value": "blue"
},
{
"trait_type": "body",
"value": "pink"
},
{
"trait_type": "clothing",
"value": "teamred"
},
{
"trait_type": "hat",
"value": "pilot"
},
{
"trait_type": "visor",
"value": "krieghaus"
},
{
"trait_type": "pet",
"value": "minimateorange"
}
],
"compiler": "HashLips Art Engine"
},
{
"name": "amongnfts #2",
"description": "This is one member of the AmongNFT collection. It's unique and only exists once, but may be a bit cooler than the others.",
"image": "/2.png",
"dna": "a05e9acf4665b9e9e5adbcb9dfc33df255e2e58f",
"edition": 2,
"date": 1645368883622,
"attributes": [
{
"trait_type": "background",
"value": "orange"
},
{
"trait_type": "body",
"value": "white"
},
{
"trait_type": "clothing",
"value": "elf"
},
{
"trait_type": "hat",
"value": "wethair"
},
{
"trait_type": "visor",
"value": "none"
},
{
"trait_type": "pet",
"value": "minimatetan"
}
],
"compiler": "HashLips Art Engine"
},
]
(Goes down for 10.000 editions so thats why I dont do it manually.)
But it should look like this:
{
"nft": [
{
"file_path": "/1.png",
"nft_name": "amongnfts #1",
"external_link": "",
"description": "This is one member of the AmongNFT collection. It's unique and only exists once, but may be a bit cooler than the others.",
"collection": "AmongNFTs",
"properties": [
{
"type": "background",
"name": "blue"
},
{
"type": "body",
"name": "pink"
},
{
"type": "clothing",
"name": "teamred"
},
{
"type": "hat",
"name": "pilot"
},
{
"type": "visor",
"name": "krieghaus"
},
{
"type": "pet",
"name": "minimateorange"
},
],
"levels": "",
"stats": "",
"unlockable_content": "",
"explicit_and_sensitive_content": "",
"supply": "1",
"blockchain": "Polygon",
"sale_type": "",
"price": 0.001,
"method": "",
"duration": "",
"specific_buyer": "",
"quantity": ""
}
]
}
Is there an easy way of converting them automatically and if so, how would the code look like?
The metadata.json file is here.
I been struggling removing set-off objects from Json file. I tried with jq json parser method but nothing has worked out. Could someone please help on this.
What am looking for is – Wherever the below key and value pair are present in a file, the entire object should be removed.
{"name": "exception"}
Input:
{
"results": [
{
"id": "a21f5193-881e-11eb-a0c1-3726f4a71fa9",
"retailerId": "1",
"category": "exception",
"context": {
"sourceEvents": [
"902bd449-881e-11eb-b603-29eb6c297e7d"
],
"entityType": "ORDER"
},
"eventStatus": "FAILED",
"attributes": [
{
"name": "exception",
"value": {
"code": 400,
"message": "Failed to execute http call",
"stackTrace": [
{
"fileName": "ReadOnlyFluentApiClient.java",
"className": "com.fluentretail.api.v2.client.ReadOnlyFluentApiClient"
}
],
"suppressed": [],
"suppressedExceptions": []
},
"type": "OBJECT"
},
{
"name": "lastRule",
"value": "ETOSUAT.base.ProposedFulfilmentWithoutInventory",
"type": "String"
},
{
"name": "lastRuleSet",
"value": "FindAndCreateDigitalFulfilment",
"type": "String"
},
{
"name": "message",
"value": "Failed to execute http call",
"type": "String"
}
],
"source": null,
"generatedBy": "Rubix User",
"generatedOn": "2021-03-18T19:17:51.517+0000"
},
{
"id": "a21f5193-881e-11eb-a0c1-3726f4a71fa9",
"retailerId": "1",
"category": "exception",
"context": {
"sourceEvents": [
"902bd449-881e-11eb-b603-29eb6c297e7d"
],
"entityType": "ORDER"
},
"eventStatus": "FAILED",
"attributes": [
{
"name": "exception",
"value": {
"code": 400,
"message": "Failed to execute http call",
"stackTrace": [
{
"fileName": "ReadOnlyFluentApiClient.java",
"className": "com.fluentretail.api.v2.client.ReadOnlyFluentApiClient"
}
],
"suppressed": [],
"suppressedExceptions": []
},
"type": "OBJECT"
},
{
"name": "lastRule",
"value": "ETOSUAT.base.ProposedFulfilmentWithoutInventory",
"type": "String"
},
{
"name": "lastRuleSet",
"value": "FindAndCreateDigitalFulfilment",
"type": "String"
},
{
"name": "message",
"value": "Failed to execute http call",
"type": "String"
}
],
"source": null,
"generatedBy": "Rubix User",
"generatedOn": "2021-03-18T19:17:51.517+0000"
}
]
}
Expected output is -
{
"results": [
{
"id": "a21f5193-881e-11eb-a0c1-3726f4a71fa9",
"retailerId": "1",
"category": "exception",
"context": {
"sourceEvents": [
"902bd449-881e-11eb-b603-29eb6c297e7d"
],
"entityType": "ORDER"
},
"eventStatus": "FAILED",
"attributes": [
{
"name": "lastRule",
"value": "ETOSUAT.base.ProposedFulfilmentWithoutInventory",
"type": "String"
},
{
"name": "lastRuleSet",
"value": "FindAndCreateDigitalFulfilment",
"type": "String"
},
{
"name": "message",
"value": "Failed to execute http call",
"type": "String"
}
],
"source": null,
"generatedBy": "Rubix User",
"generatedOn": "2021-03-18T19:17:51.517+0000"
},
{
"id": "a21f5193-881e-11eb-a0c1-3726f4a71fa9",
"retailerId": "1",
"category": "exception",
"context": {
"sourceEvents": [
"902bd449-881e-11eb-b603-29eb6c297e7d"
],
"entityType": "ORDER"
},
"eventStatus": "FAILED",
"attributes": [
{
"name": "lastRule",
"value": "ETOSUAT.base.ProposedFulfilmentWithoutInventory",
"type": "String"
},
{
"name": "lastRuleSet",
"value": "FindAndCreateDigitalFulfilment",
"type": "String"
},
{
"name": "message",
"value": "Failed to execute http call",
"type": "String"
}
],
"source": null,
"generatedBy": "Rubix User",
"generatedOn": "2021-03-18T19:17:51.517+0000"
}
]
}
del(..|select(type=="object" and .name=="exception"))
Try it at https://jqplay.org/s/il12Ribpdb
walk(if type=="object" and .name == "exception"
then empty else . end)
Equivalently:
walk(select(type=="object" and .name == "exception" | not))
I am trying to pull a few fields from the following nested json and write to a separate csv file:
{
"AccountID": "00000000-0000-0000-0000-000000000000",
"LocationID": "00000000-0000-0000-0000-000000000000",
"CreatedBy": "string",
"ModifiedBy": "string",
"Created": "string",
"Modified": "string",
"LocationData": {
"KeyFields": {},
"DisplayPoint": {
"Type": "Calculated",
"Latitude": 0.0,
"Longitude": 0.0,
"VerificationType": "Client"
},
"BusinessStatus": "Open",
"Status": "Active",
"BusinessName": {
"Name": "string",
"LongName": "string",
"Locale": "Not_set"
},
"BusinessDescription": {
"Description": "string",
"ShortDescription": "string",
"LongDescription": "string"
},
"PrimaryAddress": {
"AddressLine1": "string",
"AddressLine2": "string",
"AddressLine3": "string",
"AddressLine4": "string",
"AddressLine5": "string",
"Neighborhood": "string",
"Locality": "string",
"Region": "string",
"PostalCode": "string",
"CountryCode": "string"
},
"PhoneNumbers": {
"PrimaryPhoneNumber": "string",
"Landline": "string",
"Mobile": "string",
"Fax": "string",
"TollFree": "string"
},
"HoursOfOperationStructured": {
"Su": {
"Ranges": [
{
"StartTime": "string",
"EndTime": "string"
}
],
"State": "Open",
"AdditionalInfo": "string"
},
"Mo": {
"Ranges": [
{
"StartTime": "string",
"EndTime": "string"
}
],
"State": "Open",
"AdditionalInfo": "string"
},
"Tu": {
"Ranges": [
{
"StartTime": "string",
"EndTime": "string"
}
],
"State": "Open",
"AdditionalInfo": "string"
},
"We": {
"Ranges": [
{
"StartTime": "string",
"EndTime": "string"
}
],
"State": "Open",
"AdditionalInfo": "string"
},
"Th": {
"Ranges": [
{
"StartTime": "string",
"EndTime": "string"
}
],
"State": "Open",
"AdditionalInfo": "string"
},
"Fr": {
"Ranges": [
{
"StartTime": "string",
"EndTime": "string"
}
],
"State": "Open",
"AdditionalInfo": "string"
},
"Sa": {
"Ranges": [
{
"StartTime": "string",
"EndTime": "string"
}
],
"State": "Open",
"AdditionalInfo": "string"
},
"SpecialHours": [
{
"Date": "string",
"Ranges": [
{
"StartTime": "string",
"EndTime": "string"
}
],
"State": "Open",
"AdditionalInfo": "string"
}
]
}
}
I am able to use pandas and json_normalize to flatten the data. Then I am able to pull fields by referencing the fields I want like df['LocationData.PrimaryAddress.Locality']. This works for all the fields I need except the 'StartTime' and 'EndTime' ranges which throw a KeyError
When I try to extract the 'StartTime' or 'Endtime' ranges of any specific day by referencing it like so: df['LocationData.HoursOfOperationStructured.Su.Ranges.StartTime'] ---- it returns with a
KeyError: "['LocationData.HoursOfOperationStructured.Su.Ranges.StartTime'] not in index"
How can I access the 'StartTime'/'EndTime' columns for all of the days from this file using pandas?
The column df['LocationData.HoursOfOperationStructured.Su.Ranges'] and all other similar columns have been "undernormalized": they contain single-element lists of dictionaries with the keys "StartTime" and "EndTime". You can convert these dictionary columns to "real" columns in a loop and then concatenate with the original dataframe:
ranges = df.columns[df.columns.str.match('.*Ranges.*')]
missing = [df[r].str[0].apply(pd.Series)\
.rename(columns={'StartTime' : f"{r}.StartTime",
'EndTime' : f"{r}.EndTime"})
for r in ranges]
df = df.join(pd.concat(missing, axis=1))
It's ugly, but it works.
So I have been using this logic apps template to hit the Google Analytics API and the response is in this format
{
"reports": [
{
"columnHeader": {
"dimensions": [
"ga:date",
"ga:campaign",
"ga:country",
"ga:browser",
"ga:deviceCategory",
"ga:sourceMedium",
"ga:socialNetwork",
"ga:region"
],
"metricHeader": {
"metricHeaderEntries": [
{
"name": "ga:users",
"type": "INTEGER"
},
{
"name": "ga:sessions",
"type": "INTEGER"
},
{
"name": "ga:newUsers",
"type": "INTEGER"
},
{
"name": "ga:bounces",
"type": "INTEGER"
},
{
"name": "ga:pageviews",
"type": "INTEGER"
},
{
"name": "ga:sessionDuration",
"type": "TIME"
},
{
"name": "ga:hits",
"type": "INTEGER"
},
{
"name": "ga:goalCompletionsAll",
"type": "INTEGER"
},
{
"name": "ga:goalConversionRateAll",
"type": "PERCENT"
}
]
}
},
"data": {
"rows": [
{
"dimensions": [
"20200312",
"(not set)",
"India",
"Chrome",
"desktop",
"(direct) / (none)",
"(not set)",
"Tamil Nadu"
],
"metrics": [
{
"values": [
"4",
"4",
"4",
"0",
"111",
"5100.0",
"111",
"0",
"0.0"
]
}
]
},
{
"dimensions": [
"20200316",
"(not set)",
"India",
"Chrome",
"desktop",
"(direct) / (none)",
"(not set)",
"Tamil Nadu"
],
"metrics": [
{
"values": [
"1",
"1",
"0",
"0",
"6",
"266.0",
"6",
"0",
"0.0"
]
}
]
},
{
"dimensions": [
"20200318",
"(not set)",
"India",
"Chrome",
"desktop",
"(direct) / (none)",
"(not set)",
"Tamil Nadu"
],
"metrics": [
{
"values": [
"1",
"2",
"0",
"0",
"20",
"135.0",
"20",
"0",
"0.0"
]
}
]
}
],
"totals": [
{
"values": [
"6",
"7",
"4",
"0",
"137",
"5501.0",
"137",
"0",
"0.0"
]
}
],
"rowCount": 3,
"minimums": [
{
"values": [
"1",
"1",
"0",
"0",
"6",
"135.0",
"6",
"0",
"0.0"
]
}
],
"maximums": [
{
"values": [
"4",
"4",
"4",
"0",
"111",
"5100.0",
"111",
"0",
"0.0"
]
}
],
"isDataGolden": true
}
}
]
}
I Want to convert it and bring it in a form that the column header:dimensions and metric header entries name will become column names and their values,ie data.rows.dimensions and metrics.values become corresponding values
ga:date ga:campaign ga:country ga:browser ga:deviceCategory ga:sourceMedium ga:socialNetwork ga:region ga:users ga:sessions ga:newUsers : (column names)
20200316 (not set) India Chrome desktop (direct) / (none) (not set) Tamil Nadu 1 1 1 :(values)
If you can use an Integration account, I suggest to create a flat file schema with the desired structure, and in the logic app you can convert in xml and then apply the Flat File Encoding.
Otherwise a function app should resolve your issue
I have a json response like below. In the views array may or may not contain actions array.
In the reponse If any of view array contains actions object , then i have to validate that actions data with json schema(schema1.json)
And in the schema i mentioned the action propertes like(type,label,localizedlabel) as required ones.
But when I modify key or value type of this type,label,localizedlabel in the response does not output any errors.
Tested via( https://www.jsonschemavalidator.net/). what wrong with my schema?
How can i validate actions object only whenever it present inside any of view array?
schema1.json
{
"$id": "",
"type": "array",
"items": {
"$id": "/items",
"type": "object",
"properties": {
"name": {
"$id": "/items/properties/name",
"type": "string",
"title": "The Name Schema ",
"default": "",
"examples": [
"Preview"
]
},
"displayOrder": {
"$id": "/items/properties/displayOrder",
"type": "integer",
"title": "The Displayorder Schema ",
"default": 0,
"examples": [
1
]
},
"actions": {
"$id": "/items/properties/actions",
"type": "array",
"items": {
"$id": "/items/properties/actions/items",
"type": "object",
"properties": {
"type": {
"$id": "/items/properties/actions/items/properties/type",
"type": "string",
"title": "The Type Schema ",
"default": "",
"examples": [
"watch"
]
},
"label": {
"$id": "/items/properties/actions/items/properties/label",
"type": "string",
"title": "The Label Schema ",
"default": "",
"examples": [
"Watch"
]
},
"localizedLabel": {
"$id": "/items/properties/actions/items/properties/localizedLabel",
"type": "object",
"properties": {
"ENG": {
"$id": "/items/properties/actions/items/properties/localizedLabel/properties/ENG",
"type": "string",
"title": "The Eng Schema ",
"default": "",
"examples": [
"Watch"
]
},
"ESP": {
"$id": "/items/properties/actions/items/properties/localizedLabel/properties/ESP",
"type": "string",
"title": "The Esp Schema ",
"default": "",
"examples": [
"Ver"
]
}
}
}
},
"required": [
"type",
"label",
"localizedLabel"
]
}
},
"localizedName": {
"$id": "/items/properties/localizedName",
"type": "object",
"properties": {
"ENG": {
"$id": "/items/properties/localizedName/properties/ENG",
"type": "string",
"title": "The Eng Schema ",
"default": "",
"examples": [
"Preview"
]
},
"ESP": {
"$id": "/items/properties/localizedName/properties/ESP",
"type": "string",
"title": "The Esp Schema ",
"default": "",
"examples": [
"Adelanto"
]
}
}
}
},
"required": [
"actions"
]
}
}
response json
[{
"season": 2017,
"teamData": {
"awayTeam": {
"id": 6,
"city": "Dallas",
"name": "Mavericks",
"abbr": "DAL",
"color": "#0B51A1"
},
"homeTeam": {
"id": 8,
"city": "Detroit",
"name": "Pistons",
"abbr": "DET",
"color": "#990300"
}
},
"views": [{
"name": "Preview",
"displayOrder": 1,
"groups": [{
"type": "static",
"displayOrder": 1,
"tiles": [{
"context": "event",
"collection": "event",
"auditType": "pregame-preview",
"displayOrder": 1,
"_id": "5ac58ea21ee2112b33291f1c",
"eventId": 2018040608,
"dimensions": {
"width": 372,
"height": 375
},
"tileId": "36b154e719d7d8397da487cbc4e5f7d1",
"renderTime": "2018-04-05T02:49:05+00:00",
"dataTime": "2018-04-05T02:48:58+00:00",
"dataStamp": 1522896538,
"location": "http://test.com/2018040608/static/pre-event/pregame-preview/1522896538.png",
"tile_type": "static"
}
]
}
],
"actions": [{
"type": "watch",
"label": "Watch",
"localizedLabel": {
"ENG": "Watch",
"ESP": "Ver"
}
}, {
"type": "record",
"label": "Record",
"localizedLabel": {
"ENG": "Record",
"ESP": "Grabar"
}
}, {
"type": "tile_overlay",
"label": "Current Standings",
"tili": {
"context": "event",
"collection": "event",
"auditType": "full-standings",
"_id": "5ac6f9de2ccaf768d092c918",
"eventId": 2018040608,
"dimensions": {
"width": 1140,
"height": 660
},
"tileId": "852f92537e68dc99b54f1228459ec9ef",
"renderTime": "2018-04-06T04:38:54+00:00",
"dataTime": "2018-04-06T04:38:52+00:00",
"dataStamp": 1522989532,
"location": "http://test.com/2018040608/static/pre-event/full-standings/1522989532.png"
},
"localizedLabel": {
"ENG": "Current Standings",
"ESP": "Posición actual"
}
}, {
"type": "favorite",
"label": "Favorite",
"localizedLabel": {
"ENG": "Favorite",
"ESP": "Favorito"
}
}
],
"localizedName": {
"ENG": "Preview",
"ESP": "Adelanto"
}
}, {
"name": "Team Stats",
"displayOrder": 2,
"groups": [{
"type": "static",
"displayOrder": 1,
"tiles": [{
"context": "event",
"collection": "event",
"auditType": "pregame-team_stats",
"displayOrder": 1,
"_id": "5ac6755a4f82eb58a5eae6a6",
"eventId": 2018040608,
"dimensions": {
"width": 372,
"height": 510
},
"tileId": "1302dc16c9fe68c3e6edadd98afce2bc",
"renderTime": "2018-04-05T19:13:30+00:00",
"dataTime": "2018-04-05T19:13:28+00:00",
"dataStamp": 1522955608,
"location": "http://test.com/2018040608/static/pre-event/pregame-team_stats/1522955608.png",
"tile_type": "static"
}
]
}
],
"localizedName": {
"ENG": "Team Stats",
"ESP": "Estadísticas del equipo"
}
}, {
"name": "Leaders",
"displayOrder": 3,
"groups": [{
"type": "static",
"displayOrder": 1,
"tiles": [{
"context": "event",
"collection": "event",
"auditType": "pregame-leaders",
"displayOrder": 1,
"_id": "5ac26eb31ee2112b3328b00c",
"eventId": 2018040608,
"dimensions": {
"width": 372,
"height": 510
},
"tileId": "96abc24c47d61327426ef2b24281acbf",
"renderTime": "2018-04-02T17:55:57+00:00",
"dataTime": "2018-04-02T17:55:54+00:00",
"dataStamp": 1522691754,
"location": "http://test.com/2018040608/static/pre-event/pregame-leaders/1522691754.png",
"tile_type": "static"
}
]
}
],
"localizedName": {
"ENG": "Leaders",
"ESP": "Líderes"
}
}
]
}
]
There's nothing wrong with your schema. It should do what you need if used properly. Because it only describes the "views" part of the schema you would need to iterate through your response and pass just the "views" part of each item to the validator one at a time.
Or, you could add enough of the response structure to the schema to validate everything at once. Then you could just pass your whole response to the validator.
{
"type": "array",
"items": {
"type": "object",
"properties": {
"views": { "$ref": "schema1.json" }
}
}
}