I just seem to have a hard time figuring out the mindset of jq for more than simple extraction problems, but hopefully I can keep growing my skills...
I have data that looks like this synthetic example:
{
"accounts": [
{
"attrs": {
"name": "One"
},
"vaults": [
{
"attrs": {"name": "Private", "type": "P"},
"items": [
{
"foo": "001",
"overview": {"subtitle": "foo"}
},
{
"foo": "002",
"overview": {"subtitle": "foo", "tags": ["a tag"]}
}
]
},
{
"attrs": {"name": "Public", "type": "P"},
"items": [
{
"foo": "001",
"overview": {"subtitle": "foo"}
},
{
"foo": "002",
"overview": {"subtitle": "foo", "tags": ["a tag"]}
}
]
}
]
}
]
}
For every record in .accounts[].vaults[].items[], I need to create/replace its overview.tags value with [.Parent.Parent.attrs.name], if .Parent were to get the current element's parent in the tree.
So the desired output would be this:
{
"accounts": [
{
"attrs": {
"name": "One"
},
"vaults": [
{
"attrs": {"name": "Private", "type": "P"},
"items": [
{
"foo": "001",
"overview": {"subtitle": "foo", "tags": ["Private"]}
},
{
"foo": "002",
"overview": {"subtitle": "foo", "tags": ["Private"]}
}
]
},
{
"attrs": {"name": "Public", "type": "P"},
"items": [
{
"foo": "001",
"overview": {"subtitle": "foo", "tags": ["Public"]}
},
{
"foo": "002",
"overview": {"subtitle": "foo", "tags": ["Public"]}
}
]
}
]
}
]
}
Is there a nice way in jq to achieve this?
Background: this is the format of 1Password's .1pux export data.
One way to do it is using update-assignment |=
jq '.accounts[].vaults |= map(.items[].overview.tags = [.attrs.name])'
Related
I am trying to Extract and sum a particular value using group by from json collection in azure logic app. Trying to achieve this by applying "Condition" connector. My Json file looks like below:
"Release": [
{
"O_Id": "D_13211175",
"R_ID": "D_132111751",
"Res": [
{
"A_Id": "32323222",
"F_Qty": 1,
"I_Id": "828929",
"OL_Id": "1",
"Qty": 1,
"RL_Id": "1",
"Unique_Identifier": "5970703818904601622__1",
},
{
"A_Id": "32323223",
"F_Qty": 1,
"I_Id": "828929",
"OL_Id": "1",
"Qty": 1,
"RL_Id": "1",
"Unique_Identifier": "5970703818904692717__1",
},
]
}
]
I want to calculate the sum of Qty group by I_Id from array Res using logic app connector "Condition".
Note: As Res is an array, there can be repetition I_Id with same value, hence, the requirement to get the Qty count based on I_Id
There are few ways to achieve your requirement but here is one of the workaround that worked when reproduced from our end.
Firstly, I have initialised a temporary integer variable to iterated inside the "Res" Array using Until Loop by checking the length of "Res" using the below expression.
length(body('Parse_JSON')?['Release']?[0]?['Res'])
Inside Until I have used condition action to check if I_Id is same using the below expression to the left.
body('Parse_JSON')?['Release']?[0]?['Res']?[variables('temp')]?['I_Id']
and below expression to the right
body('Parse_JSON')?['Release']?[0]?['Res']?[add(variables('temp'),1)]?['I_Id']
Then if the condition satisfies I'm adding the Qty using the expression as below.
add(body('Parse_JSON')?['Release']?[0]?['Res']?[variables('temp')]?['Qty'],body('Parse_JSON')?['Release']?[0]?['Res']?[add(variables('temp'),1)]?['Qty'])
Here is the flow of my logic app
[Optional step]
Considering the bigger picture, I have initialzed another variable quantity and added the resultant of Compose 4 to Qty variable
RESULTS:
Considering the below sample JSON
{
"Release": [
{
"O_Id": "D_13211175",
"R_ID": "D_132111751",
"Res": [
{
"A_Id": "32323222",
"F_Qty": 1,
"I_Id": "828929",
"OL_Id": "1",
"Qty": 1,
"RL_Id": "1",
"Unique_Identifier": "5970703818904601622__1"
},
{
"A_Id": "32323223",
"F_Qty": 1,
"I_Id": "828929",
"OL_Id": "1",
"Qty": 1,
"RL_Id": "1",
"Unique_Identifier": "5970703818904692717__1"
},
{
"A_Id": "32323223",
"F_Qty": 1,
"I_Id": "828929",
"OL_Id": "1",
"Qty": 5,
"RL_Id": "1",
"Unique_Identifier": "5970703818904692717__1"
}
]
}
]
}
Here is the result
To avoid confusion try the below code view in your resource
{
"definition": {
"$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
"actions": {
"Compose": {
"inputs": {
"Release": [
{
"O_Id": "D_13211175",
"R_ID": "D_132111751",
"Res": [
{
"A_Id": "32323222",
"F_Qty": 1,
"I_Id": "828929",
"OL_Id": "1",
"Qty": 1,
"RL_Id": "1",
"Unique_Identifier": "5970703818904601622__1"
},
{
"A_Id": "32323223",
"F_Qty": 1,
"I_Id": "828929",
"OL_Id": "1",
"Qty": 1,
"RL_Id": "1",
"Unique_Identifier": "5970703818904692717__1"
},
{
"A_Id": "32323223",
"F_Qty": 1,
"I_Id": "828929",
"OL_Id": "1",
"Qty": 5,
"RL_Id": "1",
"Unique_Identifier": "5970703818904692717__1"
}
]
}
]
},
"runAfter": {},
"type": "Compose"
},
"Compose_3": {
"inputs": "#variables('Qty')",
"runAfter": {
"Until": [
"Succeeded"
]
},
"type": "Compose"
},
"Initialize_variable": {
"inputs": {
"variables": [
{
"name": "temp",
"type": "integer",
"value": 0
}
]
},
"runAfter": {
"Parse_JSON": [
"Succeeded"
]
},
"type": "InitializeVariable"
},
"Initialize_variable_2": {
"inputs": {
"variables": [
{
"name": "Qty",
"type": "integer",
"value": 0
}
]
},
"runAfter": {
"Initialize_variable": [
"Succeeded"
]
},
"type": "InitializeVariable"
},
"Parse_JSON": {
"inputs": {
"content": "#outputs('Compose')",
"schema": {
"properties": {
"Release": {
"items": {
"properties": {
"O_Id": {
"type": "string"
},
"R_ID": {
"type": "string"
},
"Res": {
"items": {
"properties": {
"A_Id": {
"type": "string"
},
"F_Qty": {
"type": "integer"
},
"I_Id": {
"type": "string"
},
"OL_Id": {
"type": "string"
},
"Qty": {
"type": "integer"
},
"RL_Id": {
"type": "string"
},
"Unique_Identifier": {
"type": "string"
}
},
"required": [
"A_Id",
"F_Qty",
"I_Id",
"OL_Id",
"Qty",
"RL_Id",
"Unique_Identifier"
],
"type": "object"
},
"type": "array"
}
},
"required": [
"O_Id",
"R_ID",
"Res"
],
"type": "object"
},
"type": "array"
}
},
"type": "object"
}
},
"runAfter": {
"Compose": [
"Succeeded"
]
},
"type": "ParseJson"
},
"Until": {
"actions": {
"Condition": {
"actions": {
"Compose_2": {
"inputs": "#add(outputs('Compose_4'),variables('Qty'))",
"runAfter": {
"Compose_4": [
"Succeeded"
]
},
"type": "Compose"
},
"Compose_4": {
"inputs": "#add(body('Parse_JSON')?['Release']?[0]?['Res']?[variables('temp')]?['Qty'],body('Parse_JSON')?['Release']?[0]?['Res']?[add(variables('temp'),1)]?['Qty'])",
"runAfter": {},
"type": "Compose"
},
"Set_variable": {
"inputs": {
"name": "Qty",
"value": "#outputs('Compose_2')"
},
"runAfter": {
"Compose_2": [
"Succeeded"
]
},
"type": "SetVariable"
}
},
"expression": {
"and": [
{
"equals": [
"#body('Parse_JSON')?['Release']?[0]?['Res']?[variables('temp')]?['I_Id']",
"#body('Parse_JSON')?['Release']?[0]?['Res']?[add(variables('temp'),1)]?['I_Id']"
]
}
]
},
"runAfter": {},
"type": "If"
},
"Increment_variable": {
"inputs": {
"name": "temp",
"value": 1
},
"runAfter": {
"Condition": [
"Succeeded"
]
},
"type": "IncrementVariable"
}
},
"expression": "#equals(variables('temp'), length(body('Parse_JSON')?['Release']?[0]?['Res']))",
"limit": {
"count": 60,
"timeout": "PT1H"
},
"runAfter": {
"Initialize_variable_2": [
"Succeeded"
]
},
"type": "Until"
}
},
"contentVersion": "1.0.0.0",
"outputs": {},
"parameters": {},
"triggers": {
"manual": {
"inputs": {
"schema": {}
},
"kind": "Http",
"type": "Request"
}
}
},
"parameters": {}
}
I am going to write json schema to verify tree data.
Schema consisting of top root and block below.
There may be another block below the block.
Schema for validation.
schema = {
"$schema": "http://json-schema.org/draft-04/schema",
"$ref": "#/definitions/root",
"definitions":{
"root": {
"properties": {
"name": {
"type": "string"
},
"children": {
"type": "array",
"items": [
{"$ref":"#/definitions/block"}
]
}
},
"required": ["name", "children"]
},
"block": {
"properties": {
"name": {
"type": "string"
},
"children": {
"type": "array",
"items": [
{"$ref":"#/definitions/block"}
]
}
},
"required": ["name"]
}
}
}
Below is incorrect data for testing. The last name properties do not exist.
{
"name": "group8",
"children": [
{
"name": "group7",
"children": [
{
"name": "group6",
"children": [
{
"name": "group5",
"children": [
{ ###### wrong
"children": []
}
]
}
]
}
]
}
]
}
This data validates well, but it doesn't work on a slightly complex tree.
# Error: ValidationError: file /home/gulliver/.local/lib/python2.7/site-packages/jsonschema/validators.py line 934: 'name' is a required property #
{
"name": "group8",
"children": [
{
"name": "group7",
"children": [
{
"name": "group6",
"children": [
{
"name": "group12",
"children": [
{
"name": "group11",
"children": [
{
"name": "group10",
"children": []
}
]
}
]
},
{
"name": "group9",
"children": [
{
"name": "group5",
"children": [
{ ####### wrong
"children": []
}
]
}
]
}
]
}
]
},
{
"name": "group13",
"children": [
{
"name": "null1",
"children": []
}
]
}
]
}
It does not work when the data at the bottom of the tree is invalid.
My guess is that the branch splits and this happens, does anyone know why or how to fix it?
I tested using python and jsonschema.
When items is an array, it applies the subschema values to the same index location in the array in the instance.
For example, where you define...
"items": [
{"$ref":"#/definitions/block"}
]
only the first item in the array will be tested. It has nothing to do with deep nesting. For example, the follwing data is valid according to your schema...
{
"name": "group8",
"children": [
{
"name": "group7"
},
{
"something": "else",
"Not": "name"
}
]
}
(Live demo: https://jsonschema.dev/s/etFGE)
If you modify your use of items, then it will work like you expect:
"items": {"$ref":"#/definitions/block"}
(do this for both uses)
Live demo: https://jsonschema.dev/s/rk1OD
I have a single table in database like database table. I want to search a child from database and return a hierarchical JSON to a front end in order to create a tree. How can I do that in FLASK.
My expected JSON for mat should be like expected JSON
Since you have tagged your question with flask, this post assumes you are using Python as well. To format your database values in JSON string, you can query the db and then use recursion:
import sqlite3, collections
d = list(sqlite3.connect('file.db').cursor().execute("select * from values"))
def get_tree(vals):
_d = collections.defaultdict(list)
for a, *b in vals:
_d[a].append(b)
return [{'name':a, **({} if not (c:=list(filter(None, b))) else {'children':get_tree(b)})} for a, b in _d.items()]
import json
print(json.dumps(get_tree(d), indent=4))
Output:
[
{
"name": "AA",
"children": [
{
"name": "BB",
"children": [
{
"name": "EE",
"children": [
{
"name": "JJ",
"children": [
{
"name": "EEV"
},
{
"name": "FFW"
}
]
},
{
"name": "KK",
"children": [
{
"name": "HHX"
}
]
}
]
}
]
},
{
"name": "CC",
"children": [
{
"name": "FF",
"children": [
{
"name": "LL",
"children": [
{
"name": "QQY"
}
]
},
{
"name": "MM",
"children": [
{
"name": "RRV"
}
]
}
]
},
{
"name": "GG",
"children": [
{
"name": "NN",
"children": [
{
"name": "SSW"
}
]
}
]
}
]
},
{
"name": "DD",
"children": [
{
"name": "HH",
"children": [
{
"name": "OO",
"children": [
{
"name": "TTZ"
}
]
}
]
},
{
"name": "II",
"children": [
{
"name": "PP",
"children": [
{
"name": "UUW"
}
]
}
]
}
]
}
]
}
]
Given the following JSON (oversimplified for the sake of the example), I need to order the keys by their value. In this case, the order should be id > name > type.
{
"link": [{
"attributes": [{
"value": "ConfigurationElement",
"name": "type"
}, {
"value": "NAME1",
"name": "name"
}, {
"value": "0026a8b4-ced6-410e-9213-e3fcb28b3aab",
"name": "id"
}
],
"href": "href1",
"rel": "down"
}, {
"attributes": [{
"value": "0026a8b4-ced6-410e-9213-k23g15h2u1l5",
"name": "id"
}, {
"value": "ConfigurationElement",
"name": "type"
}, {
"value": "NAME2",
"name": "name"
}
],
"href": "href2",
"rel": "down"
}
],
"total": 2
}
EXPECTED RESULT:
{
"link": [{
"attributes": [{
"value": "0026a8b4-ced6-410e-9213-e3fcb28b3aab",
"name": "id"
}, {
"value": "NAME1",
"name": "name"
}, {
"value": "ConfigurationElement",
"name": "type"
}
],
"href": "href1",
"rel": "down"
}, {
"attributes": [{
"value": "0026a8b4-ced6-410e-9213-k23g15h2u1l5",
"name": "id"
}, {
"value": "NAME2",
"name": "name"
}, {
"value": "ConfigurationElement",
"name": "type"
}
],
"href": "href2",
"rel": "down"
}
],
"total": 2
}
I would be very grateful if anyone could help me out. I tried jq with -S and -s with sort_by(), but this example is way too complex for me to figure it out with my current experience with jq. Thank you a lot!
You can do:
jq '.link[].attributes|=sort_by(.name)'
The |= takes all the paths matched by .link[].attributes, i.e. each "attributes" array, and applies the filter sort_by(.name) to each of them, leaving everything else unchanged.
just checking before i build the wheel
I need a hackjob to present an api endpoint in a database. It doesn't need to do anything fancy, just convert what the rest api spits out into a single column of json. A new row at each iteration at the root/top level would be nice but a single varchar or whatever would be ok too
Does an ODBC wrapper exist out there anywhere? Googling just brings up hits for doing the opposite (exposing databases as an api). I'm not interested in the simba etc paid stuff. The consumer is SQL server so i can just use xp_cmdshell with curl as a last resort
so for instance the output of this : http://jsonapiplayground.reyesoft.com/v2/authors could come out as a table like so (a row for each author)
|data |
---------------------------------------------------
|{
"type": "authors",
"id": "1",
"attributes": {
"name": "Madge Mohr DVM 2",
"date_of_birth": "1977-08-21",
"date_of_death": "2009-09-14"
},
"relationships": {
"photos": {
"data": []
},
"books": {
"data": [
{
"type": "books",
"id": "41"
}
]
}
},
"links": {
"self": "/v2/authors/1"
}
} |
---------------------------------------------------
|{
"type": "authors",
"id": "3",
"attributes": {
"name": "Zelma Ortiz DDS",
"date_of_birth": "1992-09-06",
"date_of_death": "2000-12-19"
},
"relationships": {
"photos": {
"data": [
{
"type": "photos",
"id": "3"
}
]
},
"books": {
"data": [
{
"type": "books",
"id": "36"
},
{
"type": "books",
"id": "48"
}
]
}
},
"links": {
"self": "/v2/authors/3"
}
}|
----------
|{
"type": "authors",
"id": "4",
"attributes": {
"name": "Fermin Barrows Sr.",
"date_of_birth": "1991-03-18",
"date_of_death": "1975-11-07"
},
"relationships": {
"photos": {
"data": [
{
"type": "photos",
"id": "4"
}
]
},
"books": {
"data": [
{
"type": "books",
"id": "1"
},
{
"type": "books",
"id": "26"
},
{
"type": "books",
"id": "44"
},
{
"type": "books",
"id": "46"
}
]
}
},
"links": {
"self": "/v2/authors/4"
}
}|
----------
|{
"type": "authors",
"id": "5",
"attributes": {
"name": "Terry Durgan",
"date_of_birth": "2011-03-06",
"date_of_death": "2017-04-13"
},
"relationships": {
"photos": {
"data": [
{
"type": "photos",
"id": "5"
}
]
},
"books": {
"data": [
{
"type": "books",
"id": "6"
},
{
"type": "books",
"id": "16"
},
{
"type": "books",
"id": "50"
}
]
}
},
"links": {
"self": "/v2/authors/5"
}
}|
----------
|{
"type": "authors",
"id": "6",
"attributes": {
"name": "Annalise Walsh",
"date_of_birth": "2004-11-27",
"date_of_death": "1997-07-20"
},
"relationships": {
"photos": {
"data": [
{
"type": "photos",
"id": "6"
}
]
},
"books": {
"data": [
{
"type": "books",
"id": "4"
},
{
"type": "books",
"id": "5"
},
{
"type": "books",
"id": "21"
}
]
}
},
"links": {
"self": "/v2/authors/6"
}
}|
---------