I have a JSON file something like below:
{
"name": "flare",
"children": [
{
"name": "analytics",
"children": [
{
"name": "cluster",
"children": [
{"name": "AgglomerativeCluster", "size": 3938},
{"name": "CommunityStructure", "size": 3812},
{"name": "HierarchicalCluster", "size": 6714},
{"name": "MergeEdge", "size": 743}
]
}, ....
As you can see, there is a "size" attribute at the end nodes.
I want to add the "size" attribute to other nodes as well (i.e. to "analytics", and "cluster") so what I did was:
{
"name": "flare",
"children": [
{
{"name": "analytics", "another_size":345},
"children": [
{
{"name": "cluster", "another_size":345},
"children": [
{"name": "AgglomerativeCluster", "size": 3938},
{"name": "CommunityStructure", "size": 3812},
{"name": "HierarchicalCluster", "size": 6714},
{"name": "MergeEdge", "size": 743}
]
}, ....
But it doesn't compile this way.
How do I go about achieving this? I want to colour my nodes differently for those other nodes under "another_size".
Any help appreciated!!
Try it like this:
{
"name": "flare",
"children": [
{
{"name": "analytics",
"another_size":345,
"children": [
{
{"name": "cluster",
"another_size":345,
"children": [
{"name": "AgglomerativeCluster", "size": 3938},
{"name": "CommunityStructure", "size": 3812},
{"name": "HierarchicalCluster", "size": 6714},
{"name": "MergeEdge", "size": 743}
]
}, ....
FULL EXAMPLE
{
"name": "flare",
"children": [
{
"name": "analytics",
"another_size": 345,
"children": [
{
"name": "cluster",
"another_size": 345,
"children": [
{
"name": "AgglomerativeCluster",
"size": 3938
},
{
"name": "CommunityStructure",
"size": 3812
},
{
"name": "HierarchicalCluster",
"size": 6714
},
{
"name": "MergeEdge",
"size": 743
}
]
},
{
"name": "cluster",
"another_size": 345,
"children": [
{
"name": "AgglomerativeCluster",
"size": 3938
},
{
"name": "CommunityStructure",
"size": 3812
},
{
"name": "HierarchicalCluster",
"size": 6714
},
{
"name": "MergeEdge",
"size": 743
}
]
}
]
}
]
}
Related
I have a lot of data I need to parse though.
I need to pull all pid's and price's.
`
[
{
"id": 159817,
"price": "10.69",
"stocked": true,
"store": {
"id": 809,
"nsn": "22036-0",
"pricingSource": "manual",
"lastUpdated": "2022-12-05T15:24:33.908Z"
},
"sharedFields": {
"type": "PRODUCT",
"id": 24549,
"pid": "12079",
"labels": [
{
"type": "default",
"value": "Chicken Sandwich",
"locale": "en"
},
{
"type": "fresh",
"value": "Chicken",
"locale": "en"
},
{
"type": "product_json",
"value": "Chicken",
"locale": "en"
}
],
"calMin": 600,
"calMax": 600,
"lastUpdated": "2021-12-31T13:49:22.794Z"
}
},
{
"id": 159818,
"price": "9.29",
"stocked": true,
"store": {
"id": 809,
"nsn": "22036-0",
"pricingSource": "manual",
"lastUpdated": "2022-12-05T15:24:33.908Z"
},
"sharedFields": {
"type": "PRODUCT",
"id": 25,
"pid": "1",
"labels": [
{
"type": "default",
"value": "Ham Sandwich",
"locale": "en"
},
{
"type": "fresh",
"value": "Ham",
"locale": "en"
}
],
"calMin": 540,
"calMax": 540,
"lastUpdated": "2021-07-09T19:30:00.326Z"
}
}
]
`
and I need to place them into a string like this, but on a scale of 150 products. I'd also need to change "pid" to "productId"
[{ "productId": "46238", "price": 6.09 }, { "productId": "40240", "price": 1.49 }]
I need to add a string before this data, but I'm pretty confident I can figure that part out.
I am pretty open to the easiest suggestion, whether that be VBS, Excel macro, etc.
I just seem to have a hard time figuring out the mindset of jq for more than simple extraction problems, but hopefully I can keep growing my skills...
I have data that looks like this synthetic example:
{
"accounts": [
{
"attrs": {
"name": "One"
},
"vaults": [
{
"attrs": {"name": "Private", "type": "P"},
"items": [
{
"foo": "001",
"overview": {"subtitle": "foo"}
},
{
"foo": "002",
"overview": {"subtitle": "foo", "tags": ["a tag"]}
}
]
},
{
"attrs": {"name": "Public", "type": "P"},
"items": [
{
"foo": "001",
"overview": {"subtitle": "foo"}
},
{
"foo": "002",
"overview": {"subtitle": "foo", "tags": ["a tag"]}
}
]
}
]
}
]
}
For every record in .accounts[].vaults[].items[], I need to create/replace its overview.tags value with [.Parent.Parent.attrs.name], if .Parent were to get the current element's parent in the tree.
So the desired output would be this:
{
"accounts": [
{
"attrs": {
"name": "One"
},
"vaults": [
{
"attrs": {"name": "Private", "type": "P"},
"items": [
{
"foo": "001",
"overview": {"subtitle": "foo", "tags": ["Private"]}
},
{
"foo": "002",
"overview": {"subtitle": "foo", "tags": ["Private"]}
}
]
},
{
"attrs": {"name": "Public", "type": "P"},
"items": [
{
"foo": "001",
"overview": {"subtitle": "foo", "tags": ["Public"]}
},
{
"foo": "002",
"overview": {"subtitle": "foo", "tags": ["Public"]}
}
]
}
]
}
]
}
Is there a nice way in jq to achieve this?
Background: this is the format of 1Password's .1pux export data.
One way to do it is using update-assignment |=
jq '.accounts[].vaults |= map(.items[].overview.tags = [.attrs.name])'
jq is an amazing tool and it does a lot.
as input I have
[
{
"backup": [
{
"timestamp": { "start": 1642144383, "stop": 1642144386 },
"info": { "size": 1200934840},
"type": "full"
},
{
"timestamp": {"start": 1642144388, "stop": 1642144392 },
"info": { "size": 1168586300
},
"type": "incr"
},
{
"timestamp": {"start": 1642145388, "stop": 1642145392 },
"info": { "size": 1168586330
},
"type": "incr"
}
],
"name": "dbname1"
},
{
"backup": [
{
"timestamp": { "start": 1642144383, "stop": 1642144386 },
"info": { "size": 1200934840},
"type": "full"
},
{
"timestamp": {"start": 1642144388, "stop": 1642144392 },
"info": { "size": 1168586300
},
"type": "incr"
}
],
"name": "dbname2"
}
]
and using
jq 'map([.backup[] + {name}] | max_by(.timestamp.stop))'
I get the latest timestamp.stop for a name. How should I change this to get the latest timestamp.stop for a name and group?
in SQL this would be something like max(.timestamp.stop) group by .name,.type
Hoping for output like:
[
{
"timestamp": {
"start": 1642144383,
"stop": 1642144386
},
"info": {
"size": 1200934840
},
"type": "full",
"name": "dbname1"
},
{
"timestamp": {
"start": 1642145388,
"stop": 1642145392
},
"info": {
"size": 1168586330
},
"type": "incr",
"name": "dbname1"
},
{
"timestamp": {
"start": 1642144383,
"stop": 1642144386
},
"info": {
"size": 1200934840
},
"type": "full",
"name": "dbname2"
},
{
"timestamp": {
"start": 1642144388,
"stop": 1642144392
},
"info": {
"size": 1168586300
},
"type": "incr",
"name": "dbname2"
}
]
Remove the inner brackets to flatten the array, then group_by both criteria (which makes your criteria an array), and map your max_by onto the result array:
jq 'map(.backup[] + {name}) | group_by([.name, .type]) | map(max_by(.timestamp.stop))'
[
{
"timestamp": {
"start": 1642144383,
"stop": 1642144386
},
"info": {
"size": 1200934840
},
"type": "full",
"name": "dbname1"
},
{
"timestamp": {
"start": 1642145388,
"stop": 1642145392
},
"info": {
"size": 1168586330
},
"type": "incr",
"name": "dbname1"
},
{
"timestamp": {
"start": 1642144383,
"stop": 1642144386
},
"info": {
"size": 1200934840
},
"type": "full",
"name": "dbname2"
},
{
"timestamp": {
"start": 1642144388,
"stop": 1642144392
},
"info": {
"size": 1168586300
},
"type": "incr",
"name": "dbname2"
}
]
Demo
This seems to produce the desired expected output. You need an additional grouping by the .type record, before doing the max_by
map( .backup[] + {name} ) | group_by(.name)[] |
group_by(.type) | map(max_by(.timestamp.stop))
jqplay demo
As input I have:
[
{
"backup": [
{
"timestamp": { "start": 1642144383, "stop": 1642144386 },
"info": { "size": 1200934840},
"type": "full"
},
{
"timestamp": {"start": 1642144388, "stop": 1642144392 },
"info": { "size": 1168586300
},
"type": "incr"
}
],
"name": "dbname1"
},
{
"backup": [
{
"timestamp": { "start": 1642144383, "stop": 1642144386 },
"info": { "size": 1200934840},
"type": "full"
},
{
"timestamp": {"start": 1642144388, "stop": 1642144392 },
"info": { "size": 1168586300
},
"type": "incr"
}
],
"name": "dbname2"
}
]
using
jq '.[]
| [ .backup[] + {name} ]
| max_by(.timestamp.stop)
'
(thanks #pmf) I can re-order this to
{
"timestamp": {
"start": 1642144388,
"stop": 1642144392
},
"info": {
"size": 1168586300
},
"type": "incr",
"name": "dbname1"
}
{
"timestamp": {
"start": 1642144388,
"stop": 1642144392
},
"info": {
"size": 1168586300
},
"type": "incr",
"name": "dbname2"
}
Selected is the dict containing the max timestamp and the name added to it, being the last created backup of a database. There are multiple databases possible. How can I form the output to a cleanly formatted array?
I was hoping for
[
{
"timestamp": {
"start": 1642144388,
"stop": 1642144392
},
"info": {
"size": 1168586300
},
"type": "incr",
"name": "dbname1"
},
{
"timestamp": {
"start": 1642144388,
"stop": 1642144392
},
"info": {
"size": 1168586300
},
"type": "incr",
"name": "dbname2"
}
]
And yes, I can add this using sed but I feel jq should be able to do this. So the question is how can should this be written?
Instead of .[] | … use map(…) to retain the array.
jq 'map([.backup[] + {name}] | max_by(.timestamp.stop))'
[
{
"timestamp": {
"start": 1642144388,
"stop": 1642144392
},
"info": {
"size": 1168586300
},
"type": "incr",
"name": "dbname1"
},
{
"timestamp": {
"start": 1642144388,
"stop": 1642144392
},
"info": {
"size": 1168586300
},
"type": "incr",
"name": "dbname2"
}
]
Demo
So I have been using this logic apps template to hit the Google Analytics API and the response is in this format
{
"reports": [
{
"columnHeader": {
"dimensions": [
"ga:date",
"ga:campaign",
"ga:country",
"ga:browser",
"ga:deviceCategory",
"ga:sourceMedium",
"ga:socialNetwork",
"ga:region"
],
"metricHeader": {
"metricHeaderEntries": [
{
"name": "ga:users",
"type": "INTEGER"
},
{
"name": "ga:sessions",
"type": "INTEGER"
},
{
"name": "ga:newUsers",
"type": "INTEGER"
},
{
"name": "ga:bounces",
"type": "INTEGER"
},
{
"name": "ga:pageviews",
"type": "INTEGER"
},
{
"name": "ga:sessionDuration",
"type": "TIME"
},
{
"name": "ga:hits",
"type": "INTEGER"
},
{
"name": "ga:goalCompletionsAll",
"type": "INTEGER"
},
{
"name": "ga:goalConversionRateAll",
"type": "PERCENT"
}
]
}
},
"data": {
"rows": [
{
"dimensions": [
"20200312",
"(not set)",
"India",
"Chrome",
"desktop",
"(direct) / (none)",
"(not set)",
"Tamil Nadu"
],
"metrics": [
{
"values": [
"4",
"4",
"4",
"0",
"111",
"5100.0",
"111",
"0",
"0.0"
]
}
]
},
{
"dimensions": [
"20200316",
"(not set)",
"India",
"Chrome",
"desktop",
"(direct) / (none)",
"(not set)",
"Tamil Nadu"
],
"metrics": [
{
"values": [
"1",
"1",
"0",
"0",
"6",
"266.0",
"6",
"0",
"0.0"
]
}
]
},
{
"dimensions": [
"20200318",
"(not set)",
"India",
"Chrome",
"desktop",
"(direct) / (none)",
"(not set)",
"Tamil Nadu"
],
"metrics": [
{
"values": [
"1",
"2",
"0",
"0",
"20",
"135.0",
"20",
"0",
"0.0"
]
}
]
}
],
"totals": [
{
"values": [
"6",
"7",
"4",
"0",
"137",
"5501.0",
"137",
"0",
"0.0"
]
}
],
"rowCount": 3,
"minimums": [
{
"values": [
"1",
"1",
"0",
"0",
"6",
"135.0",
"6",
"0",
"0.0"
]
}
],
"maximums": [
{
"values": [
"4",
"4",
"4",
"0",
"111",
"5100.0",
"111",
"0",
"0.0"
]
}
],
"isDataGolden": true
}
}
]
}
I Want to convert it and bring it in a form that the column header:dimensions and metric header entries name will become column names and their values,ie data.rows.dimensions and metrics.values become corresponding values
ga:date ga:campaign ga:country ga:browser ga:deviceCategory ga:sourceMedium ga:socialNetwork ga:region ga:users ga:sessions ga:newUsers : (column names)
20200316 (not set) India Chrome desktop (direct) / (none) (not set) Tamil Nadu 1 1 1 :(values)
If you can use an Integration account, I suggest to create a flat file schema with the desired structure, and in the logic app you can convert in xml and then apply the Flat File Encoding.
Otherwise a function app should resolve your issue