Parse JSON Nested SubValues in Powershell to Table - json

I converted the JSON string to Powershell in v5. The original json string is below:
$j = #'
[{
"id": "1",
"Members": [
"A",
"B",
"C"
]
}, {
"id": "2",
"Members": [
"A",
"C"
]
}, {
"id": "3",
"Members": [
"A",
"D"
]
}]
'#
$json = $j | ConvertFrom-Json
I would like the result set to look like the result set below. Eventually I will export to SQL:
id Members
----- --------
1 A
1 B
1 C
2 A
2 C
3 A
3 D


try this
$json | % {
$id = $_.id
$_.members | select #{n='id';e={$id}}, #{n='members';e={$_}}
}

Related

how to format the result with jq

I have the following printout,
{
"metric": {
"container": "container1",
"namespace": "namespace1",
"pod": "pod1"
},
"values": [
[
1664418600,
"1"
],
[
1664418900,
"2"
],
[
1664419200,
"6"
],
[
1664419500,
"8"
],
[
1664419800,
"7"
],
[
1664420100,
"9"
]
]
}
{
"metric": {
"container": "container2",
"namespace": "namespace2",
"pod": "pod2"
},
"values": [
[
1664420100,
"1"
]
]
}
What I want:
container=container1,namespace=namespace1,pod=pod1
1 1664418600
2 1664418900
6 1664419200
8 1664419500
7 1664419800
9 1664420100
container=container2,namespace=namespace2,pod=pod2
1 1664420100

Build it from two JSON programs:
Header lines: .metric | to_entries | map(join("=")) | join(",")
Get metric object: .metric
Convert to an array of key-value pairs: to_entries
Map each key-value pair object to a string "key=value": map(join("="))
Join all pairs by comma: join(",")
Value lists: .values[] | [last,first] | join(" ")
Stream values: .values[]
Reverse each two-valued array: [last,first]
Join items by blank: join(" ")
An alternative for 2.2. and 2.3. could be "\(last) \(first)", i.e. values[] | "\(last) \(first)". Or [last,first] could be replaced with reverse: .values[] | reverse | join(" ").
Putting the two programs together:
(.metric | to_entries | map(join("=")) | join(",")),
(.values[] | [last,first] | join(" "))
And then execute with raw output enabled: jq -r (.metrics|to_entries…
Output:
container=container1,namespace=namespace1,pod=pod1
1 1664418600
2 1664418900
6 1664419200
8 1664419500
7 1664419800
9 1664420100
container=container2,namespace=namespace2,pod=pod2
1 1664420100

how to extract and modify inner array objects with parent object data in jq

We are tying to format a json similar to this:
[
{"id": 1,
"type": "A",
"changes": [
{"id": 12},
{"id": 13}
],
"wanted_key": "good",
"unwanted_key": "aaa"
},
{"id": 2,
"type": "A",
"unwanted_key": "aaa"
},
{"id": 3,
"type": "B",
"changes": [
{"id": 31},
{"id": 32}
],
"unwanted_key": "aaa",
"unwanted_key2": "aaa"
},
{"id": 4,
"type": "B",
"unwanted_key3": "aaa"
},
null,
null,
{"id": 7}
]
into something like this:
[
{
"id": 1,
"type": "A",
"wanted_key": true # every record must have this key/value
},
{
"id": 12, # note: this was in the "changes" property of record id 1
"type": "A", # type should be the same type than record id 1
"wanted_key": true
},
{
"id": 13, # note: this was in the "changes" property of record id 1
"type": "A", # type should be the same type than record id 1
"wanted_key": true
},
{
"id": 2,
"type": "A",
"wanted_key": true
},
{
"id": 3,
"type": "B",
"wanted_key": true
},
{
"id": 31, # note: this was in the "changes" property of record id 3
"type": "B", # type should be the same type than record id 3
"wanted_key": true
},
{
"id": 32, # note: this was in the "changes" property of record id 3
"type": "B", # type should be the same type than record id 3
"wanted_key": true
},
{
"id": 4,
"type": "B",
"wanted_key": true
},
{
"id": 7,
"type": "UNKN", # records without a type should have this type
"wanted_key": true
}
]
So far, I've been able to:
remove null records
obtain the keys we need with their default
give records without a type a default type
What we are missing:
from records having a changes key, create new records with the type of their parent record
join all records in a single array
Unfortunately we are not entirely sure how to proceed... Any help would be appreciated.
So far our jq goes like this:
del(..|nulls) | map({id, type: (.type // "UNKN"), wanted_key: (true)}) | del(..|nulls)
Here's our test code:
https://jqplay.org/s/eLAWwP1ha8P

The following should work:
map(select(values))
| map(., .type as $type | (.changes[]? + {$type}))
| map({id, type: (.type // "UNKN"), wanted_key: true})
Only select non-null values
Return the original items followed by their inner changes array (+ outer type)
Extract 3 properties for output
Multiple map calls can usually be combined, so this becomes:
map(
select(values)
| ., (.type as $type | (.changes[]? + {$type}))
| {id, type: (.type // "UNKN"), wanted_key: true}
)
Another option without variables:
map(
select(values)
| ., .changes[]? + {type}
| {id, type: (.type // "UNKN"), wanted_key: true}
)
# or:
map(select(values))
| map(., .changes[]? + {type})
| map({id, type: (.type // "UNKN"), wanted_key: true})
or even with a separate normalization step for the unknown type:
map(select(values))
| map(.type //= "UNKN")
| map(., .changes[]? + {type})
| map({id, type, wanted_key: true})
# condensed to a single line:
map(select(values) | .type //= "UNKN" | ., .changes[]? + {type} | {id, type, wanted_key: true})
Explanation:
Select only non-null values from the array
If type is not set, create the property with value "UNKN"
Produce the original array items, followed by their nested changes elements extended with the parent type
Reshape objects to only contain properties id, type, and wanted_key.

Here's one way:
map(
select(values)
| (.type // "UNKN") as $type
| ., .changes[]?
| {id, $type, wanted_key: true}
)
[
{
"id": 1,
"type": "A",
"wanted_key": true
},
{
"id": 12,
"type": "A",
"wanted_key": true
},
{
"id": 13,
"type": "A",
"wanted_key": true
},
{
"id": 2,
"type": "A",
"wanted_key": true
},
{
"id": 3,
"type": "B",
"wanted_key": true
},
{
"id": 31,
"type": "B",
"wanted_key": true
},
{
"id": 32,
"type": "B",
"wanted_key": true
},
{
"id": 4,
"type": "B",
"wanted_key": true
},
{
"id": 7,
"type": "UNKN",
"wanted_key": true
}
]
Demo

Something like below should work
map(
select(type == "object") |
( {id}, {id : ( .changes[]? .id )} ) +
{ type: (.type // "UNKN"), wanted_key: true }
)
jq play - demo

jq - select objects and print null for missing

I'm trying to generate a CSV of sort from json file, the files are as below
cat role1.json
{
"Tags": [
{
"Key": "Name",
"Value": "Role1Name"
},
{
"Key": "ID",
"Value": "Role1ID"
},
{
"Key": "Manager",
"Value": "Role1Manager"
},
{
"Key": "User",
"Value": "Role1User"
},
{
"Key": "Country",
"Value": "USA"
}
]
}
cat role2.json
{
"Tags": [
{
"Key": "Name",
"Value": "Role2Name"
},
{
"Key": "ID",
"Value": "Role2ID"
},
{
"Key": "City",
"Value": "NewYork"
},
{
"Key": "Creator",
"Value": "Role2Creator"
},
{
"Key": "User",
"Value": "Role2User"
}
]
}
cat role3.json
{
"Tags": [
{
"Key": "Name",
"Value": "Role3Name"
},
{
"Key": "ID",
"Value": "Role3ID"
},
{
"Key": "Creator",
"Value": "Role3Creator"
},
{
"Key": "ZIP",
"Value": 82378
},
{
"Key": "Manager",
"Value": "Role3Manager"
},
{
"Key": "User",
"Value": "Role3User"
}
]
}
I want to generate lines from each of these to be later used as CSV, something like:
Role1Name, Role1ID, null, Role1Manager, Role1User
Role2Name, Role2ID, Role2Creator, null, Role2User
Role3Name, Role3ID, Role3Creator, Role3Manager, Role3User
For the header line
Name, ID, Creator, Manager, User
I'm able to get all the "Value" but not able to print null for missing "Key"
$cat role1.json | jq -rc '[.Tags[] | select(.Key == ("Name","ID","Creator","Manager","User")) | .Value]'
["Role1Name","Role1ID","Role1Manager","Role1User"]
$cat role2.json | jq -rc '[.Tags[] | select(.Key == ("Name","ID","Creator","Manager","User")) | .Value]'
["Role2Name","Role2ID","Role2Creator","Role2User"]
$cat role3.json | jq -rc '[.Tags[] | select(.Key == ("Name","ID","Creator","Manager","User")) | .Value]'
["Role3Name","Role3ID","Role3Creator","Role3Manager","Role3User"]
Can someone share with me how this can be done using jq.
Also, how can we enforce the order.
Thanks!

The key (ha!) is
[ .[ $keys[] ] ]
Had you looked at other answers to questions relating to CSV, you might have noticed the first step taken is to get the list of keys. This is often done by collecting the keys of the input objects. (Example) In your case, you have a hard-coded list, so it's even simpler.
If you wanted actual CSV, you could use
jq -sr '
[ "Name", "ID", "Creator", "Manager", "User" ] as $keys |
(
$keys,
( .[].Tags | from_entries | [ .[ $keys[] ] ] )
) |
#csv
' role*.json
This produces
"Name","ID","Creator","Manager","User"
"Role1Name","Role1ID",,"Role1Manager","Role1User"
"Role2Name","Role2ID","Role2Creator",,"Role2User"
"Role3Name","Role3ID","Role3Creator","Role3Manager","Role3User"
jqplay
Without a header:
jq -r '.Tags | from_entries | [ .["Name","ID","Creator","Manager","User"] ] | #csv' role*.json
jqplay
To get the specific output you posted (which isn't CSV), you could use
jq -sr '
[ "Name", "ID", "Creator", "Manager", "User" ] as $keys |
(
$keys,
( .[].Tags | from_entries | [ .[ $keys[] ] | . // "null" ] )
) |
join(", ")
' role*.json
This produces
Name, ID, Creator, Manager, User
Role1Name, Role1ID, null, Role1Manager, Role1User
Role2Name, Role2ID, Role2Creator, null, Role2User
Role3Name, Role3ID, Role3Creator, Role3Manager, Role3User
jqplay
Without a header:
jq -r '.Tags | from_entries | [ .["Name","ID","Creator","Manager","User"] | . // "null" ] | join(", ")' role*.json
jqplay

Got an answer from another forum, might be useful for others
$jq -rc '.Tags | from_entries | [.Name, .ID, .Creator, .Manager, .User]' role*.json
["Role1Name","Role1ID",null,"Role1Manager","Role1User"]
["Role2Name","Role2ID","Role2Creator",null,"Role2User"]
["Role3Name","Role3ID","Role3Creator","Role3Manager","Role3User"]

T-SQL OpenJson Nested Array

Iam Struggling with following JSOn Structure
Declare #Json Nvarchar(max)
Set #Json = '
{
"entities": [
{
"Fields": [
{
"Name": "test-id",
"values": [
{
"value": "1851"
}
]
},
{
"Name": "test-name",
"values": [
{
"value": "01_DUMMY"
}
]
}
],
"Type": "run",
"children-count": 0
},
{
"Fields": [
{
"Name": "test-id",
"values": [
{
"value": "1852"
}
]
},
{
"Name": "test-name",
"values": [
{
"value": "02_DUMMY"
}
]
}
],
"Type": "run",
"children-count": 0
}
],
"TotalResults": 2
}'
My Output should look like this:
test-id|test-name|Type|Children-count
1851 |01_DUMMY |run |0
1852 |02_DUMMY |run |0
I tried to use the Examples posted here but none is matching my Needs.
My closest apporach was this T-SQL Syntax
Select
*
From OPENJSON (#JSON,N'$.entities') E
CROSS APPLY OPENJSON (E.[value]) F
CROSS APPLY OPENJSON (F.[value],'$') V where F.type = 4
My next idea was to use this SQL CODE to open the next nested Array but iam always getting an error msg(
Lookup Error - SQL Server Database Error: Incorrect syntax near the
keyword 'CROSS'.
)
Select
*
From OPENJSON (#JSON,N'$.entities') E
CROSS APPLY OPENJSON (E.[value]) F
CROSS APPLY OPENJSON (F.[value]) V where F.type = 4
CROSS APPLY OPENJSON (V.[value]) N
Iam not sure how to get Closer to my needed Output.
To be honest I just started with T-SQL and never worked before with JSON Files.
Regards Johann

This is rather deeply nested. I think, you've got the right idea to dive deeper and deeper using a serie of OPENJSON. Try it like this to get your values:
Declare #Json Nvarchar(max)
Set #Json = '
{
"entities": [
{
"Fields": [
{
"Name": "test-id",
"values": [
{
"value": "1851"
}
]
},
{
"Name": "test-name",
"values": [
{
"value": "01_DUMMY"
}
]
}
],
"Type": "run",
"children-count": 0
},
{
"Fields": [
{
"Name": "test-id",
"values": [
{
"value": "1852"
}
]
},
{
"Name": "test-name",
"values": [
{
"value": "02_DUMMY"
}
]
}
],
"Type": "run",
"children-count": 0
}
],
"TotalResults": 2
}';
--This is the query
WITH ReadJson AS
(
SELECT A.TotalResults
,C.[Type]
,C.[children-count]
,D.[Name]
,E.*
FROM OPENJSON(#Json)
WITH(TotalResults INT, entities NVARCHAR(MAX) AS JSON) A
CROSS APPLY OPENJSON(A.entities) B
CROSS APPLY OPENJSON(B.[value])
WITH(Fields NVARCHAR(MAX) AS JSON,[Type] VARCHAR(100),[children-count] INT) C
CROSS APPLY OPENJSON(C.Fields)
WITH([Name] VARCHAR(100),[values] NVARCHAR(MAX) AS JSON) D
CROSS APPLY OPENJSON(D.[values])
WITH([value] VARCHAR(100)) E
)
SELECT * FROM ReadJson;
The result
+---+-----+---+-----------+----------+
| 2 | run | 0 | test-id | 1851 |
+---+-----+---+-----------+----------+
| 2 | run | 0 | test-name | 01_DUMMY |
+---+-----+---+-----------+----------+
| 2 | run | 0 | test-id | 1852 |
+---+-----+---+-----------+----------+
| 2 | run | 0 | test-name | 02_DUMMY |
+---+-----+---+-----------+----------+
Do you think you can manage the rest?

How to generate continuing indices for multiple objects in nested arrays that are in an array

Given
[{
"objects": [{
"key": "value"
},{
"key": "value"
}]
}, {
"objects": [{
"key": "value"
}, {
"key": "value"
}]
}]
How do I generate
[{
"objects": [{
"id": 0,
"key": "value"
},{
"id": 1,
"key": "value"
}]
}, {
"objects": [{
"id": 2,
"key": "value"
}, {
"id": 3,
"key": "value"
}]
}]
Using jq?
I tried to use this one, but ids are all 0:
jq '[(-1) as $i | .[] | {objects: [.objects[] | {id: ($i + 1 as $i | $i), key}]}]'

The key to a simple solution here is to break the problem down into easy pieces. This can be accomplished by defining a helper function, addId/1. Once that is done, the rest is straightforward:
# starting at start, add {id: ID} to each object in the input array
def addId(start):
reduce .[] as $o
([];
length as $l
| .[length] = ($o | (.id = start + $l)));
reduce .[] as $o
( {start: -1, answer: []};
(.start + 1) as $next
| .answer += [$o | (.objects |= addId($next))]
| .start += ($o.objects | length) )
| .answer

Inspired by #peak answer, I came up with this solution. Not much difference, just shorter way to generate IDs and opt for foreach instead of reduce since there is intermediate result involved.
def addIdsStartWith($start):
[to_entries | map((.value.id = .key + $start) | .value)];
[foreach .[] as $set (
{start: 0};
.set = $set |
.start as $start | .set.objects |= addIdsStartWith($start) |
.start += ($set.objects | length);
.set
)]