Parsing Multi-Level JSON with PowerShell [duplicate] - json

I have a json file, simplified version of it looks like this:
{
"Location": "EU",
"Country": {
"City": "xxx",
"Town": "xxx"
},
"Transport": {
"Train": "xxx"
}
}
I have run the ConvertFrom-Json command to convert to PSObject:
$conversion = Get-Content $path | ConvertFrom-Json
This will give me an output like this:
Location : EU
Country : #{City="xxx"; Town="xxx"}
Transport : #{Train="xxx"}
Question
How can I get the nested values to print out separately? I would want them all to print out like the "Location:EU" one
Is there a different command to ConvertFrom-Json that i should be using for this? Or do I just need to mess around with ConvertFrom-Json command a bit?
To note:
I am not just looking for a pretty print out - I would need them all separately for a script I am writing that will be looping through all the key/value pairs
I have read about the -Depth flag when using ConvertFrom-Json and does not seem to fix anything here - it seemed this was more relevant for ConvertTo-Json

In order to report all leaf properties as name-value pairs (i.e. those properties that contain primitive JSON values as opposed to containing nested objects with properties and / or arrays), you need to recursively walk the object graph:
Find helper function Get-LeafProperty below; assuming you have already defined it, you can call it as follows:
#'
{
"Location": "EU",
"Country": {
"City": "xxx",
"Town": "xxy"
},
"Transport": {
"Train": "xxz"
}
}
'# |
ConvertFrom-Json |
Get-LeafProperty
Output (the display formatting of [pscustomobject] instances with .Name and .Value properties representing all the leaf properties):
Name Value
---- -----
Location EU
City xxx
Town xxy
Train xxz
Get-LeafProperty source code:
# Walks a potentially nested [pscustomobject] graph
# as returned by ConvertFrom-Json and outputs all
# leaf properties as name-value custom objects.
function Get-LeafProperty {
param([Parameter(ValueFromPipeline)] [object] $InputObject)
process {
if ($InputObject -is [array]) { # array as single input object -> recurse
foreach ($o in $InputObject) { Get-LeafProperty $o }
}
else {
# Assumed to be a (potentially nested) [pscustomobject] instance:
# Recursively process its properties.
foreach ($p in $InputObject.psobject.properties) {
if ($p.Value -is [array]) { # array -> recurse
foreach ($o in $p.Value) { Get-LeafProperty $o }
} elseif ($p.Value -is [System.Management.Automation.PSCustomObject] ) { # nested [pscustomobject] -> recurse
Get-LeafProperty $p.Value
} else { # leaf property reached -> output name-value pair
[pscustomobject] #{ Name = $p.Name; Value = $p.Value }
}
}
}
}
}
Note: A variant of this function that outputs property name paths (e.g. Country.City) instead of just their names (e.g. City) can be found in this answer.

Related

How to convert json string into specified hashtable format with 5.1 powershell version?

I'm trying to convert json string into desired format of hashtable with powershell 5.1 version.
Used this code to convert json into hashtable but getting the o/p in the below mentioned format. How to convert this json into specified format of hashtable ?
My code :
function ConvertTo-Hashtable {
[CmdletBinding()]
[OutputType('hashtable')]
param (
[Parameter(ValueFromPipeline)]
$InputObject
)
process {
## Return null if the input is null. This can happen when calling the function
## recursively and a property is null
if ($null -eq $InputObject) {
return $null
}
## Check if the input is an array or collection. If so, we also need to convert
## those types into hash tables as well. This function will convert all child
## objects into hash tables (if applicable)
if ($InputObject -is [System.Collections.IEnumerable] -and $InputObject -isnot [string]) {
$collection = #(
foreach ($object in $InputObject) {
ConvertTo-Hashtable -InputObject $object
}
)
## Return the array but don't enumerate it because the object may be pretty complex
Write-Output -NoEnumerate $collection
} elseif ($InputObject -is [psobject]) { ## If the object has properties that need enumeration
## Convert it to its own hash table and return it
$hash = #{}
foreach ($property in $InputObject.PSObject.Properties) {
$hash[$property.Name] = ConvertTo-Hashtable -InputObject $property.Value
}
$hash
} else {
## If the object isn't an array, collection, or other object, it's already a hash table
## So just return it.
$InputObject
}
}
}
json :
$json = '[
{
"type": "IP",
"Fields": [
{
"Column": "FileHash",
"Id": "Address"
}
]
}
]'
Actual Format with my code:
Name Value
---- -----
Fields {System.Collections.Hashtable}
type IP
Desired Format :
Key : type
Value : IP
Name : type
Key : Fields
Value : {FileHash}
Name : Fields

Populate collection of objects from one JSON file to the collection of another one with PowerShell

I have two JSON files and want to transfer collection of objects from one file to another. Suppose, the from.json file contains property which represents collection of clients:
"Clients":
[
{
"Name": "Name1",
"Age": "12"
},
{
"Name": "Name2",
"Age": "14"
}
]
to.json file contains an empty collection, "Objects: []" ,which must be filled with objects from from.json. Each objects in toJson variable must contain additional property - Id, so eventually, my "to.json" file should look like this:
"Objects":
[
{
"Id": "{new-id}",
"Name": "Name1",
"Age": "12"
},
{
"Id": "{new-id}",
"Name": "Name1",
"Age": "12"
}
]
I've converted two files into variables:
$fromJson = (Get-Content -Raw -Path {fromPath}) | ConvertFrom-Json
$toJson = (Get-Content -Raw -Path {toPath}) | ConvertFrom-Json
I know that objects from fromJson to toJson can be transferred in the following manner:
toJson.Objects += fromJson.Clients, but that's not enough in my case. I think that it could be done by iterating through fromJson.Clients array but have no idea how to create an object and add it into toJson.Objects collection.
Here's a more efficient solution, based on:
Use of a calculated property with Select-Object, which allows you to place the new property first in the output objects.
Instead of building the array one by one with += (which is inefficient, because a new array must technically be created behind the scenes in every iteration), the solution below lets PowerShell collect the output objects of the Select-Object call in an array automatically (the [array] type constraint is needed to ensure that an array is created even if only one object happens to be output.)
# Sample input.
$fromJson = ConvertFrom-Json '{"Clients":[{"Name":"Name1","Age":"12"},{"Name":"Name2","Age":"14"}]}'
$toJson = ConvertFrom-Json '{ "Objects": [] }'
[array] $toJson.Objects =
$fromJson.Clients |
Select-Object #{ Name='Id'; Expression = { [string] (New-Guid) } }, *
$toJson | ConvertTo-Json -Depth 3 # append | Set-Content as needed.
Kind of new to the PowerShell, but after a bit of investigation came up with the following solution:
fromJson.Clients | ForEach-Object {
$_ | Add-Member -MemberType NoteProperty -Name 'Id' -Value ([guid]::NewGuid().Guid.ToString())
$toJson += $_
}
...
$toJson | ConvertTo-Json | Out-File {to.json_path}
Frankly, don't know if that is a 'proper' way to do that, but generally it works for that particular case. For now, see no other solution.

powershell - iterate over json keys that have similar name

I have a json block containing keys that have a similar name, each is numbered. I want to iterate over those keys. How can this be achieved?
Eg
$json = #"
{
"output": [
{
"AIeventCheck1": "A",
"AIeventCheck2": "B",
"AIeventCheck3": "C"
}
]
}
"#
$config = $json | ConvertFrom-Json
ForEach ($AIeventCheck in $config.output) {
Write-host AIeventCheck value: $AIeventCheck
}
target output:
A
B
C
Use the psobject memberset to access the individual properties of the object(s):
foreach($AIeventCheck in $config.output){
$AIEventCheckValues = $AIEventCheck.psobject.Properties |Where Name -like 'AIeventCheck*' |ForEach-Object Value
Write-Host AIeventCheck value: $AIeventCheckValues
}

Store to a powershell array, the value for each json key (with given name) regardless of json structure

Given a json file with any content/structure, where there are plenty of keys called "bob", how can I retrieve to an array, the values relating to all key instances called bob?
Eg:
Build list of all found keys called 'bob'
For each key
obtain the corresponding value
store it in array
# Sample JSON input with "bob" properties at various levels of the object graph.
$json = #'
{
"bob": "carol",
"foo": [
{
"bob": "ted"
},
{
"bar": {
"bob": "alice"
}
}
]
}
'#
# Collect all "bob" property values in an array.
[array] $bobValues =
$json | ConvertFrom-Json | ForEach-Object {
# Helper script block that walks the object graph and outputs
# every "bob" property value.
$sb = {
foreach ($el in $args[0]) { # iterate over elements (if an array)
foreach ($prop in $el.psobject.Properties) { # iterate over properties
if ($prop.Name -eq 'bob') { $prop.Value } # output the value
else { & $sb $prop.Value } # recurse
}
}
}
# Call the script block with the input object.
& $sb $_
}
# Print the array.
$bobValues
The above yields:
carol
ted
alice

Split Period-Delimited Nodes To JSON Object

I have many string entries (this are namespace/class trees) that look like the following:
appsystem
appsystem.applications
appsystem.applications.APPactivities
appsystem.applications.APPmanager
appsystem.applications.APPmodels
appsystem.applications.MAPmanager
appsystem.applications.MAPmanager.maphub
appsystem.applications.MAPmanager.mapmanager
appsystem.applications.pagealertsmanager
appsystem.authentication
appsystem.authentication.manager
appsystem.authentication.manager.encryptionmanager
appsystem.authentication.manager.sso
appsystem.authentication.manager.tokenmanager
But, I need the final output to be like:
{
"name": "appsystem",
"children": [
{
"name": "applications",
"children": [
{"name": "APPactivities"},
{"name": "APPmanager"},
{"name": "APPmodels"},
{"name": "MAPmanager",
"children": [
{"name": "maphub"},
{"name": "mapmanager"}
]},
{"name": "pagealertsmanager"}
]
},
{
"name": "authentication",
"children": [
{"name": "manager",
"children": [
{"name": "encryptionmanager"},
{"name": "sso"},
{"name": "tokenmanager"}
]}
]
}
]
}
The total nodes can be any number.
I am assuming I am going to need recursion but I am at a loss on where even to begin.
This builds up nested lists, PowerShell ConvertTo-JSON flattens the outer list.
You can change the $Line in $s to $line in (Get-Content input.txt).
But I think this does it:
$s = #'
appsystem
appsystem.applications
appsystem.applications.APPactivities
appsystem.applications.APPmanager
appsystem.applications.APPmodels
appsystem.applications.MAPmanager
appsystem.applications.MAPmanager.maphub
appsystem.applications.MAPmanager.mapmanager
appsystem.applications.pagealertsmanager
appsystem.authentication
appsystem.authentication.manager
appsystem.authentication.manager.encryptionmanager
appsystem.authentication.manager.sso
appsystem.authentication.manager.tokenmanager
'# -split "`r`n"
$TreeRoot = New-Object System.Collections.ArrayList
foreach ($Line in $s) {
$CurrentDepth = $TreeRoot
$RemainingChunks = $Line.Split('.')
while ($RemainingChunks)
{
# If there is a dictionary at this depth then use it, otherwise create one.
$Item = $CurrentDepth | Where-Object {$_.name -eq $RemainingChunks[0]}
if (-not $Item)
{
$Item = #{name=$RemainingChunks[0]}
$null = $CurrentDepth.Add($Item)
}
# If there will be child nodes, look for a 'children' node, or create one.
if ($RemainingChunks.Count -gt 1)
{
if (-not $Item.ContainsKey('children'))
{
$Item['children'] = New-Object System.Collections.ArrayList
}
$CurrentDepth = $Item['children']
}
$RemainingChunks = $RemainingChunks[1..$RemainingChunks.Count]
}
}
$TreeRoot | ConvertTo-Json -Depth 1000
Edit: It's too slow? I tried some random pausing profiling and found (not too surprisingly) that it's the inner nested loop, which searches children arrays for matching child nodes, which is being hit too many times.
This is a redesigned version which still builds the tree, and this time it also builds a TreeMap hashtable of shortcuts into the tree, to all the previously build nodes, so it can jump right too them instead of searching the children lists for them.
I made a testing file, some 20k random lines. Original code processed it in 108 seconds, this one does it in 1.5 seconds and the output matches.
$TreeRoot = New-Object System.Collections.ArrayList
$TreeMap = #{}
foreach ($line in (Get-Content d:\out.txt)) {
$_ = ".$line" # easier if the lines start with a dot
if ($TreeMap.ContainsKey($_)) # Skip duplicate lines
{
continue
}
# build a subtree from the right. a.b.c.d.e -> e then d->e then c->d->e
# keep going until base 'a.b' reduces to something already in the tree, connect new bit to that.
$LineSubTree = $null
$TreeConnectionPoint = $null
do {
$lastDotPos = $_.LastIndexOf('.')
$leaf = $_.Substring($lastDotPos + 1)
$_ = $_.Substring(0, $lastDotPos)
# push the leaf on top of the growing subtree
$LineSubTree = if ($LineSubTree) {
#{"name"=$leaf; "children"=([System.Collections.ArrayList]#($LineSubTree))}
} else {
#{"name"=$leaf}
}
$TreeMap["$_.$leaf"] = $LineSubTree
} while (!($TreeConnectionPoint = $TreeMap[$_]) -and $_)
# Now we have a branch built to connect in to the existing tree
# but is there somewhere to put it?
if ($TreeConnectionPoint)
{
if ($TreeConnectionPoint.ContainsKey('children'))
{
$null = $TreeConnectionPoint['children'].Add($LineSubTree)
} else {
$TreeConnectionPoint['children'] = [System.Collections.ArrayList]#($LineSubTree)
}
} else
{ # nowhere to put it, this is a new root level connection
$null = $TreeRoot.Add($LineSubTree)
}
}
$TreeRoot | ConvertTo-Json -Depth 100
(#mklement0's code takes 103 seconds and produces a wildly different output - 5.4M characters of JSON instead of 10.1M characters of JSON. [Edit: because my code allows multiple root nodes in a list which my test file has, and their code does not allow that])
Auto-generated PS help links from my codeblock (if available):
New-Object (in module Microsoft.PowerShell.Utility)
Get-Content (in module Microsoft.PowerShell.Management)
ConvertTo-Json (in module Microsoft.PowerShell.Utility)
To complement TessellatingHeckler's great answer with an alternative implementation that uses a recursive function.
The emphasis is on modularity and terseness, not performance.[1]
# Outer function that loops over all paths and builds up a one or more nested
# hashtables reflecting the path hierarchy, which are converted to JSON on output.
# Note that only a single JSON object is output if all paths share the same root
# component; otherwise, a JSON *array* is output.
function convert-PathsToNestedJsonObject([string[]] $paths) {
$hts = New-Object Collections.ArrayList
$paths.ForEach({
$rootName = $_.split('.')[0]
$ht = $hts.Where({ $_.name -eq $rootName }, 'First')[0]
if (-not $ht) { [void] $hts.Add(($ht = #{})) }
convert-PathToNestedHashtable $ht $_
})
$hts | ConvertTo-Json -Depth 100
}
# Recursive helper function that takes a path such as "appsystem.applications"
# and converts it into a nested hashtable with keys "name" and "children" to
# reflect the path hierarchy.
function convert-PathToNestedHashtable([hashtable] $ht, [string] $path) {
$name, $childName, $rest = $path -split '\.', 3
$ht.name = $name
if ($childName) {
if ($ht.children) {
$htChild = $ht.children.Where({ $_.name -eq $childName }, 'First')[0]
} else {
$ht.children = New-Object Collections.ArrayList
$htChild = $null
}
if (-not $htChild) {
[void] $ht.children.Add(($htChild = #{}))
}
convert-PathToNestedHashtable $htChild "$childName.$rest"
}
}
# Call the outer function with the input paths (assumed to be stored in $paths).
convert-PathsToNestedJsonObject $paths
[1] One deliberate type of optimization is applied, which, however, still keeps the code terse:
PSv4+ offers the (little-known) array methods .ForEach() and .Where(), which are not only noticeably faster than their cmdlet counterparts ForEach-Object and Where-Object, but also offer additional features.
Specifically:
$paths.ForEach({ ... }) is used instead of
$paths | ForEach-Object { ... }
$ht.children.Where({ $_.name -eq $childName }, 'First')[0] is used instead of
$ht.children | Where-Object { $_.name -eq $childName } | Select-Object -First 1