Access JSON nested property using dynamic proprty name [duplicate] - json

I'm trying to dynamically parse & build-up a data structure of some incoming JSON files I'm to be supplied with (that'll be in non-standard structure) via Powershell to then process the data in those files & hand them over to the next step.
As part of that, I'm trying to build up the data structure of the JSON file into essentially a list of of data-paths for me to parse through & grab the data out of, so that I can cope with arrays, nested JSON objects and so on. So far so good.
Where I fall into some sort of Powershell peculiarity is in handling 2+ levels of depth via a variable. Let me give you a nice code-block to demonstrate the problem...
# Generate a Quick JSON file with different data types & levels
[object]$QuickJson = #'
{
"Name" : "I am a JSON",
"Version" : "1.2.3.4",
"SomeBool" : true,
"NULLValue" : null,
"ArrayOfVersions" : [1.0,2.0,3.0],
"MyInteger" : 69,
"NestedJSON" : {
"Version" : 5.0,
"IsReady" : false
},
"DoubleNestedJSON" : {
"FirstLevel" : 1,
"DataValue" : "I am at first nested JSON level!",
"Second_JSON_Level" : {
"SecondLevel" : 2,
"SecondDataValue" : "I am on the 2nd nested level"
}
}
}
'#
# Import our JSON file into Powershell
[object]$MyPSJson = ConvertFrom-Json -InputObject $QuickJson
# Two quick string variables to access our JSON data paths
[string]$ShortJsonPath = "Name"
[string]$NestedJsonPath = "NestedJson.Version"
# Long string to access a double-nested JSON object
[string]$LongNestedJsonPath = "DoubleNestedJSON.Second_JSON_Level.SecondDataValue"
# Both of these work fine
Write-Host ("JSON Name (Direct) ==> " + $MyPSJson.Name)
Write-Host ("JSON Name (via Variable) ==> " + $MyPSJson.$ShortJsonPath)
# The following way to access a single nested Json Path works fine
Write-Host ("Nested JSON Version (via direct path) ==> " + $MyPSJson.NestedJson.Version)
# And THIS returns an empty line / is where I fall afoul of something in Powershell
Write-Host ("Nested JSON Version (via variable) ==> " + $MyPSJson.$NestedJsonPath)
# Other things I tried -- all returning an empty line / failing in effect
Write-Host ("Alternate Nested JSON Version ==> " + $($MyPSJson.$NestedJsonPath))
Write-Host ("Alternate Nested JSON Version ==> " + $MyPSJson.$($NestedJsonPath))
Write-Host ("Alternate Nested JSON Version ==> " + $($MyPSJson).$($NestedJsonPath))
# Similarly, while THIS works...
$MyPSJson | select-object -Property NestedJSON
# This will fail / return me nothing
$MyPSJson | select-object -Property NestedJSON.Version
... in doing a bunch of research around this, I came across a suggestion to transform this into a Hashtable -- but that has the same problem, sadly. So with the above code-snippet, the following will transform the JSON object into a hashtable.
# Same problem with a hash-table if constructed from the JSON file...
[hashtable]$MyHash = #{}
# Populate $MyHash with the data from our quickie JSON file...
$QuickJson | get-member -MemberType NoteProperty | Where-Object{ -not [string]::IsNullOrEmpty($QuickJson."$($_.name)")} | ForEach-Object {$MyHash.add($_.name, $QuickJson."$($_.name)")}
# ... and even then -- $MyHash."$($NestedJsonPath)" -- fails, while a single level deep string works fine in the variable! :(
So it's pretty clear that I'm running into "something" of a Powershell internal logic problem, but I can't get Powershell to be overly helpful in WHY that is. Adding a '-debug' or similar in an attempt to increase verbosity hasn't helped shed light on this.
I suspect it's something akin to the items raised in this article here ( https://blogs.technet.microsoft.com/heyscriptingguy/2011/10/16/dealing-with-powershell-hash-table-quirks/ ) but just specific with variables.
I've not had any luck in finding anything obvious in the Powershell language specification (3.0 still being the latest from here as far as I can tell -- https://www.microsoft.com/en-usdownload/details.aspx?id=36389 ) either. It may be in there, I may just miss it.
Any advice in how to get Powershell to play nice with this would be greatly appreciated. I'm not sure how / why Powershell is fine with a simple string but seems to have issues with a 'something.somethingelse' type string here.
Thank you.
Further notes & addenda to the original:
It seems there are several issues to attack. One is "dealing with a single nested level". The "quick fix" for that seems to be using "Invoke-Expression" to resolve the statement, so for instance (IMPORTANT - take note of the back-tick with the first variable!):
iex "`$MyPSJson.$NestedJsonPath"
That use of Invoke-Expression also works with multi-nested situations:
iex "`$MyPSJson.$LongNestedJsonPath"
An alternative approach that was mentioned is the use of multiple select statements ... but I've not been able to get that to work with multi-nested objects (Powershell seems to not resolve those properly for some reason).
So for instance in this scenario:
($MyComp | select $_.DoubleNestedJSON | select FirstLevel)
Powershell returns
FirstLevel
----------
... instead of the actual data value. So - for now, it seems that selects won't work with multi-level nested objects due to Powershell apparently not resolving them?

When you write something like
$MyPSJson.Name
this will attempt to retrieve the member named Name from the object $MyPSJson. If there is no such member, you'll get $null.
Now, when you do that with variables for the member name:
$MyPSJson.$ShortJsonPath
this works pretty much identical in that the member with the name stored in $ShortJsonPath is looked up and its value retrieved. No surprises here.
When you try that with a member that doesn't exist on the object, such as
$MyPSJson.$NestedJsonPath
# equivalent to
# $MyPSJson.'NestedJSON.Version'
you'll get $null, as detailed before. The . operator will only ever access a member of the exact object that is the result of its left-hand-side expression. It will never go through a member hierarchy in the way you seem to expect it to do. To be frank, I'm not aware of a language that works that way.
The reason it works with Invoke-Expression is, that you effectively converting the $NestedJsonPath string into part of an expression resulting in:
$MyPSJson.NestedJSON.Version
which Invoke-Expression then evaluates.
You can, of course, define your own function that works that way (and I'd much prefer that instead of using Invoke-Expression, a cmdlet that should rarely, if ever, used (heck, it's eval for PowerShell – few languages with eval advocate its use)):
function Get-DeepProperty([object] $InputObject, [string] $Property) {
$path = $Property -split '\.'
$obj = $InputObject
$path | %{ $obj = $obj.$_ }
$obj
}
PS> Get-DeepProperty $MyPSJson NestedJson.Version
5,0
You could even make it a filter, so you can use it more naturally on the pipeline:
filter Get-DeepProperty([string] $Property) {
$path = $Property -split '\.'
$obj = $_
$path | %{ $obj = $obj.$_ }
$obj
}
PS> $MyPSJson | Get-DeepProperty nestedjson.version
5,0

Why this doesn't work
When you provide the properties that you'd like within a string, like this
[string]$NestedJsonPath = "NestedJson.Version"
Powershell looks for a property called NestedJSon.Version. It's not actually traversing the properties, but looking for a string literal which contains a period. In fact, if I add a property like that to your JSON like so.
[object]$QuickJson = #'
{
"Name" : "I am a JSON",
"Version" : "1.2.3.4",
"SomeBool" : true,
"NULLValue" : null,
"ArrayOfVersions" : [1.0,2.0,3.0],
"MyInteger" : 69,
"NestedJSON.Version" : 69,
"NestedJSON" : {
"Version" : 5.0,
"IsReady" : false
}
}
I now get a value back, like so:
>$MyPSJson.$NestedJsonPath
69
The best way to get your values back is to use two separate variables, like this.
$NestedJson = "NestedJson"
$property = "Version"
>$MyPSJson.$NestedJson.$property
5.0
Or, alternatively, you could use select statements, as seen in the original answer below.
$MyPSJson | select $_.NestedJSON | select Version
Version
-------
1.2.3.4
If you use multiple Select-Object statements, they'll discard the other properties and allow you to more easily drill down to the value you'd like.

I followed Joey's filter example. However, I found it did not support accessing arrays.
Sharing the code that I got to work for this. Hopefully it will help others as well. Awesome thread!
filter Get-DeepProperty([string] $Property) {
$path = $Property -split '\.'
$obj = $_
foreach($node in $path){
if($node -match '.*\[\d*\]'){
$keyPieces = $node -split ('\[')
$arrayKey = $keyPieces[0]
$arrayIndex = $keyPieces[1] -replace ('\]','')
$obj = $obj.$arrayKey[$arrayIndex]
} else {
$obj = $obj.$node
}
}
$obj
}
Example usage:
$path = "nested.nestedtwo.steps[2]"
$payload | Get-DeepProperty $path

I had the same problem, so I wrote a function that does the trick.
It enables accessing any level of the json by variable path (string):
function getNestedJsonValue() {
param(
[Parameter(Mandatory = $true, ValueFromPipeline)] [PSCustomObject] $inputObj,
[Parameter(Mandatory = $true)] [string] $valuePath
)
if (($valuePath -eq $null) -or ($valuePath.length -eq 0) -or ($inputObj -eq $null)) {
return $inputObj
}
[System.Array] $nodes = "$valuePath" -split '\.'
foreach ($node in $nodes) {
if (($node -ne $null) -and ($node.length -gt 0) -and ($inputObj -ne $null)) {
$inputObj = $inputObj.$node
} else {
return $inputObj
}
}
return $inputObj
}
Usage: getNestedJsonValue -valuePath $nestedValuePath -inputObj $someJson
Pipe usage: $someJson | getNestedJsonValue -valuePath $nestedValuePath
An example nestedValuePath would be $nestedValuePath="some.nested.path"

Credit to wOxxOm for getting things on the right track.
Invoke-Expression does seem to work perfectly for this situation (if somewhat expensive, but that's fine in my personal example & situation), and it can cope with multiple levels of nesting.
So as examples for the above code snippet, the following will resolve just fine (Key point - pay attention to the initial back-tick. That caught me off guard):
Write-Host ("Single level JSON test ==> " + (iex "`$MyPSJson.$NestedJsonPath"))
Write-Host ("Double level JSON test ==> " + (iex "`$MyPSJson.$LongNestedJsonPath"))
That'll return our desired results:
Single level JSON test ==> 5.0
Double level JSON test ==> I am on the 2nd nested level
FoxDeploy's answer of using multi-level selects doesn't seem to work with 2+ levels of nesting, unfortunately for some bizarre reason.
Using:
($MyPSJson | select $_.DoubleNestedJSON | select FirstLevel)
We get the following back from Powershell:
FirstLevel
----------
... it seems that Powershell doesn't resolve nested objects in its entirety? We get a similar results if we intentionally use something that doesn't exist:
($MyPSJson | select $_.DoubleNestedJSON | select Doesnotexist)
... also simply returns:
Doesnotexist
------------
So - for now - it seems as if "Invoke-Expression" works most reliably (and most easily, as it's just a case of handing it a variable with the path'ed string).
I still can't explain the WHY of any of this so far (since I've used 'dotwalk'-ing with multiple variables through arrays quite happily), but at least there's a solution for now ... and that is Invoke-Expression !
The best (/least bad?) explanations for Invoke-Expression I've found so far are here (Microsoft's own description of the cmdlet doesn't really make a great job of hinting that it'd help in situations such as this):
http://ss64.com/ps/invoke-expression.html
https://www.adminarsenal.com/powershell/invoke-expression/

Related

Fetching JSON with PowerShell - How do I fetch all keys under multiple arrays?

I'm trying to build a json file that'll be used in our diverse scripts to fetch our server details.
The same script will be used across all our environments that contains multiple products and components. Here's the kind of json file I'm working with:
{
"DEV" : {
"Product1" : [
{"serverName" : "hostname1", "isWebServer" : "true"},
{"serverName" : "hostname2", "isWebServer" : "false"}
],
"Product2" : [
{"serverName" : "hostname3", "isWebServer" : "false"},
{"serverName" : "hostname4", "isWebServer" : "true"}
]
}
}
JSON file is imported into a variable and I'm able to use it successfully.
$jsonFile = Get-Content -Path "C:\temp\test.json" -Raw | ConvertFrom-Json
What I'm trying to do in my PowerShell script is fetch all serverName that is configured with key "isWebServer" : "true"
I can successfully achieve my goal if I'm drilling down on each arrays directly.
($jsonFile.DEV.Product1 | Where-Object {$_.isWebServer -eq "true"}).serverName
However, I'm unable to achieve the same success when I want to fetch all web servers from the whole DEV environment. I'm looking for something like this but that will actually return something.
($jsonFile.DEV | Where-Object {$_.isWebServer -eq "true"}).serverName
How can I fetch everything under all arrays within an object ? I'm used to XPATHs in which I can use wildcards for that kind of scenario but I haven't found the equivalent in JSON.
Here is a possible solution:
$json.DEV.PSObject.Properties.Value.Where{ $_.isWebServer -eq 'true' }.serverName
The hidden (intrinsic) PSObject member allows us to access properties (Product1, Product2, ...) of the DEV object, without having to know their names in advance.
Then, using member access enumeration, we get an array of all Value members of the PSPropertyInfo objects stored in PSObject.Properties.
To filter the server objects, the intrinsic method Where is used to return only the servers matching the given condition. Feel free to use Where-Object instead:
$json.DEV.PSObject.Properties.Value |
Where-Object isWebServer -eq 'true' |
ForEach-Object serverName
Though I prefer the intrinsic methods as they are faster than the pipeline (less overhead) and more succinct to write.
It looks similar to what I did a long time ago. See if this helps.

How to set json property on powershell using a parameter value as the property name [duplicate]

Say I have JSON like:
{
"a" : {
"b" : 1,
"c" : 2
}
}
Now ConvertTo-Json will happily create PSObjects out of that. I want to access an item I could do $json.a.b and get 1 - nicely nested properties.
Now if I have the string "a.b" the question is how to use that string to access the same item in that structure? Seems like there should be some special syntax I'm missing like & for dynamic function calls because otherwise you have to interpret the string yourself using Get-Member repeatedly I expect.
No, there is no special syntax, but there is a simple workaround, using iex, the built-in alias[1] for the Invoke-Expression cmdlet:
$propertyPath = 'a.b'
# Note the ` (backtick) before $json, to prevent premature expansion.
iex "`$json.$propertyPath" # Same as: $json.a.b
# You can use the same approach for *setting* a property value:
$newValue = 'foo'
iex "`$json.$propertyPath = `$newValue" # Same as: $json.a.b = $newValue
Caveat: Do this only if you fully control or implicitly trust the value of $propertyPath.
Only in rare situation is Invoke-Expression truly needed, and it should generally be avoided, because it can be a security risk.
Note that if the target property contains an instance of a specific collection type and you want to preserve it as-is (which is not common) (e.g., if the property value is a strongly typed array such as [int[]], or an instance of a list type such as [System.Collections.Generic.List`1]), use the following:
# "," constructs an aux., transient array that is enumerated by
# Invoke-Expression and therefore returns the original property value as-is.
iex ", `$json.$propertyPath"
Without the , technique, Invoke-Expression enumerates the elements of a collection-valued property and you'll end up with a regular PowerShell array, which is of type [object[]] - typically, however, this distinction won't matter.
Note: If you were to send the result of the , technique directly through the pipeline, a collection-valued property value would be sent as a single object instead of getting enumerated, as usual. (By contrast, if you save the result in a variable first and the send it through the pipeline, the usual enumeration occurs). While you can force enumeration simply by enclosing the Invoke-Expression call in (...), there is no reason to use the , technique to begin with in this case, given that enumeration invariably entails loss of the information about the type of the collection whose elements are being enumerated.
Read on for packaged solutions.
Note:
The following packaged solutions originally used Invoke-Expression combined with sanitizing the specified property paths in order to prevent inadvertent/malicious injection of commands. However, the solutions now use a different approach, namely splitting the property path into individual property names and iteratively drilling down into the object, as shown in Gyula Kokas's helpful answer. This not only obviates the need for sanitizing, but turns out to be faster than use of Invoke-Expression (the latter is still worth considering for one-off use).
The no-frills, get-only, always-enumerate version of this technique would be the following function:
# Sample call: propByPath $json 'a.b'
function propByPath { param($obj, $propPath) foreach ($prop in $propPath.Split('.')) { $obj = $obj.$prop }; $obj }
What the more elaborate solutions below offer: parameter validation, the ability to also set a property value by path, and - in the case of the propByPath function - the option to prevent enumeration of property values that are collections (see next point).
The propByPath function offers a -NoEnumerate switch to optionally request preserving a property value's specific collection type.
By contrast, this feature is omitted from the .PropByPath() method, because there is no syntactically convenient way to request it (methods only support positional arguments). A possible solution is to create a second method, say .PropByPathNoEnumerate(), that applies the , technique discussed above.
Helper function propByPath:
function propByPath {
param(
[Parameter(Mandatory)] $Object,
[Parameter(Mandatory)] [string] $PropertyPath,
$Value, # optional value to SET
[switch] $NoEnumerate # only applies to GET
)
Set-StrictMode -Version 1
# Note: Iteratively drilling down into the object turns out to be *faster*
# than using Invoke-Expression; it also obviates the need to sanitize
# the property-path string.
$props = $PropertyPath.Split('.') # Split the path into an array of property names.
if ($PSBoundParameters.ContainsKey('Value')) { # SET
$parentObject = $Object
if ($props.Count -gt 1) {
foreach ($prop in $props[0..($props.Count-2)]) { $parentObject = $parentObject.$prop }
}
$parentObject.($props[-1]) = $Value
}
else { # GET
$value = $Object
foreach ($prop in $props) { $value = $value.$prop }
if ($NoEnumerate) {
, $value
} else {
$value
}
}
}
Instead of the Invoke-Expression call you would then use:
# GET
propByPath $obj $propertyPath
# GET, with preservation of the property value's specific collection type.
propByPath $obj $propertyPath -NoEnumerate
# SET
propByPath $obj $propertyPath 'new value'
You could even use PowerShell's ETS (extended type system) to attach a .PropByPath() method to all [pscustomobject] instances (PSv3+ syntax; in PSv2 you'd have to create a *.types.ps1xml file and load it with Update-TypeData -PrependPath):
'System.Management.Automation.PSCustomObject',
'Deserialized.System.Management.Automation.PSCustomObject' |
Update-TypeData -TypeName { $_ } `
-MemberType ScriptMethod -MemberName PropByPath -Value { #`
param(
[Parameter(Mandatory)] [string] $PropertyPath,
$Value
)
Set-StrictMode -Version 1
$props = $PropertyPath.Split('.') # Split the path into an array of property names.
if ($PSBoundParameters.ContainsKey('Value')) { # SET
$parentObject = $this
if ($props.Count -gt 1) {
foreach ($prop in $props[0..($props.Count-2)]) { $parentObject = $parentObject.$prop }
}
$parentObject.($props[-1]) = $Value
}
else { # GET
# Note: Iteratively drilling down into the object turns out to be *faster*
# than using Invoke-Expression; it also obviates the need to sanitize
# the property-path string.
$value = $this
foreach ($prop in $PropertyPath.Split('.')) { $value = $value.$prop }
$value
}
}
You could then call $obj.PropByPath('a.b') or $obj.PropByPath('a.b', 'new value')
Note: Type Deserialized.System.Management.Automation.PSCustomObject is targeted in addition to System.Management.Automation.PSCustomObject in order to also cover deserialized custom objects, which are returned in a number of scenarios, such as using Import-CliXml, receiving output from background jobs, and using remoting.
.PropByPath() will be available on any [pscustomobject] instance in the remainder of the session (even on instances created prior to the Update-TypeData call [2]); place the Update-TypeData call in your $PROFILE (profile file) to make the method available by default.
[1] Note: While it is generally advisable to limit aliases to interactive use and use full cmdlet names in scripts, use of iex to me is acceptable, because it is a built-in alias and enables a concise solution.
[2] Verify with (all on one line) $co = New-Object PSCustomObject; Update-TypeData -TypeName System.Management.Automation.PSCustomObject -MemberType ScriptMethod -MemberName GetFoo -Value { 'foo' }; $co.GetFoo(), which outputs foo even though $co was created before Update-TypeData was called.
This workaround is maybe useful to somebody.
The result goes always deeper, until it hits the right object.
$json=(Get-Content ./json.json | ConvertFrom-Json)
$result=$json
$search="a.c"
$search.split(".")|% {$result=$result.($_) }
$result
You can have 2 variables.
$json = '{
"a" : {
"b" : 1,
"c" : 2
}
}' | convertfrom-json
$a,$b = 'a','b'
$json.$a.$b
1

How to serialize an object in powershell to json and get identical result in PS desktop and core?

Prolog
It turns out that in my case it is important to understand the source of the objects - it is a JSON payload from a REST API response. Unfortunately, JSON -> Object conversion produces different results on PS desktop vs PS core. On desktop the numbers are deserialized into Int32 types, but on core - to Int64 types. From that it follows that I cannot use Export-CliXml, because the binary layout of the objects is different.
Main question
I have a unit test that needs to compare the actual result with an expected. The expected result is saved in a json file, so the procedure is:
Convert the actual result to json string
Read the expected result from disk to string
Compare the actual and the expected as strings
Unfortunately, this scheme does not work because PS desktop ConvertTo-Json and PS core ConvertTo-Json do not produce identical results. So, if the expected result was saved on desktop and the test runs on core - boom, failure. And vice versa.
One way is to keep two versions of jsons. Another way is to use a library to create the json.
First I tried the Newtonsoft-Json powershell module, but it just does not work. I think the problem is that whatever C# library we use, it must be aware of PSCustomObject and alike and treat them specially. So, we cannot just take any C# JSON library.
At this point I am left with having two jsons - one per PS edition, which is kind of sad.
Are there better options?
EDIT 1
I guess I can always read the json, convert to object and then back to json again. That sucks.
EDIT 2
I tried to use ConvertTo-Json -Compress. This eliminates the difference in spacing, but the problem is that for some reason the desktop version translates all the non characters to \u000... representation. The core version does not do it.
Please, observe:
Desktop
C:\> #{ x = "'a'" } |ConvertTo-Json -Compress
{"x":"\u0027a\u0027"}
C:\>
Core
C:\> #{ x = "'a'" } |ConvertTo-Json -Compress
{"x":"'a'"}
C:\>
Now the core version has the flag -EscapeHandling, so:
C:\> #{ x = "'a'" } |ConvertTo-Json -Compress -EscapeHandling EscapeHtml
{"x":"\u0027a\u0027"}
C:\>
Bingo! Same result. But now this code does not run on the desktop version, which does not have this flag. More massaging is needed. I will check if that is the only problem.
EDIT 3
It is impossible to reconcile the differences between the core and the desktop versions without expensive post processing. Please, observe:
Desktop
C:\> #{ x = '"a"';y = "'b'" } |ConvertTo-Json -Compress
{"y":"\u0027b\u0027","x":"\"a\""}
C:\>
Core
C:\> #{ x = '"a"';y = "'b'" } |ConvertTo-Json -Compress -EscapeHandling EscapeHtml
{"y":"\u0027b\u0027","x":"\u0022a\u0022"}
C:\> #{ x = '"a"';y = "'b'" } |ConvertTo-Json -Compress
{"y":"'b'","x":"\"a\""}
C:\>
Any suggestions on how to salvage the json approach?
EDIT 4
The Export-CliXml approach does not work too, because of the differences between the PS versions.
Desktop
C:\> ('{a:1}' | ConvertFrom-Json).a.gettype()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Int32 System.ValueType
C:\>
Core
C:\> ('{a:1}' | ConvertFrom-Json).a.gettype()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Int64 System.ValueType
C:\>
So the same JSON is represented using different numeric types - Int32 in desktop and Int64 in core. That puts to bed the option of using Export-CliXml.
Unless I am missing something.
I believe there is no other choice, but do the double conversion - json -> object -> json and then I will have two jsons created on the same PS edition. That sucks big time.
On converting from the original JSON, use the third-party Newtonsoft.Json PowerShell wrapper's ConvertFrom-JsonNewtonsoft cmdlet - this should ensure cross-edition compatibility (which the built-in ConvertFrom-Json does not guarantee across PowerShell editions, because Windows PowerShell uses a custom parser, whereas PowerShell [Core] v6+ uses Newtonsoft.json up to at least v7.1, though a move to the new(ish) .NET Core System.Text.Json API is coming).
Important: ConvertFrom-JsonNewtonsoft returns (arrays of) nested ordered hashtables ([ordered] #{ ... }, System.Collections.Specialized.OrderedDictionary), unlike the nested [pscustomobject] graphs that the built-in ConvertFrom-Json outputs. Similarly, ConvertTo-JsonNewtonsoft expects only (arrays of) hashtables (dictionaries) as input, and notably does not support [pscustomobject] instances, as you've learned yourself.
Caveat: As of this writing, the wrapper module was last updated in May 2019, and the version of the underlying bundled Newtonsoft.Json.dll assembly is quite old (8.0, whereas 12.0 is current as of this writing). See the module's source code.
Note that in order to parse JSON obtained from a RESTful web service manually, you mustn't use Invoke-RestMethod, as it implicitly parses and returns [pscustomobject] object graphs. Instead, use Invoke-WebRequest and access the returned response's .Content property.
On converting to a format suitable for storing on disk, you have two options:
(A) If you do need the serialized format to be JSON also, you must convert all [pscustomobject] graphs to (ordered) hashtables before passing them to ConvertTo-JsonNewtonsoft.
See below for function ConvertTo-OrderedHashTable, which does just that.
(B) If the specific serialization format isn't important, i.e. if all that matters is that the formats are identical across PowerShell editions for the purpose of comparison, no extra works is needed: use the built-in Export-Clixml cmdlet, which can handle any type and produces PowerShell's native, XML-based serialization format called CLIXML (as notably used in PowerShell remoting), which should be cross-edition-compatible (at least with v5.1 on the Windows PowerShell side and as of PowerShell [Core] v7.1, both of which use the same version of the serialization protocol, 1.1.0.1, as reported by $PSVersionTable.SerializationVersion).
While you could re-convert such a persisted file to objects with Import-Clixml, the potential loss of type fidelity on deserialization makes comparing the serialized (CLIXML) representations advisable.
Also note that as of PowerShell v7.1 there is no cmdlet-based way to create an in-memory CLIXML representation, so you'll have to use the PowerShell API directly for now: System.Management.Automation.PSSerializer.Serialize. However, providing in-memory counterparts to Import-CliXml / Export-CliXml in the form of ConvertFrom-CliXml / ConvertTo-CliXml cmdlets has been green-lighted as a future enhancement.
Re (A): Here's function ConvertTo-OrderedHashtable, which converts (potentially nested) [pscustomobject] objects to ordered hashtables while passing other types through, so you should be able to simply insert it into a pipeline as follows:
# CAVEAT: ConvertTo-JsonNewtonSoft only accepts a *single* input object.
[pscustomobject] #{ foo = 1 }, [pscustomobject] #{ foo = 2 } |
ConvertTo-OrderedHashtable |
ForEach-Object { ConvertTo-JsonNewtonSoft $_ }
function ConvertTo-OrderedHashtable {
<#
.SYNOPSIS
Converts custom objects to ordered hashtables.
.DESCRIPTION
Converts PowerShell custom objects (instances of [pscustomobject]) to
ordered hashtables (instances of [System.Collections.Specialized.OrderedDictionary]),
which is useful for to-JSON serialization via the Newtonsoft.JSON library.
Note:
* Custom objects are processed recursively.
* Any scalar non-custom objects are passed through as-is.
* Any (non-dictionary) collections in property values are converted to
[object[]] arrays.
.EXAMPLE
1, [pscustomobject] #{ foo = [pscustomobject] #{ bar = 'none' }; other = 2 } | ConvertTo-OrderedHashtable
Passes integer 1 through, and converts the custom object to a nested ordered
hashtable.
#>
[CmdletBinding()]
param(
[Parameter(ValueFromPipeline)] $InputObject
)
begin {
# Recursive helper function
function convert($obj) {
if ($obj -is [System.Management.Automation.PSCustomObject]) {
# a custom object: recurse on its properties
$oht = [ordered] #{ }
foreach ($prop in $obj.psobject.Properties) {
$oht.Add($prop.Name, (convert $prop.Value))
}
return $oht
}
elseif ($obj -isnot [string] -and $obj -is [System.Collections.IEnumerable] -and $obj -isnot [System.Collections.IDictionary]) {
# A collection of sorts (other than a string or dictionary (hash table)), recurse on its elements.
return #(foreach ($el in $obj) { convert $el })
}
else {
# a non-custom object, including .NET primitives and strings: use as-is.
return $obj
}
}
}
process {
convert $InputObject
}
}
Re (B): A demonstration of the Export-CliXml approach (you can run this code from either PS edition):
$sb = {
Install-Module -Scope CurrentUser Newtonsoft.json
if (-not $IsCoreClr) {
# Workaround for PS Core's $env:PSModulePath overriding WinPS'
Import-Module $HOME\Documents\WindowsPowerShell\Modules\newtonsoft.json
}
#'
{
"results": {
"users": [
{
"userId": 1,
"emailAddress": "jane.doe#example.com",
"date": "2020-10-05T08:08:43.743741-04:00",
"attributes": {
"height": 165,
"weight": 60
}
},
{
"userId": 2,
"emailAddress": "john.doe#example.com",
"date": "2020-10-06T08:08:43.743741-04:00",
"attributes": {
"height": 180,
"weight": 72
}
}
]
}
}
'# | ConvertFrom-JsonNewtonsoft | Export-CliXml "temp-$($PSVersionTable.PSEdition).xml"
}
# Execute the script block in both editions
Write-Verbose -vb 'Running in Windows PowerShell...'
powershell -noprofile $sb
Write-Verbose -vb 'Running in PowerShell Core...'
pwsh -noprofile $sb
# Compare the resulting CLIXML files.
Write-Verbose -vb "Comparing the resulting files: This should produce NO output,`n indicating that the files have identical content."
Compare-Object (Get-Content 'temp-Core.xml') (Get-Content 'temp-Desktop.xml')
Write-Verbose -vb 'Cleaning up...'
Remove-Item 'temp-Core.xml', 'temp-Desktop.xml'
You should see the following verbose output:
VERBOSE: Running in Windows PowerShell...
VERBOSE: Running in PowerShell Core...
VERBOSE: Comparing the resulting files: This should produce NO output,
indicating that the files have identical content.
VERBOSE: Cleaning up...

Problems parsing / accessing nested JSON / Hashtable data via variables in Powershell

I'm trying to dynamically parse & build-up a data structure of some incoming JSON files I'm to be supplied with (that'll be in non-standard structure) via Powershell to then process the data in those files & hand them over to the next step.
As part of that, I'm trying to build up the data structure of the JSON file into essentially a list of of data-paths for me to parse through & grab the data out of, so that I can cope with arrays, nested JSON objects and so on. So far so good.
Where I fall into some sort of Powershell peculiarity is in handling 2+ levels of depth via a variable. Let me give you a nice code-block to demonstrate the problem...
# Generate a Quick JSON file with different data types & levels
[object]$QuickJson = #'
{
"Name" : "I am a JSON",
"Version" : "1.2.3.4",
"SomeBool" : true,
"NULLValue" : null,
"ArrayOfVersions" : [1.0,2.0,3.0],
"MyInteger" : 69,
"NestedJSON" : {
"Version" : 5.0,
"IsReady" : false
},
"DoubleNestedJSON" : {
"FirstLevel" : 1,
"DataValue" : "I am at first nested JSON level!",
"Second_JSON_Level" : {
"SecondLevel" : 2,
"SecondDataValue" : "I am on the 2nd nested level"
}
}
}
'#
# Import our JSON file into Powershell
[object]$MyPSJson = ConvertFrom-Json -InputObject $QuickJson
# Two quick string variables to access our JSON data paths
[string]$ShortJsonPath = "Name"
[string]$NestedJsonPath = "NestedJson.Version"
# Long string to access a double-nested JSON object
[string]$LongNestedJsonPath = "DoubleNestedJSON.Second_JSON_Level.SecondDataValue"
# Both of these work fine
Write-Host ("JSON Name (Direct) ==> " + $MyPSJson.Name)
Write-Host ("JSON Name (via Variable) ==> " + $MyPSJson.$ShortJsonPath)
# The following way to access a single nested Json Path works fine
Write-Host ("Nested JSON Version (via direct path) ==> " + $MyPSJson.NestedJson.Version)
# And THIS returns an empty line / is where I fall afoul of something in Powershell
Write-Host ("Nested JSON Version (via variable) ==> " + $MyPSJson.$NestedJsonPath)
# Other things I tried -- all returning an empty line / failing in effect
Write-Host ("Alternate Nested JSON Version ==> " + $($MyPSJson.$NestedJsonPath))
Write-Host ("Alternate Nested JSON Version ==> " + $MyPSJson.$($NestedJsonPath))
Write-Host ("Alternate Nested JSON Version ==> " + $($MyPSJson).$($NestedJsonPath))
# Similarly, while THIS works...
$MyPSJson | select-object -Property NestedJSON
# This will fail / return me nothing
$MyPSJson | select-object -Property NestedJSON.Version
... in doing a bunch of research around this, I came across a suggestion to transform this into a Hashtable -- but that has the same problem, sadly. So with the above code-snippet, the following will transform the JSON object into a hashtable.
# Same problem with a hash-table if constructed from the JSON file...
[hashtable]$MyHash = #{}
# Populate $MyHash with the data from our quickie JSON file...
$QuickJson | get-member -MemberType NoteProperty | Where-Object{ -not [string]::IsNullOrEmpty($QuickJson."$($_.name)")} | ForEach-Object {$MyHash.add($_.name, $QuickJson."$($_.name)")}
# ... and even then -- $MyHash."$($NestedJsonPath)" -- fails, while a single level deep string works fine in the variable! :(
So it's pretty clear that I'm running into "something" of a Powershell internal logic problem, but I can't get Powershell to be overly helpful in WHY that is. Adding a '-debug' or similar in an attempt to increase verbosity hasn't helped shed light on this.
I suspect it's something akin to the items raised in this article here ( https://blogs.technet.microsoft.com/heyscriptingguy/2011/10/16/dealing-with-powershell-hash-table-quirks/ ) but just specific with variables.
I've not had any luck in finding anything obvious in the Powershell language specification (3.0 still being the latest from here as far as I can tell -- https://www.microsoft.com/en-usdownload/details.aspx?id=36389 ) either. It may be in there, I may just miss it.
Any advice in how to get Powershell to play nice with this would be greatly appreciated. I'm not sure how / why Powershell is fine with a simple string but seems to have issues with a 'something.somethingelse' type string here.
Thank you.
Further notes & addenda to the original:
It seems there are several issues to attack. One is "dealing with a single nested level". The "quick fix" for that seems to be using "Invoke-Expression" to resolve the statement, so for instance (IMPORTANT - take note of the back-tick with the first variable!):
iex "`$MyPSJson.$NestedJsonPath"
That use of Invoke-Expression also works with multi-nested situations:
iex "`$MyPSJson.$LongNestedJsonPath"
An alternative approach that was mentioned is the use of multiple select statements ... but I've not been able to get that to work with multi-nested objects (Powershell seems to not resolve those properly for some reason).
So for instance in this scenario:
($MyComp | select $_.DoubleNestedJSON | select FirstLevel)
Powershell returns
FirstLevel
----------
... instead of the actual data value. So - for now, it seems that selects won't work with multi-level nested objects due to Powershell apparently not resolving them?
When you write something like
$MyPSJson.Name
this will attempt to retrieve the member named Name from the object $MyPSJson. If there is no such member, you'll get $null.
Now, when you do that with variables for the member name:
$MyPSJson.$ShortJsonPath
this works pretty much identical in that the member with the name stored in $ShortJsonPath is looked up and its value retrieved. No surprises here.
When you try that with a member that doesn't exist on the object, such as
$MyPSJson.$NestedJsonPath
# equivalent to
# $MyPSJson.'NestedJSON.Version'
you'll get $null, as detailed before. The . operator will only ever access a member of the exact object that is the result of its left-hand-side expression. It will never go through a member hierarchy in the way you seem to expect it to do. To be frank, I'm not aware of a language that works that way.
The reason it works with Invoke-Expression is, that you effectively converting the $NestedJsonPath string into part of an expression resulting in:
$MyPSJson.NestedJSON.Version
which Invoke-Expression then evaluates.
You can, of course, define your own function that works that way (and I'd much prefer that instead of using Invoke-Expression, a cmdlet that should rarely, if ever, used (heck, it's eval for PowerShell – few languages with eval advocate its use)):
function Get-DeepProperty([object] $InputObject, [string] $Property) {
$path = $Property -split '\.'
$obj = $InputObject
$path | %{ $obj = $obj.$_ }
$obj
}
PS> Get-DeepProperty $MyPSJson NestedJson.Version
5,0
You could even make it a filter, so you can use it more naturally on the pipeline:
filter Get-DeepProperty([string] $Property) {
$path = $Property -split '\.'
$obj = $_
$path | %{ $obj = $obj.$_ }
$obj
}
PS> $MyPSJson | Get-DeepProperty nestedjson.version
5,0
Why this doesn't work
When you provide the properties that you'd like within a string, like this
[string]$NestedJsonPath = "NestedJson.Version"
Powershell looks for a property called NestedJSon.Version. It's not actually traversing the properties, but looking for a string literal which contains a period. In fact, if I add a property like that to your JSON like so.
[object]$QuickJson = #'
{
"Name" : "I am a JSON",
"Version" : "1.2.3.4",
"SomeBool" : true,
"NULLValue" : null,
"ArrayOfVersions" : [1.0,2.0,3.0],
"MyInteger" : 69,
"NestedJSON.Version" : 69,
"NestedJSON" : {
"Version" : 5.0,
"IsReady" : false
}
}
I now get a value back, like so:
>$MyPSJson.$NestedJsonPath
69
The best way to get your values back is to use two separate variables, like this.
$NestedJson = "NestedJson"
$property = "Version"
>$MyPSJson.$NestedJson.$property
5.0
Or, alternatively, you could use select statements, as seen in the original answer below.
$MyPSJson | select $_.NestedJSON | select Version
Version
-------
1.2.3.4
If you use multiple Select-Object statements, they'll discard the other properties and allow you to more easily drill down to the value you'd like.
I followed Joey's filter example. However, I found it did not support accessing arrays.
Sharing the code that I got to work for this. Hopefully it will help others as well. Awesome thread!
filter Get-DeepProperty([string] $Property) {
$path = $Property -split '\.'
$obj = $_
foreach($node in $path){
if($node -match '.*\[\d*\]'){
$keyPieces = $node -split ('\[')
$arrayKey = $keyPieces[0]
$arrayIndex = $keyPieces[1] -replace ('\]','')
$obj = $obj.$arrayKey[$arrayIndex]
} else {
$obj = $obj.$node
}
}
$obj
}
Example usage:
$path = "nested.nestedtwo.steps[2]"
$payload | Get-DeepProperty $path
I had the same problem, so I wrote a function that does the trick.
It enables accessing any level of the json by variable path (string):
function getNestedJsonValue() {
param(
[Parameter(Mandatory = $true, ValueFromPipeline)] [PSCustomObject] $inputObj,
[Parameter(Mandatory = $true)] [string] $valuePath
)
if (($valuePath -eq $null) -or ($valuePath.length -eq 0) -or ($inputObj -eq $null)) {
return $inputObj
}
[System.Array] $nodes = "$valuePath" -split '\.'
foreach ($node in $nodes) {
if (($node -ne $null) -and ($node.length -gt 0) -and ($inputObj -ne $null)) {
$inputObj = $inputObj.$node
} else {
return $inputObj
}
}
return $inputObj
}
Usage: getNestedJsonValue -valuePath $nestedValuePath -inputObj $someJson
Pipe usage: $someJson | getNestedJsonValue -valuePath $nestedValuePath
An example nestedValuePath would be $nestedValuePath="some.nested.path"
Credit to wOxxOm for getting things on the right track.
Invoke-Expression does seem to work perfectly for this situation (if somewhat expensive, but that's fine in my personal example & situation), and it can cope with multiple levels of nesting.
So as examples for the above code snippet, the following will resolve just fine (Key point - pay attention to the initial back-tick. That caught me off guard):
Write-Host ("Single level JSON test ==> " + (iex "`$MyPSJson.$NestedJsonPath"))
Write-Host ("Double level JSON test ==> " + (iex "`$MyPSJson.$LongNestedJsonPath"))
That'll return our desired results:
Single level JSON test ==> 5.0
Double level JSON test ==> I am on the 2nd nested level
FoxDeploy's answer of using multi-level selects doesn't seem to work with 2+ levels of nesting, unfortunately for some bizarre reason.
Using:
($MyPSJson | select $_.DoubleNestedJSON | select FirstLevel)
We get the following back from Powershell:
FirstLevel
----------
... it seems that Powershell doesn't resolve nested objects in its entirety? We get a similar results if we intentionally use something that doesn't exist:
($MyPSJson | select $_.DoubleNestedJSON | select Doesnotexist)
... also simply returns:
Doesnotexist
------------
So - for now - it seems as if "Invoke-Expression" works most reliably (and most easily, as it's just a case of handing it a variable with the path'ed string).
I still can't explain the WHY of any of this so far (since I've used 'dotwalk'-ing with multiple variables through arrays quite happily), but at least there's a solution for now ... and that is Invoke-Expression !
The best (/least bad?) explanations for Invoke-Expression I've found so far are here (Microsoft's own description of the cmdlet doesn't really make a great job of hinting that it'd help in situations such as this):
http://ss64.com/ps/invoke-expression.html
https://www.adminarsenal.com/powershell/invoke-expression/

Merging JSON Objects at runtime in Powershell

Following on from a question yesterday (JSON and references to other JSON objects).
Is it possible to merge JSON objects at runtime in a similar fashion?
In my test.json I wish to insert the $Defaults.wimos object into WIMS.wimos at runtime similar to what I did for the $Paths.drive value in $Defaults.
{
Paths: {drive: "W:"},
Defaults: {wimos: {dstdrive: "$($Paths.drive)"}
},
WIMS: {
winos: "$($Defaults.wimos)",
wimre: {dstdrive: "$($Paths.drive)"}
}
}
In the following code I cannot work out the syntax to have the object replaced at runtime.
$JSONConfig="test.json"
$rawJSON = (Get-Content $JSONConfig -Raw)
$pathJSON = $rawJSON | ConvertFrom-Json
#
# Load Paths from JSON into $Paths
#
$Paths=$pathJSON.Paths
#
# Merge JSON objects to have the defaults replaced
#
$DefaultsJSON=($ExecutionContext.InvokeCommand.ExpandString($rawJSON)) | ConvertFrom-Json
#
# Load Defaults into $Defaults
#
$Defaults=$DefaultsJSON.Defaults
#
# Merge JSON objects to have the System replaced
#
$JSON = ($ExecutionContext.InvokeCommand.ExpandString($rawJSON)) | ConvertFrom-Json
$JSON.WIMS
write-host ("JSON.WIMS.wimre.dstdrive =" + $JSON.WIMS.wimre.dstdrive)
write-host ("JSON.WIMS.wimos.dstdrive =" + $JSON.WIMS.wimos.dstdrive) #This fails to access and print "W:"
I would like to be able to replace $($Defaults.wimos) and be able to query the dstdrive member.
Is this possible to achieve this in powershell? Any suggestions on how?
Thanks
Stuart
There's a couple of issues in your code. First, the value that would be inserted into your template would be interpolated as a string and not a json-object.
To do this, you could use the ConvertTo-json within the template. I'll show you how in a minute.
If you're gonna be recreating and using different "templates" within the same file (like path, defaults etc), I would create a helper function for recreating / filling out the template on your session variables all the time (in order to avoid repeating code).
Create the following method:
function Get-ParsedJsonTemplate(){
return ($ExecutionContext.InvokeCommand.ExpandString((Get-Content $JSONConfig -Raw))) | ConvertFrom-Json
}
Change your template json to the following (showing you that you can use powershell code within your template file:
{
Paths: {drive: "W:"},
Defaults: {wimos: {dstdrive: "$($Paths.drive)"}},
WIMS: {
winos: $(
if($Defaults -and $Defaults.wimos){
$Defaults.wimos|ConvertTo-Json -Compress
} else {
#{"dstdrive"=""}|ConvertTo-Json -compress
}
),
wimre: {dstdrive: "$($Paths.drive)"}
}
}
You also had a typo in either your template or your printout (I see that you used wimos in your "printout" in powershell and "winos" in your template. I modified the code and the following should work for you (given the above test.json):
$JSONConfig="c:\tmp\test.json"
function Get-ParsedJsonTemplate(){
return ($ExecutionContext.InvokeCommand.ExpandString((Get-Content $JSONConfig -Raw))) | ConvertFrom-Json
}
$pathJSON = Get-ParsedJsonTemplate;
$Paths=$pathJSON.Paths
$DefaultsJSON=Get-ParsedJsonTemplate;
$Defaults=$DefaultsJSON.Defaults
$JSON = Get-ParsedJsonTemplate;
$JSON.WIMS
write-host ("JSON.WIMS.wimre.dstdrive =" + $JSON.WIMS.wimre.dstdrive)
write-host ("JSON.WIMS.wimos.dstdrive =" + $JSON.WIMS.winos.dstdrive)
The reason your script was failing was due to the fact that $Defaults.wimos was not the string { dstdrive: "W:" } it was instead a PSCustomObject which would be interpolated to a the value #{drive=W:}. To obtain the correct behavior, you need to convert the $Defaults.wimos value back to a json string.
$Defaults = $DefaultsJSON.Defaults
$Defaults.wimos = $Defaults.wimos | ConvertTo-Json -Compress