When loading an object from a json file one can normally set the value on properties and write the file back out like so:
$manifest = (gc $manifestPath) | ConvertFrom-Json -AsHashtable
$manifest.name = "$($manifest.name)-sxs"
$manifest | ConvertTo-Json -depth 100 | Out-File $manifestPath -Encoding utf8NoBOM
But if the source json file contains comments, the object's properties can't be set:
// *******************************************************
// GENERATED FILE - DO NOT EDIT DIRECTLY
// *******************************************************
{
"name": "PublishBuildArtifacts"
}
Running the code above throws an error:
$manifest
id : 1D341BB0-2106-458C-8422-D00BCEA6512A
name : PublishBuildArtifacts
friendlyName : ms-resource:loc.friendlyName
description : ms-resource:loc.description
category : Build
visibility : {Build}
author : Microsoft Corporation
version : #{Major=0; Minor=1; Patch=71}
demands : {}
inputs : {#{name=CopyRoot; type=filePath; label=ms-resource:loc.input.label.CopyRoot; defaultValue=;
required=False; helpMarkDown=Root folder to apply copy patterns to. Empty is the root of the
repo.}, #{name=Contents; type=multiLine; label=ms-resource:loc.input.label.Contents;
defaultValue=; required=True; helpMarkDown=File or folder paths to include as part of the
artifact.}, #{name=ArtifactName; type=string; label=ms-resource:loc.input.label.ArtifactName;
defaultValue=; required=True; helpMarkDown=The name of the artifact to create.},
#{name=ArtifactType; type=pickList; label=ms-resource:loc.input.label.ArtifactType;
defaultValue=; required=True; helpMarkDown=The name of the artifact to create.; options=}…}
instanceNameFormat : Publish Artifact: $(ArtifactName)
execution : #{PowerShell=; Node=}
$manifest.name
PublishBuildArtifacts
$manifest.name = "sxs"
InvalidOperation: The property 'name' cannot be found on this object. Verify that the property exists and can be set.
When I strip the comments, I can overwrite the property.
Is there a way I can coax PowerShell to ignore the comments while loading the json file/convert the object and generate a writable object?
I'm not sure if this is intended, but seems like ConvertFrom-Json is treating the comments on the Json as $null when converting it to an object. This only happens if it's receiving an object[] from pipeline, with a string or multi-line string it works fine.
A simple way to demonstrate this using the exact same Json posted in the question:
$contentAsArray = Get-Content test.json | Where-Object {
-not $_.StartsWith('/')
} | ConvertFrom-Json -AsHashtable
$contentAsArray['name'] = 'hello' # works
Here you can see the differences and the workaround, it is definitely recommended to use -Raw on Get-Content so you're passing a multi-line string to ConvertFrom-Json:
$contentAsString = Get-Content test.json -Raw | ConvertFrom-Json -AsHashtable
$contentAsArray = Get-Content test.json | ConvertFrom-Json -AsHashtable
$contentAsString.PSObject, $contentAsArray.PSObject | Select-Object TypeNames, BaseObject
TypeNames BaseObject
--------- ----------
{System.Collections.Hashtable, System.Object} {name}
{System.Object[], System.Array, System.Object} {$null, System.Collections.Hashtable}
$contentAsArray['name'] # null
$null -eq $contentAsArray[0] # True
$contentAsArray[1]['name'] # PublishBuildArtifacts
$contentAsArray[1]['name'] = 'hello'
$contentAsArray[1]['name'] # hello
Santiago Squarzon's helpful answer shows an effective solution and analyzes the symptom. Let me complement it with some background information.
tl;dr
You're seeing a variation of a known bug, still present as of PowerShell 7.2.1, where a blank line or a single-line comment as the first input object unexpectedly causes $null to be emitted (first) - see GitHub issue #12229.
Using -Raw with Get-Content isn't just a workaround, it is the right - and faster - thing to do when piping a file containing JSON to be parsed as whole to ConvertFrom-Json
For multiple input objects (strings), ConverFrom-Json has a(n unfortunate) heuristic built in that tries to infer whether the multiple strings represent either (a) the lines of a single JSON document or (b) separate, individual JSON documents, each on its own line, as follows:
If the first input string is valid JSON by itself, (b) is assumed, and an object representing the parsed JSON ([pscustomobject] or , with -AsHashtable, [hastable], or an array of either) is output for each input string.
Otherwise, (a) is assumed and all input strings are collected first, in a multi-line string, which is then parsed.
The aforementioned bug is that if the first string is an empty/blank line or a single-line comment[1] (applies to both // ... comments, which are invariably single-line, and /* ... */ if they happen to be single-line), (b) is (justifiably) assumed, but an extraneous $null is emitted before the remaining (non-blank, non-comment) lines are parsed and their object representation(s) are output.
As a result an array is invariably returned, whose first element is $null - which isn't obvious, but results in subtle changes in behavior, as you've experienced:
Notably, attempting to set a property on what is presumed to be a single object then fails, because the fact that an array is being accessed makes the property access an instance of member-access enumeration - implicitly applying property access to all elements of a collection rather than the collection itself - which only works for getting property values - and fails obscurely when setting is attempted - see this answer for details.
A simplified example:
# Sample output hashtable parsed from JSON
$manifest = #{ name = 'foo' }
# This is what you (justifiably) THOUGHT you were doing.
$manifest.name = 'bar' # OK
# Due to the bug, this is what you actually attempted.
($null, $manifest).name = 'bar' # !! FAILS - member-access enumeration doesn't support setting.
As noted above, the resulting error message - The property 'name' cannot be found on this object. ... - is unhelpful, as it doesn't indicate the true cause of the problem.
Improving it would be tricky, however, as the user's intent is inherently ambiguous: the property name may EITHER be a situationally unsuccessful attempt to reference a nonexistent property of the collection itself OR, as in this case, a fundamentally unsupported attempt at setting properties via member-access enumeration.
Conceivably, the following would help if the target object is a collection (enumerable) from PowerShell's perspective: The property 'name' cannot be found on this collection, and setting a collection's elements' properties by member-access enumeration isn't supported.
[1] Note that comments in JSON are supported in PowerShell (Core) v6+ only.
Related
I'm trying to setup an IIS application pool via PowerShell 7.1.1.
I read configuration from a JSON file into the variable $configuration which is hand over to Windows Powershell because of WebAdministration module which isn't natively supported PS 7.1.1.
A script block is defined in the top level function, the configuration is injected as PSCustomObject into the script block and executed in Windows PowerShell.
function Set-AxisAppPool
{
Write-Message 'Setting up a resource pool for Axis...'
$executeInWindowsPowerShellForCompatibilityReasons = {
param (
[Parameter(Mandatory)]
[ValidateNotNullOrEmpty()]
[PSCustomObject]
$Configuration
)
Import-Module WebAdministration
Remove-WebAppPool -Name $Configuration.AppPool.Name -Confirm:$false -ErrorAction:SilentlyContinue
New-WebAppPool -Name $Configuration.AppPool.Name -Force | Write-Verbose
$params = #{
Path = "IIS:\AppPools\$($Configuration.AppPool.Name)"
Name = 'processModel'
Value = #{
userName = $Configuration.AxisUser.Name
password = $Configuration.AxisUser.Password
identitytype = 'SpecificUser'
}
}
Set-ItemProperty #params
}
powershell -NoLogo -NoProfile $executeInWindowsPowerShellForCompatibilityReasons -Args $configuration # This is a line 546
}
When the configuration JSON file exceeds a certain level, PowerShell can't pass through this deserialized JSON, the PSCustomObject, into Windows PowerShell.
Program 'powershell.exe' failed to run: The Process object must have the UseShellExecute property set to false in order to use environment
| variables.At C:\Users\JohnDoe\Desktop\Localhost automatization\Set-AxisEnvironment.ps1:546 char:5 + powershell -NoLogo -NoProfile
| $executeInWindowsPowerShellForCompa … +
It literally work with level n of objects in the JSON and it doesn't with n+1 level of objects in the configuration JSON. The JSON schema is validated, deserialization works as expected.
When I use Start-Process for invoking Windows PowerShell, I receive a different problem. Does anybody have any hint on this one?
Update
This seems to be a bug in PowerShell.
I suspect it is the size of the argument list overflowing into other fields, thus giving you weird error messages. From Start Process:
The length of the string assigned to the Arguments property must
be less than 32,699.
If you are passing a configuration that is larger than 32,699 characters (including spaces), then that likely may be your problem. It would likely take those first 32,699 characters then continue to the next field, -UseShellExecute which would receive a character which is not zero or false, and thus true. This would trip the "wrong", and misleading error message.
Prolog
It turns out that in my case it is important to understand the source of the objects - it is a JSON payload from a REST API response. Unfortunately, JSON -> Object conversion produces different results on PS desktop vs PS core. On desktop the numbers are deserialized into Int32 types, but on core - to Int64 types. From that it follows that I cannot use Export-CliXml, because the binary layout of the objects is different.
Main question
I have a unit test that needs to compare the actual result with an expected. The expected result is saved in a json file, so the procedure is:
Convert the actual result to json string
Read the expected result from disk to string
Compare the actual and the expected as strings
Unfortunately, this scheme does not work because PS desktop ConvertTo-Json and PS core ConvertTo-Json do not produce identical results. So, if the expected result was saved on desktop and the test runs on core - boom, failure. And vice versa.
One way is to keep two versions of jsons. Another way is to use a library to create the json.
First I tried the Newtonsoft-Json powershell module, but it just does not work. I think the problem is that whatever C# library we use, it must be aware of PSCustomObject and alike and treat them specially. So, we cannot just take any C# JSON library.
At this point I am left with having two jsons - one per PS edition, which is kind of sad.
Are there better options?
EDIT 1
I guess I can always read the json, convert to object and then back to json again. That sucks.
EDIT 2
I tried to use ConvertTo-Json -Compress. This eliminates the difference in spacing, but the problem is that for some reason the desktop version translates all the non characters to \u000... representation. The core version does not do it.
Please, observe:
Desktop
C:\> #{ x = "'a'" } |ConvertTo-Json -Compress
{"x":"\u0027a\u0027"}
C:\>
Core
C:\> #{ x = "'a'" } |ConvertTo-Json -Compress
{"x":"'a'"}
C:\>
Now the core version has the flag -EscapeHandling, so:
C:\> #{ x = "'a'" } |ConvertTo-Json -Compress -EscapeHandling EscapeHtml
{"x":"\u0027a\u0027"}
C:\>
Bingo! Same result. But now this code does not run on the desktop version, which does not have this flag. More massaging is needed. I will check if that is the only problem.
EDIT 3
It is impossible to reconcile the differences between the core and the desktop versions without expensive post processing. Please, observe:
Desktop
C:\> #{ x = '"a"';y = "'b'" } |ConvertTo-Json -Compress
{"y":"\u0027b\u0027","x":"\"a\""}
C:\>
Core
C:\> #{ x = '"a"';y = "'b'" } |ConvertTo-Json -Compress -EscapeHandling EscapeHtml
{"y":"\u0027b\u0027","x":"\u0022a\u0022"}
C:\> #{ x = '"a"';y = "'b'" } |ConvertTo-Json -Compress
{"y":"'b'","x":"\"a\""}
C:\>
Any suggestions on how to salvage the json approach?
EDIT 4
The Export-CliXml approach does not work too, because of the differences between the PS versions.
Desktop
C:\> ('{a:1}' | ConvertFrom-Json).a.gettype()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Int32 System.ValueType
C:\>
Core
C:\> ('{a:1}' | ConvertFrom-Json).a.gettype()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Int64 System.ValueType
C:\>
So the same JSON is represented using different numeric types - Int32 in desktop and Int64 in core. That puts to bed the option of using Export-CliXml.
Unless I am missing something.
I believe there is no other choice, but do the double conversion - json -> object -> json and then I will have two jsons created on the same PS edition. That sucks big time.
On converting from the original JSON, use the third-party Newtonsoft.Json PowerShell wrapper's ConvertFrom-JsonNewtonsoft cmdlet - this should ensure cross-edition compatibility (which the built-in ConvertFrom-Json does not guarantee across PowerShell editions, because Windows PowerShell uses a custom parser, whereas PowerShell [Core] v6+ uses Newtonsoft.json up to at least v7.1, though a move to the new(ish) .NET Core System.Text.Json API is coming).
Important: ConvertFrom-JsonNewtonsoft returns (arrays of) nested ordered hashtables ([ordered] #{ ... }, System.Collections.Specialized.OrderedDictionary), unlike the nested [pscustomobject] graphs that the built-in ConvertFrom-Json outputs. Similarly, ConvertTo-JsonNewtonsoft expects only (arrays of) hashtables (dictionaries) as input, and notably does not support [pscustomobject] instances, as you've learned yourself.
Caveat: As of this writing, the wrapper module was last updated in May 2019, and the version of the underlying bundled Newtonsoft.Json.dll assembly is quite old (8.0, whereas 12.0 is current as of this writing). See the module's source code.
Note that in order to parse JSON obtained from a RESTful web service manually, you mustn't use Invoke-RestMethod, as it implicitly parses and returns [pscustomobject] object graphs. Instead, use Invoke-WebRequest and access the returned response's .Content property.
On converting to a format suitable for storing on disk, you have two options:
(A) If you do need the serialized format to be JSON also, you must convert all [pscustomobject] graphs to (ordered) hashtables before passing them to ConvertTo-JsonNewtonsoft.
See below for function ConvertTo-OrderedHashTable, which does just that.
(B) If the specific serialization format isn't important, i.e. if all that matters is that the formats are identical across PowerShell editions for the purpose of comparison, no extra works is needed: use the built-in Export-Clixml cmdlet, which can handle any type and produces PowerShell's native, XML-based serialization format called CLIXML (as notably used in PowerShell remoting), which should be cross-edition-compatible (at least with v5.1 on the Windows PowerShell side and as of PowerShell [Core] v7.1, both of which use the same version of the serialization protocol, 1.1.0.1, as reported by $PSVersionTable.SerializationVersion).
While you could re-convert such a persisted file to objects with Import-Clixml, the potential loss of type fidelity on deserialization makes comparing the serialized (CLIXML) representations advisable.
Also note that as of PowerShell v7.1 there is no cmdlet-based way to create an in-memory CLIXML representation, so you'll have to use the PowerShell API directly for now: System.Management.Automation.PSSerializer.Serialize. However, providing in-memory counterparts to Import-CliXml / Export-CliXml in the form of ConvertFrom-CliXml / ConvertTo-CliXml cmdlets has been green-lighted as a future enhancement.
Re (A): Here's function ConvertTo-OrderedHashtable, which converts (potentially nested) [pscustomobject] objects to ordered hashtables while passing other types through, so you should be able to simply insert it into a pipeline as follows:
# CAVEAT: ConvertTo-JsonNewtonSoft only accepts a *single* input object.
[pscustomobject] #{ foo = 1 }, [pscustomobject] #{ foo = 2 } |
ConvertTo-OrderedHashtable |
ForEach-Object { ConvertTo-JsonNewtonSoft $_ }
function ConvertTo-OrderedHashtable {
<#
.SYNOPSIS
Converts custom objects to ordered hashtables.
.DESCRIPTION
Converts PowerShell custom objects (instances of [pscustomobject]) to
ordered hashtables (instances of [System.Collections.Specialized.OrderedDictionary]),
which is useful for to-JSON serialization via the Newtonsoft.JSON library.
Note:
* Custom objects are processed recursively.
* Any scalar non-custom objects are passed through as-is.
* Any (non-dictionary) collections in property values are converted to
[object[]] arrays.
.EXAMPLE
1, [pscustomobject] #{ foo = [pscustomobject] #{ bar = 'none' }; other = 2 } | ConvertTo-OrderedHashtable
Passes integer 1 through, and converts the custom object to a nested ordered
hashtable.
#>
[CmdletBinding()]
param(
[Parameter(ValueFromPipeline)] $InputObject
)
begin {
# Recursive helper function
function convert($obj) {
if ($obj -is [System.Management.Automation.PSCustomObject]) {
# a custom object: recurse on its properties
$oht = [ordered] #{ }
foreach ($prop in $obj.psobject.Properties) {
$oht.Add($prop.Name, (convert $prop.Value))
}
return $oht
}
elseif ($obj -isnot [string] -and $obj -is [System.Collections.IEnumerable] -and $obj -isnot [System.Collections.IDictionary]) {
# A collection of sorts (other than a string or dictionary (hash table)), recurse on its elements.
return #(foreach ($el in $obj) { convert $el })
}
else {
# a non-custom object, including .NET primitives and strings: use as-is.
return $obj
}
}
}
process {
convert $InputObject
}
}
Re (B): A demonstration of the Export-CliXml approach (you can run this code from either PS edition):
$sb = {
Install-Module -Scope CurrentUser Newtonsoft.json
if (-not $IsCoreClr) {
# Workaround for PS Core's $env:PSModulePath overriding WinPS'
Import-Module $HOME\Documents\WindowsPowerShell\Modules\newtonsoft.json
}
#'
{
"results": {
"users": [
{
"userId": 1,
"emailAddress": "jane.doe#example.com",
"date": "2020-10-05T08:08:43.743741-04:00",
"attributes": {
"height": 165,
"weight": 60
}
},
{
"userId": 2,
"emailAddress": "john.doe#example.com",
"date": "2020-10-06T08:08:43.743741-04:00",
"attributes": {
"height": 180,
"weight": 72
}
}
]
}
}
'# | ConvertFrom-JsonNewtonsoft | Export-CliXml "temp-$($PSVersionTable.PSEdition).xml"
}
# Execute the script block in both editions
Write-Verbose -vb 'Running in Windows PowerShell...'
powershell -noprofile $sb
Write-Verbose -vb 'Running in PowerShell Core...'
pwsh -noprofile $sb
# Compare the resulting CLIXML files.
Write-Verbose -vb "Comparing the resulting files: This should produce NO output,`n indicating that the files have identical content."
Compare-Object (Get-Content 'temp-Core.xml') (Get-Content 'temp-Desktop.xml')
Write-Verbose -vb 'Cleaning up...'
Remove-Item 'temp-Core.xml', 'temp-Desktop.xml'
You should see the following verbose output:
VERBOSE: Running in Windows PowerShell...
VERBOSE: Running in PowerShell Core...
VERBOSE: Comparing the resulting files: This should produce NO output,
indicating that the files have identical content.
VERBOSE: Cleaning up...
The command gci env:ApiSecret | ConvertTo-Json works to return a long string, the API secret for Twitter, which is truncated without the pipe to JSON.
However, the JSON is rather spammy.
Is there a "goldilocks" way to get the lengthy string value without the extraneous details?
(Unfortunately, gci env: truncates the key)
Get-ChildItem is for retrieving all or a subset of items from a container. Note that it outputs an object with Name and Value properties (substituting Path as another lengthy environment variable value)...
PS> gci env:Path
Name Value
---- -----
Path C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\WINDO...
Get-Item yields the same result...
PS> gi env:Path
Name Value
---- -----
Path C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\WINDO...
Either way, the object retrieved is a DictionaryEntry...
PS> gi env:Path | Get-Member
TypeName: System.Collections.DictionaryEntry
Name MemberType Definition
---- ---------- ----------
Name AliasProperty Name = Key
Equals Method bool Equals(System.Object obj)
GetHashCode Method int GetHashCode()
GetType Method type GetType()
ToString Method string ToString()
PSDrive NoteProperty PSDriveInfo PSDrive=Env
PSIsContainer NoteProperty bool PSIsContainer=False
PSPath NoteProperty string PSPath=Microsoft.PowerShell.Core\Environment::path
PSProvider NoteProperty ProviderInfo PSProvider=Microsoft.PowerShell.Core\Environment
Key Property System.Object Key {get;set;}
Value Property System.Object Value {get;set;}
...and when you pipe that to ConvertTo-Json it will include all kinds of undesirable properties from that class.
In short, don't use ConvertTo-Json for this. Since you know the exact item you want, just retrieve it directly using variable syntax...
PS> $env:Path
C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;...
Equivalent code using the .NET API would be...
PS> [Environment]::GetEnvironmentVariable('Path')
C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;...
If you really wanted to use a Get-*Item cmdlet you'd just need to specify that it's the Value property you want using property syntax...
PS> (gi env:Path).Value
C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;...
...or Select-Object...
PS> gi env:Path | Select-Object -ExpandProperty 'Value'
C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;...
All of the above commands will output only a [String] containing the entirety of that environment variable value. I inserted trailing ellipses since showing my entire Path value is not useful here; in practice, those commands will output the entire environment variable with no truncation.
The simplest way to inspect the value of environment variables in full is to use the $env:<varName> (namespace variable notation) syntax, which in your case means: $env:ApiSecret (if the variable name contains special characters, enclose everything after the $ in {...}; e.g., ${env:ApiSecret(1)})
That way, environment-variable values (which are invariably strings) that are longer than your terminal's (console's) width simply continue on subsequent lines.
To demonstrate:
# Simulate a long value (200 chars.)
$env:ApiSecret = 'x' * 199 + '!'
# Output the long value
$env:ApiSecret
With an 80-char. wide terminal, you'd see output as follows:
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx!
If you do want to use Get-Item (or Get-ChildItem, which acts the same in this case), you have two options:
# Format-List shows each property on its own line,
# with values wrapping across multiple lines
Get-Item env:ApiSecret | Format-List
# Format-Table -Wrap causes values to wrap as well.
Get-Item env:ApiSecret | Format-Table -Wrap
Your statement does not strip anything away. However, for console display purpose, it truncate the output that you view in the console.
If you assign the result to a variable or pipe to a file, nothing will be truncated.
Therefore, my assumption on your question is that you want to view the result in the console without the console truncating your stuff there.
For that, you could write the results to the host yourself.
Here's a simple example that do just that.
$envvars = gci env:
$Max = ($envvars.name| Measure-Object -Property length -Maximum).Maximum + 3
$envvars | % {Write-Host $_.name.padright($Max,' ') -ForegroundColor Cyan -NoNewline;Write-Host $_.value}
Result — As you can see, the path variable value is no longer truncated.
I'm quite new to powershell and just need it for a small task so please excuse my complete and utter ineptitude for the language. I was wondering if it were possible to form a json object based off environment variables and a variable that has already been declared earlier in my script. The variable that was already declared is based off a json config named optionsConfig.json and the contents of that file are here.
{"test1": ["options_size", "options_connection", "options_object"],
"test2":["options_customArgs", "options_noUDP", "options_noName"]}
The purpose of the $Options variable in the code below is to take each element in the list value for the respective test and assume that those elements are environment variables in the system, then find their values and form a dictionary object that will be used in the json.
Here is what I have so far.
# Read the JSON file into a custom object.
$configObj = Get-Content -Raw optionsConfig.json |
ConvertFrom-Json
# Retrieve the environment variables whose
# names are listed in the $env:test property
# as name-value pairs.
Get-Item -Path env:* -Include $configObj.$env:testTool
$Options = Get-Item -Path env:* -Include $configObj.$env:testTool |
% {$hash = #{}} {$hash[$_.Name]=$_.Value} {$hash}
The $Options variable looks like so when converted to json
{
"options_size": "default",
"options_object": "forward open",
"options_connection": "connected"
}
I have a few other environment variable values that I would like to be a part of the json object. Those 3 other environment variables I would like the value of are listed below.
$Env.testTool = "test1"
$Env.RecordName = "Record1"
$Env.Target = "Target1"
How would I construct a powershell statement to get the json object to be formatted like this? -
data = {"test": $Env.testTool, "target": "$Env.Target",
"options": "$Options", "RecordName': "$Env.RecordName"}
The keys are all predefined strings and $Options is the dict object from up above. Am I able to form a Json object like this in powershell and how would it be done? Any help would be appreciated. This appears to be the last step in my struggle with powershell.
Here is what I have done.
$jObj = [ordered]#{test= $Env:testTool}
When I change this variable to $jObj = [ordered]#{test= $Env:testTool,options= $Options} I get an error saying missing expression after ','
When I change this variable to $jObj = [ordered]#{test= $Env:testTool,options= $Options} I get an error saying missing expression after ','
Entries of a hashtable literal (#{ ... } or, in its ordered form, [ordered] #{ ... }) must be separated:
either by newlines (each entry on its own line)
or by ; if placed on the same line.
Thus, the following literals are equivalent:
# Multiline form
#{
test= $env:testTool
RecordName= $env:RecordName
Target= $env.Target
options=$Options
}
# Single-line form; separator is ";"
#{ test= $env:testTool; RecordName= $env:RecordName; Target= $env.Target; options=$Options }
Get-Help about_Hashtables has more information.
$jObj = #{test= $env:testTool
RecordName= $env:RecordName
Target= $env.Target
options=$Options}
$jObj | ConvertTo-Json | Set-Content jsonStuff.json
JsonStuff.json is the new json file for the new json object. This syntax for forming $jObj seems to have done the trick.
I'm trying to dynamically parse & build-up a data structure of some incoming JSON files I'm to be supplied with (that'll be in non-standard structure) via Powershell to then process the data in those files & hand them over to the next step.
As part of that, I'm trying to build up the data structure of the JSON file into essentially a list of of data-paths for me to parse through & grab the data out of, so that I can cope with arrays, nested JSON objects and so on. So far so good.
Where I fall into some sort of Powershell peculiarity is in handling 2+ levels of depth via a variable. Let me give you a nice code-block to demonstrate the problem...
# Generate a Quick JSON file with different data types & levels
[object]$QuickJson = #'
{
"Name" : "I am a JSON",
"Version" : "1.2.3.4",
"SomeBool" : true,
"NULLValue" : null,
"ArrayOfVersions" : [1.0,2.0,3.0],
"MyInteger" : 69,
"NestedJSON" : {
"Version" : 5.0,
"IsReady" : false
},
"DoubleNestedJSON" : {
"FirstLevel" : 1,
"DataValue" : "I am at first nested JSON level!",
"Second_JSON_Level" : {
"SecondLevel" : 2,
"SecondDataValue" : "I am on the 2nd nested level"
}
}
}
'#
# Import our JSON file into Powershell
[object]$MyPSJson = ConvertFrom-Json -InputObject $QuickJson
# Two quick string variables to access our JSON data paths
[string]$ShortJsonPath = "Name"
[string]$NestedJsonPath = "NestedJson.Version"
# Long string to access a double-nested JSON object
[string]$LongNestedJsonPath = "DoubleNestedJSON.Second_JSON_Level.SecondDataValue"
# Both of these work fine
Write-Host ("JSON Name (Direct) ==> " + $MyPSJson.Name)
Write-Host ("JSON Name (via Variable) ==> " + $MyPSJson.$ShortJsonPath)
# The following way to access a single nested Json Path works fine
Write-Host ("Nested JSON Version (via direct path) ==> " + $MyPSJson.NestedJson.Version)
# And THIS returns an empty line / is where I fall afoul of something in Powershell
Write-Host ("Nested JSON Version (via variable) ==> " + $MyPSJson.$NestedJsonPath)
# Other things I tried -- all returning an empty line / failing in effect
Write-Host ("Alternate Nested JSON Version ==> " + $($MyPSJson.$NestedJsonPath))
Write-Host ("Alternate Nested JSON Version ==> " + $MyPSJson.$($NestedJsonPath))
Write-Host ("Alternate Nested JSON Version ==> " + $($MyPSJson).$($NestedJsonPath))
# Similarly, while THIS works...
$MyPSJson | select-object -Property NestedJSON
# This will fail / return me nothing
$MyPSJson | select-object -Property NestedJSON.Version
... in doing a bunch of research around this, I came across a suggestion to transform this into a Hashtable -- but that has the same problem, sadly. So with the above code-snippet, the following will transform the JSON object into a hashtable.
# Same problem with a hash-table if constructed from the JSON file...
[hashtable]$MyHash = #{}
# Populate $MyHash with the data from our quickie JSON file...
$QuickJson | get-member -MemberType NoteProperty | Where-Object{ -not [string]::IsNullOrEmpty($QuickJson."$($_.name)")} | ForEach-Object {$MyHash.add($_.name, $QuickJson."$($_.name)")}
# ... and even then -- $MyHash."$($NestedJsonPath)" -- fails, while a single level deep string works fine in the variable! :(
So it's pretty clear that I'm running into "something" of a Powershell internal logic problem, but I can't get Powershell to be overly helpful in WHY that is. Adding a '-debug' or similar in an attempt to increase verbosity hasn't helped shed light on this.
I suspect it's something akin to the items raised in this article here ( https://blogs.technet.microsoft.com/heyscriptingguy/2011/10/16/dealing-with-powershell-hash-table-quirks/ ) but just specific with variables.
I've not had any luck in finding anything obvious in the Powershell language specification (3.0 still being the latest from here as far as I can tell -- https://www.microsoft.com/en-usdownload/details.aspx?id=36389 ) either. It may be in there, I may just miss it.
Any advice in how to get Powershell to play nice with this would be greatly appreciated. I'm not sure how / why Powershell is fine with a simple string but seems to have issues with a 'something.somethingelse' type string here.
Thank you.
Further notes & addenda to the original:
It seems there are several issues to attack. One is "dealing with a single nested level". The "quick fix" for that seems to be using "Invoke-Expression" to resolve the statement, so for instance (IMPORTANT - take note of the back-tick with the first variable!):
iex "`$MyPSJson.$NestedJsonPath"
That use of Invoke-Expression also works with multi-nested situations:
iex "`$MyPSJson.$LongNestedJsonPath"
An alternative approach that was mentioned is the use of multiple select statements ... but I've not been able to get that to work with multi-nested objects (Powershell seems to not resolve those properly for some reason).
So for instance in this scenario:
($MyComp | select $_.DoubleNestedJSON | select FirstLevel)
Powershell returns
FirstLevel
----------
... instead of the actual data value. So - for now, it seems that selects won't work with multi-level nested objects due to Powershell apparently not resolving them?
When you write something like
$MyPSJson.Name
this will attempt to retrieve the member named Name from the object $MyPSJson. If there is no such member, you'll get $null.
Now, when you do that with variables for the member name:
$MyPSJson.$ShortJsonPath
this works pretty much identical in that the member with the name stored in $ShortJsonPath is looked up and its value retrieved. No surprises here.
When you try that with a member that doesn't exist on the object, such as
$MyPSJson.$NestedJsonPath
# equivalent to
# $MyPSJson.'NestedJSON.Version'
you'll get $null, as detailed before. The . operator will only ever access a member of the exact object that is the result of its left-hand-side expression. It will never go through a member hierarchy in the way you seem to expect it to do. To be frank, I'm not aware of a language that works that way.
The reason it works with Invoke-Expression is, that you effectively converting the $NestedJsonPath string into part of an expression resulting in:
$MyPSJson.NestedJSON.Version
which Invoke-Expression then evaluates.
You can, of course, define your own function that works that way (and I'd much prefer that instead of using Invoke-Expression, a cmdlet that should rarely, if ever, used (heck, it's eval for PowerShell – few languages with eval advocate its use)):
function Get-DeepProperty([object] $InputObject, [string] $Property) {
$path = $Property -split '\.'
$obj = $InputObject
$path | %{ $obj = $obj.$_ }
$obj
}
PS> Get-DeepProperty $MyPSJson NestedJson.Version
5,0
You could even make it a filter, so you can use it more naturally on the pipeline:
filter Get-DeepProperty([string] $Property) {
$path = $Property -split '\.'
$obj = $_
$path | %{ $obj = $obj.$_ }
$obj
}
PS> $MyPSJson | Get-DeepProperty nestedjson.version
5,0
Why this doesn't work
When you provide the properties that you'd like within a string, like this
[string]$NestedJsonPath = "NestedJson.Version"
Powershell looks for a property called NestedJSon.Version. It's not actually traversing the properties, but looking for a string literal which contains a period. In fact, if I add a property like that to your JSON like so.
[object]$QuickJson = #'
{
"Name" : "I am a JSON",
"Version" : "1.2.3.4",
"SomeBool" : true,
"NULLValue" : null,
"ArrayOfVersions" : [1.0,2.0,3.0],
"MyInteger" : 69,
"NestedJSON.Version" : 69,
"NestedJSON" : {
"Version" : 5.0,
"IsReady" : false
}
}
I now get a value back, like so:
>$MyPSJson.$NestedJsonPath
69
The best way to get your values back is to use two separate variables, like this.
$NestedJson = "NestedJson"
$property = "Version"
>$MyPSJson.$NestedJson.$property
5.0
Or, alternatively, you could use select statements, as seen in the original answer below.
$MyPSJson | select $_.NestedJSON | select Version
Version
-------
1.2.3.4
If you use multiple Select-Object statements, they'll discard the other properties and allow you to more easily drill down to the value you'd like.
I followed Joey's filter example. However, I found it did not support accessing arrays.
Sharing the code that I got to work for this. Hopefully it will help others as well. Awesome thread!
filter Get-DeepProperty([string] $Property) {
$path = $Property -split '\.'
$obj = $_
foreach($node in $path){
if($node -match '.*\[\d*\]'){
$keyPieces = $node -split ('\[')
$arrayKey = $keyPieces[0]
$arrayIndex = $keyPieces[1] -replace ('\]','')
$obj = $obj.$arrayKey[$arrayIndex]
} else {
$obj = $obj.$node
}
}
$obj
}
Example usage:
$path = "nested.nestedtwo.steps[2]"
$payload | Get-DeepProperty $path
I had the same problem, so I wrote a function that does the trick.
It enables accessing any level of the json by variable path (string):
function getNestedJsonValue() {
param(
[Parameter(Mandatory = $true, ValueFromPipeline)] [PSCustomObject] $inputObj,
[Parameter(Mandatory = $true)] [string] $valuePath
)
if (($valuePath -eq $null) -or ($valuePath.length -eq 0) -or ($inputObj -eq $null)) {
return $inputObj
}
[System.Array] $nodes = "$valuePath" -split '\.'
foreach ($node in $nodes) {
if (($node -ne $null) -and ($node.length -gt 0) -and ($inputObj -ne $null)) {
$inputObj = $inputObj.$node
} else {
return $inputObj
}
}
return $inputObj
}
Usage: getNestedJsonValue -valuePath $nestedValuePath -inputObj $someJson
Pipe usage: $someJson | getNestedJsonValue -valuePath $nestedValuePath
An example nestedValuePath would be $nestedValuePath="some.nested.path"
Credit to wOxxOm for getting things on the right track.
Invoke-Expression does seem to work perfectly for this situation (if somewhat expensive, but that's fine in my personal example & situation), and it can cope with multiple levels of nesting.
So as examples for the above code snippet, the following will resolve just fine (Key point - pay attention to the initial back-tick. That caught me off guard):
Write-Host ("Single level JSON test ==> " + (iex "`$MyPSJson.$NestedJsonPath"))
Write-Host ("Double level JSON test ==> " + (iex "`$MyPSJson.$LongNestedJsonPath"))
That'll return our desired results:
Single level JSON test ==> 5.0
Double level JSON test ==> I am on the 2nd nested level
FoxDeploy's answer of using multi-level selects doesn't seem to work with 2+ levels of nesting, unfortunately for some bizarre reason.
Using:
($MyPSJson | select $_.DoubleNestedJSON | select FirstLevel)
We get the following back from Powershell:
FirstLevel
----------
... it seems that Powershell doesn't resolve nested objects in its entirety? We get a similar results if we intentionally use something that doesn't exist:
($MyPSJson | select $_.DoubleNestedJSON | select Doesnotexist)
... also simply returns:
Doesnotexist
------------
So - for now - it seems as if "Invoke-Expression" works most reliably (and most easily, as it's just a case of handing it a variable with the path'ed string).
I still can't explain the WHY of any of this so far (since I've used 'dotwalk'-ing with multiple variables through arrays quite happily), but at least there's a solution for now ... and that is Invoke-Expression !
The best (/least bad?) explanations for Invoke-Expression I've found so far are here (Microsoft's own description of the cmdlet doesn't really make a great job of hinting that it'd help in situations such as this):
http://ss64.com/ps/invoke-expression.html
https://www.adminarsenal.com/powershell/invoke-expression/