How to traverse JSON properties with Powershell - json

I am trying to access a particular property value of a JSON object with Powershell. Unfortunately I do not know the keys of some of the parent properties within the structure, so I cannot do this in a straightforward way. Also as the JSON is not an array I am unable to access via index position.
The context is that I am querying a list of running tasks from elasticsearch and need to get the ID of the task (I know that there will only be one) so I can make subsequent calls to discover its completion status.
I've researched some querying methods but am unsure on how to apply them (PowerShell syntax is quite new to me).
The JSON response I am working with looks like this;
"nodes" : {
"oTUltX4IQMOUUVeiohTt8A" : {
"name" : "H5dfFeA",
"transport_address" : "127.0.0.1:9300",
"host" : "127.0.0.1",
"ip" : "127.0.0.1:9300",
"tasks" : {
"oTUltX4IQMOUUVeiohTt8A:124" : {
"node" : "oTUltX4IQMOUUVeiohTt8A",
"id" : 124,
"type" : "direct",
"action" : "cluster:monitor/tasks/lists[n]",
"start_time_in_millis" : 1458585884904,
"running_time_in_nanos" : 47402,
"cancellable" : false,
"parent_task_id" : "oTUltX4IQMOUUVeiohTt8A:123"
}
}
}
}
}
With this given structure, I would like to be able to access the ID property of the first 'task'.
So if I knew the prop keys it would be:
nodes.oTUltX4IQMOUUVeiohTt8A.tasks.oTUltX4IQMOUUVeiohTt8A:124.id
How can I access this value without knowing the keys beforehand?
Any help very appreciated.
Thanks
Nick

The following code defines and uses function Get-FirstPropertyValue, which performs a recursive, depth-first search for the first property inside an object graph that has a given name and returns its value, assuming the value is non-null:
# Function that returns the value of the first property with the given
# name found during recursive depth-first traversal of the given object.
# Note that null-valued properties are ignored.
function Get-FirstPropertyValue($obj, $propName) {
$propNames = $obj.psobject.properties.Name
if ($propName -in $propNames) {
$obj.$propName
} else {
foreach ($iterPropName in $propNames) {
if ($null -ne ($val = Get-FirstPropertyValue $obj.$iterPropName $propName)) {
return $val
}
}
}
}
# Input JSON
$json = #'
{
"nodes": {
"oTUltX4IQMOUUVeiohTt8A": {
"name": "H5dfFeA",
"transport_address": "127.0.0.1:9300",
"host": "127.0.0.1",
"ip": "127.0.0.1:9300",
"tasks": {
"oTUltX4IQMOUUVeiohTt8A:124": {
"node": "oTUltX4IQMOUUVeiohTt8A",
"id": 124,
"type": "direct",
"action": "cluster:monitor/tasks/lists[n]",
"start_time_in_millis": 1458585884904,
"running_time_in_nanos": 47402,
"cancellable": false,
"parent_task_id": "oTUltX4IQMOUUVeiohTt8A:123"
}
}
}
}
}
'#
# Convert the JSON to a [pscustomobject] graph with ConvertFrom-Json.
$objFromJson = $json | ConvertFrom-Json
# Using the function defined above, get the first 'tasks' object found
# during recursive depth-first traversal.
$tasks = Get-FirstPropertyValue $objFromJson 'tasks'
# Get the name of the resulting object's first property.
$propName = #($tasks.psobject.properties.Name)[0]
# Extract the .id property from the object stored in the first property.
$tasks.$propName.id
The above yields:
124
A more concise, but more obscure and presumably slower alternative is to convert the JSON input to XML and then use XPath to query it:
# Input JSON
$json = #'
{
"nodes": {
"oTUltX4IQMOUUVeiohTt8A": {
"name": "H5dfFeA",
"transport_address": "127.0.0.1:9300",
"host": "127.0.0.1",
"ip": "127.0.0.1:9300",
"tasks": {
"oTUltX4IQMOUUVeiohTt8A:124": {
"node": "oTUltX4IQMOUUVeiohTt8A",
"id": 124,
"type": "direct",
"action": "cluster:monitor/tasks/lists[n]",
"start_time_in_millis": 1458585884904,
"running_time_in_nanos": 47402,
"cancellable": false,
"parent_task_id": "oTUltX4IQMOUUVeiohTt8A:123"
}
}
}
}
}
'#
$parent = 'tasks'
$prop = 'id'
$propType = 'int'
$json |
ConvertFrom-Json |
ConvertTo-Xml -Depth ([int]::MaxValue) |
Select-Xml "//Property[#Name='$parent']/*/*[#Name='$prop']/text()" |
ForEach-Object { $_.Node.InnerText -as $propType }

There are two ways I know of that you can achieve this, both look a bit gnarly.
For these examples, I will load the JSON you have provided into $json.
$json = #'
{
"nodes": {
"oTUltX4IQMOUUVeiohTt8A": {
"name": "H5dfFeA",
"transport_address": "127.0.0.1:9300",
"host": "127.0.0.1",
"ip": "127.0.0.1:9300",
"tasks": {
"oTUltX4IQMOUUVeiohTt8A:124": {
"node": "oTUltX4IQMOUUVeiohTt8A",
"id": 124,
"type": "direct",
"action": "cluster:monitor/tasks/lists[n]",
"start_time_in_millis": 1458585884904,
"running_time_in_nanos": 47402,
"cancellable": false,
"parent_task_id": "oTUltX4IQMOUUVeiohTt8A:123"
}
}
}
}
}
'# | ConvertFrom-Json
The first is to use Select-Object to select the first item and then expand the properties.
(($json.nodes | Select-Object -First 1 -ExpandProperty *).tasks | Select-Object -First 1 -ExpandProperty *).id
A more robust method, is to use the hidden PSObject property Value, as the JSON is parsed by PowerShell into a PSCustomObject.
PS C:\Windows\system32> $json.nodes.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True False PSCustomObject System.Object
The Properties of .PSObject
PS C:\Windows\system32> $json.nodes.PSObject.Properties
MemberType : NoteProperty
IsSettable : True
IsGettable : True
Value : #{name=H5dfFeA; transport_address=127.0.0.1:9300; host=127.0.0.1; ip=127.0.0.1:9300; tasks=}
TypeNameOfValue : Selected.System.Management.Automation.PSCustomObject
Name : oTUltX4IQMOUUVeiohTt8A
IsInstance : True
The full command to access the ID value:
$json.nodes.PSObject.Properties.Value.tasks.PSObject.Properties.Value.id

Related

JSON file - how to delete entire node that contains a string

I have a JSON file:
{
"JSONS" : [
{
"id" : "ToRemove",
"First" : [
{
"id" : "geo",
"Name" : "Person1",
"model" : [
],
"adjustments" : [
{
"uid" : "3",
"name" : "4s",
"value" : "1"
},
{
"uid" : "5",
"name" : "3s",
"value" : "6"
}
]
},
{
"id" : "Meters",
"Dictionary" : "4.2"
},
{
"id" : "Moon",
"Filter" : "0.5",
"Saturn" : {
"s" : "0",
"v" : "1"
}
}
]
}
]
}
I would like to delete entire node, if the "id", in this example, contains "ToRemove" string. Everyting between { and }, including those lines also, to make the final JSON consistent.
This is a screenshot what I want to get rid of.
I only found how to delete properties, but not entire nodes. I've tried to appli something like this:
$ToRemove = Get-Content $SourceFile | ConvertFrom-Json
$ToRemove.PSObject.Object.Remove('id:','ToRemove')
$ToRemove | ConvertTo-Json -Depth 100 | Out-File $DestFile
but of course it didn't work.
How to delete the entire node? I would love to use an array to put all strings I would like to delete.
Based on your comment, you can remove that object having the property id = ToRemove by filtering where id is not equal to ToRemove and assigning that result to the .JSONS property:
$json = Get-Content path\to\json.json -Raw | ConvertFrom-Json
$json.JSONS = #($json.JSONS.Where{ $_.id -ne 'ToRemove' })
$json | ConvertTo-Json
The end result in this case would be an empty array for the .JSONS property:
{
"JSONS": []
}
.PSObject.Properties.Remove(...) wouldn't be useful in this case because what it does is remove properties from one object but what you want to do is filter out an entire object based on a condition.
You should be able to use just plain PowerShell, like this:
{
"JSONS" : [
{
"id" : "ToRemove",
"First" : [
{
"id" : "geo",
"Name" : "Person1",
"model" : [
]
},
{
"id" : "Meters",
"Dictionary" : "4.2"
}
]
},
{
"id" : "DontRemove",
"First" : []
}
]
}
$json = Get-Content -Path $SourceFile | ConvertFrom-Json
$json.JSONS = $json.JSONS | Where-Object { $_.Id -ne "ToRemove" }
$json | ConvertTo-Json -Depth 100 | Out-File -Path $DestFile

(PowerShell) How to remove double quotes from the JSON output?

Here's an excerpt from my PowerShell script, that writes JSON data to a file:
$jsonData = #"
{
"FunctionName" :
{
"description": "",
"parameters": [],
"signature" : ""
}
}
"#
$jsonParameterData = #"
{
`t`t`t`t`t`t`t`t`t`t"description": "",
`t`t`t`t`t`t`t`t`t`t"name": ""
`t`t`t`t`t`t`t`t`t }
"#
$jsonData = $jsonData | ConvertFrom-Json
In the code above I have defined the JSON template that I will use to write to the file. I'm using jsonData to describe the function FunctionName with the keys of description, parameters, and signature.
On the other hand, I'm using jsonParameterData to describe each argument of the function FunctionName with the keys of description and name.
My PowerShell script pulls the data from other files, and writes this JSON information to a .json file.
For example, if the signature of the function I want to extract data from is Function FunctionName(Parameter1, Parameter2), in the .json file I would expect to see this:
"FunctionName": {
"description": "This function does this and that",
"parameters": [
{
"description": "This is the first parameter",
"name": "Parameter1"
},
{
"description": "This is the second parameter",
"name": "Parameter2"
}
],
"signature": "Function FunctionName(Parameter1, Parameter2)"
}
Somewhere in the code, I store the number of function arguments in the variable NumberOfArguments; the arguments of the function in ArrayOfArguments, and the argument descriptions in ArrayOfArgumentDescriptions.
The PowerShell code below is used to add jsonParameterData to jsonData:
for($i=0; $i -lt $NumberOfArguments; $i++){
$SelectedArgument = $ArrayOfArguments[$i]
$SelectedDescription = $ArrayOfArgumentDescriptions[$i]
$jsonData.AplusWait.parameters += $jsonParameterData
}
Then, I write jsonData to the file like this:
$jsonData = $jsonData | ConvertTo-Json | ForEach-Object { [System.Text.RegularExpressions.Regex]::Unescape($_) }
$jsonData.Substring(1, $jsonData.length - 2) | Add-Content -Path $OutputFile
Write-Output "`nFinished writing JSON data to file.`n"
However, the resulting output slightly does not match the intended output. The output looks like this:
"FunctionName": {
"description": "This function does this and that",
"parameters": [
"{
"description": "This is the first parameter",
"name": "Parameter1"
}",
"{
"description": "This is the second parameter",
"name": "Parameter2"
}"
],
"signature": "Function FunctionName(Parameter1, Parameter2)"
}
If you look closely, the output displays the parameters array as an array of strings (note the double quotes), when it should actually be an array of JSON objects.
How can I transform this:
"{
"description": "This is the first parameter",
"name": "Parameter1"
}"
into this:
{
"description": "This is the first parameter",
"name": "Parameter1"
}
?
Having the following JSONs:
$jsonData = #"
{
"foo": {
"des": "foo-des",
"par": [],
"sig": "foo-sig"
}
}
"#
$jsonParamData = #"
{
"des": "param-des",
"name": "param-name"
}
"#
$jsonObj = $jsonData | ConvertFrom-Json
$jsonParamObj = $jsonParamData | ConvertFrom-Json
# add param
$jsonObj.foo.par += $jsonParamObj
# Pay attention to the -Depth parameter otherwise the final json will be truncated
$jsonResult = $jsonObj | ConvertTo-Json -Depth 5
Write-Host $jsonResult
Should print:
{
"foo": {
"des": "foo-des",
"par": [
{
"des": "param-des",
"name": "param-name"
}
],
"sig": "foo-sig"
}
}
When converting objects back to json you should pay attention to the -Depth parameter since it will truncate your result.

How to get perl json access hash reference

I am trying to get remote host value which is remotehost . I decoded the json and able to get $name and $id values. but getting
Bad index while coercing array into hash while accessing 'remote -> host value which is remotehost.
my $json input is
[
{
"auth":{
"req": "1234",
"link": "http://localhost"
},
"host": "localhost",
"name": "mytest",
"remote": [
{
"host": "remotehost",
"name": "remotetest"
}
]
}
]
#My code
use JSON qw( decode_json );
use Data::Dumper;
my $list = decode_json($json) ;
my #array = #{$list};
foreach(#array)
{
my %obj = %{$_};
my $id = $obj{'auth'}{'link'};
my $name = $obj{'name'};
my $remotehost = $obj{'auth'}->{'remote'}{'host'}; #getting error
}
"remote": [
{
"host": "remotehost",
"name": "remotetest"
}
]
This bit is an array of objects (hashes in Perl). You cannot coerce it. You need to access the 1st element.
# A
# | B
# | | C
# | | |
$obj{'remote'}->[0]->{'host'}
Note there's no auth in this, because that's on the same level as remote (marked as A).
{
"auth" : {
"req": "1234",
"link": "http://localhost"
},
"host" : "localhost",
"name" : "mytest",
// A
"remote" : [ // B
{ // C
"host": "remotehost",
"name": "remotetest"
}
]
}

To delete loop inside a variable using powershell

I did invoke-restmethod and stored the output in variable $a and used convertTo-json and i want to remove the variables and values which are not required.
I have used $a -replace "variables" -replace "value" but it's not working
Don't try to manipulate JSON as text (as a string).
It's easier and more robust to transform the _objects ([pscustomobject] instances) representing the input JSON that Invoke-RestMethod returns:
# Assume that $fromJson was obtained as follows:
# $fromJson = Invoke-RestMethod ...
$fromJson.variables | ForEach-Object {
# Replace the property values with the current value's .value property
# value.
foreach ($prop in $_.psobject.Properties) {
$_.($prop.Name) = $prop.Value.value
}
$_ # Output the modified object.
} | ConvertTo-Json
$json.variables uses member-access enumeration to return an array of the variables property values, and the ForEach-Object command transforms the resulting objects by replacing their property values with their .value property value.
.psobject.Properties is a way of reflecting on any object's properties, and each property-information object returned has a .Name and a .Value property.
ConvertTo-Json converts the modified objects back to JSON
Given the following sample JSON input:
[
{
"variables": {
"appdata": {
"value": "x1"
},
"appinsta": {
"value": "y1"
}
}
},
{
"variables": {
"appdata": {
"value": "x2"
},
"appinsta": {
"value": "y2"
}
}
}
]
the above outputs:
[
{
"appdata": "x1",
"appinsta": "y1"
},
{
"appdata": "x2",
"appinsta": "y2"
}
]

Cannot convert PSCustomObjects within array back to JSON correctly

I'm trying to ingest a JSON file into Powershell, append a block of JSON to an existing node (Components), then convert the PSCustomObject back to JSON and save the file. The JSON I'm playing with looks something like Figure 1.
As you see in my code, I run ConvertTo-Json to cast the data into a PSCustomObject, and I then append a new object to the Components node. If I view the object, $configFile in this case it all looks fine, but when I convert back to JSON the items in the Components node, are treated as strings and not evaluated into JSON (see last snippet). I imagine this is because ConvertTo-JSON treats arrays literally, but not 100% sure.
If someone can advise how to ensure the PSCustomObjects in the Components node get casted back to JSON properly I would be grateful, thank you.
Figure 1 - the original JSON:
{
"EngineConfiguration": {
"PollInterval": "00:00:15",
"Components": [
{
"Id": "ApplicationEventLog",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "Application",
"Levels": "1"
}
},
{
"Id": "SystemEventLog",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "System",
"Levels": "7"
}
}
],
"Flows": {
"Flows":
[
"(ApplicationEventLog,SystemEventLog),CloudWatchLogs"
]
}
}
}
Figure 2 - my code:
#Requires -Version 3.0
$configFile = "C:\Program Files\Amazon\EC2ConfigService\Settings\AWS.EC2.Windows.CloudWatch.json"
$configToPSObject = ConvertFrom-Json "$(Get-Content $configFile)"
$configToPSObject.EngineConfiguration.Components += New-Object -Type PSObject -Property ([ordered]#{
"Id" = "IISRequestQueueSize"
"FullName" = "AWS.EC2.Windows.CloudWatch.PerformanceCounterComponent.PerformanceCounterInputComponent,AWS.EC2.Windows.CloudWatch"
"Parameters" = [PSCustomObject]#{
"CategoryName" = "HTTP Service Request Queues"
"CounterName" = "CurrentQueueSize"
"InstanceName" = "_Total"
"MetricName" = "IISRequestQueueSize"
"Unit" = ""
"DimensionName" = ""
"DimensionValue" = ""
}
})
$configJson = ConvertTo-Json -Depth 5 $configToPSObject
Set-Content -Path $configFile -Value $configJson
Figure 3 - the JSON output:
{
"EngineConfiguration": {
"PollInterval": "00:00:15",
"Components": [
"#{Id=ApplicationEventLog; FullName=AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch; Parameters=}",
"#{Id=SystemEventLog; FullName=AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch; Parameters=}",
"#{Id=IISRequestQueueSize; FullName=AWS.EC2.Windows.CloudWatch.PerformanceCounterComponent.PerformanceCounterInputComponent,AWS.EC2.Windows.CloudWatch; Parameters=}"
],
"Flows": {
"Flows":
"(ApplicationEventLog,SystemEventLog),CloudWatchLogs"
}
}
}
If I increase the depth to say, 8 or beyond, the JSON comes out as follows:
{
"EngineConfiguration": {
"PollInterval": "00:00:15",
"Components": [
"#{Id=ApplicationEventLog; FullName=AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch; Parameters=}",
"#{Id=SystemEventLog; FullName=AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch; Parameters=}",
"Id": "IISRequestQueueSize",
"FullName": "AWS.EC2.Windows.CloudWatch.PerformanceCounterComponent.PerformanceCounterInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"CategoryName": "HTTP Service Request Queues",
"CounterName": "CurrentQueueSize",
"InstanceName": "_Total",
"MetricName": "IISRequestQueueSize",
"Unit": "",
"DimensionName": "",
"DimensionValue": ""
}
}
],
"Flows": {
"Flows": "(ApplicationEventLog,SystemEventLog),CloudWatchLogs"
}
}
}
The ConvertTo-Json cmdlet also has a Depth parameter, beyond which an object is treated with toString() instead of going deeper with recursion. So just setting that parameter to whatever max depth of objects you have should result in a correctly formed JSON.
$configJson = ConvertTo-Json $configToPSObject -Depth 8
# your JSON has depth of 5, get some extra
You have to supply the depth for the ConvertTo-Json commandlet.
Otherwise it only does the first level and leaves the subnodes as is and converts them to a string apparently.
$configJson = ConvertTo-Json $obj -Depth 3