Related
Trying to create the following JSON structure through bash. There will be a max of 4 environments that I want to be shown even if there are no content within them, and example output can be found below the structure.
Input Text File:
DEV,Middleware,Mqwerty,Mqwerty
DEV,Middleware,Mqwerty,Mqwerty
DEV,Middleware,Mqwerty,Mqwerty
DEV,System,Sqwerty,Sqwerty
DEV,Application,Aqwerty,Aqwerty,Aqwerty
UAT,Application,Aqwerty,Aqwerty,Aqwerty
DEV,Utility,Uqwerty,Uqwerty,Uqwerty
PROD,Middleware,Mqwerty,Mqwerty
DEV,Middleware,Mqwerty,Mqwerty
Desired JSON Structure:
{
"ENV": {
"DEV": {
"Middleware": [
{
"name": "Mqwerty",
"release": "Mqwerty"
},
{
"name": "Mqwerty",
"release": "Mqwerty"
},
{
"name": "Mqwerty",
"release": "Mqwerty"
}
],
"System": [
{
"name": "Sqwerty",
"tag": "Sqwerty"
}
],
"Application": [
{
"domain": "Aqwerty",
"host": "Aqwerty",
"user": "Aqwerty"
},
{
"domain": "Aqwerty",
"host": "Aqwerty",
"user": "Aqwerty"
}
],
"Utility": [
{
"domain": "Uqwerty",
"health": "Uqwerty",
"version": "Uqwerty"
}
]
},
"SIT": {
"Middleware": [],
"System": [],
"Application": [],
"Utility": []
},
"UAT": {
"Middleware": [
{
"name": "Mqwerty",
"release": "Mqwerty"
},
{
"name": "Mqwerty",
"release": "Mqwerty"
}
],
"System": [],
"Application": [],
"Utility": []
},
"PROD": {
"Middleware": [],
"System": [],
"Application": [],
"Utility": []
}
}
}
Some key notes, even in environments that don't have information, the 'template' of middleware, system, application and utility (lets call these categories) is still there. The categories also have a predefined key:value structure that follows:
Application (keys): domain, host, user
Utility: domain, health, version
Middleware: name, release
System: name, tag
This is the code I've been able to get so far, however its unable to add a particular set of keys for each category (Application, Utility, Middleware and System) and also isn't able to add all the values as well.
#!/usr/bin/jq -Rnf
reduce inputs as $line
( .ENV
["DEV", "SIT", "UAT", "PROD"]
["Middleware", "System", "Application", "Utility"] = []
; ($line | split(",")) as $elements
| .ENV [$elements[0]] [$elements[1]] +=
[ $elements[2:]
| with_entries(.key |= "value\(.+1)")
]
)
I really do appreciate any help and thank you for taking you time reading this questions, apologies for being a long one. Also any good resources regarding jq would be appreciated.
Here's one way to build up a solution from easily understood pieces. In this case, jq would be invoked with -nR.
def initial:
null
| .["DEV", "SIT", "UAT", "PROD"]["Middleware", "System", "Application", "Utility"] = [];
def objectify($keys):
. as $in
| reduce range(0; $keys|length) as $i ({}; .[$keys[$i]] = ($in[$i]) );
def object:
.[0] as $top
| .[1:]
| if $top == "Middleware" then objectify(["name", "release"])
elif $top == "System" then objectify(["domain", "tag"])
elif $top == "Application" then objectify(["domain", "host", "user"])
elif $top == "Utility" then objectify(["domain", "health", "version"])
else objectify( map(tostring) ) # or raise an error, or ...
end;
reduce (inputs | split(",")) as $line (initial;
getpath($line[0:2]) as $v
| setpath($line[0:2]; $v + [$line[1:] | object] ))
| {ENV: .}
Here's a DRYer and more declarative version of my other solution on this page. It also handles the anomalous case slightly differently.
< input.txt jq -nR '
def categories:
{ "Middleware": ["name", "release"],
"System": ["domain", "tag"],
"Application": ["domain", "host", "user"],
"Utility": ["domain", "health", "version"] };
def initial:
null
| .["DEV", "SIT", "UAT", "PROD"][ categories | keys[]] = [];
def objectify($keys):
. as $in
| reduce range(0; $keys|length) as $i ({}; .[$keys[$i]] = ($in[$i]) );
def object:
categories[.[0]] as $keys
| .[1:]
| objectify($keys // [range(0;length) | tostring]);
reduce (inputs | split(",")) as $line (initial;
getpath($line[0:2]) as $v
| setpath($line[0:2]; $v + [$line[1:] | object] ))
| {ENV: .}
I am trying to access a particular property value of a JSON object with Powershell. Unfortunately I do not know the keys of some of the parent properties within the structure, so I cannot do this in a straightforward way. Also as the JSON is not an array I am unable to access via index position.
The context is that I am querying a list of running tasks from elasticsearch and need to get the ID of the task (I know that there will only be one) so I can make subsequent calls to discover its completion status.
I've researched some querying methods but am unsure on how to apply them (PowerShell syntax is quite new to me).
The JSON response I am working with looks like this;
"nodes" : {
"oTUltX4IQMOUUVeiohTt8A" : {
"name" : "H5dfFeA",
"transport_address" : "127.0.0.1:9300",
"host" : "127.0.0.1",
"ip" : "127.0.0.1:9300",
"tasks" : {
"oTUltX4IQMOUUVeiohTt8A:124" : {
"node" : "oTUltX4IQMOUUVeiohTt8A",
"id" : 124,
"type" : "direct",
"action" : "cluster:monitor/tasks/lists[n]",
"start_time_in_millis" : 1458585884904,
"running_time_in_nanos" : 47402,
"cancellable" : false,
"parent_task_id" : "oTUltX4IQMOUUVeiohTt8A:123"
}
}
}
}
}
With this given structure, I would like to be able to access the ID property of the first 'task'.
So if I knew the prop keys it would be:
nodes.oTUltX4IQMOUUVeiohTt8A.tasks.oTUltX4IQMOUUVeiohTt8A:124.id
How can I access this value without knowing the keys beforehand?
Any help very appreciated.
Thanks
Nick
The following code defines and uses function Get-FirstPropertyValue, which performs a recursive, depth-first search for the first property inside an object graph that has a given name and returns its value, assuming the value is non-null:
# Function that returns the value of the first property with the given
# name found during recursive depth-first traversal of the given object.
# Note that null-valued properties are ignored.
function Get-FirstPropertyValue($obj, $propName) {
$propNames = $obj.psobject.properties.Name
if ($propName -in $propNames) {
$obj.$propName
} else {
foreach ($iterPropName in $propNames) {
if ($null -ne ($val = Get-FirstPropertyValue $obj.$iterPropName $propName)) {
return $val
}
}
}
}
# Input JSON
$json = #'
{
"nodes": {
"oTUltX4IQMOUUVeiohTt8A": {
"name": "H5dfFeA",
"transport_address": "127.0.0.1:9300",
"host": "127.0.0.1",
"ip": "127.0.0.1:9300",
"tasks": {
"oTUltX4IQMOUUVeiohTt8A:124": {
"node": "oTUltX4IQMOUUVeiohTt8A",
"id": 124,
"type": "direct",
"action": "cluster:monitor/tasks/lists[n]",
"start_time_in_millis": 1458585884904,
"running_time_in_nanos": 47402,
"cancellable": false,
"parent_task_id": "oTUltX4IQMOUUVeiohTt8A:123"
}
}
}
}
}
'#
# Convert the JSON to a [pscustomobject] graph with ConvertFrom-Json.
$objFromJson = $json | ConvertFrom-Json
# Using the function defined above, get the first 'tasks' object found
# during recursive depth-first traversal.
$tasks = Get-FirstPropertyValue $objFromJson 'tasks'
# Get the name of the resulting object's first property.
$propName = #($tasks.psobject.properties.Name)[0]
# Extract the .id property from the object stored in the first property.
$tasks.$propName.id
The above yields:
124
A more concise, but more obscure and presumably slower alternative is to convert the JSON input to XML and then use XPath to query it:
# Input JSON
$json = #'
{
"nodes": {
"oTUltX4IQMOUUVeiohTt8A": {
"name": "H5dfFeA",
"transport_address": "127.0.0.1:9300",
"host": "127.0.0.1",
"ip": "127.0.0.1:9300",
"tasks": {
"oTUltX4IQMOUUVeiohTt8A:124": {
"node": "oTUltX4IQMOUUVeiohTt8A",
"id": 124,
"type": "direct",
"action": "cluster:monitor/tasks/lists[n]",
"start_time_in_millis": 1458585884904,
"running_time_in_nanos": 47402,
"cancellable": false,
"parent_task_id": "oTUltX4IQMOUUVeiohTt8A:123"
}
}
}
}
}
'#
$parent = 'tasks'
$prop = 'id'
$propType = 'int'
$json |
ConvertFrom-Json |
ConvertTo-Xml -Depth ([int]::MaxValue) |
Select-Xml "//Property[#Name='$parent']/*/*[#Name='$prop']/text()" |
ForEach-Object { $_.Node.InnerText -as $propType }
There are two ways I know of that you can achieve this, both look a bit gnarly.
For these examples, I will load the JSON you have provided into $json.
$json = #'
{
"nodes": {
"oTUltX4IQMOUUVeiohTt8A": {
"name": "H5dfFeA",
"transport_address": "127.0.0.1:9300",
"host": "127.0.0.1",
"ip": "127.0.0.1:9300",
"tasks": {
"oTUltX4IQMOUUVeiohTt8A:124": {
"node": "oTUltX4IQMOUUVeiohTt8A",
"id": 124,
"type": "direct",
"action": "cluster:monitor/tasks/lists[n]",
"start_time_in_millis": 1458585884904,
"running_time_in_nanos": 47402,
"cancellable": false,
"parent_task_id": "oTUltX4IQMOUUVeiohTt8A:123"
}
}
}
}
}
'# | ConvertFrom-Json
The first is to use Select-Object to select the first item and then expand the properties.
(($json.nodes | Select-Object -First 1 -ExpandProperty *).tasks | Select-Object -First 1 -ExpandProperty *).id
A more robust method, is to use the hidden PSObject property Value, as the JSON is parsed by PowerShell into a PSCustomObject.
PS C:\Windows\system32> $json.nodes.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True False PSCustomObject System.Object
The Properties of .PSObject
PS C:\Windows\system32> $json.nodes.PSObject.Properties
MemberType : NoteProperty
IsSettable : True
IsGettable : True
Value : #{name=H5dfFeA; transport_address=127.0.0.1:9300; host=127.0.0.1; ip=127.0.0.1:9300; tasks=}
TypeNameOfValue : Selected.System.Management.Automation.PSCustomObject
Name : oTUltX4IQMOUUVeiohTt8A
IsInstance : True
The full command to access the ID value:
$json.nodes.PSObject.Properties.Value.tasks.PSObject.Properties.Value.id
I have a string like this:
$string = "PackageName1,1,C:\Path
PackageName2,12,C:\Path2
PackageName3,3,C:\Path3"
(is a file with multilines, when I get the content I have the string above)
I want to convert this string to Json:
[
{
Pacakge: "PackageName1",
Branch: = "1",
LocalPath = "C:\Path"
}
{
Pacakge: "PackageName2",
Branch: = "2",
LocalPath = "C:\Path2"
}
]
I can get the values with this code:
$spiltted = $string.Split(' ')
ForEach ($s in $splitted)
{
$values = $s.Split(',')
}
How can I add the Json keys to each value and convert it to Json object?
Thank you.
As your String looks like a csv without headers:
$string | ConvertFrom-Csv -Header Package,Branch,LocalPath|ConvertTo-Json
Sample output
[
{
"Package": "PackageName1",
"Branch": "1",
"LocalPath": "C:\\Path"
},
{
"Package": "PackageName2",
"Branch": "12",
"LocalPath": "C:\\Path2"
},
{
"Package": "PackageName3",
"Branch": "3",
"LocalPath": "C:\\Path3"
}
]
I have below sample nested json response and I need to convert this response with specific values into CSV file. Below is the sample nested Json response:
{
"transaction": {
"id": "TestTransID",
"testCode": "NEW",
"TestStatus": "SUCCESS",
"client": {
"TestNumber": "112112111"
},
"subject": {
"individual": {
"additionalAttributes": {
"extraid": "787877878"
},
"addressList": [
{
"city": "New York",
"country": {
"name": "United States"
},
"postalCode": "123456789",
"stateOrProvince": {
"codeValue": "NY"
}
}
],
"gender": "F",
"identificationDocumentList": [
{
"number": "1214558520",
"type": "TestId"
}
],
"name": {
"firstName": "Qusay TestFull",
"lastName": "TestLast",
"middleName": "Middle 3"
}
}
},
"PROCESSConfiguration": {
"id": 1
},
"testProductList": [
{
"product": {
"id": 00,
"name": "Test PROCESS",
"productCode": "EFG",
"disclaimer": "TestDisclaimer"
},
"testSourceResponseList": [
{
"testSource": {
"id": 1,
"name": "TEST"
},
"testSourceRecordList": [
{
"type": "TestRecord",
"alertReasonCode": "TESTS",
"alertReasonDescription": "ACTION LIST HIT - TEST",
"testSource": "TEST",
"varListNameFull": "TEST FULL NAME",
"varListNameShort": "TEST SHORT",
"varProgList": [
"SHORT"
],
"varListId": "3421",
"subject": {
"individual": {
"TestScore": {
"TestScore": 100,
"triggeredRule": "TestRule"
},
"aNameList": [
{
"fullName": " TestNameA",
"lastName": "TestNameA"
},
{
"firstName": "TestFirst",
"fullName": "TestFirst HUSAYN",
"lastName": "TestLast"
},
{
"firstName": "TestFirst",
"fullName": "TestFull",
"lastName": "TestLast"
},
{
"firstName": "TestFirst",
"fullName": "TestFull",
"lastName": "TestLast"
}
],
"birthList": [
{
"dateOfBirth": "12 Apr 1910",
"dateOfBirthVerified": "true"
}
],
"name": {
"firstName": "TestFirst",
"fullName": "TestFull",
"lastName": "TestLast"
},
"varNationality": [
{
"verified": "true"
}
],
"remarks": "remark1"
}
}
},
{
"testSource": "TEST",
"varListNameFull": "TEST FULL",
"varListNameShort": "TEST SHORT",
"varProgList": [
"XYZ"
],
"varListId": "1234",
"subject": {
"individual": {
"overallScore": {
"TestScore": 100,
"triggeredRule": "Testing"
},
"birthList": [
{
"dateOfBirth": "1965",
},
{
"dateOfBirth": "1966",
}
],
"name": {
"firstName": "TestFirst",
"fullName": "TestFull",
"lastName": "TestLast",
},
"varNationality": [
{
"verified": "true"
}
],
"remarks": "REMARK2"
}
}
}
]
}
]
}
],
}
}
I need to take response from ""PROCESSConfiguration": {
"id": 1"
from row # 40. If u'll take above code in notepad ++.
Also I need the response with respect to var value like first name, last name full name, DOB etc.
I am still unsure what is being asked for. Let me assume that you want a fully qualified path for each of the elements in the JSON file.
I started by making some small adjustments to the JSON based on http://jsonlint.com/.
Based on that, I did a proof of concept that works for ONE OBJECT like the JSON you posted. It works for THIS CASE. I wrapped the logic to deal with multiple objects making assumptions about when one object started and the next began. Also, logic would should be added to deal with nested series/array containing multiple properties (i.e. Threads in Get-Process) to handle a generic case. A single element with an array of values (like .transaction.varProgList in this case) is handled by putting a ‘;’ between them.
CSV files assume that all the objects are symmetric (have the same properties). There is no check to see that the properties for each object align with the properties of the other objects. Note the handling of nested series is related to this. You may see an example of this by uncommenting the [System.Collections.ICollection] section and trying something like (Get-Process r*) | Select-Object Name,Threads | Expand-NestedProperty | Out-File .\t.csv -Width 100000
The repro goes as follows where $a is the adjusted JSON content and the function is saved as Expand-NestedProperty.ps1.
# Load the function
. .\Expand-NestedProperty.ps1
# Create PowerShell object(s) based on the JSON
$b = $a | ConvertFrom-Json
# Create a file with the CSV contents.
$b | Expand-NestedProperty | Out-File -FilePath .\my.csv -Width 100000
Save this as Expand-NestedProperty.ps1
function Expand-NestedProperty {
[CmdletBinding()]
param (
[Parameter( Position=0,
Mandatory=$true,
ValueFromPipeline=$true,
ValueFromPipelineByPropertyName=$true,
ValueFromRemainingArguments=$false,
HelpMessage='Object required...' )]
$InputObject
)
begin {
function ExpandNestedProperty {
[CmdletBinding()]
param (
[Parameter( Position=0,
Mandatory=$true,
ValueFromPipeline=$true,
ValueFromPipelineByPropertyName=$true,
ValueFromRemainingArguments=$false,
HelpMessage='Object required...' )]
$InputObject,
[Parameter( Position=1,
Mandatory=$false,
ValueFromPipeline=$false,
ValueFromPipelineByPropertyName=$true,
ValueFromRemainingArguments=$false,
HelpMessage='String required...' )]
[string]
$FullyQualifiedName = ""
)
begin {
$localResults =#()
$FQN = $FullyQualifiedName
$nestedProperties = $null
}
process {
foreach ($obj in $InputObject.psobject.Properties) {
if ($(try {$obj.Value[0] -is [PSCustomObject]} catch {$false})) { # Catch 'Cannot index into a null array' for null values
# Nested properties
$FQN = "$($FullyQualifiedName).$($obj.Name)"
$nestedProperties = $obj.value | ExpandNestedProperty -FullyQualifiedName $FQN
}
elseif ($obj.Value -is [array]) {
# Array property
$FQN = "$($FullyQualifiedName).$($obj.Name)"
[psobject]$nestedProperties = #{
$FQN = ($obj.Value -join ';')
}
}
# Example of how to deal with generic case.
# This needed for the Get-Process values ([System.Collections.ReadOnlyCollectionBase] and [System.Diagnostics.FileVersionInfo]) that are not [array] collection type.
<#
elseif ($obj.Value -is [System.Collections.ICollection]) {
# Nested properties
$FQN = "$($FullyQualifiedName).$($obj.Name)"
$nestedProperties = $obj.value | ExpandNestedProperty -FullyQualifiedName $FQN
}
#>
else { # ($obj -is [PSNoteProperty]) for this case, but could be any type
$FQN = "$($FullyQualifiedName).$($obj.Name)"
[psobject]$nestedProperties = #{
$FQN = $obj.Value
}
}
$localResults += $nestedProperties
} #foreach $obj
}
end {
[pscustomobject]$localResults
}
} # function ExpandNestedProperty
$objectNumber = 0
$firstObject = #()
$otherObjects = #()
}
process {
if ($objectNumber -eq 0) {
$objectNumber++
$firstObject = $InputObject[0] | ExpandNestedProperty
}
else {
if ($InputObject -is [array]) {
foreach ($nextInputObject in $InputObject[1..-1]) {
$objectNumber++
$otherObjects += ,($nextInputObject | ExpandNestedProperty)
}
}
else {
$objectNumber++
$otherObjects += ,($InputObject | ExpandNestedProperty)
}
}
}
end {
# Output CSV header and a line for each object which was the specific requirement here.
# Could create an array of objects using $firstObject + $otherObjects that is then piped to Export-CSV if we want a generic case.
Write-Output "`"$($firstObject.Keys -join '","')`""
Write-Output "`"$($firstObject.Values -join '","')`""
foreach ($otherObject in $otherObjects) {
Write-Output "`"$($otherObject.Values -join '","')`""
}
}
} # function Expand-NestedProperty
I have a json of this structure:
{
"nodes": {
"60e327ee58a0": {
"nodeinfo": {
"network": {
"mesh": {
"bat0": {
"interfaces": {
"wireless": [
"<mac-address-removed>"
],
"tunnel": [
"<mac-address-removed>"
]
}
}
},
"mac": "<mac removed>",
"addresses": [
"<ipv6 removed>",
"<ipv6 removed>"
]
},
"hardware": {
"model": "TP-Link TL-WR841N/ND v10",
"nproc": 1
},
"software": {
"batman-adv": {
"compat": 15,
"version": "2015.1"
},
"autoupdater": {
"branch": "stable",
"enabled": true
},
"firmware": {
"release": "v2016.1+1.0.1",
"base": "gluon-v2016.1"
},
"status-page": {
"api": 1
},
"fastd": {
"enabled": true,
"version": "v17"
}
},
"hostname": "Antoniusweg12",
"system": {
"site_code": "ffmsd03"
},
"node_id": "60e327ee58a0"
},
"lastseen": "2016-04-14T12:39:04",
"flags": {
"gateway": false,
"online": true
},
"firstseen": "2016-03-16T15:14:04",
"statistics": {
"clients": 1,
"gateway": "de:ad:be:ef:43:02",
"rootfs_usage": 0.6041666666666667,
"loadavg": 0.09,
"uptime": 1822037.41,
"memory_usage": 0.8124737210932025,
"traffic": {
"rx": {
"packets": 50393821,
"bytes": 5061895206
},
"forward": {
"packets": 173,
"bytes": 17417
},
"mgmt_rx": {
"packets": 47453745,
"bytes": 6623785282
},
"tx": {
"packets": 1205695,
"bytes": 173509528,
"dropped": 5683
},
"mgmt_tx": {
"packets": 37906725,
"bytes": 11475209742
}
}
}
},
"30b5c2b042f4": {
<next block...>
And I want to query it with jq for the hostname, the mac or the IPv6.
cat nodes.json |jq -c '.nodes[] | select(.nodes[]| contains("Antoniusweg12"))'
Most examples do not fit this kind of json structure as the objects have an index
Thanks for help in advance.
If you're going to filter, you need to drill down to the property that you want to check for and see if it matches your criteria. You can't expect to just give a name and you'll magically be presented with the results you want.
Searching by hostname, it is found on the .nodeinfo.hostname property of each node:
$ jq -c --arg hostname "Antoniusweg12" \
'.nodes[] | select(.nodeinfo.hostname == $hostname)' nodes.json
Similarly for the mac address, it's found on the .nodeinfo.network.mac property:
$ jq -c --arg mac "aa:bb:cc:dd:ee:ff" \
'.nodes[] | select(.nodeinfo.network.mac == $mac)' nodes.json
For the ip addresses, there's an array of them but it's not that much different in the query. They're found on the .nodeinfo.network.addresses property:
$ jq -c --arg ip "aaaa:bbbb:cccc:dddd::1" \
'.nodes[] | select(.nodeinfo.network.addresses[] == $ip)' nodes.json
Here's another take on the question. Suppose you want to find all occurrences of the key "hostname" for which the value is "Antoniusweg12",
no matter where the key/value combination occurs.
The following will reveal the path to the key/value combination of interest:
paths as $p
| select ( $p[-1] == "hostname" and getpath($p) == "Antoniusweg12" )
| $p
The result for the given input JSON:
[
"nodes",
"60e327ee58a0",
"nodeinfo",
"hostname"
]
If you wanted the path to the containing object, then replace the final $p with $p[0:-1]; and if you want the containing object itself: getpath($p[0:-1])
Here is a solution which searches for nodes where the specified $needle is present in any of the addresses, mac or hostname fields.
"<ipv6 removed>" as $needle # set to whatever you like
| foreach (.nodes|keys[]) as $k (
.
; .
; ( .nodes[$k].nodeinfo.network.addresses?
+ [ .nodes[$k].nodeinfo.network.mac?
, .nodes[$k].nodeinfo.hostname?
]
) as $haystack
| if $haystack | index($needle)
then {($k): .nodes[$k]}
else empty
end
)
EDIT: I now realize a filter of the form foreach E as $X (.; .; R) can almost always be rewritten as E as $X | R so the above is really just
"<ipv6 removed>" as $needle
| (.nodes|keys[]) as $k
| ( .nodes[$k].nodeinfo.network.addresses?
+ [ .nodes[$k].nodeinfo.network.mac?
, .nodes[$k].nodeinfo.hostname?
]
) as $haystack
| if $haystack | index($needle)
then {($k): .nodes[$k]}
else empty
end