JSON file - how to delete entire node that contains a string - json

I have a JSON file:
{
"JSONS" : [
{
"id" : "ToRemove",
"First" : [
{
"id" : "geo",
"Name" : "Person1",
"model" : [
],
"adjustments" : [
{
"uid" : "3",
"name" : "4s",
"value" : "1"
},
{
"uid" : "5",
"name" : "3s",
"value" : "6"
}
]
},
{
"id" : "Meters",
"Dictionary" : "4.2"
},
{
"id" : "Moon",
"Filter" : "0.5",
"Saturn" : {
"s" : "0",
"v" : "1"
}
}
]
}
]
}
I would like to delete entire node, if the "id", in this example, contains "ToRemove" string. Everyting between { and }, including those lines also, to make the final JSON consistent.
This is a screenshot what I want to get rid of.
I only found how to delete properties, but not entire nodes. I've tried to appli something like this:
$ToRemove = Get-Content $SourceFile | ConvertFrom-Json
$ToRemove.PSObject.Object.Remove('id:','ToRemove')
$ToRemove | ConvertTo-Json -Depth 100 | Out-File $DestFile
but of course it didn't work.
How to delete the entire node? I would love to use an array to put all strings I would like to delete.

Based on your comment, you can remove that object having the property id = ToRemove by filtering where id is not equal to ToRemove and assigning that result to the .JSONS property:
$json = Get-Content path\to\json.json -Raw | ConvertFrom-Json
$json.JSONS = #($json.JSONS.Where{ $_.id -ne 'ToRemove' })
$json | ConvertTo-Json
The end result in this case would be an empty array for the .JSONS property:
{
"JSONS": []
}
.PSObject.Properties.Remove(...) wouldn't be useful in this case because what it does is remove properties from one object but what you want to do is filter out an entire object based on a condition.

You should be able to use just plain PowerShell, like this:
{
"JSONS" : [
{
"id" : "ToRemove",
"First" : [
{
"id" : "geo",
"Name" : "Person1",
"model" : [
]
},
{
"id" : "Meters",
"Dictionary" : "4.2"
}
]
},
{
"id" : "DontRemove",
"First" : []
}
]
}
$json = Get-Content -Path $SourceFile | ConvertFrom-Json
$json.JSONS = $json.JSONS | Where-Object { $_.Id -ne "ToRemove" }
$json | ConvertTo-Json -Depth 100 | Out-File -Path $DestFile

Related

Transform JSON to a compact format

I'm trying to Transform the following json
{ "application" : [
{ "name" : "app1",
"policies" : [
{ "name" : "pol_1",
"orderNumber" : "10"
},
{ "name" : "pol_2",
"orderNumber" : "20"
}
]
},
{ "name" : "app2",
"policies" : [
{ "name" : "pol_A",
"orderNumber" : "10"
},
{ "name" : "pol_B",
"orderNumber" : "20"
}
]
}
]
}
To the following
{ "pol_1":"10", "pol_2":"20" }
Using
jq -r ".application[] | select(.name==\"app1\") | .policies[] | {\".name\" : .orderNumber}"
I was able to get
{
"pol_1":"10"
}
{
"pol_2":"20"
}
Any idea how I can merge them. Am I missing something Or am I doing it the wrong way?
You were almost there. Use map to create a single array instead of two independent objects, then use add to merge its contents.
jq '.application[]
| select(.name == "app1")
| .policies
| map({ (.name) : .orderNumber } )
| add' file.json

Trying to convert complex JSON to CSV with Powershell

I'm fairly new to PowerShell and I am trying to convert this JSON file to CSV.
Using some simple code like this:
$sourceFilePath = "Records.json"
$destinationFilePath = "Records.csv"
((Get-Content -Path $sourceFilePath -Raw) | ConvertFrom-Json) | Export-CSV $destinationFilePath -NoTypeInformation
Results in:
Because I am getting System.Object[] I think I need to do some kind of expansion or "ForEach" of the contact_ids, site_ids, first name aliases, and aliases. I've done some searching and I can't seem to wrap my head around the examples I've been given. Also, "first name aliases" has spaces in it that add to the complexity.
I don't have any control over the source JSON. Here it is as an example:
[
{
"mdm id" : "947b2a12-3aac-480a-a0bf-626aff106f48",
"first name" : "Claude",
"last name" : "Bones",
"contact_ids" : [
"CTP70499"
],
"site_ids" : [
"5015"
],
"first name aliases" : [
"Claude"
],
"aliases" : [
"Claude Bones"
],
"createdDate" : "2020-06-03T19:59:08Z",
"updatedDate" : "2020-06-03T20:48:27Z",
"veevaID" : "V0B000000000569"
},
{
"mdm id" : "bce21b05-0b28-4ebb-a34d-761c1a397821",
"first name" : "Daniel",
"last name" : "Smith",
"contact_ids" : [
"CTP699"
],
"site_ids" : [
"5015"
],
"first name aliases" : [
"Dan",
"Danial",
"Danne",
"Danny",
"Daniel"
],
"aliases" : [
"Daniel Smith"
],
"createdDate" : "2020-06-03T19:59:08Z",
"updatedDate" : "2020-06-03T20:48:27Z",
"veevaID" : "V0B000000000566"
}
]
The values that are arrays will need to be combined into a single value. Regardless of how many values it may have, if it's designated as an array [] in json, it will need to be manipulated. There are several articles and custom functions written on the internet. Your example could be handled with this bit of code.
$JSONdata = #'
[
{
"mdm id" : "947b2a12-3aac-480a-a0bf-626aff106f48",
"first name" : "Claude",
"last name" : "Bones",
"contact_ids" : [
"CTP70499"
],
"site_ids" : [
"5015"
],
"first name aliases" : [
"Claude"
],
"aliases" : [
"Claude Bones"
],
"createdDate" : "2020-06-03T19:59:08Z",
"updatedDate" : "2020-06-03T20:48:27Z",
"veevaID" : "V0B000000000569"
},
{
"mdm id" : "bce21b05-0b28-4ebb-a34d-761c1a397821",
"first name" : "Daniel",
"last name" : "Smith",
"contact_ids" : [
"CTP699"
],
"site_ids" : [
"5015"
],
"first name aliases" : [
"Dan",
"Danial",
"Danne",
"Danny",
"Daniel"
],
"aliases" : [
"Daniel Smith"
],
"createdDate" : "2020-06-03T19:59:08Z",
"updatedDate" : "2020-06-03T20:48:27Z",
"veevaID" : "V0B000000000566"
}
]
'# | ConvertFrom-Json
$JSONdata | foreach {
$record = [ordered]#{}
foreach($property in $_.psobject.Properties)
{
if($property.value -is [string])
{
$record.Add($property.name,$property.value)
}
else
{
$record.Add($property.name,($property.value -join ', '))
}
}
[PSCustomObject]$record
} | ConvertTo-Csv -NoTypeInformation
Output
"mdm id","first name","last name","contact_ids","site_ids","first name aliases","aliases","createdDate","updatedDate","veevaID"
"947b2a12-3aac-480a-a0bf-626aff106f48","Claude","Bones","CTP70499","5015","Claude","Claude Bones","2020-06-03T19:59:08Z","2020-06-03T20:48:27Z","V0B000000000569"
"bce21b05-0b28-4ebb-a34d-761c1a397821","Daniel","Smith","CTP699","5015","Dan, Danial, Danne, Danny, Daniel","Daniel Smith","2020-06-03T19:59:08Z","2020-06-03T20:48:27Z","V0B000000000566"
Simply change it to Export-Csv

Parse and Map 2 Arrays with jq

I am working with a JSON file similar to the one below:
{ "Response" : {
"TimeUnit" : [ 1576126800000 ],
"metaData" : {
"errors" : [ ],
"notices" : [ "query served by:1"]
},
"stats" : {
"data" : [ {
"identifier" : {
"names" : [ "apiproxy", "response_status_code", "target_response_code", "target_ip" ],
"values" : [ "IO", "502", "502", "7.1.143.6" ]
},
"metric" : [ {
"env" : "dev",
"name" : "sum(message_count)",
"values" : [ 0.0]
} ]
} ]
} } }
My object is to display a mapping of the identifier and values like :
apiproxy=IO
response_status_code=502
target_response_code=502
target_ip=7.1.143.6
I have been able to parse both names and values with
.[].stats.data[] | (.identifier.names[]) and .[].stats.data[] | (.identifier.values[])
but I need help with the jq way to map the values.
The whole thing can be done in jq using the -r command-line option:
.[].stats.data[]
| [.identifier.names, .identifier.values]
| transpose[]
| "\(.[0])=\(.[1])"

How to traverse JSON properties with Powershell

I am trying to access a particular property value of a JSON object with Powershell. Unfortunately I do not know the keys of some of the parent properties within the structure, so I cannot do this in a straightforward way. Also as the JSON is not an array I am unable to access via index position.
The context is that I am querying a list of running tasks from elasticsearch and need to get the ID of the task (I know that there will only be one) so I can make subsequent calls to discover its completion status.
I've researched some querying methods but am unsure on how to apply them (PowerShell syntax is quite new to me).
The JSON response I am working with looks like this;
"nodes" : {
"oTUltX4IQMOUUVeiohTt8A" : {
"name" : "H5dfFeA",
"transport_address" : "127.0.0.1:9300",
"host" : "127.0.0.1",
"ip" : "127.0.0.1:9300",
"tasks" : {
"oTUltX4IQMOUUVeiohTt8A:124" : {
"node" : "oTUltX4IQMOUUVeiohTt8A",
"id" : 124,
"type" : "direct",
"action" : "cluster:monitor/tasks/lists[n]",
"start_time_in_millis" : 1458585884904,
"running_time_in_nanos" : 47402,
"cancellable" : false,
"parent_task_id" : "oTUltX4IQMOUUVeiohTt8A:123"
}
}
}
}
}
With this given structure, I would like to be able to access the ID property of the first 'task'.
So if I knew the prop keys it would be:
nodes.oTUltX4IQMOUUVeiohTt8A.tasks.oTUltX4IQMOUUVeiohTt8A:124.id
How can I access this value without knowing the keys beforehand?
Any help very appreciated.
Thanks
Nick
The following code defines and uses function Get-FirstPropertyValue, which performs a recursive, depth-first search for the first property inside an object graph that has a given name and returns its value, assuming the value is non-null:
# Function that returns the value of the first property with the given
# name found during recursive depth-first traversal of the given object.
# Note that null-valued properties are ignored.
function Get-FirstPropertyValue($obj, $propName) {
$propNames = $obj.psobject.properties.Name
if ($propName -in $propNames) {
$obj.$propName
} else {
foreach ($iterPropName in $propNames) {
if ($null -ne ($val = Get-FirstPropertyValue $obj.$iterPropName $propName)) {
return $val
}
}
}
}
# Input JSON
$json = #'
{
"nodes": {
"oTUltX4IQMOUUVeiohTt8A": {
"name": "H5dfFeA",
"transport_address": "127.0.0.1:9300",
"host": "127.0.0.1",
"ip": "127.0.0.1:9300",
"tasks": {
"oTUltX4IQMOUUVeiohTt8A:124": {
"node": "oTUltX4IQMOUUVeiohTt8A",
"id": 124,
"type": "direct",
"action": "cluster:monitor/tasks/lists[n]",
"start_time_in_millis": 1458585884904,
"running_time_in_nanos": 47402,
"cancellable": false,
"parent_task_id": "oTUltX4IQMOUUVeiohTt8A:123"
}
}
}
}
}
'#
# Convert the JSON to a [pscustomobject] graph with ConvertFrom-Json.
$objFromJson = $json | ConvertFrom-Json
# Using the function defined above, get the first 'tasks' object found
# during recursive depth-first traversal.
$tasks = Get-FirstPropertyValue $objFromJson 'tasks'
# Get the name of the resulting object's first property.
$propName = #($tasks.psobject.properties.Name)[0]
# Extract the .id property from the object stored in the first property.
$tasks.$propName.id
The above yields:
124
A more concise, but more obscure and presumably slower alternative is to convert the JSON input to XML and then use XPath to query it:
# Input JSON
$json = #'
{
"nodes": {
"oTUltX4IQMOUUVeiohTt8A": {
"name": "H5dfFeA",
"transport_address": "127.0.0.1:9300",
"host": "127.0.0.1",
"ip": "127.0.0.1:9300",
"tasks": {
"oTUltX4IQMOUUVeiohTt8A:124": {
"node": "oTUltX4IQMOUUVeiohTt8A",
"id": 124,
"type": "direct",
"action": "cluster:monitor/tasks/lists[n]",
"start_time_in_millis": 1458585884904,
"running_time_in_nanos": 47402,
"cancellable": false,
"parent_task_id": "oTUltX4IQMOUUVeiohTt8A:123"
}
}
}
}
}
'#
$parent = 'tasks'
$prop = 'id'
$propType = 'int'
$json |
ConvertFrom-Json |
ConvertTo-Xml -Depth ([int]::MaxValue) |
Select-Xml "//Property[#Name='$parent']/*/*[#Name='$prop']/text()" |
ForEach-Object { $_.Node.InnerText -as $propType }
There are two ways I know of that you can achieve this, both look a bit gnarly.
For these examples, I will load the JSON you have provided into $json.
$json = #'
{
"nodes": {
"oTUltX4IQMOUUVeiohTt8A": {
"name": "H5dfFeA",
"transport_address": "127.0.0.1:9300",
"host": "127.0.0.1",
"ip": "127.0.0.1:9300",
"tasks": {
"oTUltX4IQMOUUVeiohTt8A:124": {
"node": "oTUltX4IQMOUUVeiohTt8A",
"id": 124,
"type": "direct",
"action": "cluster:monitor/tasks/lists[n]",
"start_time_in_millis": 1458585884904,
"running_time_in_nanos": 47402,
"cancellable": false,
"parent_task_id": "oTUltX4IQMOUUVeiohTt8A:123"
}
}
}
}
}
'# | ConvertFrom-Json
The first is to use Select-Object to select the first item and then expand the properties.
(($json.nodes | Select-Object -First 1 -ExpandProperty *).tasks | Select-Object -First 1 -ExpandProperty *).id
A more robust method, is to use the hidden PSObject property Value, as the JSON is parsed by PowerShell into a PSCustomObject.
PS C:\Windows\system32> $json.nodes.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True False PSCustomObject System.Object
The Properties of .PSObject
PS C:\Windows\system32> $json.nodes.PSObject.Properties
MemberType : NoteProperty
IsSettable : True
IsGettable : True
Value : #{name=H5dfFeA; transport_address=127.0.0.1:9300; host=127.0.0.1; ip=127.0.0.1:9300; tasks=}
TypeNameOfValue : Selected.System.Management.Automation.PSCustomObject
Name : oTUltX4IQMOUUVeiohTt8A
IsInstance : True
The full command to access the ID value:
$json.nodes.PSObject.Properties.Value.tasks.PSObject.Properties.Value.id

JQ add properties to nested object in nested array

I have the following json:
{
"first": {
"second" : "A"
},
"array": [
{
"name" : "AAA",
"something": {
"hola": "hi"
}
},
{
"name" : "BBB",
"something": {
"hola": "hi"
}
}
]
}
I would like to trasform it adding a property to the something object, using the value from the name property of the parent, like:
I have the following json:
{
"first": {
"second" : "A"
},
"array": [
{
"name" : "AAA",
"something": {
"hola": "hi",
"NEW_PROPERTY": "AAA"
}
},
{
"name" : "BBB",
"something": {
"hola": "hi",
"NEW_PROPERTY": "BBB"
}
}
]
}
Which jq expression can do this?
Try this jq script:
<file jq '.array = [ .array[] | .something.NEW_PROPERTY = .name ]'
This is replacing the array by another one that is the same as the original one with one more key NEW_PROPERTY.
You could simply use the filter:
.array |= map(.something.NEW_PROPERTY = .name)
or if map's not your thing (or if you want to save typing one character):
.array[] |= (.something.NEW_PROPERTY = .name)