I have two different JSONs and I would like to remove streets from the JSON object if only it exists under Address which is an array. I am trying to do this in powershell. I can get my script working and remove the streets but I only want to run the exclude line of command if the address has the streets property.
{
"Customer": [{
"id": "123"
}],
"address": [{
"$type": "Home",
"name": "Houston",
"streets": [{
"name": "Union",
"postalCode": "10"
}]
},
{
"$type": "Office",
"name": "Hawai",
"streets": [{
"name": "Rock",
"postalCode": "11"
}]
}
]
}
2nd JSON - Do not want to run the exclude line for 2nd JSON because there are no streets
{
"Customer": [{
"id": "123"
}],
"address": [{
"$type": "Home",
"name": "Houston"
},
{
"$type": "Office",
"name": "Hawai"
}
]
}
Powershell script
$FileContent = Get-Content -Path "Test.json" -Raw | ConvertFrom-Json
#Only want to run for address objects that contains streets
$FileContent.address = $FileContent.address | Select-Object * -ExcludeProperty streets #Only run for 1st json and not for 2nd json
$FileContent | ConvertTo-Json
If you want to execute the code only if the address has the member streets you can test for just that:
if (
($FileContent.address | Get-Member -MemberType NoteProperty -Name "streets") -ne $null
){
$FileContent.address = $FileContent.address | Select-Object * -ExcludeProperty streets
}
T-Me's helpful answer is the most robust approach, because it looks for the presence of the property itself rather than non-null values.
If you're willing to assume that the absence of a value also means the absence of the property itself, you can take the following shortcut, which performs better:
$hasAtLeastOneStreet = 0 -ne
(#((Get-Content Test.json -Raw | ConvertFrom-Json).address.streets) -ne $null).Count
.address.streets uses member-access enumeration to extract all streets values, #(...) ensures that the result is an array, -ne $null filters out any $null values from that array, and .Count counts its elements.
Note: This expression should be simpler:
$null -ne (Get-Content Test.json -Raw | ConvertFrom-Json).address.streets
but due to a bug currently cannot - see the bottom section.
To demonstrate (the input strings are compressed, single-line versions of your JSON documents):
'{"Customer":[{"id":"123"}],"address":[{"$type":"Home","name":"Houston","streets":[{"name":"Union","postalCode":"10"}]},{"$type":"Office","name":"Hawai","streets":[{"name":"Rock","postalCode":"11"}]}]}',
'{"Customer":[{"id":"123"}],"address":[{"$type":"Home","name":"Houston"},{"$type":"Office","name":"Hawai"}]}' |
foreach {
"has street values: " +
(0 -ne #(((ConvertFrom-Json $_).address.streets) -ne $null).Count)
}
The above yields, showing that the first JSON document had street values, whereas the second one did not.
has street values: True
has street values: False
Note: You should be able to simplify the test expression to the following, but this doesn't work due to a bug present up to at least PowerShell 7.0:
# !! SHOULD worm, but as of PowerShell 7.0, DOESN'T, due to a bug relating
# to the presence of two or more [pscustomobject] instances in the address array:
$hasAtLeastOneStreet =
$null -ne (Get-Content Test.json -Raw | ConvertFrom-Json).address.streets
Normally, the absence of any streets property values should result in $null, but with two or more [pscustomobject] instances present in the .address array, an array of $null values is unexpectedly returned.
See GitHub issue #13752.
Related
How can I make my code a lot more performant when I wish to make an easily/fast parseable object/PSCustomObject from a JSON payload $JSON?
An example of the structure of the PAYLOAD I receive is:
[
{
"name": "system.enablenetflow",
"value": "false"
},
{
"name": "system.deviceGroupId",
"value": "186,3060"
},
{
"name": "system.prefcollectorid",
"value": "144"
},
{
"name": "system.collectorplatform",
"value": "windows"
}
]
As you can see its in a very annoying format.
Note that the payloads I attempt to parse are much larger and variable in count from 500 of these Name/Value objects to 50000, rather than just the 4 listed above.
###########################################################################
MY GOAL
To have this turn into a key:value pair scenario for easier parsing later
NOT This:
With the JSON I have to do $JSON.where({$_.name -eq "system.enablenetflow"}).value
YES THIS:
I want the end state to be that the new variable $obj I create will let me get the value with $obj."system.enablenetflow"
###########################################################################
MY CURRENT ATTEMPT THAT IS SUPER SLOW
I did the following:
Create an Empty PSCustomObject and saved it as variable $obj
Did a foreach method on the $JSON variable which iterated through the JSON Array
Add-Member to $obj with setting the 'name' as PropertyName and 'value' as PropertyValue
Heres a sample of my code:
$obj = [PSCustomObject] #{}
$json.foreach({
$thisitem = $_
$obj | Add-member -NotePropertyName $($thisitem.name) -NotePropertyValue $($thisitem.name)
})
HOW CAN I MAKE THIS FASTER?
# Sample input JSON.
$json = #'
[
{
"name": "system.enablenetflow",
"value": "false"
},
{
"name": "system.deviceGroupId",
"value": "186,3060"
},
{
"name": "system.prefcollectorid",
"value": "144"
},
{
"name": "system.collectorplatform",
"value": "windows"
}
]
'#
# Initialize the (ordered) result hashtable.
$result = [ordered] #{}
# Note: In PowerShell (Core) 7+, add -AsHashTable to the ConvertFrom-Json
# call for additional performance gain, combined with -AsArray,
# in which case you don't need the `(...)` around the call anymore.
foreach ($element in (ConvertFrom-Json $json)) {
$result[$element.name] = $element.value
}
The above creates an (ordered) hashtable instead of a [pscustomobject] instance - especially if the latter are iteratively constructed via Add-Member calls.
Hashtables are lighter-weight and faster to construct than [pscustomobject] instances.
Using a foreach loop rather than processing the ConvertFrom-Json output in a pipeline via ForEach-Object also speeds up processing.
PowerShell allows you to use the familiar dot notation also with hashtables; so, for instance, after running the above, you'll get:
PS> $result.'system.collectorplatform'
windows
If you do need $result to be a [pscustomobject] instance, you can simply cast the fully populated hashtable to that type:
PS> $obj = [pscustomobject] $result; $obj.'system.collectorplatform'
windows
I have following JSON and I would like to remove streets from the JSON object if only it exists under Address which is an array. I am trying to do this in powershell. I can get my script working and remove the streets but I only want to run the exclude line of command if the address has the streets property. Is that possible?
{
"Customer": [{
"id": "123"
}],
"Nationality": [{
"name": "US",
"id": "456"
}],
"address": [{
"$type": "Home",
"name": "Houston",
"streets": [{
"name": "Union",
"postalCode": "10"
}]
},
{
"$type": "Home5",
"name": "Houston5"
},
{
"$type": "Office",
"name": "Hawai",
"streets": [{
"name": "Rock",
"postalCode": "11"
}]
}
]
}
Powershell script
$FileContent = Get-Content -Path "Test.json" -Raw | ConvertFrom-Json
#Only want to run for address objects that contains streets
$FileContent.address = $FileContent.address | Select-Object * -ExcludeProperty streets #Only would like to run if object address has streets property
$FileContent | ConvertTo-Json
Note:
This answer performs the same operation as in the question, only more succinctly, in a single pipeline.
It is benign to run Select-Object * -ExcludeProperty streets against all objects in array address, because the call is an effective no-op for those objects that already lack a streets property (though a copy of such objects is created too).
You need an assignment to modify your objects in-place before outputting them, which requires a ForEach-Object call:
Get-Content -Raw Test.json | ConvertFrom-Json |
ForEach-Object {
[array] $_.address = $_.address | select * -exclude streets; $_
}
Note how each object parsed from the JSON input is first modified via the assignment ($_.address = ...), and then passed out ($_).
A more efficient, but a little more obscure variant:
Get-Content -Raw Test.json | ConvertFrom-Json |
ForEach-Object {
$_.address.ForEach({ $_.psobject.Properties.Remove('streets') }); $_
}
With your sample JSON input, both commands output the following:
Customer Nationality address
-------- ----------- -------
{#{id=123}} {#{name=US; id=456}} {#{$type=Home; name=Houston}, #{$type=Home5; name=Houston5}, #{$type=Office; name=Hawai}}
Note how the objects in the address column no longer have a streets property.
Caveat: Note that ConvertTo-Json limits the serialization depth to 2 by default, which is sufficient in this case, but in other cases you may have to pass a -Depth argument to prevent data loss - see this post.
I have a JSON data structured as following (there may be some mistakes here, the data I'm using is fine):
[{
"id": 12345,
"itemName": "some string",
"sellerId": 123,
"seller": "",
"categoryId": ,
"categoryPath": [
{
//more data
},
{
//more data
}
]},
{"id": 12346,
"itemName": "some other string",
"sellerId": 234,
"seller": "",
"categoryId": ,
"categoryPath": [
{
//more data
},
{
//more data
}
]
}]
I would like to convert it to csv so that the selected property names become csv headers and their value (depth 1 only) become data.
e.g
id,itemName,sellerId
12345,"some string",123
12346,"some other string",234
I've tried using hundreds of variations of
cat file.json | convertfrom-json | convertto-csv
but none have worked. All I get is csv data with objects names/types and I can't figure out how to make it use only selected properties of each object from json data.
In short you need to do something like this:
(Get-Content file.json -Raw | ConvertFrom-Json) | Select id,itemName,sellerId | Convertto-CSV -NoTypeInformation
The first problem was that Get-Content was passing individual lines to ConvertFrom-Json which is not what it wants. Using the -Raw switch passes it in its entirety.
The (Get-Content file.json -Raw | ConvertFrom-Json) needs to be in parentheses as that allows us to continue with the pipe. The properties are not accessible without doing this. It looks like it is trying to pass the entire object instead of its individual parts down the pipe.
-NoTypeInformation removes lines like this
#TYPE Selected.System.Management.Automation.PSCustomObject
Overview
From a powershell 3 prompt,, I want to call a RESTful service, get some JSON, and pretty-print it. I discovered that if I convert the data to a powershell object, and then convert the powershell object back to json, I get back a nice pretty-printed string. However, if I combine the two conversions into a one-liner with a pipe I will get a different result.
TL;DR: this:
PS> $psobj = $orig | ConvertFrom-JSON
PS> $psobj | ConvertTo-JSON
... gives me different result than this:
PS> $orig | ConvertFrom-JSON | ConvertTo-JSON
Original data
[
{
"Type": "1",
"Name": "QA"
},
{
"Type": "2",
"Name": "whatver"
}
]
Doing the conversion in two steps
I'm going to remove the whitespace (so it fits on one line...), convert it to a powershell object, and then convert it back to JSON. This works well, and gives me back the correct data:
PS> $orig = '[{"Type": "1","Name": "QA"},{"Type": "2","Name": "DEV"}]'
PS> $psobj = $orig | ConvertFrom-JSON
PS> $psobj | ConvertTo-JSON
[
{
"Type": "1",
"Name": "QA"
},
{
"Type": "2",
"Name": "DEV"
}
]
Combining the two steps with a pipe
However, if I combine those last two statements into a one-liner, I get a different result:
PS> $orig | ConvertFrom-JSON | ConvertTo-JSON
{
"value": [
{
"Type": "1",
"Name": "QA"
},
{
"Type": "2",
"Name": "DEV"
}
],
"Count": 2
}
Notice the addition of the keys "value" and "Count". Why is there a difference? I'm sure it has something to do with the desire to return JSON object rather than a JSON array, but I don't understand why the way I do the conversion affects the end result.
Note: The problem still exists as of Windows PowerShell v5.1, but PowerShell Core (v6+) is not affected.
The existing answers provide an effective workaround - enclosing $orig | ConvertFrom-JSON in (...) - but do not explain the problem correctly; also, the workaround cannot be used in all situations.
As for why use of an intermediate variable did not exhibit the problem:
The in-pipeline distinction between emitting an array's elements one by one vs. the array as a whole (as a single object) is nullified if you collect the output in a variable; e.g., $a = 1, 2 is effectively equivalent to $a = Write-Output -NoEnumerate 1, 2, even though the latter originally emits array 1, 2 as a single object; however, the distinction matters if further pipeline segments process the objects - see below.
The problematic behavior is a combination of two factors:
ConvertFrom-Json deviates from normal output behavior by sending arrays as single objects through the pipeline. That is, with a JSON string representing an array, ConvertFrom-Json sends the resulting array of objects as a single object through the pipeline.
You can verify ConvertFrom-Json's surprising behavior as follows:
PS> '[ "one", "two" ]' | ConvertFrom-Json | Get-Member
TypeName: System.Object[] # !! should be: System.String
...
If ConvertFrom-Json passed its output through the pipeline one by one - as cmdlets normally do - Get-Member would instead return the (distinct) types of the items in the collection, which is [System.String] in this case.
Enclosing a command in (...) forces enumeration of its output, which is why ($orig | ConvertFrom-Json) | ConvertTo-Json is an effective workaround.
Whether this behavior - which is still present in PowerShell Core too - should be changed is being debated in this GitHub issue.
The System.Array type - the base type for all arrays - has a .Count property defined for it via PowerShell's ETS (Extended Type System - see Get-Help about_Types.ps1xml), which causes ConvertTo-Json to include that property in the JSON string it creates, with the array elements included in a sibling value property.
This happens only when ConvertTo-Json sees an array as a whole as an input object, as produced by ConvertFrom-Json in this case; e.g., , (1, 2) | ConvertTo-Json surfaces the problem (a nested array whose inner array is sent as a single object), but
1, 2 | ConvertTo-Json does not (the array elements are sent individually).
This ETS-supplied .Count property was effectively obsoleted in PSv3, when arrays implicitly gained a .Count property due to PowerShell now surfacing explicitly implemented interface members as well, which surfaced the ICollection.Count property (additionally, all objects were given an implicit .Count property in an effort to unify the handling of scalars and collections).
Sensibly, this ETS property has therefore been removed in PowerShell Core, but is still present in Windows PowerShell v5.1 - see below for a workaround.
Workaround (not needed in PowerShell Core)
Tip of the hat, as many times before, to PetSerAl.
Note: This workaround is PSv3+ by definition, because the Convert*-Json cmdlets were only introduced in v3.
Given that the ETS-supplied .Count property is (a) the cause of the problem and (b) effectively obsolete in PSv3+, the solution is to simply remove it before calling ConvertTo-Json - it is sufficient to do this once in a session, and it should not affect other commands:
Remove-TypeData System.Array # Remove the redundant ETS-supplied .Count property
With that, the extraneous .Count and .value properties should have disappeared:
PS> '[ "one", "two" ]' | ConvertFrom-Json | ConvertTo-Json
[
"one",
"two"
]
The above workaround also fixes the problem for array-valued properties; e.g.:
PS> '' | Select-Object #{ n='prop'; e={ #( 1, 2 ) } } | ConvertTo-Json
{
"prop": [
1,
2
]
}
Without the workaround, the value of "prop" would include the extraneous .Count and .value properties as well.
The solution is to wrap the first two operations with parenthesis:
PS C:\> ($orig | ConvertFrom-JSON) | ConvertTo-JSON
[
{
"Type": "1",
"Name": "QA"
},
{
"Type": "2",
"Name": "DEV"
}
]
The parenthesis allow you to grab the output of the first two operations all at once. Without them, powershell will attempt to parse any objects its gets separately. The collection of PSCustomObject resulting from $orig | ConvertFrom-JSON contains two PSCustomObjects for the 1/QA and 2/DEV pairs, so by piping the output of that collection powershell attempts to handle the key/value pairs one-at-a-time.
Using parenthesis is a shorter way of "grouping" that output and allows you to operate on it without making a variable.
First off, why is this happening?
PowerShell automatically wraps multiple objects into a collection called a PSMemberSet that has a Count property on it. It's basically how PowerShell manages arbitrary arrays of objects. What's happening is that the Count property is getting added to the resulting JSON, yielding the undesirable results that you're seeing.
We can prove what I just stated above by doing the following:
$Json = #"
[
{
"Type": "1",
"Name": "QA"
},
{
"Type": "2",
"Name": "DEV"
}
]
"#;
# Deserialize the JSON into an array of "PSCustomObject" objects
$Deserialized = ConvertFrom-Json -InputObject $Json;
# Examine the PSBase property of the PowerShell array
# Note the .NET object type name: System.Management.Automation.PSMemberSet
$Deserialized.psbase | Get-Member;
Here is the output from the above
TypeName: System.Management.Automation.PSMemberSet
Name MemberType Definition
---- ---------- ----------
Add Method int IList.Add(System.Object value)
Address Method System.Object&, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 Address(int )
Clear Method void IList.Clear()
......
......
Count Property int Count {get;}
You can work around this behavior by referencing the SyncRoot property of the PSMemberSet (which implements the ICollection .NET interface), and passing the value of that property to ConvertTo-Json.
Here is a complete, working example:
$Json = #"
[
{
"Type": "1",
"Name": "QA"
},
{
"Type": "2",
"Name": "DEV"
}
]
"#;
($Json | ConvertFrom-Json) | ConvertTo-Json;
The correct (expected) output will be displayed, similar to the following:
[
{
"Type": "1",
"Name": "QA"
},
{
"Type": "2",
"Name": "DEV"
}
]
[
{
"Type": "1",
"Name": "QA"
},
{
"Type": "2",
"Name": "DEV"
}
]
I've encountered the same problem here. I was able to resolve it by using a ForEach-Object and a PSCustomObject. Hope it helps you.
'' | ForEach-Object { [PSCustomObject]#{ prop= #( 1, 2 ) }} | ConvertTo-Json
It gives this, which is the expected behavior and using cleaner approach:
{
"prop": [
1,
2
]
}
Consider JSON in this format :
"Stuffs": [
{
"Name": "Darts",
"Type": "Fun Stuff"
},
{
"Name": "Clean Toilet",
"Type": "Boring Stuff"
}
]
In PowerShell 3, we can obtain a list of Stuffs :
$JSON = Get-Content $jsonConfigFile | Out-String | ConvertFrom-Json
Assuming we don't know the exact contents of the list, including the ordering of the objects, how can we retrieve the object(s) with a specific value for the Name field ?
Brute force, we could iterate through the list :
foreach( $Stuff in $JSON.Stuffs ) {
But I am hopeful there exists a more direct mechanism ( similar to Lync or Lambda expressions in C# ).
$json = #"
{
"Stuffs":
[
{
"Name": "Darts",
"Type": "Fun Stuff"
},
{
"Name": "Clean Toilet",
"Type": "Boring Stuff"
}
]
}
"#
$x = $json | ConvertFrom-Json
$x.Stuffs[0] # access to Darts
$x.Stuffs[1] # access to Clean Toilet
$darts = $x.Stuffs | where { $_.Name -eq "Darts" } #Darts
I just asked the same question here: https://stackoverflow.com/a/23062370/3532136
It has a good solution. I hope it helps ^^.
In resume, you can use this:
The Json file in my case was called jsonfile.json:
{
"CARD_MODEL_TITLE": "OWNER'S MANUAL",
"CARD_MODEL_SUBTITLE": "Configure your download",
"CARD_MODEL_SELECT": "Select Model",
"CARD_LANG_TITLE": "Select Language",
"CARD_LANG_DEVICE_LANG": "Your device",
"CARD_YEAR_TITLE": "Select Model Year",
"CARD_YEAR_LATEST": "(Latest)",
"STEPS_MODEL": "Model",
"STEPS_LANGUAGE": "Language",
"STEPS_YEAR": "Model Year",
"BUTTON_BACK": "Back",
"BUTTON_NEXT": "Next",
"BUTTON_CLOSE": "Close"
}
Code:
$json = (Get-Content "jsonfile.json" -Raw) | ConvertFrom-Json
$json.psobject.properties.name
Output:
CARD_MODEL_TITLE
CARD_MODEL_SUBTITLE
CARD_MODEL_SELECT
CARD_LANG_TITLE
CARD_LANG_DEVICE_LANG
CARD_YEAR_TITLE
CARD_YEAR_LATEST
STEPS_MODEL
STEPS_LANGUAGE
STEPS_YEAR
BUTTON_BACK
BUTTON_NEXT
BUTTON_CLOSE
Thanks to mjolinor.
David Brabant's answer led me to what I needed, with this addition:
x.Stuffs | where { $_.Name -eq "Darts" } | Select -ExpandProperty Type
Hows about this:
$json=Get-Content -Raw -Path 'my.json' | Out-String | ConvertFrom-Json
$foo="TheVariableYourUsingToSelectSomething"
$json.SomePathYouKnow.psobject.properties.Where({$_.name -eq $foo}).value
which would select from json structured
{"SomePathYouKnow":{"TheVariableYourUsingToSelectSomething": "Tada!"}
This is based on this accessing values in powershell SO question
. Isn't powershell fabulous!
In regards to PowerShell 5.1 ...
Operating off the assumption that we have a file named jsonConfigFile.json with the following content from your post:
{
"Stuffs": [
{
"Name": "Darts",
"Type": "Fun Stuff"
},
{
"Name": "Clean Toilet",
"Type": "Boring Stuff"
}
]
}
This will create an ordered hashtable from a JSON file to help make retrieval easier:
$json = [ordered]#{}
(Get-Content "jsonConfigFile.json" -Raw | ConvertFrom-Json).PSObject.Properties |
ForEach-Object { $json[$_.Name] = $_.Value }
$json.Stuffs will list a nice hashtable, but it gets a little more complicated from here. Say you want the Type key's value associated with the Clean Toilet key, you would retrieve it like this:
$json.Stuffs.Where({$_.Name -eq "Clean Toilet"}).Type
It's a pain in the ass, but if your goal is to use JSON on a barebones Windows 10 installation, this is the best way to do it as far as I've found.
This is my json data:
[
{
"name":"Test",
"value":"TestValue"
},
{
"name":"Test",
"value":"TestValue"
}
]
Powershell script:
$data = Get-Content "Path to json file" | Out-String | ConvertFrom-Json
foreach ($line in $data) {
$line.name
}