Pass one of the member as array for JSON in powershell - json

Here is a small Powershell code snippet:
$users = New-Object System.Collections.ArrayList
$userAsJson = '
{
"name" : "abc",
"companies" : ["facebook", "google"]
}'
$user = $userAsJson | ConvertFrom-Json
$null = $users.Add($user)
$users | ConvertTo-Json -Depth 5
It gives me the following expected output:
{
"name": "abc",
"companies": [
"facebook",
"google"
]
}
Now, I'm dynamically trying to create the companies list. I tried all possible things which I can think of. Here is what I have tried:
$company = New-Object System.Collections.ArrayList
$null = $company.Add('facebook')
$null = $company.Add('google')
$b = $company.ToArray()
$users = New-Object System.Collections.ArrayList
$userAsJson = '
{
"name" : "abc",
"companies" : $b
}'
$user = $userAsJson | ConvertFrom-Json
$null = $users.Add($user)
$users | ConvertTo-Json -Depth 5
Can anyone suggest me what is the best way to achieve it?

PowerShell's strength is in staying in the realm objects, until the time comes to interface with the outside world, such as when writing to a file or creating a string representation of these objects.
In your case that means:
# Array of companies; statically constructed here, but creating it
# dynamically works just as well.
$company = (
'facebook',
'google'
)
# Initialize the output collection.
# Note: Creating a [System.Collections.ArrayList] instance is
# advisable for building up *large* arrays *incrementally*.
# For smallish arrays, using regular PowerShell arrays will do; e.g.:
# $users = #() # initialize array
# $users += ... # append to array, but be aware that a *new* array
# is created behind the scenes every time.
$users = New-Object System.Collections.ArrayList
# Add a user based on the $company array defined above as
# a [pscustomobject]
$null = $users.Add(
[pscustomobject] #{
name = 'abc'
companies = $company
}
)
# After all users have been added *as objects*, convert them to JSON.
$users | ConvertTo-Json -Depth 5
The above yields (based on a single object having been added; with more, a JSON array would be output):
{
"name": "abc",
"companies": [
"facebook",
"google"
]
}

Related

Error adding to array: Cannot find an overload for "Add" and the argument count: "1"

I can't for the life of me work out how to add elements to an array inside a foreach loop.
I keep getting the error:
8 | $logs.Add(#{
| ~~~~~~~~~~~~
| Cannot find an overload for "Add" and the argument count: "1".
MethodException: C:\Users\michael.dawson\OneDrive - Sage Software, Inc\Documents\Source_Control\secrets-dashboard-script\function\Calculate\recreate-error2.ps1:8:5
I can get it to work outside of a foreach, but I've been scratching my head for ages :-(
Here's the code:
$logs = New-Object System.Collections.ArrayList
$logs = $null
$logs = #{}
Get-Service | ForEach-Object {
$appCreds = $_
$logs.Add(#{
"test" = $appCreds.DisplayName;
"message" = "blah"
})
}
$logs | ConvertTo-Json -Depth 10
Appreciate any help guys. Cheers.
I'm expecting some JSON a bit like this:
{
"message": "blah",
"test": "Optimise drives"
},
{
"message": "blah",
"test": "Dell Client Management Service"
},
{
"message": "blah",
"test": "DeviceAssociationBroker_1592b90"
}
Try the following:
$logs =
Get-Service |
ForEach-Object {
#{
test = $_.DisplayName;
message = "blah"
}
}
$logs | ConvertTo-Json -Depth 10
The above takes advantage of:
Being able to use the results of entire commands as expressions, relying on PowerShell implicitly collecting the command's output objects in the target variable, $logs - as-is in the case of a single output object, and as an array (of type [object[]] in the case of two ore more output objects.
PowerShell's implicit output behavior, where the output (result) from a command or expression is implicitly output.
As for what you tried:
$logs = New-Object System.Collections.ArrayList
$logs = $null
$logs = #{}
Each statement overrides the previous one, so that $logs ends up containing #{}, i.e. an empty hashtable.
Thus, a later $logs.Add(...) call with a single argument predictably fails, because a hashtable's .Add() method requires two arguments, namely a key and and a value, separately - which isn't what you intended, however.
While you could have avoided this problem by making do with just $logs = New-Object System.Collections.ArrayList as the variable initialization, given that that type's .Add() method accepts any single object to append to the list, the solution at the top shows a PowerShell-idiomatic solution that is both simpler and more efficient.
I am in rush but, I want to help. Try the example below.
$logs = New-Object System.Collections.ArrayList
Get-Service | ForEach-Object {
$appCreds = $_
$logs.Add(#{
"test" = $appCreds.DisplayName;
"message" = "blah"
}) | out-null
}
$logs | ConvertTo-Json -Depth 10

I want to create json file by substituting values from environment variables in a json template file

One requirement of mine is - Using windows, not use any tools not already available as part of aws cli or windows
For example, I have this json file test.json with below content:
"My number is $myvar"
I read this into a powershell variable like so:
$myobj=(get-content .\test.json | convertfrom-json)
$myvar=1
From here, I would like to do something with this $myobj which will enable me to get this output:
$myobj | tee json_with_values_from_environment.json
My number is 1
I got some limited success with iex, but not sure if it can be made to work for this example
You can use $ExecutionContext.InvokeCommand.ExpandString()
$myobj = '{test: "My number is $myvar"}' | ConvertFrom-Json
$myvar = 1
$ExecutionContext.InvokeCommand.ExpandString($myobj.test)
Output
My number is 1
Here is one way to do it using the Parser to find all VariableExpressionAst and replace them with the values in your session.
Given the following test.json:
{
"test1": "My number is $myvar",
"test2": {
"somevalue": "$env:myothervar",
"someothervalue": "$anothervar !!"
}
}
We want to find and replace $myvar, $myothervar and $anothervar with their corresponding values defined in the current session, so the code looks like this (note that we do the replacement before converting the Json string into an object, this way is much easier):
using namespace System.Management.Automation.Language
$isCore7 = $PSVersionTable.PSVersion -ge '7.2'
# Define the variables here
$myvar = 10
$env:myothervar = 'hello'
$anothervar = 'world'
# Read the Json
$json = Get-Content .\test.json -Raw
# Now parse it
$ast = [Parser]::ParseInput($json, [ref] $null, [ref] $null)
# Find all variables in it, and enumerate them
$ast.FindAll({ $args[0] -is [VariableExpressionAst] }, $true) |
Sort-Object { $_.Extent.Text } -Unique | ForEach-Object {
# now replace the text with the actual value
if($isCore7) {
# in PowerShell Core is very easy
$json = $json.Replace($_.Extent.Text, $_.SafeGetValue($true))
return
}
# in Windows PowerShell not so much
$varText = $_.Extent.Text
$varPath = $_.VariablePath
# find the value of the var (here we use the path)
$value = $ExecutionContext.SessionState.PSVariable.GetValue($varPath.UserPath)
if($varPath.IsDriveQualified) {
$value = $ExecutionContext.SessionState.InvokeProvider.Item.Get($varPath.UserPath).Value
}
# now replace the text with the actual value
$json = $json.Replace($varText, $value)
}
# now we can safely convert the string to an object
$json | ConvertFrom-Json
If we were to convert it back to Json to see the result:
{
"test1": "My number is 10",
"test2": {
"somevalue": "hello",
"someothervalue": "world !!"
}
}

Find and Replace Nested JSON Values with Powershell

I have an appsettings.json file that I would like to transform with a PowerShell script in a VSTS release pipeline PowerShell task. (BTW I'm deploying a netstandard 2 Api to IIS). The JSON is structured like the following:
{
"Foo": {
"BaseUrl": "http://foo.url.com",
"UrlKey": "12345"
},
"Bar": {
"BaseUrl": "http://bar.url.com"
},
"Blee": {
"BaseUrl": "http://blee.url.com"
}
}
I want to replace BaseUrl and, if it exists, the UrlKey values in each section which are Foo, Bar and Blee. (Foo:BaseUrl, Foo:UrlKey, Bar:BaseUrl, etc.)
I'm using the following JSON structure to hold the new values:
{
"##{FooUrl}":"$(FooUrl)",
"##{FooUrlKey}":"$(FooUrlKey)",
"##{BarUrl}":"$(BarUrl)",
"##{BleeUrl}":"$(BleeUrl)"
}
So far I have the following script:
# Get file path
$filePath = "C:\mywebsite\appsettings.json"
# Parse JSON object from string
$jsonString = "$(MyReplacementVariablesJson)"
$jsonObject = ConvertFrom-Json $jsonString
# Convert JSON replacement variables object to HashTable
$hashTable = #{}
foreach ($property in $jsonObject.PSObject.Properties) {
$hashTable[$property.Name] = $property.Value
}
# Here's where I need some help
# Perform variable replacements
foreach ($key in $hashTable.Keys) {
$sourceFile = Get-Content $filePath
$sourceFile -replace $key, $hashTable[$key] | Set-Content $filePath
Write-Host 'Replaced key' $key 'with value' $hashTable[$key] 'in' $filePath
}
Why are you defining your replacement values as a JSON string? That's just going to make your life more miserable. If you're defining the values in your script anyway just define them as hashtables right away:
$newUrls = #{
'Foo' = 'http://newfoo.example.com'
'Bar' = 'http://newbaz.example.com'
'Blee' = 'http://newblee.example.com'
}
$newKeys = #{
'Foo' = '67890'
}
Even if you wanted to read them from a file you could make that file a PowerShell script containing those hashtables and dot-source it. Or at least define the values as lists of key=value lines in text files, which can easily be turned into hashtables:
$newUrls = Get-Content 'new_urls.txt' | Out-String | ConvertFrom-StringData
$newKeys = Get-Content 'new_keys.txt' | Out-String | ConvertFrom-StringData
Then iterate over the top-level properties of your input JSON data and replace the nested properties with the new values:
$json = Get-Content $filePath | Out-String | ConvertFrom-Json
foreach ($name in $json.PSObject.Properties) {
$json.$name.BaseUrl = $newUrls[$name]
if ($newKeys.ContainsKey($name)) {
$json.$name.UrlKey = $newKeys[$name]
}
}
$json | ConvertTo-Json | Set-Content $filePath
Note that if your actual JSON data has more than 2 levels of hierarchy you'll need to tell ConvertTo-Json via the parameter -Depth how many levels it's supposed to convert.
Side note: piping the Get-Content output through Out-String is required because ConvertFrom-Json expects JSON input as a single string, and using Out-String makes the code work with all PowerShell versions. If you have PowerShell v3 or newer you can simplify the code a little by replacing Get-Content | Out-String with Get-Content -Raw.
Thank you, Ansgar for your detailed answer, which helped me a great deal. Ultimately, after having no luck iterating over the top-level properties of my input JSON data, I settled on the following code:
$json = (Get-Content -Path $filePath) | ConvertFrom-Json
$json.Foo.BaseUrl = $newUrls["Foo"]
$json.Bar.BaseUrl = $newUrls["Bar"]
$json.Blee.BaseUrl = $newUrls["Blee"]
$json.Foo.Key = $newKeys["Foo"]
$json | ConvertTo-Json | Set-Content $filePath
I hope this can help someone else.
To update values of keys at varying depth in the json/config file, you can pass in the key name using "." between the levels, e.g. AppSettings.Setting.Third to represent:
{
AppSettings = {
Setting = {
Third = "value I want to update"
}
}
}
To set the value for multiple settings, you can do something like this:
$file = "c:\temp\appSettings.json"
# define keys and values in hash table
$settings = #{
"AppSettings.SettingOne" = "1st value"
"AppSettings.SettingTwo" = "2nd value"
"AppSettings.SettingThree" = "3rd value"
"AppSettings.SettingThree.A" = "A under 3rd"
"AppSettings.SettingThree.B" = "B under 3rd"
"AppSettings.SettingThree.B.X" = "Z under B under 3rd"
"AppSettings.SettingThree.B.Y" = "Y under B under 3rd"
}
# read config file
$data = Get-Content $file -Raw | ConvertFrom-Json
# loop through settings
$settings.GetEnumerator() | ForEach-Object {
$key = $_.Key
$value = $_.Value
$command = "`$data.$key = $value"
Write-Verbose $command
# update value of object property
Invoke-Expression -Command $command
}
$data | ConvertTo-Json -Depth 10 | Out-File $file -Encoding "UTF8"

How to clone a PowerShell PSCustomObject variable to disconnect it from another variable? [duplicate]

I have a powershell script in which I do the following
$somePSObjectHashtables = New-Object Hashtable[] $somePSObject.Length;
$somePSObjects = Import-CSV $csvPath
0..($somePSObject.Length - 1) | ForEach-Object {
$i = $_;
$somePSObjectHashtables[$i] = #{};
$somePSObject[$_].PSObject.Properties | ForEach-Object {
$somePSObjectHashtables[$i][$_.Name] = $_.Value;
}
}
I need to do this because I want to make several distinct copies of the data in the CSV to perform several distinct manipulations. In a sense I'm performing an "INNER JOIN" on the resulting array of PSObject. I can easily iterate through $somePSObjectHashtables with a ForEach-Object and call Hashtable.Clone() on each member of the array. I can then use New-Object PSObject -Property $someHashTable[$i] to get a deep copy of the PSObject.
My question is, is there some easier way of making the deep copy, without an intermediary Hashtable?
Note that here is a shorter, maybe a bit cleaner version of this (that I quite enjoy):
$data = Import-Csv .\test.csv
$serialData = [System.Management.Automation.PSSerializer]::Serialize($data)
$data2 = [System.Management.Automation.PSSerializer]::Deserialize($serialData)
Note:
However, weirdly, it does not keep the ordering of ordered hashtables.
$data = [ordered] #{
1 = 1
2 = 2
}
$serialData = [System.Management.Automation.PSSerializer]::Serialize($data)
$data2 = [System.Management.Automation.PSSerializer]::Deserialize($serialData)
$data2
Will output:
Name Value
---- -----
2 2
1 1
While with other types it works just fine:
$data = [PsCustomObject] #{
1 = 1
2 = 2
}
$data = #(1, 2, 3)
For getting really deep copies we can use binary serialization (assuming that all data are serializable; this is definitely the case for data that come from CSV):
# Get original data
$data = Import-Csv ...
# Serialize and Deserialize data using BinaryFormatter
$ms = New-Object System.IO.MemoryStream
$bf = New-Object System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
$bf.Serialize($ms, $data)
$ms.Position = 0
$data2 = $bf.Deserialize($ms)
$ms.Close()
# Use deep copied data
$data2
Here's an even shorter one that I use as a function:
using namespace System.Management.Automation
function Clone-Object ($InputObject) {
<#
.SYNOPSIS
Use the serializer to create an independent copy of an object, useful when using an object as a template
#>
[psserializer]::Deserialize(
[psserializer]::Serialize(
$InputObject
)
)
}

Split Period-Delimited Nodes To JSON Object

I have many string entries (this are namespace/class trees) that look like the following:
appsystem
appsystem.applications
appsystem.applications.APPactivities
appsystem.applications.APPmanager
appsystem.applications.APPmodels
appsystem.applications.MAPmanager
appsystem.applications.MAPmanager.maphub
appsystem.applications.MAPmanager.mapmanager
appsystem.applications.pagealertsmanager
appsystem.authentication
appsystem.authentication.manager
appsystem.authentication.manager.encryptionmanager
appsystem.authentication.manager.sso
appsystem.authentication.manager.tokenmanager
But, I need the final output to be like:
{
"name": "appsystem",
"children": [
{
"name": "applications",
"children": [
{"name": "APPactivities"},
{"name": "APPmanager"},
{"name": "APPmodels"},
{"name": "MAPmanager",
"children": [
{"name": "maphub"},
{"name": "mapmanager"}
]},
{"name": "pagealertsmanager"}
]
},
{
"name": "authentication",
"children": [
{"name": "manager",
"children": [
{"name": "encryptionmanager"},
{"name": "sso"},
{"name": "tokenmanager"}
]}
]
}
]
}
The total nodes can be any number.
I am assuming I am going to need recursion but I am at a loss on where even to begin.
This builds up nested lists, PowerShell ConvertTo-JSON flattens the outer list.
You can change the $Line in $s to $line in (Get-Content input.txt).
But I think this does it:
$s = #'
appsystem
appsystem.applications
appsystem.applications.APPactivities
appsystem.applications.APPmanager
appsystem.applications.APPmodels
appsystem.applications.MAPmanager
appsystem.applications.MAPmanager.maphub
appsystem.applications.MAPmanager.mapmanager
appsystem.applications.pagealertsmanager
appsystem.authentication
appsystem.authentication.manager
appsystem.authentication.manager.encryptionmanager
appsystem.authentication.manager.sso
appsystem.authentication.manager.tokenmanager
'# -split "`r`n"
$TreeRoot = New-Object System.Collections.ArrayList
foreach ($Line in $s) {
$CurrentDepth = $TreeRoot
$RemainingChunks = $Line.Split('.')
while ($RemainingChunks)
{
# If there is a dictionary at this depth then use it, otherwise create one.
$Item = $CurrentDepth | Where-Object {$_.name -eq $RemainingChunks[0]}
if (-not $Item)
{
$Item = #{name=$RemainingChunks[0]}
$null = $CurrentDepth.Add($Item)
}
# If there will be child nodes, look for a 'children' node, or create one.
if ($RemainingChunks.Count -gt 1)
{
if (-not $Item.ContainsKey('children'))
{
$Item['children'] = New-Object System.Collections.ArrayList
}
$CurrentDepth = $Item['children']
}
$RemainingChunks = $RemainingChunks[1..$RemainingChunks.Count]
}
}
$TreeRoot | ConvertTo-Json -Depth 1000
Edit: It's too slow? I tried some random pausing profiling and found (not too surprisingly) that it's the inner nested loop, which searches children arrays for matching child nodes, which is being hit too many times.
This is a redesigned version which still builds the tree, and this time it also builds a TreeMap hashtable of shortcuts into the tree, to all the previously build nodes, so it can jump right too them instead of searching the children lists for them.
I made a testing file, some 20k random lines. Original code processed it in 108 seconds, this one does it in 1.5 seconds and the output matches.
$TreeRoot = New-Object System.Collections.ArrayList
$TreeMap = #{}
foreach ($line in (Get-Content d:\out.txt)) {
$_ = ".$line" # easier if the lines start with a dot
if ($TreeMap.ContainsKey($_)) # Skip duplicate lines
{
continue
}
# build a subtree from the right. a.b.c.d.e -> e then d->e then c->d->e
# keep going until base 'a.b' reduces to something already in the tree, connect new bit to that.
$LineSubTree = $null
$TreeConnectionPoint = $null
do {
$lastDotPos = $_.LastIndexOf('.')
$leaf = $_.Substring($lastDotPos + 1)
$_ = $_.Substring(0, $lastDotPos)
# push the leaf on top of the growing subtree
$LineSubTree = if ($LineSubTree) {
#{"name"=$leaf; "children"=([System.Collections.ArrayList]#($LineSubTree))}
} else {
#{"name"=$leaf}
}
$TreeMap["$_.$leaf"] = $LineSubTree
} while (!($TreeConnectionPoint = $TreeMap[$_]) -and $_)
# Now we have a branch built to connect in to the existing tree
# but is there somewhere to put it?
if ($TreeConnectionPoint)
{
if ($TreeConnectionPoint.ContainsKey('children'))
{
$null = $TreeConnectionPoint['children'].Add($LineSubTree)
} else {
$TreeConnectionPoint['children'] = [System.Collections.ArrayList]#($LineSubTree)
}
} else
{ # nowhere to put it, this is a new root level connection
$null = $TreeRoot.Add($LineSubTree)
}
}
$TreeRoot | ConvertTo-Json -Depth 100
(#mklement0's code takes 103 seconds and produces a wildly different output - 5.4M characters of JSON instead of 10.1M characters of JSON. [Edit: because my code allows multiple root nodes in a list which my test file has, and their code does not allow that])
Auto-generated PS help links from my codeblock (if available):
New-Object (in module Microsoft.PowerShell.Utility)
Get-Content (in module Microsoft.PowerShell.Management)
ConvertTo-Json (in module Microsoft.PowerShell.Utility)
To complement TessellatingHeckler's great answer with an alternative implementation that uses a recursive function.
The emphasis is on modularity and terseness, not performance.[1]
# Outer function that loops over all paths and builds up a one or more nested
# hashtables reflecting the path hierarchy, which are converted to JSON on output.
# Note that only a single JSON object is output if all paths share the same root
# component; otherwise, a JSON *array* is output.
function convert-PathsToNestedJsonObject([string[]] $paths) {
$hts = New-Object Collections.ArrayList
$paths.ForEach({
$rootName = $_.split('.')[0]
$ht = $hts.Where({ $_.name -eq $rootName }, 'First')[0]
if (-not $ht) { [void] $hts.Add(($ht = #{})) }
convert-PathToNestedHashtable $ht $_
})
$hts | ConvertTo-Json -Depth 100
}
# Recursive helper function that takes a path such as "appsystem.applications"
# and converts it into a nested hashtable with keys "name" and "children" to
# reflect the path hierarchy.
function convert-PathToNestedHashtable([hashtable] $ht, [string] $path) {
$name, $childName, $rest = $path -split '\.', 3
$ht.name = $name
if ($childName) {
if ($ht.children) {
$htChild = $ht.children.Where({ $_.name -eq $childName }, 'First')[0]
} else {
$ht.children = New-Object Collections.ArrayList
$htChild = $null
}
if (-not $htChild) {
[void] $ht.children.Add(($htChild = #{}))
}
convert-PathToNestedHashtable $htChild "$childName.$rest"
}
}
# Call the outer function with the input paths (assumed to be stored in $paths).
convert-PathsToNestedJsonObject $paths
[1] One deliberate type of optimization is applied, which, however, still keeps the code terse:
PSv4+ offers the (little-known) array methods .ForEach() and .Where(), which are not only noticeably faster than their cmdlet counterparts ForEach-Object and Where-Object, but also offer additional features.
Specifically:
$paths.ForEach({ ... }) is used instead of
$paths | ForEach-Object { ... }
$ht.children.Where({ $_.name -eq $childName }, 'First')[0] is used instead of
$ht.children | Where-Object { $_.name -eq $childName } | Select-Object -First 1