In a devops pipeline Im using a powershell script in a YAML file to read a json parameter file and push these values into YAML variables.
I can successfully read and extract the parameters
ScriptType: InlineScript
Inline: |
$params = Get-Content main.parameters.json -Raw | ConvertFrom-Json
$clientname = Out-String -InputObject $params.parameters.clientname
$clientname = $clientname.ToLower()
but when i try to set the variables using
Write-Host "##vso[task.setvariable variable=testvar2]$clientname"
it does not set the testvar2 to the $clientname value - however if i explicitly assign a string value to the variable $clientname = 'pickles' - this does update the variable.
The $clientname type returns as a string. Is there something to do with how the json keys are returned?
full code img
output img
EDIT:
I can read the json absolutely fine - $clientname is populated with the correct value - the issue is when i try to assign it to a YAML variable using task.setvariable. If you look at the full code the $newclientname works as expected e.g populates the yAML variable - $clientname does not.
Related
I have a myJson.json that look like this:
{
"FirewallGroupsToEnable": [
"Remote Event Log Management",
"Windows Remote Management",
"Performance Logs and Alerts",
"File and Printer Sharing",
"Windows Management Instrumentation (WMI)"
],
"MemoryStartupBytes": "3GB"
}
I'd like to serialize it as a string and then set it as a variable to be used by other tasks. If there is a better way to use this file inside a pipeline please let me know.
I'm serializing it and setting it like this:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
$Configs= Get-Content -Path $(Build.SourcesDirectory)\sources\myJson.json -Raw | ConvertFrom-Json
Write-Host "##vso[task.setvariable variable=Configs]$Configs"
In the following task, I am running a PowerShell script.
- task: PowerShell#2
displayName: 'myTask'
inputs:
targetType: 'filePath'
filePath: 'sources\myScript.ps1'
pwsh: true
I'm using the variable in my script like this:
$env:Configs
[Configs]$envConfigs = ConvertFrom-Json -InputObject $env:Configs -ErrorAction Stop
The Configs is a class that is being imported at the top of the script like so Using module .\Configs.psm1. I know it's being read because if it wasn't the error would be about a missing type.
Configs.psm1
class Configs
{
[string[]]$FirewallGroupsToEnable
[string]$MemoryStartupBytes
}
This is what I get in the pipeline.
##[debug]Processed: ##vso[task.setvariable variable=Configs]#{FirewallGroupsToEnable=System.Object[]; MemoryStartupBytes=3GB}
#{FirewallGroupsToEnable=System.Object[]; MemoryStartupBytes=3GB}
Cannot convert the "#{FirewallGroupsToEnable=System.Object[]; MemoryStartupBytes=3GB}" value of type "System.String" to type "Configs".
I've always casted a deserialized JSON into custom types like this and it always worked. But right now there is something wrong!
I tried to remove ConvertFrom-Json while serializing the JSON (before setting it as a variable) but it doesn't serialize it right. It shows like this in the pipeline:
It looks like it's only getting the first curly braces!
So, how do I serialize a JSON regardless of its depth into the pipeline to be used in later tasks inside a script file?
You could pass the json data via a file, here is my YAML snippet that can pass the full JSON content successfully in next task.
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
$Configs= Get-Content -Path $(Build.SourcesDirectory)\wit.json
$Configs | Out-File Test.json
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
$Input = Get-Content Test.json
Write-Host $Input
Here are similar ticket about pass Json variable for reference.
As it turns out, you can't Write-Host multi-line variables even if they were strings due to the fact of PowerShell formatting. Hence, I had to ConvertFrom-Json and then ConvertTo-Json -Compress to get my string in one line.
$Configs= Get-Content -Path $(Build.SourcesDirectory)\sources\myJson.json -Raw | ConvertFrom-Json | ConvertTo-Json -Compress
When loading an object from a json file one can normally set the value on properties and write the file back out like so:
$manifest = (gc $manifestPath) | ConvertFrom-Json -AsHashtable
$manifest.name = "$($manifest.name)-sxs"
$manifest | ConvertTo-Json -depth 100 | Out-File $manifestPath -Encoding utf8NoBOM
But if the source json file contains comments, the object's properties can't be set:
// *******************************************************
// GENERATED FILE - DO NOT EDIT DIRECTLY
// *******************************************************
{
"name": "PublishBuildArtifacts"
}
Running the code above throws an error:
$manifest
id : 1D341BB0-2106-458C-8422-D00BCEA6512A
name : PublishBuildArtifacts
friendlyName : ms-resource:loc.friendlyName
description : ms-resource:loc.description
category : Build
visibility : {Build}
author : Microsoft Corporation
version : #{Major=0; Minor=1; Patch=71}
demands : {}
inputs : {#{name=CopyRoot; type=filePath; label=ms-resource:loc.input.label.CopyRoot; defaultValue=;
required=False; helpMarkDown=Root folder to apply copy patterns to. Empty is the root of the
repo.}, #{name=Contents; type=multiLine; label=ms-resource:loc.input.label.Contents;
defaultValue=; required=True; helpMarkDown=File or folder paths to include as part of the
artifact.}, #{name=ArtifactName; type=string; label=ms-resource:loc.input.label.ArtifactName;
defaultValue=; required=True; helpMarkDown=The name of the artifact to create.},
#{name=ArtifactType; type=pickList; label=ms-resource:loc.input.label.ArtifactType;
defaultValue=; required=True; helpMarkDown=The name of the artifact to create.; options=}…}
instanceNameFormat : Publish Artifact: $(ArtifactName)
execution : #{PowerShell=; Node=}
$manifest.name
PublishBuildArtifacts
$manifest.name = "sxs"
InvalidOperation: The property 'name' cannot be found on this object. Verify that the property exists and can be set.
When I strip the comments, I can overwrite the property.
Is there a way I can coax PowerShell to ignore the comments while loading the json file/convert the object and generate a writable object?
I'm not sure if this is intended, but seems like ConvertFrom-Json is treating the comments on the Json as $null when converting it to an object. This only happens if it's receiving an object[] from pipeline, with a string or multi-line string it works fine.
A simple way to demonstrate this using the exact same Json posted in the question:
$contentAsArray = Get-Content test.json | Where-Object {
-not $_.StartsWith('/')
} | ConvertFrom-Json -AsHashtable
$contentAsArray['name'] = 'hello' # works
Here you can see the differences and the workaround, it is definitely recommended to use -Raw on Get-Content so you're passing a multi-line string to ConvertFrom-Json:
$contentAsString = Get-Content test.json -Raw | ConvertFrom-Json -AsHashtable
$contentAsArray = Get-Content test.json | ConvertFrom-Json -AsHashtable
$contentAsString.PSObject, $contentAsArray.PSObject | Select-Object TypeNames, BaseObject
TypeNames BaseObject
--------- ----------
{System.Collections.Hashtable, System.Object} {name}
{System.Object[], System.Array, System.Object} {$null, System.Collections.Hashtable}
$contentAsArray['name'] # null
$null -eq $contentAsArray[0] # True
$contentAsArray[1]['name'] # PublishBuildArtifacts
$contentAsArray[1]['name'] = 'hello'
$contentAsArray[1]['name'] # hello
Santiago Squarzon's helpful answer shows an effective solution and analyzes the symptom. Let me complement it with some background information.
tl;dr
You're seeing a variation of a known bug, still present as of PowerShell 7.2.1, where a blank line or a single-line comment as the first input object unexpectedly causes $null to be emitted (first) - see GitHub issue #12229.
Using -Raw with Get-Content isn't just a workaround, it is the right - and faster - thing to do when piping a file containing JSON to be parsed as whole to ConvertFrom-Json
For multiple input objects (strings), ConverFrom-Json has a(n unfortunate) heuristic built in that tries to infer whether the multiple strings represent either (a) the lines of a single JSON document or (b) separate, individual JSON documents, each on its own line, as follows:
If the first input string is valid JSON by itself, (b) is assumed, and an object representing the parsed JSON ([pscustomobject] or , with -AsHashtable, [hastable], or an array of either) is output for each input string.
Otherwise, (a) is assumed and all input strings are collected first, in a multi-line string, which is then parsed.
The aforementioned bug is that if the first string is an empty/blank line or a single-line comment[1] (applies to both // ... comments, which are invariably single-line, and /* ... */ if they happen to be single-line), (b) is (justifiably) assumed, but an extraneous $null is emitted before the remaining (non-blank, non-comment) lines are parsed and their object representation(s) are output.
As a result an array is invariably returned, whose first element is $null - which isn't obvious, but results in subtle changes in behavior, as you've experienced:
Notably, attempting to set a property on what is presumed to be a single object then fails, because the fact that an array is being accessed makes the property access an instance of member-access enumeration - implicitly applying property access to all elements of a collection rather than the collection itself - which only works for getting property values - and fails obscurely when setting is attempted - see this answer for details.
A simplified example:
# Sample output hashtable parsed from JSON
$manifest = #{ name = 'foo' }
# This is what you (justifiably) THOUGHT you were doing.
$manifest.name = 'bar' # OK
# Due to the bug, this is what you actually attempted.
($null, $manifest).name = 'bar' # !! FAILS - member-access enumeration doesn't support setting.
As noted above, the resulting error message - The property 'name' cannot be found on this object. ... - is unhelpful, as it doesn't indicate the true cause of the problem.
Improving it would be tricky, however, as the user's intent is inherently ambiguous: the property name may EITHER be a situationally unsuccessful attempt to reference a nonexistent property of the collection itself OR, as in this case, a fundamentally unsupported attempt at setting properties via member-access enumeration.
Conceivably, the following would help if the target object is a collection (enumerable) from PowerShell's perspective: The property 'name' cannot be found on this collection, and setting a collection's elements' properties by member-access enumeration isn't supported.
[1] Note that comments in JSON are supported in PowerShell (Core) v6+ only.
I'm trying to setup an IIS application pool via PowerShell 7.1.1.
I read configuration from a JSON file into the variable $configuration which is hand over to Windows Powershell because of WebAdministration module which isn't natively supported PS 7.1.1.
A script block is defined in the top level function, the configuration is injected as PSCustomObject into the script block and executed in Windows PowerShell.
function Set-AxisAppPool
{
Write-Message 'Setting up a resource pool for Axis...'
$executeInWindowsPowerShellForCompatibilityReasons = {
param (
[Parameter(Mandatory)]
[ValidateNotNullOrEmpty()]
[PSCustomObject]
$Configuration
)
Import-Module WebAdministration
Remove-WebAppPool -Name $Configuration.AppPool.Name -Confirm:$false -ErrorAction:SilentlyContinue
New-WebAppPool -Name $Configuration.AppPool.Name -Force | Write-Verbose
$params = #{
Path = "IIS:\AppPools\$($Configuration.AppPool.Name)"
Name = 'processModel'
Value = #{
userName = $Configuration.AxisUser.Name
password = $Configuration.AxisUser.Password
identitytype = 'SpecificUser'
}
}
Set-ItemProperty #params
}
powershell -NoLogo -NoProfile $executeInWindowsPowerShellForCompatibilityReasons -Args $configuration # This is a line 546
}
When the configuration JSON file exceeds a certain level, PowerShell can't pass through this deserialized JSON, the PSCustomObject, into Windows PowerShell.
Program 'powershell.exe' failed to run: The Process object must have the UseShellExecute property set to false in order to use environment
| variables.At C:\Users\JohnDoe\Desktop\Localhost automatization\Set-AxisEnvironment.ps1:546 char:5 + powershell -NoLogo -NoProfile
| $executeInWindowsPowerShellForCompa … +
It literally work with level n of objects in the JSON and it doesn't with n+1 level of objects in the configuration JSON. The JSON schema is validated, deserialization works as expected.
When I use Start-Process for invoking Windows PowerShell, I receive a different problem. Does anybody have any hint on this one?
Update
This seems to be a bug in PowerShell.
I suspect it is the size of the argument list overflowing into other fields, thus giving you weird error messages. From Start Process:
The length of the string assigned to the Arguments property must
be less than 32,699.
If you are passing a configuration that is larger than 32,699 characters (including spaces), then that likely may be your problem. It would likely take those first 32,699 characters then continue to the next field, -UseShellExecute which would receive a character which is not zero or false, and thus true. This would trip the "wrong", and misleading error message.
I'm quite new to powershell and just need it for a small task so please excuse my complete and utter ineptitude for the language. I was wondering if it were possible to form a json object based off environment variables and a variable that has already been declared earlier in my script. The variable that was already declared is based off a json config named optionsConfig.json and the contents of that file are here.
{"test1": ["options_size", "options_connection", "options_object"],
"test2":["options_customArgs", "options_noUDP", "options_noName"]}
The purpose of the $Options variable in the code below is to take each element in the list value for the respective test and assume that those elements are environment variables in the system, then find their values and form a dictionary object that will be used in the json.
Here is what I have so far.
# Read the JSON file into a custom object.
$configObj = Get-Content -Raw optionsConfig.json |
ConvertFrom-Json
# Retrieve the environment variables whose
# names are listed in the $env:test property
# as name-value pairs.
Get-Item -Path env:* -Include $configObj.$env:testTool
$Options = Get-Item -Path env:* -Include $configObj.$env:testTool |
% {$hash = #{}} {$hash[$_.Name]=$_.Value} {$hash}
The $Options variable looks like so when converted to json
{
"options_size": "default",
"options_object": "forward open",
"options_connection": "connected"
}
I have a few other environment variable values that I would like to be a part of the json object. Those 3 other environment variables I would like the value of are listed below.
$Env.testTool = "test1"
$Env.RecordName = "Record1"
$Env.Target = "Target1"
How would I construct a powershell statement to get the json object to be formatted like this? -
data = {"test": $Env.testTool, "target": "$Env.Target",
"options": "$Options", "RecordName': "$Env.RecordName"}
The keys are all predefined strings and $Options is the dict object from up above. Am I able to form a Json object like this in powershell and how would it be done? Any help would be appreciated. This appears to be the last step in my struggle with powershell.
Here is what I have done.
$jObj = [ordered]#{test= $Env:testTool}
When I change this variable to $jObj = [ordered]#{test= $Env:testTool,options= $Options} I get an error saying missing expression after ','
When I change this variable to $jObj = [ordered]#{test= $Env:testTool,options= $Options} I get an error saying missing expression after ','
Entries of a hashtable literal (#{ ... } or, in its ordered form, [ordered] #{ ... }) must be separated:
either by newlines (each entry on its own line)
or by ; if placed on the same line.
Thus, the following literals are equivalent:
# Multiline form
#{
test= $env:testTool
RecordName= $env:RecordName
Target= $env.Target
options=$Options
}
# Single-line form; separator is ";"
#{ test= $env:testTool; RecordName= $env:RecordName; Target= $env.Target; options=$Options }
Get-Help about_Hashtables has more information.
$jObj = #{test= $env:testTool
RecordName= $env:RecordName
Target= $env.Target
options=$Options}
$jObj | ConvertTo-Json | Set-Content jsonStuff.json
JsonStuff.json is the new json file for the new json object. This syntax for forming $jObj seems to have done the trick.
Following on from a question yesterday (JSON and references to other JSON objects).
Is it possible to merge JSON objects at runtime in a similar fashion?
In my test.json I wish to insert the $Defaults.wimos object into WIMS.wimos at runtime similar to what I did for the $Paths.drive value in $Defaults.
{
Paths: {drive: "W:"},
Defaults: {wimos: {dstdrive: "$($Paths.drive)"}
},
WIMS: {
winos: "$($Defaults.wimos)",
wimre: {dstdrive: "$($Paths.drive)"}
}
}
In the following code I cannot work out the syntax to have the object replaced at runtime.
$JSONConfig="test.json"
$rawJSON = (Get-Content $JSONConfig -Raw)
$pathJSON = $rawJSON | ConvertFrom-Json
#
# Load Paths from JSON into $Paths
#
$Paths=$pathJSON.Paths
#
# Merge JSON objects to have the defaults replaced
#
$DefaultsJSON=($ExecutionContext.InvokeCommand.ExpandString($rawJSON)) | ConvertFrom-Json
#
# Load Defaults into $Defaults
#
$Defaults=$DefaultsJSON.Defaults
#
# Merge JSON objects to have the System replaced
#
$JSON = ($ExecutionContext.InvokeCommand.ExpandString($rawJSON)) | ConvertFrom-Json
$JSON.WIMS
write-host ("JSON.WIMS.wimre.dstdrive =" + $JSON.WIMS.wimre.dstdrive)
write-host ("JSON.WIMS.wimos.dstdrive =" + $JSON.WIMS.wimos.dstdrive) #This fails to access and print "W:"
I would like to be able to replace $($Defaults.wimos) and be able to query the dstdrive member.
Is this possible to achieve this in powershell? Any suggestions on how?
Thanks
Stuart
There's a couple of issues in your code. First, the value that would be inserted into your template would be interpolated as a string and not a json-object.
To do this, you could use the ConvertTo-json within the template. I'll show you how in a minute.
If you're gonna be recreating and using different "templates" within the same file (like path, defaults etc), I would create a helper function for recreating / filling out the template on your session variables all the time (in order to avoid repeating code).
Create the following method:
function Get-ParsedJsonTemplate(){
return ($ExecutionContext.InvokeCommand.ExpandString((Get-Content $JSONConfig -Raw))) | ConvertFrom-Json
}
Change your template json to the following (showing you that you can use powershell code within your template file:
{
Paths: {drive: "W:"},
Defaults: {wimos: {dstdrive: "$($Paths.drive)"}},
WIMS: {
winos: $(
if($Defaults -and $Defaults.wimos){
$Defaults.wimos|ConvertTo-Json -Compress
} else {
#{"dstdrive"=""}|ConvertTo-Json -compress
}
),
wimre: {dstdrive: "$($Paths.drive)"}
}
}
You also had a typo in either your template or your printout (I see that you used wimos in your "printout" in powershell and "winos" in your template. I modified the code and the following should work for you (given the above test.json):
$JSONConfig="c:\tmp\test.json"
function Get-ParsedJsonTemplate(){
return ($ExecutionContext.InvokeCommand.ExpandString((Get-Content $JSONConfig -Raw))) | ConvertFrom-Json
}
$pathJSON = Get-ParsedJsonTemplate;
$Paths=$pathJSON.Paths
$DefaultsJSON=Get-ParsedJsonTemplate;
$Defaults=$DefaultsJSON.Defaults
$JSON = Get-ParsedJsonTemplate;
$JSON.WIMS
write-host ("JSON.WIMS.wimre.dstdrive =" + $JSON.WIMS.wimre.dstdrive)
write-host ("JSON.WIMS.wimos.dstdrive =" + $JSON.WIMS.winos.dstdrive)
The reason your script was failing was due to the fact that $Defaults.wimos was not the string { dstdrive: "W:" } it was instead a PSCustomObject which would be interpolated to a the value #{drive=W:}. To obtain the correct behavior, you need to convert the $Defaults.wimos value back to a json string.
$Defaults = $DefaultsJSON.Defaults
$Defaults.wimos = $Defaults.wimos | ConvertTo-Json -Compress