How to pass hashtables / objects from template to runbooks - json

I have written a template and accompanying runbook. The template triggers the runbook. The template and the runbook are working fine, until I try and pass an object to one of the runbook's parameters. The Azure error is:
Error
"content":
{
"status": "Failed",
"error":
{
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.",
"details":
[
{
"code": "BadRequest",
"message": "{\r\n \"code\": \"BadRequest\",\r\n \"message\": \"{\\\"Message\\\":\\\"The request is invalid.\\\",\\\"ModelState\\\":{\\\"job.properties.parameters.MyTags\\\":[\\\"An error has occurred.\\\"]}}\"\r\n}"
}
]
}
}
I'm not sure what is meant by ModelState, however the job.properties.parameters.MyTags is pointing to the troublesome parameter.
I am sure it is something to do with the datatype. It may be because the parameter is being passed as a JSON object and the runbook cannot understand it. I'm more used to passing objects from PowerShell to templates.
Runbook (Update-ResourceGroupTags.ps1)
For testing, the runbook looks like:
param
(
[string]$ResourceGroupId,
$MyTags
)
Write-Output "ResourceGroupId: $ResourceGroupId"
Write-Output "MyTags: $($MyTags | Out-String)"
Write-Output "MyTagsType: $($MyTags.GetType() | Out-String)"
No authentication as it's not required yet and I have deliberately not typed $MyTags, although I have tried [object] and using `[parameter(Mandatory = $true)] just in case there's an unwritten / missed rule.
Goal
In brief, I create a complex object as a variable in the template and I want to pass it to a runbook's parameter as a single object. The values are dynamic and not known until deploy-time. Having to specify each parameter (test 8) in the runbook breaks this requirement.
I start a template with the New-AzureRmResourceGroupDeployment cmdlet.
I create a "complex object" variable:
"rolesTagObject": {
"db": "TestVm1",
"Server": "TestVm1",
"Client": "TestVm1"
}
The template runs a resource of "type" : "jobs".
The Runbook, run by jobs has a parameter MyTags which needs to take roleTagObject which I pass to jobs.properties.parameters.MyTags thus:
,
"parameters": {
"MyTags": "[variable('roleTagObject')]"
}
This is what's not working. If I break it down into each key (Test 8) it works.
My initial thought is to convert it to a single string and pass it to the runbook using json() function, but I don't know of a way to do that in a template.
Template (testRunRunbook.json)
I've put the template in a GIST so as not to make this question any longer.
https://gist.github.com/arcotek-ltd/7c606540980a45a3a7915ccae2e0b140
The template has been written so I can copy out the "resources":[] section into a different template. Hence why some variables and parameters may appear to be oddly named. As I said, the template works, apart from this issue.
PowerShell
I am calling the template with PowerShell, thus:
$Ticks = (Get-Date).Ticks.ToString()
$RGID = (Get-AzureRmResourceGroup -Name "MyResourceGroup").ResourceId
$MyTags = #{"TestTag2"="TestValue2"}
$JsonTagsHash = ($MyTags | ConvertTo-Json -Depth 20 | Out-String) -replace '\s',''
$TemplateParametersObject = #{
currentDateTimeInTicks = $Ticks
runbookParameters = #{
ResourceGroupId = $RGID
#"MyTags" = $MyTags #$JsonTagsHash
}
}
New-AzureRmResourceGroupDeployment `
-Name "Test_Runbook" `
-ResourceGroupName "MyResourceGroup" `
-Mode Incremental `
-DeploymentDebugLogLevel All `
-Force `
-TemplateFile "D:temp\testRunRunbook.json" `
-Verbose `
-TemplateParameterObject $TemplateParametersObject
I've tried the following tests:
Test 1
Uncomment $TemplateParametersObject.runbookParameters.MyTags:
$MyTags = #{"TestTag1"="TestValue1"}
$TemplateParametersObject = #{
currentDateTimeInTicks = $Ticks
runbookParameters = #{
ResourceGroupId = $RGID
MyTags = $MyTags
}
}
Result Fail - See error above.
Test 2
Replace $MyTags with $JsonTagsHash:
$MyTags = #{"TestTag2"="TestValue2"}
$JsonTagsHash = ($MyTags | ConvertTo-Json -Depth 20 | Out-String) -replace '\s',''
$TemplateParametersObject = #{
currentDateTimeInTicks = $Ticks
runbookParameters = #{
ResourceGroupId = $RGID
MyTags = $JsonTagsHash
}
}
Result: PASS Works as expected. Parameters are passed to runbook.
Test 2 works, but I need to be able to pass the parameters to the runbook that are generated inside the template at runtime. In other words, I can't use PowerShell. Inside the template at job.properties.parameters (Line 103 in the gist)
Test 3
To prove it's MyTags causing the issue, take it out completely:
"parameters": {
"ResourceGroupId": "[parameters('runbookParameters').ResourceGroupId]"
}
Result: No error, however, myTags are not passed (obviously).
Test 4
Create a variable object and pass that to the parameter:
"variables" : {
"rolesTagObject": {
"db": "TestVm1",
"Server": "TestVm1",
"Client": "TestVm1"
}
}
And back in job.properties.parameters:
"parameters": {
"ResourceGroupId": "[parameters('runbookParameters').ResourceGroupId]",
"MyTags": "[variables('rolesTagObject')]"
}
Result Fail - See error above.
Test 5
Try directly:
"parameters": {
"ResourceGroupId": "[parameters('runbookParameters').ResourceGroupId]",
"MyTags": {
"testTag5" : "testValue5"
}
}
Result Fail - See error above.
Test 6
Use the json() template function. Not that I expect it to work as it needs a string.
"parameters": {
"ResourceGroupId": "[parameters('runbookParameters').ResourceGroupId]",
"MyTags": "[json(variables('rolesTagObject'))]"
}
Result Fail. As predicted:
'The template language function 'json' expects an argument of type 'string'. The provided value is of type 'Object'.
Test 7
Try with the example provided my MS for json() template function:
"parameters": {
"ResourceGroupId": "[parameters('runbookParameters').ResourceGroupId]",
"MyTags": "[json('{\"a\": \"b\"}')]"
}
Result Fail- See error above. Interesting! But why? The working PowerShell test suggests it needs to be a JSON object. Is this not what json() does?
Test 8
Breaking out the object into individual key / values:
"parameters": {
"db": "[variables('rolesTagObject').db]",
"server": "[variables('rolesTagObject').server]",
"client": "[variables('rolesTagObject').client]"
}
I also have to change the parameters in the runbook from one to three.
Result Pass - but the idea of being able to pass an object is that it can have any number of key / value pairs. This way, not only does the variable and parameter have to be hardcoded, so does the runbook. Not very flexible.
I'm out of ideas. Any suggestions please?

The automation job parameters does not understand complex objects so you would want to pass in the parameter as a string and then convert back to Json within your runbook.
In your example, you could put quotes around the json object $MyTags so that it comes through as a string.
From
$TemplateParametersObject = #{
currentDateTimeInTicks = $Ticks
runbookParameters = #{
ResourceGroupId = $RGID
"MyTags" = $MyTags
}
}
to
$TemplateParametersObject = #{
currentDateTimeInTicks = $Ticks
runbookParameters = #{
ResourceGroupId = $RGID
"MyTags" = "$MyTags"
}
}
Then inside your runbook you could do
$MyTags = ConvertFrom-Json $MyTags
to get it back to a json object that you can use as needed.
If you are converting an object inside a template to a string, then you can just use the string function https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-group-template-functions-string#string
Example as shown in the comments:
"myParams": {
"Hello": "World",
"Location": {
"Country": "USA",
"State": "WA"
}
},
and then you can pass in this template object by just converting it to a string:
"properties": {
"runbook": {
"name": "[variables('runbookName')]"
},
"parameters": {
"ResourceGroupId": "[variables('myParams').Hello]",
"MyTags": "[String(variables('myParams'))]"
}
If the string function doesn't work, please let me know.
Hope this help,
Eamon

Related

JSON Prettifier Using Azure Function w/ PowerShell and HTTP Trigger

Thought this would be pretty simple, but alas, I can't figure it out. It appears that PowerShell will prettify JSON with a single cmdlet.
Goal: Prettify JSON using a PowerShell Azure Function app
Using Microsoft Flow, send an HTTP request (POST) to an Azure Function w/ "ugly", serialized JSON file in body
Azure Function reads file in (then uses ConvertToJson cmdlet to prettify?) and outputs the file back to Flow
Questions:
What do I put in the run.ps1 area of the Azure Function to make this happen?
What do I put in the functions.json area of the Azure Function to make this happen?
I have taken below serialize string
'{ "baz": "quuz", "cow": [ "moo", "cud" ], "foo": "bar" }'
which was mentioned in Prettify json in powershell 3
Here is my function which i used with HttpPost and send the request:
using namespace System.Net
# Input bindings are passed in via param block.
param($Request, $TriggerMetadata)
# Write to the Azure Functions log stream.
Write-Host "PowerShell HTTP trigger function processed a request."
# Interact with query parameters or the body of the request.
$name = $Request.Query.baz
if (-not $name) {
$name = $Request.Body.baz
}
if ($name) {
$status = [HttpStatusCode]::OK
$body = "Hello $name"
}
else {
$status = [HttpStatusCode]::BadRequest
$body = "Please pass a name on the query string or in the request body."
}
# Associate values to output bindings by calling 'Push-OutputBinding'.
Push-OutputBinding -Name Response -Value ([HttpResponseContext]#{
StatusCode = $status
Body = $body
})
and below you can see , i am able to read it from the string which i posted.
You can use ConvertFrom-Json to convert it but i wondering if you even need it as you can access it by doing below:
$name = $Request.Query.baz
my binding is same as yours. Hope it helps.
Let me know if you still need any help.
Are you looking for something like this?
using namespace System.Net
param($Request, $TriggerMetadata)
Push-OutputBinding -Name Response -Value ([HttpResponseContext]#{
StatusCode = [HttpStatusCode]::OK
Body = $Request.RawBody | ConvertFrom-Json | ConvertTo-Json
})

Powershell Core Invoke-RestMethod giving Unexpected character ('M' (code 77)): expected a valid value

This is driving me crazy I have spent the last few days trying to find some answers in the forums but not seen anything which can help me here. Maybe I am just blind.
Here is my problem. I am trying to update one of our Confluence Wiki pages using the API provided to update a page.
I have three scripts or functions:
Deploy script or controller script which calls the Create and with the response (Json) calls Update
Create Confluence Json
Update page
This is Create function
function Create-WikiPage
[CmdletBinding()]
param(
[Parameter(Mandatory)]
[ValidateNotNullOrEmpty()]
[int]$CurrentPageRevisionNumber
)
# Creates the ID for new page
$NextPageID=$CurrentPageRevisionNumber + 1
# Creates the json body of the Wiki page
$ToolsPage= #{
"version"= #{
"number"= $NextPageID
};
"title"= "Windows Build Agent Tool Set";
"type"= "page";
"body"= #{
"storage"= #{
"value"= "<p><table><tr><th>Vendor-Application</th><th>Version</th></tr></table></p>";
"representation"= "storage"
}
}
} | ConvertTo-Json
$ToolsPage
}
The update function looks like this:
Update-WikiPage {
[CmdletBinding()]
param(
[ValidateNotNullOrEmpty()]
[string]$Server,
[Parameter(Mandatory)]
[ValidateNotNullOrEmpty()]
[string]$Username,
[ValidateNotNullOrEmpty()]
[string]$Password,
[ValidateNotNullOrEmpty()]
[long]$PageId,
[Parameter(Mandatory)]
[ValidateNotNullOrEmpty()]
$Data
)
[Net.ServicePointManager]::SecurityProtocol = 'Tls12, Tls11'
$Encoded = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes("$($Username):$Password"))
$Header = #{"Authorization" = "Basic $Encoded"
}
Write-Information ($Data | Out-String)
$Data.GetType()
# Updates the Wiki page
Invoke-RestMethod "$($Server)/rest/api/content/$($PageId)" -Method PUT -Headers $Header -ContentType "application/json" -Body $Data -PreserveAuthorizationOnRedirect
}
As you can see I have get a print out of the JSon object as part of the update function. This is the print out:
{
"version": {
"number": 9
},
"body": {
"storage": {
"value": "<p><table><tr><th>Vendor-Application</th><th>Version</th></tr></table></p>",
"representation": "storage"
}
},
"title": "Windows Build Agent Tool Set",
"type": "page"
}
And this is the powershell error I get:
VERBOSE: received -byte response of content type application/json
Invoke-RestMethod : {"statusCode":500,"message":"org.codehaus.jackson.JsonParseException: Unexpected character ('M' (code 77)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n at [Source: com.atlassian.confluence.plugins.restapi.filters.LimitingRequestFilter$1#6b6831ec; line: 1, column: 2]"}
At C:\Users\ChildsC\Documents\Git\PowerShellModules\Wiki\Update-WikiPage.ps1:65 char:2
+ Invoke-RestMethod "$($Server)/rest/api/content/$($PageId)" -Metho ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (Method: PUT, Re...ication/json
}:HttpRequestMessage) [Invoke-RestMethod], HttpResponseException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
EDIT
I have noticed that he issue happens when I am passing the json object from the Create Json -> Deploy -> Update
if I create the json in the Deploy script and pass it to the Update it works without issues.
Ok
So for this it turns out I was an idiot. And no-one else would see the problem because I did not post the complete script.
Earlier in the Create function I had a line to post params to the console:
$PsBoundParameters | Out-String
So when I thought the script was returning only the $ToolsPage value it was actually also returning another object with it.
Therefore the reason why the Update Script is because I am explicitly passing on the $ToolsPage variable.
I found this out by setting Parameters, which should hold the value of the object to $Global:variable_name and then compared the values from my console.

How to replace a property in a json file in powershell

I'm having a difficult time reading in a JSON file in powershell, replacing a value and writing back to the file or a file.
Given the following:
[Object]$QuickJson = #'
{
"swagger": "1.0",
"info": {
"version": "0.0.1",
"title": "this is a title",
"description": "this is a description"
},
"basePath": "/test",
"paths" : {
"/GetSomething": {
"get": {
"name" : "test01"
}
},
"/GetSomethingElse" : {
"get": {
"name" : "test02"
}
},
"/GetAnotherThing": {
"get": {
"name" : "test03"
}
}
}
}
'#
What I'm interested in here is replacing the values in the "paths" object with something else. I read the file and can access the object however with what I'm trying:
[object]$MyPSJson = ConvertFrom-Json -InputObject $QuickJson
foreach($info in $MyPSJson.paths.PSObject.Properties)
{
$path = $info.Name
Write-Host "path: $path"
$info.Name = "$path?code=fsfsfsfsfsfdfds"
Write-Host $info.Name
}
I don't know what those "paths" will be, I need to take the existing value and append a code value to it so I need to be able to iterate through all the paths and do this replacement. When I try the above I get an error:
path: /GetSomething
-- 'Name' is a ReadOnly property. At C:\scripts\test.ps1:44 char:5
+ $info.Name = "$path?code=fsfsfsfsfsfdfds"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : PropertyAssignmentException
I've tried several different things and yet to come up with a good, or workable for that matter, solution. Any help or pointing in the right direction is greatly appreciated.
You could do something very similar to what you have now, just instead of changing the existing properties you record the initial properties, add new ones with the modifications you want, then set $MyPSJson.paths to itself excluding the old properties, so all it has are the new properties.
#Find initial paths
$Paths=$MyPSJson.paths.psobject.Properties.Name
#Add new paths with the modification to the name, and set the value to the same as the old path
$Paths|%{Add-Member -InputObject $MyPSJson.paths -NotePropertyName "$_`?code=fsfsfsfsfsfdfds" -NotePropertyValue $MyPSJson.paths.$_}
#Set the paths object to itself, excluding the original paths
$MyPSJson.paths = $MyPSJson.paths|Select * -ExcludeProperty $Paths
Try the following (PSv3+):
$MyPSJson.paths = $MyPSJson.paths.psobject.properties |
ForEach-Object { $renamedProps = [ordered] #{} } {
$renamedProps[$_.Name + '?foo=bar&baz=bam'] = $_.Value
} { [pscustomobject] $renamedProps }
Of technically necessity, this recreates the .paths object with modified property names.
It does so by enumerating the original properties, adding their values under the modified name to an ordered hashtable, which on completion of the pipeline is converted to a custom object ([pscustomobject]).
As for what you tried:
A given object's properties cannot be renamed - only its properties' values can be changed.
Therefore, you must create a new object with the desired new property names and the same values as the original.
As an aside: an [object] cast is pointless in PowerShell - it is an effective no-op.
By contrast, a [pscustomobject] cast can be used to create [pscustomobject] instances from [ordered] hashtables, which is the technique used above; casting from other types is again a virtual no-op.
In addition to the accepted answer which worked for me I also found another way which was initially suggested which was to use a hashtable:
function Convert-JsonFileToHashTable([string]$file) {
$text = [IO.File]::ReadAllText($file)
$parser = New-Object Web.Script.Serialization.JavaScriptSerializer
$parser.MaxJsonLength = $text.length
$hashtable = $parser.Deserialize($text, #{}.GetType())
return $hashtable
}
$hashtable = Convert-JsonFileToHashTable (Resolve-Path -Path $DefinitionFile)
$keys = New-Object System.Collections.ArrayList($null)
$keys.AddRange($hashtable["paths"].Keys)
foreach ($key in $keys)
{
$newPath = $key + "?code=$Code"
$hashtable["paths"].$newPath = $hashtable["paths"].$key
$hashtable["paths"].Remove($key) | Out-Null
}
$json = $hashtable | ConvertTo-Json -Depth 20
Write-Output $json | Out-File $newFile -Encoding utf8

How to pass variables into an escaped json string?

I'm trying to pass json into a REST api call in order to start builds on VSTS.
The body has two parameters that are passed in as an escaped string and i'm struggling to update the parameters before invoking my request. An example of my body is below:
$body ='
{
"definition":
{
"id": ""
},
"parameters": "{\"environment\":\"uat\", \"browser\": \"ie\"}"
}
'
This is passed into the following where I update the definition id successfully:
$bodyJson=$body | ConvertFrom-Json
$bodyjson.definition.id = $buildDefId
$bodyString=$bodyJson | ConvertTo-Json -Depth 100
This works successfully but I can't access the parameters element of the json in order to pass the browser and environments in as variables.
Do you have any suggestions on how to do this?
I have tried the following without success:
$params = $bodyJson.Paramaters
$params -replace "uat","test"
$bodyString=$bodyJson | ConvertTo-Json -Depth 100
This updates the parameter in $params but it isn't passed back into the json when converted. I feel that i'm close but obviously missing a step.
Apparently you have a Json (parameters) string embedded in another Json string.
Meaning that you have to ConvertFrom-Json twice to deserialize everything and ConvertTo-Json twice to serialize it back with the new parameters:
(Note that I swapped the variable names $body and $bodyJson because $body is an object and $bodyJson is actually your Json string)
$bodyJson = '
{
"definition": {
"id": ""
},
"parameters": "{\"environment\":\"uat\", \"browser\": \"ie\"}"
}
'
$body = $bodyJson | ConvertFrom - Json
$paramJson = $body.parameters
$parameters = $paramJson | ConvertFrom - Json
$parameters
environment browser
----------- -------
uat ie
Change the parameter:
$parameters.environment = "test"
And rebuild the Json string:
$paramJson = $parameters | ConvertTo-Json -Compress
$body.parameters = $paramJson
$bodyJson = $body | ConvertTo-Json
$bodyJson
{
"definition": {
"id": ""
},
"parameters": "{\"environment\":\"test\",\"browser\":\"ie\"}"
}

ConvertFrom-Json : Cannot convert the JSON string because a dictionary that was converted from the string contains the duplicated keys

The following JSON is getting returned from OData API service:
{
"d": {
"results": [
{
"FileSystemObjectType": 0,
"Id": 1,
"ContentTypeId": "0x0100BC97B2F575CB0C42B79549F3BABD32A8",
"Title": "Nokia California",
"Address": "200 South Matilda Avenue\nW Washington Ave\n94086 Sunnyvale, California\nUnited States of America",
"ID": 1,
"Modified": "2014-02-24T10:06:39Z",
"Created": "2014-02-24T10:06:39Z",
"AuthorId": 12,
"EditorId": 12,
"OData__UIVersionString": "1.0",
"Attachments": false,
"GUID": "d12aafad-502a-4968-a69e-36a7ea05ec80"
}
]
}
}
and saved as a string into variable named $data
An attempt to convert a JSON-formatted string to a custom object using ConvertFrom-Json cmdlet:
$results = $data | ConvertFrom-Json
gives the following error:
ConvertFrom-Json : Cannot convert the JSON string because a dictionary
that was converted from the string contains the duplicated keys 'Id'
and 'ID'.
Is there any way to convert the specified JSON-formatted string in PowerShell?
This is how I have done with it:
$results = $data.ToString().Replace("ID", "_ID") | ConvertFrom-Json
Note, both examples assume the JSON is stored in the $jsonstring variable.
In PowerShell Core, ConvertFrom-Json -AsHashtable is the easiest alternative:
$json = $jsonstring | ConvertFrom-Json -AsHashtable
$json['d']['results']
Name Value
---- -----
Modified 2/24/2014 10:06:39 AM
Title Nokia California
Attachments False
ID 1
ContentTypeId 0x0100BC97B2F575CB0C42B79549F3BABD32A8
GUID d12aafad-502a-4968-a69e-36a7ea05ec80
Created 2/24/2014 10:06:39 AM
EditorId 12
AuthorId 12
Address 200 South Matilda Avenue…
Id 1
OData__UIVersionString 1.0
FileSystemObjectType 0
In Windows PowerShell, you can use the Deserialize(String, Type) method from the JavaScriptSerializer Class.
Add-Type -AssemblyName System.Web.Extensions
$serializer = [Web.Script.Serialization.JavaScriptSerializer]::new()
$json = $serializer.Deserialize($jsonstring, [hashtable])
$json['d']['results']
In PowerShell V1.0, or in PowerShell V2.0 when the JSON is too big, I still use a convertion to XML :
Add-Type -AssemblyName System.ServiceModel.Web, System.Runtime.Serialization
function Convert-JsonToXml
{
PARAM([Parameter(ValueFromPipeline=$true)][string[]]$json)
BEGIN
{
$mStream = New-Object System.IO.MemoryStream
}
PROCESS
{
$json | Write-String -stream $mStream
}
END
{
$mStream.Position = 0
try
{
$jsonReader = [System.Runtime.Serialization.Json.JsonReaderWriterFactory]::CreateJsonReader($mStream,[System.Xml.XmlDictionaryReaderQuotas]::Max)
$xml = New-Object Xml.XmlDocument
$xml.Load($jsonReader)
$xml
}
finally
{
$jsonReader.Close()
$mStream.Dispose()
}
}
}
Using this code you can loop thru your items you can test :
$a = Get-Content C:\temp\jsontest.txt
$b.root.d.results.Item
$b.root.d.results.Item[7].Id[0].InnerText
(Edited)
In you case I would only replace the expected duplicate ID/Id
$data = Get-Content C:\temp\jsontest.txt -Raw
$datacorrected = $a -creplace '"Id":','"Id-minus":'
$psJsonIn = $datacorrected | ConvertFrom-Json
If really you've got unexpected duplicate you can write a function that trap the convertion error due to duplicated key and replace one.
ConvertFrom-JSON it going to try to create a PS Custom Object from the JSON string. PowerShell object property names are case-insensitive, so "ID" and "id" represent the same property name. You're going to have to do something with those duplicate property names in your JSON before you try to do that conversion, or it's going to fail.
I used the ToLower() before converting the json to object, resolved my issue.
$sdf = $data.ToLower() | ConvertFrom-Json