Powershell ConvertFrom-Json eats node - json

ConvertFrom-Json is failing to successfully parse my Json into object, despite my object being legal Json.
I'm using the latest version of Powershell.
When I try parsing a google request JSON body like this:
{
"reportScope": {
"agencyId": "11111111111111111",
"advertiserId": "22222222222222222"
},
"reportType": "adGroup",
"columns": [
{
"columnName": "status"
},
{
"columnName": "date"
},
{
"columnName": "account"
},
{
"columnName": "accountType"
},
{
"columnName": "campaign"
},
{
"columnName": "adGroup"
},
{
"columnName": "deviceSegment"
},
{
"columnName": "impr"
},
{
"columnName": "clicks"
},
{
"columnName": "dfaRevenue"
},
{
"columnName": "cost"
},
{
"columnName": "avgPos"
}
],
"timeRange": {
"startDate": "2017-09-13",
"endDate": "2017-08-14"
},
"filters": [
{
"column": {"columnName": "impr"},
"operator": "greaterThan",
"values": "0"
}
],
"downloadFormat": "csv",
"maxRowsPerFile": 6000000,
"statisticsCurrency": "agency",
"verifySingleTimeZone": "false",
"includeRemovedEntities": "false"
}
I get an object like this:
[DBG]: PS Microsoft.PowerShell.Core\FileSystem::\\psDashboard>> $objJson
reportScope : #{agencyId=11111111111111111; advertiserId=22222222222222222}
reportType : adGroup
columns : {#{columnName=status}, #{columnName=date}, #{columnName=account}, #{columnName=accountType}...}
timeRange : #{startDate=2017-09-13; endDate=2017-08-14}
filters : {#{column=; operator=greaterThan; values=0}}
downloadFormat : csv
maxRowsPerFile : 6000000
statisticsCurrency : agency
verifySingleTimeZone : false
includeRemovedEntities : false
[DBG]: PS Microsoft.PowerShell.Core\FileSystem::\\psDashboard>> $objJson.filters
column operator values
------ -------- ------
greaterThan 0
Notice column should contain #{columnNane=imgr} but it instead contains nothing.
Code used for parsing:
$objJson = Get-Content -Path $FileName -Raw | ConvertFrom-Json
Any ideas?

i took your code and was able to get to the object you are looking for I first put it in a Here string #""# and was able to get to it.. then like you I put it in a variable and piped to convertfrom-json and got the ame results.
$test = #"
{
"reportScope": {
"agencyId": "11111111111111111",
"advertiserId": "22222222222222222"
},
"reportType": "adGroup",
"columns": [
{
"columnName": "status"
},
{
"columnName": "date"
},
{
"columnName": "account"
},
{
"columnName": "accountType"
},
{
"columnName": "campaign"
},
{
"columnName": "adGroup"
},
{
"columnName": "deviceSegment"
},
{
"columnName": "impr"
},
{
"columnName": "clicks"
},
{
"columnName": "dfaRevenue"
},
{
"columnName": "cost"
},
{
"columnName": "avgPos"
}
],
"timeRange": {
"startDate": "2017-09-13",
"endDate": "2017-08-14"
},
"filters": [
{
"column": {"columnName": "impr"},
"operator": "greaterThan",
"values": "0"
}
],
"downloadFormat": "csv",
"maxRowsPerFile": 6000000,
"statisticsCurrency": "agency",
"verifySingleTimeZone": "false",
"includeRemovedEntities": "false"
}
"#
$t = $test | ConvertFrom-Json
$t.filters.column
PS C:\WINDOWS\system32> $t.filters
column operator values
------ -------- ------
#{columnName=impr} greaterThan 0
PS C:\WINDOWS\system32> $t.filters.column
columnName
----------
impr
Using get content and using convertfrom-json.
PS C:\WINDOWS\system32> $test.filters
column operator values
------ -------- ------
#{columnName=impr} greaterThan 0
PS C:\WINDOWS\system32> $test.filters.column
columnName
----------
impr
could it be your version of convertFrom-json my module version is:
PS C:\WINDOWS\system32> gcm convertfrom-json
CommandType Name Version Source
----------- ---- ------- ------
Cmdlet ConvertFrom-Json 3.1.0.0 Microsoft.PowerShell.Utility

Related

Add nested records of JSON body into one column while exporting to CSV in powershell

I have JSON including multiple nested records. I want to add records with comma separated and store it in a CSV file.
JSON Body
{
"projectVitals": {
"productName": "Enterprise",
"name": "WhiteSource Bolt",
"token": "61a48eab05356f149828c0e",
"creationDate": "2022-10-17 09:08:46",
"lastUpdatedDate": "2023-01-25 06:37:32"
},
"libraries": [
{
"keyUuid": "a89b-40759d783dc3",
"keyId": 145110423,
"type": "NUGET_PACKAGE_MODULE",
"languages": "Nuget",
"references": {
"url": "https://api.nuget.org/packages/system.text.encodings.web.5.0.1.nupkg",
"homePage": "https://github.com/dotnet/runtime",
"genericPackageIndex": "https://api.nuget.org/packages/System.Text.Encodings.Web/5.0.1"
},
"matchType": "SHA1",
"sha1": "05cd84c678cddd1de0c",
"name": "system.text.encodings.web.5.0.1.nupkg",
"artifactId": "system.text.encodings.web.5.0.1.nupkg",
"version": "5.0.1",
"groupId": "System.Text.Encodings.Web",
"licenses": [
{
"name": "MIT",
"url": "http://www.opensource.org/licenses/MIT",
"profileInfo": {
"copyrightRiskScore": "THREE",
"patentRiskScore": "ONE",
"copyleft": "NO",
"royaltyFree": "YES"
},
"references": [
{
"referenceType": "NuGet package (details available in nuget gallery)",
"reference": "https://index.whitesourcesoftware.com/gri/app/reader/resource/content/asString/33131621-c9e5-4c87-ac1d-b988bbef1e0a"
}
]
}
],
"vulnerabilities": []
},
{
"keyUuid": "936f-5daddbcc37b2",
"keyId": 69037902,
"type": "DOT_NET_AS_GENERIC_RESOURCE",
"languages": ".NET",
"references": {
"url": "https://api.nuget.org/packages/system.runtime.interopservices.runtimeinformation.4.3.0.nupkg",
"genericPackageIndex": ""
},
"matchType": "SHA1",
"sha1": "32d3122a48aa379904",
"name": "System.Runtime.InteropServices.RuntimeInformation-4.6.24705.01.dll",
"artifactId": "System.Runtime.InteropServices.RuntimeInformation-4.6.24705.01.dll",
"version": "4.6.24705.01",
"groupId": "System.Runtime.InteropServices.RuntimeInformation",
"licenses": [
{
"name": "Microsoft .NET Library",
"url": "http://microsoft.com/web/webpi/eula/aspnetcomponent_rtw_enu.htm",
"riskLevel": "unknown",
"references": [
{
"referenceType": "Details available in GitHub repository",
"reference": "https://dot.net/"
},
{
"referenceType": "Details available in GitHub repository",
"reference": "https://dotnet.microsoft.com/"
}
]
},
{
"name": "MIT",
"url": "http://www.opensource.org/licenses/MIT",
"profileInfo": {
"copyrightRiskScore": "THREE",
"patentRiskScore": "ONE",
"copyleft": "NO",
"royaltyFree": "YES"
},
"references": [
{
"referenceType": "Details available in GitHub repository",
"reference": "https://dot.net/"
}
]
}
],
"vulnerabilities": []
}
]
}
Powershell Script
$pathToInputJsonFile = "C:\Users\abc\Downloads\test.json"
$pathToOutputCSVFile = "C:\Users\abc\Downloads\License3.csv"
$jsonFileContent = Get-Content -Raw -Path $pathToInputJsonFile | Out-String | ConvertFrom-Json
$libraries = $jsonFileContent.libraries
foreach($obj in $libraries)
{
$LibraryName = $obj.name
$LibraryVersion = $obj.version
$LibraryType = $obj.type
$LibraryLanguage = $obj.languages
$LibraryURL = $obj.references.url
$LicenseName = $obj.licenses.name
$LicenseURL = $obj.licenses.url
[PSCustomObject]#{
LibraryName = $LibraryName
LibraryVersion = $LibraryVersion
LibraryType = $LibraryType
LibraryLanguage = $LibraryLanguage
LibraryURL = $LibraryURL
LicenseName = $LicenseName
LicenseURL = $LicenseURL
} | Export-Csv $pathToOutputCSVFile -notype -Append
}
Actual Result
Expected Result
Use the -join operator to join 1 or more strings together with a given separator:
[PSCustomObject]#{
# ...
LicenseName = $LicenseName -join ', '
LicenseURL = $LicenseURL -join ', '
}

Check if Json field Exist, giving default value with (select) jq command

I have some JSON data and i want to push some of them to DB , but sometimes the json values not exists for specific devices:
from all of the following data I want just to pull , "ICCID","MDN","MSISDN","MCC","MNC","FeatureTypes","RatePlanCode","RatePlanDescription","DeviceState","BillingCycleStartDate","BillingCycleEndDate","CurrentBillCycleDataUnRatedUsage"
and if any one not exist print not-exist .
{
"categories": [{
"categoryName": "DeviceIdentifier",
"extendedAttributes": [{
"key": "ICCID",
"value": "89148000"
},
{
"key": "IMSI",
"value": "31148094"
},
{
"key": "MDN",
"value": "5514048068"
},
{
"key": "MEID",
"value": "A0000000005006"
},
{
"key": "MIN",
"value": "5514041185"
}
]
},
{
"categoryName": "DeviceAttributes",
"extendedAttributes": [{
"key": "MCC",
"value": "311"
},
{
"key": "MNC",
"value": "480"
},
{
"key": "FeatureCodes",
"value": "75802,84777,54307"
},
{
"key": "FeatureNames",
"value": "75802,84777,54307"
},
{
"key": "FeatureTypes",
"value": "4G Public Dynamic"
},
{
"key": "RatePlanCode",
"value": "4G5G "
},
{
"key": "RatePlanDescription",
"value": "4G5G"
},
{
"key": "Services",
"value": "null"
}
]
},
{
"categoryName": "Provisioning",
"extendedAttributes": [{
"key": "LastActivationDate",
"value": "2022-03-01T19:38:52Z"
},
{
"key": "CreatedAt",
"value": "2021-12-01T21:22:55Z"
},
{
"key": "DeviceState",
"value": "active"
},
{
"key": "LastDeactivationDate",
"value": "2021-12-01T21:22:55Z"
}
]
},
{
"categoryName": "Connectivity",
"extendedAttributes": [{
"key": "Connected",
"value": "true"
},
{
"key": "LastConnectionDate",
"value": "2022-09-08T03:38:55Z"
},
{
"key": "LastDisconnectDate",
"value": "2022-09-08T03:25:15Z"
}
]
},
{
"categoryName": "Billing",
"extendedAttributes": [{
"key": "BillingCycleStartDate",
"value": "2022-09-02T00:00:00Z"
},
{
"key": "BillingCycleEndDate",
"value": "2022-10-01T00:00:00Z"
},
{
"key": "DefaultRatePlan",
"value": "0"
}
]
},
{
"categoryName": "Usage",
"extendedAttributes": [{
"key": "CurrentRatedUsageRecordDate",
"value": "2022-09-04T00:00:00Z"
}, {
"key": "CurrentUnRatedUsageRecordDate",
"value": "2022-09-08T01:25:15Z"
},
{
"key": "CurrentBillCycleDataUnRatedUsage",
"value": "1698414605"
}
]
}
]
}
i'm not pushing all fields to db so i'm selecting a specific keys from that,
(what i'am selecting its fixed not changed) so the select will not change and always will be :
Expected output :
"89148000"
"5514048068"
"not-exist"
"4G Public Dynamic"
"4G5G"
"4G5G"
"active"
"2022-09-02T00:00:00Z"
"2022-10-01T00:00:00Z"
"2022-09-08T01:25:15Z"
I would like to check if the value of key is missing for this case "MSISDN" ,if not will print for me not-exist or null
any help ?
.categories[].Attributes[] |
if (.key | IN(["AAA","BBB","DDD","EEE"][]))
then .value
else "NOT-EXIST"
end
Gives the following output
"111"
"222"
"NOT-EXIST"
"444"
"555"
First we loop over the Attributes
Then we use an if to;
Check if key exist in ["AAA","BBB","DDD","EEE"]
TRUE: use .value
FALSE: use NOT-EXIST as value
Demo
Another approach, using with_entries() to update the .value before looping over all the objects to show just the value, gives the same output as above:
.categories[].Attributes[]
| select(.key | IN(["AAA","BBB","DDD","EEE"][]) | not).value = "NOT-EXIST"
| .value
Demo
I hope I understood your requirements correctly, but here is a solution that looks simple enough to understand and should be somewhat efficient. If you always expect the same 5 keys in your input, you can try:
.categories[].Attributes | from_entries as $attr
| ["AAA", "BBB", "CCC", "DDD", "EEE"]
| map($attr[.] // "NOT-EXIST")
Input:
{"categories": [
{
"categoryName": "Device",
"Attributes": [
{
"key": "AAA",
"value": "111"
},
{
"key": "BBB",
"value": "222"
},
{
"key": "DDD",
"value": "444"
},
{
"key": "EEE",
"value": "555"
}
]
}]}
Output:
[
"111",
"222",
"NOT-EXIST",
"444",
"555"
]
If you require only the values, add [] or | .[] at the end of the script or rewrite to:
.categories[].Attributes | from_entries as $attr
| "AAA", "BBB", "CCC", "DDD", "EEE"
| $attr[.] // "NOT-EXIST"
With the input from updated question, you intend to first merge all extendedAttributes array into one big array, convert to an object and then use this complete object to look up your values:
.categories | map(.extendedAttributes[]) | from_entries as $attr
| "ICCID", "MDN", "MSISDN", "MCC", "MNC", "FeatureTypes", "RatePlanCode", "RatePlanDescription", "DeviceState", "BillingCycleStartDate", "BillingCycleEndDate", "CurrentBillCycleDataUnRatedUsage"
| $attr[.] // "NOT-EXIST"
.categories | map(.extendedAttributes[]) can be rewritten as [.categories[].extendedAttributes[]] or .categories | map(.extendedAttributes) | add, which might be easier to grok.
Output:
"89148000"
"5514048068"
"NOT-EXIST"
"311"
"480"
"4G Public Dynamic"
"4G5G "
"4G5G"
"active"
"2022-09-02T00:00:00Z"
"2022-10-01T00:00:00Z"
"1698414605"
A version without an intermediate variable is also possible:
[
.categories | map(.extendedAttributes[]) | from_entries[
"ICCID",
"MDN",
"MSISDN",
"MCC",
"MNC",
"FeatureTypes",
"RatePlanCode",
"RatePlanDescription",
"DeviceState",
"BillingCycleStartDate",
"BillingCycleEndDate",
"CurrentBillCycleDataUnRatedUsage"
]
]
| map(. // "NOT-EXIST")
or
[
.categories | map(.extendedAttributes[]) | from_entries
| .["ICCID", "MDN", "MSISDN", "MCC", "MNC", "FeatureTypes", "RatePlanCode", "RatePlanDescription", "DeviceState", "BillingCycleStartDate", "BillingCycleEndDate", "CurrentBillCycleDataUnRatedUsage"]
]
| map(. // "NOT-EXIST")

DataWeave JSON Transformation/ Extract and concatenate values

I'm looking to go from a JSON structure that looks something like this:
{
"id": "955559665",
"timestamp": "2022-04-21 00:00:19",
"user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15",
"remote_addr": "123.456.789.012",
"read": "0",
"data": {
"80928111": {
"field": "80928111",
"value": "Z01234567",
"flat_value": "Z01234567",
"label": "ID",
"type": "text"
},
"90924321": {
"field": "90924321",
"value": {
"first": "Jane",
"last": "Doe"
},
"flat_value": "first = Jane\nlast = Doe",
"label": "Name",
"type": "name"
},
"88888770": {
"field": "88888770",
"value": "jdoe001#gmail.com",
"flat_value": "jdoe001#gmail.com",
"label": "Email",
"type": "email"
},
"12345678": {
"field": "12345678",
"value": "https://www.google.com/subdomain/attachment/file.txt",
"flat_value": "https://www.google.com/subdomain/attachment/file.txt",
"label": "Choose File",
"type": "file"
}
}
}
Ultimately to something like this:
{
"name_val":"Name: first = Jane\nlast = Doe\nEmail: jdoe001#gmail.com\n",
"file": {
"id": "12345678C",
"name": "file.txt"
}
}
In the original JSON, the 'data' object represents a form submission. Each sub object represents a field on the submitted form. The only distinction I'm interested in is the 'type' of field identified as 'file'.
Every response that is not of 'file' type, I want to concatenate into one large String value that looks like: 'label1: flat_value1\nlabel2: flat_value2...'
Note, the number of actual fields is variable.
Then I need a second object to show the field of type 'file', by identifying the 'field' id, and the name of the file.
I've gotten pieces of this to work. For example, using pluck and filter, I've been able to separate the types of fields.
Something like this:
%dw 2.0
output application/json
---
[
"fields": payload.data pluck(
{
"field": $."label",
"value": $."flat_value",
"type": $."type"
}
) filter ($."type" != "file") default "",
"files": payload.data pluck(
{
"type": $."type",
"fieldId": $."field"
}
) filter ($."type" == "file") default ""
]
Gives me:
[
{
"fields": [
{
"field": "ID",
"value": "Z01234567",
"type": "text"
},
{
"field": "Name",
"value": "first = Jane\nlast = Doe",
"type": "name"
},
{
"field": "Email",
"value": "jdoe001#gmail.com",
"type": "email"
}
]
},
{
"files": [
{
"type": "file",
"fieldId": "12345678"
}
]
}
]
And playing around with a modified JSON input, I was able to easily see concatenation similar to how I want to see it, but not quite there:
%dw 2.0
output application/json
var inputJson = [
{
"field": "ID",
"value": "Z01234567",
"type": "text"
},
{
"field": "Name",
"value": "first = Jane\nlast = Doe",
"type": "name"
}
]
---
inputJson map ((value, index) -> value.field ++ ': ' ++ value.value)
Gives me:
[
"ID: Z01234567",
"Name: first = Jane\nlast = Doe"
]
But I can't seem to put it all together and go from Beginning to End.
There are several ways to implement this. I recommend to try to encapsulate the parts that you get working and use them as building blocks.
%dw 2.0
output application/json
fun fields(x) = x.data pluck(
{
"field": $."label",
"value": $."flat_value",
"type": $."type"
}
) filter ($."type" != "file") default ""
fun files(x) = x.data pluck(
{
"type": $."type",
"fieldId": $."field"
}
) filter ($."type" == "file") default ""
---
{
name_val: fields(payload) reduce ((item,acc="") -> acc ++ item.field ++ ': ' ++ item.value ++ "\n"),
files: files(payload)[0]
}
Output:
{
"name_val": "ID: Z01234567\nName: first = Jane\nlast = Doe\nEmail: jdoe001#gmail.com\n",
"files": {
"type": "file",
"fieldId": "12345678"
}
}

Azure Powershell - Deployment Template Validation Failed - Jtoken

I'm trying to deploy multiple virtual machines one at a time with Powershell and a template file but I keep getting this error:
New-AzResourceGroupDeployment : 9:27:40 AM - Error: Code=InvalidTemplate; Message=Deployment template validation
failed: 'Template parameter JToken type is not valid. Expected 'String, Uri'. Actual 'Object'. Please see
https://aka.ms/resource-manager-parameter-files for usage details.'.
At C:\Users\jackk\OneDrive\Desktop\AzureAutomation\main.ps1:193 char:9
+ New-AzResourceGroupDeployment #parameters
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [New-AzResourceGroupDeployment], Exception
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.NewAzureResourceGroupDep
loymentCmdlet
New-AzResourceGroupDeployment : The deployment validation failed
At C:\Users\jackk\OneDrive\Desktop\AzureAutomation\main.ps1:193 char:9
+ New-AzResourceGroupDeployment #parameters
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [New-AzResourceGroupDeployment], InvalidOperationException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.NewAzureResourceGroupDep
loymentCmdlet
Here is the relevant part of my powershell script:
$location = "eastus2"
$networkInterfaceName = ""
$networkSecurityGroupName = ""
#$networkSecurityGroupRules = #()
$subnetName = $SubnetSelection
$virtualNetworkId = "/subscriptions/1234-1234-123-123-123/resourceGroups/MainGroup/providers/Microsoft.Network/virtualNetworks/Main_Network"
$virtualMachineName = ""
$virtualMachineComputerName = ""
$virtualMachineRG = $resourceGroupSelection
$osDiskType = "StandardSSD_LRS"
$virtualMachineSize = "Standard_B2s"
$adminUsername = "ADMIN"
$adminPassword = ConvertTo-SecureString "Password1234" -AsPlainText -Force
$diagnosticsStorageAccountName = $StorageAccountName
$diagnosticsStorageAccountId = $StorageAccountID
$imageName = $ImageSelection
foreach ($i in 1..5) {
$virtualMachineName = "test-$i".ToString()
$virtualMachineComputerName = "test-$i".ToString()
$rand = Get-Random -Maximum 1000
$networkInterfaceName = "test-$i$rand".ToString()
$networkSecurityGroupName = "test-$i-nsg".ToString()
$paramObject = #{
'location' = $location
'networkInterfaceName' = $networkInterfaceName
'networkSecurityGroupName' = $networkSecurityGroupName
'subnetName' = $SubnetSelection
'virtualNetworkId' = $virtualNetworkId
'virtualMachineName' = $virtualMachineName
'virtualMachineComputerName' = $virtualMachineComputerName
'virtualMachineRG' = $resourceGroupSelection
'osDiskType' = $osDiskType
'virtualMachineSize' = $virtualMachineSize
'adminUsername' = $adminUsername
'adminPassword' = $adminPassword
'diagnosticsStorageAccountName' = $diagnosticsStorageAccountName
'diagnosticsStorageAccountId' = $diagnosticsStorageAccountId
'imageName' = $imageName
}
$parameters = #{
'ResourceGroupName' = $resourceGroupSelection
'TemplateFile' = $PathTemplate
'TemplateParameterObject' = $paramObject
'Verbose' = $True
}
New-AzResourceGroupDeployment #parameters
}
Assume any variables not defined in this snippet are strings defined elsewhere with valid information.
The Template file I am using:
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"location": {
"type": "String"
},
"networkInterfaceName": {
"type": "String"
},
"networkSecurityGroupName": {
"type": "String"
},
"networkSecurityGroupRules": {
"type": "Array",
"defaultValue": []
},
"subnetName": {
"type": "String"
},
"virtualNetworkId": {
"type": "String"
},
"virtualMachineName": {
"type": "String"
},
"virtualMachineComputerName": {
"type": "String"
},
"virtualMachineRG": {
"type": "String"
},
"osDiskType": {
"type": "String"
},
"virtualMachineSize": {
"type": "String"
},
"adminUsername": {
"type": "String"
},
"adminPassword": {
"type": "SecureString"
},
"diagnosticsStorageAccountName": {
"type": "String"
},
"diagnosticsStorageAccountId": {
"type": "String"
},
"imageName": {
"type": "String"
}
},
"variables": {
"nsgId": "[resourceId(resourceGroup().name, 'Microsoft.Network/networkSecurityGroups', parameters('networkSecurityGroupName'))]",
"vnetId": "[parameters('virtualNetworkId')]",
"subnetRef": "[concat(variables('vnetId'), '/subnets/', parameters('subnetName'))]"
},
"resources": [
{
"type": "Microsoft.Network/networkInterfaces",
"apiVersion": "2019-07-01",
"name": "[parameters('networkInterfaceName')]",
"location": "[parameters('location')]",
"dependsOn": [
"[concat('Microsoft.Network/networkSecurityGroups/', parameters('networkSecurityGroupName'))]"
],
"properties": {
"ipConfigurations": [
{
"name": "ipconfig1",
"properties": {
"subnet": {
"id": "[variables('subnetRef')]"
},
"privateIPAllocationMethod": "Dynamic"
}
}
],
"networkSecurityGroup": {
"id": "[variables('nsgId')]"
}
}
},
{
"type": "Microsoft.Network/networkSecurityGroups",
"apiVersion": "2019-02-01",
"name": "[parameters('networkSecurityGroupName')]",
"location": "[parameters('location')]",
"properties": {
"securityRules": "[parameters('networkSecurityGroupRules')]"
}
},
{
"type": "Microsoft.Compute/virtualMachines",
"apiVersion": "2019-07-01",
"name": "[parameters('virtualMachineName')]",
"location": "[parameters('location')]",
"dependsOn": [
"[concat('Microsoft.Network/networkInterfaces/', parameters('networkInterfaceName'))]"
],
"properties": {
"hardwareProfile": {
"vmSize": "[parameters('virtualMachineSize')]"
},
"storageProfile": {
"osDisk": {
"createOption": "fromImage",
"managedDisk": {
"storageAccountType": "[parameters('osDiskType')]"
}
},
"imageReference": {
"id": "[parameters('imageName')]"
}
},
"networkProfile": {
"networkInterfaces": [
{
"id": "[resourceId('Microsoft.Network/networkInterfaces', parameters('networkInterfaceName'))]"
}
]
},
"osProfile": {
"computerName": "[parameters('virtualMachineComputerName')]",
"adminUsername": "[parameters('adminUsername')]",
"adminPassword": "[parameters('adminPassword')]",
"windowsConfiguration": {
"enableAutomaticUpdates": true,
"provisionVmAgent": true
}
},
"diagnosticsProfile": {
"bootDiagnostics": {
"enabled": true,
"storageUri": "[concat('https://', parameters('diagnosticsStorageAccountName'), '.blob.core.windows.net/')]"
}
}
}
}
],
"outputs": {
"adminUsername": {
"type": "String",
"value": "[parameters('adminUsername')]"
}
}
}
I have tried everything I could think of and I think I've narrowed the issue down to networkSecurityGroupRules. I took this template file from a valid deployment in Azure and the parameter file that came with networkSecurityGroupRules had the value of the that set to []. So I created a default value for it but it does not work. I've also tried declaring it as a empty array in powershell and passing it through with the other paramObjects but I get the same error. I've tried ArrayList as well to no avail. I have also filled in a parameter.json file with the exact values I pass to it in $paramObjects and it works flawlessly. If I change the defaultValue to null, then on deployment it will ask me to fill in networkSecurityGroupRules[0], networkSecurityGroupRules[1], etc.. I really don't know where the issue is so any help would be really appreciated, thanks.
Found the issue. Everything in the 'paramObject' must be in quotes. For example, 'location' = $location must be 'location' = "$location"

Can't run powershall script

I have a powershell script like that:
Param(
[Parameter(Mandatory=$True)]
[String]
$NEWPARAMVALUE,
[Parameter(Mandatory=$True)]
[String]
$PARAM_NAME
)
$FILEPATH = “arm_template.json”
$JSON = Get-Content $FILEPATH | Out-String | ConvertFrom-Json
$JSON.$PARAM_NAME = $NEWPARAMVALUE
$JSON | ConvertTo-Json -Depth 10 | Set-Content $FILEPATH
When I used this script without $PARAM_NAME variable (I just put a parameter as a plain text, example: $JSON.resources.properties.parameters.daasServiceBaseUrl.defaultValue) - it worked.
I tried different ways:
$JSON.resources.$PARAM_NAME (where $PARAM_NAME = 'properties.parameters.daasServiceBaseUrl.defaultValue')
$JSON"."$PARAM_NAME"
I think that's because of the point between two env variables. Could please someone help with it?
Update:
Error that I get from Azure DevOps:
The property
| 'properties.parameters.parametername.defaultValue' cannot
| be found on this object. Verify that the property exists and
| can be set.
How I tried it manually:
$env:PARAM_NAME="resources.properties.parameters.daasServiceBaseUrl.defaultValue"
./test.ps1 -NEWPARAMVALUE "[parameters('DaaS_Contract_Daily_Trigger_properties_Daas-UI-to-Contract_parameters_daasServiceBaseUrl')]" -PARAM_NAME $env:PARAM_NAME
What I got:
Line |
12 | $JSON.$PARAM_NAME = $NEWPARAMVALUE
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Exception setting "resources.properties.parameters.daasServiceBaseUrl.defaultValue": "The property
| 'resources.properties.parameters.daasServiceBaseUrl.defaultValue' cannot be found on this object. Verify that the property exists and can be
| set."
My JSON:
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"dataFactoryName": {
"type": "string",
"defaultValue": ""
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]",
"metadata": {
"description": "Data Factory location matches Resource group location"
}
},
"project": {
"type": "string",
"defaultValue": "ct",
"metadata": {
"description": "Azure Tag used to track projects"
}
},
"environment": {
"type": "string",
"defaultValue": "default",
"metadata": {
"description": "Azure Tag used to track environments"
}
},
"sleepTime": {
"type": "int",
"defaultValue": "1"
},
"DaaS_Contract_Daily_Trigger_properties_Daas-UI-to-Contract_parameters_daasServiceBaseUrl": {
"type": "string",
"defaultValue": "http://digital-daas-service-om-sqa.nplabsusk8s.com/daas"
}
},
"resources": [
{
"apiVersion": "2018-06-01",
"name": "[parameters('dataFactoryName')]",
"location": "[parameters('location')]",
"tags": {
"project": "[parameters('project')]",
"environment": "[parameters('environment')]"
},
"type": "Microsoft.DataFactory/factories",
"identity": {
"type": "SystemAssigned"
},
"properties": {}
},
{
"name": "[concat(parameters('dataFactoryName'), '/pipeline1')]",
"type": "Microsoft.DataFactory/factories/pipelines",
"apiVersion": "2018-06-01",
"properties": {
"activities": [
{
"name": "Wait1",
"type": "Wait",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"waitTimeInSeconds": "[parameters('sleepTime')]"
}
},
{
"name": "Copy Delivery Request",
"type": "Copy",
"dependsOn": [
{
"activity": "Delete old Req File",
"dependencyConditions": [
"Succeeded"
]
}
],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "RestSource",
"httpRequestTimeout": "00:01:40",
"requestInterval": "00.00:00:00.010",
"requestMethod": "GET"
},
"sink": {
"type": "JsonSink",
"storeSettings": {
"type": "AzureBlobFSWriteSettings"
},
"formatSettings": {
"type": "JsonWriteSettings",
"quoteAllText": true
}
},
"enableStaging": false
},
"inputs": [
{
"referenceName": "Rest_Json_File",
"type": "DatasetReference",
"parameters": {
"relativeURL": {
"value": "#{pipeline().parameters.daasServiceRelURL}/#{variables('createdDate')}",
"type": "Expression"
},
"BaseUrl": {
"value": "#pipeline().parameters.daasServiceBaseUrl",
"type": "Expression"
}
}
}
]
}
],
"parameters": {
"deliveryReqFileName": {
"type": "string",
"defaultValue": "delivery_request.json"
},
"daasServiceBaseUrl": {
"type": "string",
"defaultValue": "http://digital-daas-service-om-sqa.nplabsusk8s.com/daas"
}
},
"annotations": []
},
"dependsOn": []
}
]
}
When you pass a string "a.b.c.d" as the right-hand operand of the . member access operator, PowerShell will attempt to resolve the exact member name a.b.c.d, not "a, then b, then c, then d".
While this could be solved by executing Invoke-Expression against "${JSON}.${PARAM_NAME}", that also opens your script up to arbitrary execution of whatever the user might pass in as $PARAM_NAME.
To avoid that, you'll need to manually:
Split $PARAM_NAME into individual member names, like $names = $PARAM_NAME.Split('.'), then
Resolve each property name except for the last one manually
Assign to $resolvedObject."$last"
Here's how that might look in a generic helper function that takes care of step 2 and 3:
function Set-PropertyChained
{
param(
[Parameter(Mandatory = $true, ValueFromPipeline = $true, Position = 0)]
[ValidateScript({$_ -isnot [ValueType]})]
[object]$RootObject,
[Parameter(Mandatory = $true, Position = 1)]
[string[]]$MemberChain,
[Parameter(Mandatory = $true, Position = 2)]
[object]$Value,
[switch]$PassThru
)
begin {
$resolutionChain = #($MemberChain |Select -SkipLast 1)
$terminalMember = $($MemberChain |Select -Last 1)
}
process {
$obj = $RootObject
# Member resolution up until the parent of the last property
foreach($memberName in $resolutionChain){
Write-Vebose "Resolving $memberName on $obj"
$obj = $obj.$memberName
}
# Set the value of the last property on its parent
Write-Vebose "Setting $memberName on $obj to $Value"
$obj.$terminalMember = $Value
if($PassThru){
# Write modified object back if requested
Write-Output $RootObject -NoEnumerate
}
}
}
Then in your script:
Param(
[Parameter(Mandatory=$True)]
[String]
$NEWPARAMVALUE,
[Parameter(Mandatory=$True)]
[String]
$PARAM_NAME
)
$FILEPATH = “arm_template.json”
$JSON = Get-Content $FILEPATH | Out-String | ConvertFrom-Json
# Step 1, split the resolution chain into individual components
$MemberNames = $PARAM_NAME.Split('.')
# Step 2 + 3, resolve n-2 members in chain, then assign to n-1
$JSON | Set-PropertyChained -MemberChain $MemberNames -Value $NEWPARAMVALUE
$JSON | ConvertTo-Json -Depth 10 | Set-Content $FILEPATH