ARM Template DSC: configuration does not 'see' protectedSettings.configurationArguments - json

I need to protect a DSC configuration parameter ([pscredential]RegistrationKey), so I have put it under "settings.protectedSettings.configurationData" thus:
"protectedSettings": {
"configurationArguments": {
"RegistrationKey": {
"UserName": "PLACEHOLDER_DONOTUSE",
"Password": "[parameters('dscAutomationRegistrationKey')]"
}
},
"configurationUrlSasToken": "[parameters('artifactsLocationSasToken')]"
}
I get the error:
"VM has reported a failure when processing extension 'Microsoft.Powershell.DSC'. Error message: \"The DSC Extension failed to execute: Mandatory
parameter RegistrationKey is missing.
If I move RegistrationKey out of "settings.protectedSettings.configurationArguments", into "settings.configurationArguments", it works, therefore, I assume there is nothing wrong with the syntax, so I believe it is to do with PsDscAllowPlainTextPassword = $true that wasn't included in the DSC configuration.
(I tried to include the configuration block in the PS1 file, but this threw an error, suggesting this can't be done)
I have now written a configurationdata .psd1 file, containing the following:
$ConfigData = #{
AllNodes = #(
#{
NodeName = "*"
PsDscAllowPlainTextPassword = $true
}
)
}
and referenced it in settings.configurationdata.url.
This now results in the same error as before: VM has reported a failure...
The ARM template is called from PowerShell:
$oAutomationAccount = Get-AzureRmAutomationAccount -ResourceGroupName $AAresourceGroupName -Name $AutomationAccountName
$RegistrationInfo = $oAutomationAccount | Get-AzureRmAutomationRegistrationInfo
$DscRegKeyString = $RegistrationInfo.PrimaryKey
$ssDscAutomationRegistrationKey = (ConvertTo-SecureString -string $DscRegKeyString -AsPlainText -Force)
#Automation Account EndPoint Uri
$DscRegistrationUrl = $RegistrationInfo.Endpoint
$params = #{
artifactsLocationSasToken = $TemplateSas
vmName = "XYZ"
dscAutomationRegistrationKey = $ssDscAutomationRegistrationKey
dscAutomationRegistrationUrl = $DscRegistrationUrl
dscNodeConfigurationName = "CreateAFolder.localhost"
dscTimeStamp = (Get-Date -f "MM/dd/yyyy H:mm:ss tt") #"MM/dd/yyyy H:mm:ss tt"
dscResourceUrl = $DscResourceUrl
dscConfigurationUrl = $DscConfigurationUrl
dscResourceScript = $DscResourceScriptName
dscResourceFunction = "ConfigureLCMforAAPull"
#sequenceId = $sequenceId
}
New-AzureRmResourceGroupDeployment #params `
-Name "$TemplateInstance-$branch" `
-ResourceGroupName $DeploymentResourceGroup.ResourceGroupName `
-Mode Incremental `
-DeploymentDebugLogLevel All `
-TemplateUri $TemplateUri `
-Verbose
Where I believe the parameters are being passed as the correct types.
What am I doing wrong?
Reference template: https://github.com/Azure/azure-quickstart-templates/blob/master/dsc-extension-azure-automation-pullserver/azuredeploy.json
Updated to use a newer DSC schema:https://blogs.msdn.microsoft.com/powershell/2016/02/26/arm-dsc-extension-settings/

this is the template I've been using for node onboarding:
{
"name": "xxx",
"type": "Microsoft.Compute/virtualMachines/extensions",
"location": "[parameters('location')]",
"apiVersion": "2015-06-15",
"dependsOn": [
"xxx"
],
"properties": {
"publisher": "Microsoft.Powershell",
"type": "DSC",
"typeHandlerVersion": "2.22",
"autoUpgradeMinorVersion": false,
"protectedSettings": {
"Items": {
"registrationKeyPrivate": "[parameters('registrationData')]"
}
},
"settings": {
"ModulesUrl": "https://github.com/Azure/azure-quickstart-templates/raw/master/dsc-extension-azure-automation-pullserver/UpdateLCMforAAPull.zip",
"SasToken": "",
"ConfigurationFunction": "UpdateLCMforAAPull.ps1\\ConfigureLCMforAAPull",
"Properties": [
{
"Name": "RegistrationKey",
"Value": {
"UserName": "PLACEHOLDER_DONOTUSE",
"Password": "PrivateSettingsRef:registrationKeyPrivate"
},
"TypeName": "System.Management.Automation.PSCredential"
},
{
"Name": "RegistrationUrl",
"Value": "xxx",
"TypeName": "System.String"
},
{
"Name": "NodeConfigurationName",
"Value": "xxx",
"TypeName": "System.String"
},
{
"Name": "ConfigurationMode",
"Value": "ApplyAndMonitor",
"TypeName": "System.String"
},
{
"Name": "ConfigurationModeFrequencyMins",
"Value": 15,
"TypeName": "System.Int32"
},
{
"Name": "RefreshFrequencyMins",
"Value": 30,
"TypeName": "System.Int32"
},
{
"Name": "RebootNodeIfNeeded",
"Value": true,
"TypeName": "System.Boolean"
},
{
"Name": "ActionAfterReboot",
"Value": "ContinueConfiguration",
"TypeName": "System.String"
},
{
"Name": "AllowModuleOverwrite",
"Value": true,
"TypeName": "System.Boolean"
},
{
"Name": "Timestamp",
"Value": "MM/dd/yyyy H:mm:ss tt",
"TypeName": "System.String"
}
]
}
}
}
I know its using an old format, but that works so, meh.

Related

How to get required json output from complex nested json format.?

My original file is in CSV format which I have converted to python JSON array to JSON Sring.
jsonfile
<class 'list'>
<class 'dict'>
[
{
"key": "timestamp",
"source": "eia007",
"turnover": "65million",
"url": "abc.com",
"record": "",
"loc.reg": "nord000",
"loc.count": "abs39i5",
"loc.town": "cold54",
"co.gdp": "nscrt77",
"co.pop.min": "min50",
"co.pop.max": "max75",
"co.rev": "",
"chain.system": "5t5t5",
"chain.type": "765ef",
"chain.strat": "",
}
]
I would like to get the output as below:
{
"timestamp001": {
"key": "timestamp001",
"phNo": "ner007",
"turnover": "65million",
"url": "abc.com",
"record": "",
"loc": {
"reg": "nord000",
"count": "abs39i5",
"town": "cold54"
},
"co": {
"form": "nscrt77",
"pop": {
"min": "min50",
"max": "max75"
},
"rev: ""
},
"chain":{
"system": "5t5t5",
"type": "765ef",
"strat": ""
}
...
}
...
}
]
I have tried different options; tried to enumerate, but cannot get the required output. Please help me with this. Thanks in advance.
You can use something like this to create the nested dict:
import json
def unflatten(somedict):
unflattened = {}
for key, value in somedict.items():
splitkey = key.split(".")
print(f"doing {key} {value} {splitkey}")
# subdict is the dict that goes deeper in the nested structure
subdict = unflattened
for subkey in splitkey[:-1]:
# if this is the first time we see this key, add it
if subkey not in subdict:
subdict[subkey] = {}
# shift the subdict a level deeper
subdict = subdict[subkey]
# add the value
subdict[splitkey[-1]] = value
return unflattened
data = {
"key": "timestamp",
"source": "eia007",
"turnover": "65million",
"url": "abc.com",
"record": "",
"loc.reg": "nord000",
"loc.count": "abs39i5",
"loc.town": "cold54",
"co.gdp": "nscrt77",
"co.pop.min": "min50",
"co.pop.max": "max75",
"co.rev": "",
"chain.system": "5t5t5",
"chain.type": "765ef",
"chain.strat": "",
}
unflattened = unflatten(data)
print(json.dumps(unflattened, indent=4))
Which produces:
{
"key": "timestamp",
"source": "eia007",
"turnover": "65million",
"url": "abc.com",
"record": "",
"loc": {
"reg": "nord000",
"count": "abs39i5",
"town": "cold54"
},
"co": {
"gdp": "nscrt77",
"pop": {
"min": "min50",
"max": "max75"
},
"rev": ""
},
"chain": {
"system": "5t5t5",
"type": "765ef",
"strat": ""
}
}
Cheers!

powershell json format variable in tmsl script

I use powershell and i want to put variable in json format into TMSL create role script. I want to put this variable into TablePermission parameter
json variable:
$filter=
[
{
"name": "DimGeography",
"filterExpression": "DimGeography[CountryRegionCode] = \"US\" "
}
]
TMSL script:
$Query='
{
"create": {
"parentObject": {
"database": "AW Internet Sales"
},
"role": {
"name": "test_filter",
"modelPermission": "read",
"tablePermissions": "'+$filter+'"
}
}
}'
This is script which I want to parametrize
{
"createOrReplace": {
"object": {
"database": "AW Internet Sales",
"role": "SalesManagerUS"
},
"role": {
"name": "SalesManagerUS",
"modelPermission": "read",
"tablePermissions": [
{
"name": "DimGeography",
"filterExpression": "DimGeography[CountryRegionCode] = \"US\" "
}
]
}
}
}
How can I put $filter to $Query to receive working script like above ?
You could try the below snippet :
$filter=
'[
{
"name": "DimGeography",
"filterExpression": "DimGeography[CountryRegionCode] = \"US\" "
}
]'
function populate-tsml ($filter)
{
$Query='
{
"create": {
"parentObject": {
"database": "AW Internet Sales"
},
"role": {
"name": "test_filter",
"modelPermission": "read",
"tablePermissions": '+$filter+'
}
}
}'
return $Query
}
$Query = populate-tsml -filter $filter
OUTPUT :
Thank you!
It wasn't necesseary to define a function, a little changes in quotation marks was enough.
Script below works:
$filter=
'[
{
"name": "DimGeography",
"filterExpression": "DimGeography[CountryRegionCode] = \"US\" "
}
]'
$Query='
{
"create": {
"parentObject": {
"database": "AW Internet Sales"
},
"role": {
"name": "test_filter",
"modelPermission": "read",
"tablePermissions": '+$filter+'
}
}
}'

PowerShell Question using Azure ARM Template Parameter array (my implementation of ConvertFrom-Json using an array fails)

I'm trying to pass in a $paramObject into a new ARM template deployment. I think I am missing something in my PowerShell to correctly convert the arrays so they can be used in the template.
$ResourceGroupName = "rg-testtemplate"
New-AzResourceGroup -Name "rg-testtemplate" -Location "West US"
$addressPrefixes = '["10.0.0.0/24"]' | ConvertFrom-Json
$subnet = '[{"name":"default","properties":{"addressPrefix":"10.0.0.0/24"}}]' | ConvertFrom-Json
$paramObject = #{
'addressPrefixes' = $addressPrefixes
'subnets' = $subnet
'virtualNetworkName' = "vnet-testtemplate"
}
$parameters = #{
'ResourceGroupName' = $ResourceGroupName
'TemplateFile' = '.\testpowershell.json'
'TemplateParameterObject' = $paramObject
'Verbose' = $true
}
New-AzResourceGroupDeployment #parameters
My template takes arrays.
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"addressPrefixes": {
"type": "array"
},
"subnets": {
"type": "array"
},
"virtualNetworkName": {
"type": "string"
}
},
"resources": [
{
"name": "[parameters('virtualNetworkName')]",
"type": "Microsoft.Network/virtualNetworks",
"apiVersion": "2019-09-01",
"location": "West US",
"properties": {
"addressSpace": {
"addressPrefixes": "[parameters('addressPrefixes')]"
},
"subnets": "[parameters('subnets')]"
}
}
]
}
But when I go to deploy, I get back:
New-AzResourceGroupDeployment : 8:35:23 AM - Error: Code=InvalidTemplate; Message=Deployment template validation
failed: 'Template parameter JToken type is not valid. Expected 'Array'. Actual 'Object'. Please see
https://aka.ms/resource-manager-parameter-files for usage details.'.
At D:\repo\foldername\TestVM\testpowershelldeploy.ps1:30 char:5
+ New-AzResourceGroupDeployment #parameters
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [New-AzResourceGroupDeployment], Exception
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.NewAzureResourceGroupDep
loymentCmdlet
New-AzResourceGroupDeployment : The deployment validation failed
At D:\repo\foldername\TestVM\testpowershelldeploy.ps1:30 char:5
+ New-AzResourceGroupDeployment #parameters
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [New-AzResourceGroupDeployment], InvalidOperationException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.NewAzureResourceGroupDeploymentCmdlet
What is the "fix" for converting my Json to a PowerShell array that my template needs?
If you want to create an array object with PowerShell, please use the command $t=#() to create.
For example
My template.json
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"addressPrefixes": {
"type": "array"
},
"subnets": {
"type": "array"
},
"virtualNetworkName": {
"type": "string"
}
},
"resources": [
{
"name": "[parameters('virtualNetworkName')]",
"type": "Microsoft.Network/virtualNetworks",
"apiVersion": "2019-09-01",
"location": "West US",
"properties": {
"addressSpace": {
"addressPrefixes": "[parameters('addressPrefixes')]"
},
"subnets": "[parameters('subnets')]"
}
}
]
}
my deploy script
Connect-AzAccount
$addressPrefixes =#( "10.0.0.0/24")
$subnet1= #{
"name"= "default"
"properties"=#{
"addressPrefix"="10.0.0.0/24"
}
}
$subnets = #($subnet1)
$paramObject = #{
'addressPrefixes' = $addressPrefixes
'subnets' = $subnets
'virtualNetworkName' = "vnet-testtemplate"
}
$parameters = #{
'ResourceGroupName' = "testtable"
'TemplateFile' = 'E:\test1.json'
'TemplateParameterObject' = $paramObject
'Verbose' = $true
}
New-AzResourceGroupDeployment #parameters

Azure ARM Inline IF not working

I am trying to build an Azure VNET via .json template.
I am trying to use an inline conditional statement to either create a 2nd subnet or skip creating the 2nd subnet. I do not think I am using json('null') correctly, or if this is even possible. My understanding is if json('null') is chosen, nothing is chosen.
Any help is appreciated!
"apiVersion": "2016-06-01",
"type": "Microsoft.Network/virtualNetworks",
"name": "My-VNET",
"location": "[resourceGroup().location]",
"properties": {
"addressSpace": {
"addressPrefixes": [
"[parameters('virtualNetworkCIDR')]"
]
},
"subnets": [{
"name": "[parameters('firstSubnetName')]",
"properties": {
"addressPrefix": "10.10.1.0/24"
}
}, {
"name": "[if(equals(parameters('createSecondSubnet'), 'Yes'), parameters('secondSubnetName'), json('null'))]",
"properties": {
"addressPrefix": "10.10.2.0/24"
}
}
]
}
I'm sure that you managed to get some kind of solution to this a while ago ... however, I have a solution that works well for this sort of thing ... it doesn't use conditional statements though.
In PowerShell create some hash-tables like so ...
# Resource group Hashtables.
$rgDev = #{
Name = "DEV-RG"
SubscriptionId = $subNonProd
Location = "desiredregion"
}
$rgUat = #{
Name = "UAT-RG"
SubscriptionId = $subNonProd
Location = "desiredregion"
}
#Vnet Hashtables
$vnetDev = #{
ResourceGroup = $rgDev
VnetName = "Dev-vnet"
CIDR = #('x.x.x.x/27')
Subnets = #(
#{
Name = "Dev-Web-subnet"
CIDR = "y.y.y.y/28"
},
#{
Name = "Dev-DB-subnet"
CIDR = "z.z.z.z/28"
})
}
$vnetUat = #{
ResourceGroup = $rgUat
VnetName = "UAT-vnet"
CIDR = #('f.f.f.f/27')
Subnets = #(
#{
Name = "UAT-Web-subnet"
CIDR = "g.g.g.g/28"
},
#{
Name = "UAT-DB-subnet"
CIDR = "h.h.h.h/28"
})
}
Next I lump the hashtables into an array and foreach over the lot. I have a little script that switches my context so that I can deploy to multiple subscriptions in the one bootstrap type script.
$vnets = #($vnetDev, $vnetUat)
ForEach ($vn in $vnets) {
$deploymentName = $vn.VnetName + "_Deployment."
.\SwitchSubscription.ps1 -subName $vn.ResourceGroup.SubscriptionName -subId $vn.ResourceGroup.SubscriptionId
New-AzureRmResourceGroupDeployment -Name $deploymentName `
-ResourceGroupName $vn.ResourceGroup.Name `
-TemplateFile .\JSONFiles\Vnets.json `
-vnet $vn
}
The Parameters section of the ARM template looks like this ...
"parameters": {
"vnet": {
"type": "object",
}
},
then the Resource Section looks like this ...
{
"name": "[concat(parameters('vnet').VnetName)]",
"type": "Microsoft.Network/virtualNetworks",
"apiVersion": "2017-10-01",
"location": "[resourceGroup().location]",
"tags": "[parameters('vnet').tags]",
"properties": {
"addressSpace": {
"addressPrefixes": "[parameters('vnet').CIDR]"
},
"copy": [
{
"name": "subnets",
"count": "[length(parameters('vnet').Subnets)]",
"input": {
"name": "[parameters('vnet').Subnets[copyIndex('Subnets')].Name]",
"properties": {
"addressPrefix": "[parameters('vnet').Subnets[copyIndex('Subnets')].CIDR]"
}
}
}
]
}
}
]
So what this all does is pass an object through to the ARM template that can have one Vnet with one or more subnets and create them all.
Hopefully this will help someone else out when / if they find this when googling.
Cheers,
Dave.
:)
Normally to conditionally create resources in a template, you can use the "Condition" property: https://learn.microsoft.com/en-us/azure/architecture/building-blocks/extending-templates/conditional-deploy
if you would like to create multiple of a single type of resource, you can use the "Copy" property: https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-multiple#resource-iteration
Unfortunately, subnets do not have a condition or a copy property, because they are a sub-resource of the "Virtual Network" resource. Because of this, the entire VNET will need to be conditional, and you can specify multiple VNETS with only 1 of them deploying. The VNETS can also not have the same name, so you will need to specify multiple VNETS in your template.
For Example:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"NumberofSubnets": {
"type": "string",
"allowedValues": ["1","2"],
"metadata": {
"description": "would you like to deploy 1 or 2 subnets?"
}
}
},
"resources": [
{
"type": "Microsoft.Network/virtualNetworks",
"apiVersion": "2016-06-01",
"name": "My-VNET1",
"location": "[resourceGroup().location]",
"condition": "[equals(parameters('NumberofSubnets'), 1)]",
"properties": {
"addressSpace": {
"addressPrefixes": ["10.10.0.0/23"]
},
"subnets": [{
"name": "Subnet1",
"properties": {
"addressPrefix": "10.10.0.0/24"
}
}
]
}
},
{
"type": "Microsoft.Network/virtualNetworks",
"apiVersion": "2016-06-01",
"name": "My-VNET2",
"location": "[resourceGroup().location]",
"condition": "[equals(parameters('NumberofSubnets'), 2)]",
"properties": {
"addressSpace": {
"addressPrefixes": ["10.10.0.0/23"]
},
"subnets": [{
"name": "Subnet1",
"properties": {
"addressPrefix": "10.10.0.0/24"
}
},
{
"name": "Subnet2",
"properties": {
"addressPrefix": "10.10.1.0/24"
}
}
]
}
}
]
}
This will solve your problem, but given a large amount of Subnets, you can see how a template can get very tedious.
The Best but most complicated way is to use linked templates. This Repository shows how you can create a dynamic number of subnets using linked templates

Search JSON for multiple values, not using a library

I'd like to be able to search the following JSON object for objects containing the key 'location' then get in return an array or json object with the 'name' of the person plus the value of location for that person.
Sample return:
var matchesFound = [{Tom Brady, New York}, {Donald Steven,Los Angeles}];
var fbData0 = {
"data": [
{
"id": "X999_Y999",
"location": "New York",
"from": {
"name": "Tom Brady", "id": "X12"
},
"message": "Looking forward to 2010!",
"actions": [
{
"name": "Comment",
"link": "http://www.facebook.com/X999/posts/Y999"
},
{
"name": "Like",
"link": "http://www.facebook.com/X999/posts/Y999"
}
],
"type": "status",
"created_time": "2010-08-02T21:27:44+0000",
"updated_time": "2010-08-02T21:27:44+0000"
},
{
"id": "X998_Y998",
"location": "Los Angeles",
"from": {
"name": "Donald Steven", "id": "X18"
},
"message": "Where's my contract?",
"actions": [
{
"name": "Comment",
"link": "http://www.facebook.com/X998/posts/Y998"
},
{
"name": "Like",
"link": "http://www.facebook.com/X998/posts/Y998"
}
],
"type": "status",
"created_time": "2010-08-02T21:27:44+0000",
"updated_time": "2010-08-02T21:27:44+0000"
}
]
};
#vsiege - you can use this javascript lib (http://www.defiantjs.com/) to search your JSON structure.
var fbData0 = {
...
},
res = JSON.search( fbData0, '//*[./location and ./from/name]' ),
str = '';
for (var i=0; i<res.length; i++) {
str += res[i].location +': '+ res[i].from.name +'<br/>';
}
document.getElementById('output').innerHTML = str;
Here is a working fiddle;
http://jsfiddle.net/hbi99/XhRLP/
DefiantJS extends the global object JSON with the method "search" and makes it possible to query JSON with XPath expressions (XPath is standardised query language). The method returns an array with the matches (empty array if none were found).
You can test XPath expressions by pasting your JSON here:
http://www.defiantjs.com/#xpath_evaluator