How do I pass secret value from keyvault using Azure ARM Template - json
I am trying to create Azure Application Gateway with ssl certificate from keyvaults. But didn't find any option to add keyvaults to ARM template with .pfx and .cer files. So I have encoded the certificate contents and added as secret in existing keyvault. Now trying to pass the secrets using ARM template. Validation passed but getting error in Deployment stage. Attached the template and parameters I am using.
Getting Error while deploying the resource
Deployment template validation failed: 'Template parameter JToken type is not valid. Expected 'String, Uri'. Actual 'Object'
"additionalInfo": [
{
"type": "TemplateViolation",
"info": {
"lineNumber": 226,
"linePosition": 33,
"path": "properties.template.parameters.appgwfesslcertsecret"
}
}
]
Updated Template:
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"namingSettings": {
"type": "object"
},
"taggingSettings": {
"type": "object"
},
"applicationGatewaySettings": {
"type": "object"
},
"appgwfesslcertsecret": {
"type": "securestring"
},
"appgwbecertsecret": {
"type": "securestring"
}
},
"variables": {
"namePrefix": "[concat(parameters('namingSettings').name.org,'-',parameters('namingSettings').name.cloud,'-',parameters('namingSettings').name.region,'-',parameters('namingSettings').name.businessUnit,'-',parameters('namingSettings').name.account,'-',parameters('namingSettings').name.app,'-',parameters('namingSettings').name.sdlc,'-')]" },
"resources": [
{
"apiVersion": "2018-11-01",
"name": "[concat(variables('namePrefix'),parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].name)]",
"type": "Microsoft.Network/applicationGateways",
"location": "[resourceGroup().location]",
"copy": {
"name": "appgwCopy",
"count": "[length(parameters('applicationGatewaySettings').settings)]"
},
"tags": "[parameters('taggingSettings').tags]",
"properties": {
"sku": {
"name": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].applicationGatewaySku]",
"tier": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].applicationGatewayTier]",
"capacity": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].applicationGatewayInstanceCount]"
},
"sslPolicy": {
"policyType": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].policyType]",
"policyName": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].policy]"
},
"copy": [
{
"name": "frontendPorts",
"count": "[length(parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].frontendPorts)]",
"input": {
"name": "[concat(variables('namePrefix'),parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].frontendPorts[copyIndex('frontendPorts')].name)]",
"properties": {
"port": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].frontendPorts[copyIndex('frontendPorts')].properties.port]"
}
}
},
{
"name": "gatewayIPConfigurations",
"count": "[length(parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].gatewayIPConfigurations)]",
"input": {
"name": "[concat(variables('namePrefix'),parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].gatewayIPConfigurations[copyIndex('gatewayIPConfigurations')].name)]",
"properties": {
"subnet": {
"id": "[resourceId(parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].gatewayIPConfigurations[copyIndex('gatewayIPConfigurations')].properties.subnet.vnetRGName,'microsoft.network/virtualnetworks/subnets', parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].gatewayIPConfigurations[copyIndex('gatewayIPConfigurations')].properties.subnet.vnetName, parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].gatewayIPConfigurations[copyIndex('gatewayIPConfigurations')].properties.subnet.subnetName)]"
}
}
}
},
{
"name": "frontendIPConfigurations",
"count": "[length(parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].frontendIPConfigurations)]",
"input": {
"name": "[concat(variables('namePrefix'),parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].frontendIPConfigurations[copyIndex('frontendIPConfigurations')].name)]",
"properties": {
"subnet": {
"id": "[resourceId(parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].gatewayIPConfigurations[copyIndex('frontendIPConfigurations')].properties.subnet.vnetRGName,'microsoft.network/virtualnetworks/subnets', parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].frontendIPConfigurations[copyIndex('frontendIPConfigurations')].properties.subnet.vnetName, parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].frontendIPConfigurations[copyIndex('frontendIPConfigurations')].properties.subnet.subnetName)]"
}
}
}
},
{
"name": "backendHttpSettingsCollection",
"count": "[length(parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].backendHttpSettingsCollection)]",
"input": {
"name": "[concat(variables('namePrefix'),parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].backendHttpSettingsCollection[copyIndex('backendHttpSettingsCollection')].name)]",
"properties": {
"port": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].backendHttpSettingsCollection[copyIndex('backendHttpSettingsCollection')].properties.port]",
"protocol": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].backendHttpSettingsCollection[copyIndex('backendHttpSettingsCollection')].properties.protocol]",
"authenticationCertificates": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].backendHttpSettingsCollection[copyIndex('backendHttpSettingsCollection')].properties.authenticationCertificates]"
}
}
},
{
"name": "backendAddressPools",
"count": "[length(parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].backendAddressPools)]",
"input": {
"name": "[concat(variables('namePrefix'),parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].backendAddressPools[copyIndex('backendAddressPools')].name)]"
}
},
{
"name": "httpListeners",
"count": "[length(parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].httpListeners)]",
"input": {
"name": "[concat(variables('namePrefix'),parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].httpListeners[copyIndex('httpListeners')].name)]",
"properties": {
"frontendIPConfiguration": {
"id": "[resourceId(resourceGroup().name, 'microsoft.network/applicationGateways/frontendIPConfigurations', concat(variables('namePrefix'),parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].name),concat(variables('namePrefix'), parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].httpListeners[copyIndex('httpListeners')].properties.frontendIPConfiguration))]"
},
"frontendPort": {
"id": "[resourceId(resourceGroup().name, 'microsoft.network/applicationGateways/frontendPorts', concat(variables('namePrefix'),parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].name),concat(variables('namePrefix'), parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].httpListeners[copyIndex('httpListeners')].properties.frontendPort))]"
},
"protocol": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].httpListeners[copyIndex('httpListeners')].properties.protocol]",
"sslCertificate": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].httpListeners[copyIndex('httpListeners')].properties.sslCertificate]"
}
}
},
{
"name": "requestRoutingRules",
"count": "[length(parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].requestRoutingRules)]",
"input": {
"name": "[concat(variables('namePrefix'),parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].requestRoutingRules[copyIndex('requestRoutingRules')].name)]",
"properties": {
"httpListener": {
"id": "[resourceId(resourceGroup().name, 'microsoft.network/applicationGateways/httpListeners', concat(variables('namePrefix'),parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].name),concat(variables('namePrefix'), parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].requestRoutingRules[copyIndex('requestRoutingRules')].properties.httpListener))]"
},
"backendAddressPool": {
"id": "[resourceId(resourceGroup().name, 'microsoft.network/applicationGateways/backendAddressPools', concat(variables('namePrefix'),parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].name),concat(variables('namePrefix'), parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].requestRoutingRules[copyIndex('requestRoutingRules')].properties.backendAddressPool))]"
},
"backendHttpSettings": {
"id": "[resourceId(resourceGroup().name, 'microsoft.network/applicationGateways/backendHttpSettingsCollection', concat(variables('namePrefix'),parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].name),concat(variables('namePrefix'), parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].requestRoutingRules[copyIndex('requestRoutingRules')].properties.backendHttpSettings))]"
}
}
}
},
{
"name": "sslCertificates",
"count": "[length(parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].sslCertificates)]",
"input": {
"name": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].sslCertificates[copyIndex('sslCertificates')].name]",
"properties": {
"data": "[parameters('appgwfesslcertsecret')]",
"password": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].sslCertificates[copyIndex('sslCertificates')].properties.password]"
}
}
},
{
"name": "authenticationCertificates",
"count": "[length(parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].authenticationCertificates)]",
"input": {
"name": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].authenticationCertificates[copyIndex('authenticationCertificates')].name]",
"properties": {
"data": "[parameters('appgwbecertsecret')]"
}
}
}
],
"probes": [],
"webApplicationFirewallConfiguration": {
"enabled": true,
"firewallMode": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].firewallMode]",
"ruleSetType": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].ruleSetType]",
"ruleSetVersion": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].ruleSetVersion]",
"requestBodyCheck": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].requestBodyCheck]",
"maxRequestBodySizeInKb": "[if(parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].requestBodyCheck, parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].maxReqBodySize, json('null'))]",
"fileUploadLimitInMb": "[int(100)]"
},
"enableHttp2": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].enableHTTP2]"
},
"resources": [
{
"type": "providers/diagnosticSettings",
"name": "[concat('Microsoft.Insights/', parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].diagname)]",
"dependsOn": [
"[concat(variables('namePrefix'),parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].name)]"
],
"apiVersion": "2017-05-01-preview",
"properties": {
"name": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].diagname]",
"logs": [
{
"category": "ApplicationGatewayAccessLog",
"enabled": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].enableApplicationGatewayAccessLog]",
"retentionPolicy": {
"days": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].applicationGatewayAccessLogRetentionDays]",
"enabled": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].enableApplicationGatewayAccessLogRetention]"
}
},
{
"category": "ApplicationGatewayPerformanceLog",
"enabled": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].enableApplicationGatewayPerformanceLog]",
"retentionPolicy": {
"days": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].applicationGatewayPerformanceLogRetentionDays]",
"enabled": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].enableApplicationGatewayPerformanceLogRetention]"
}
},
{
"category": "ApplicationGatewayFirewallLog",
"enabled": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].enableApplicationGatewayFirewallLog]",
"retentionPolicy": {
"days": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].applicationGatewayFirewallLogRetentionDays]",
"enabled": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].enableApplicationGatewayFirewallLogRetention]"
}
}
],
"metrics": [
{
"category": "AllMetrics",
"enabled": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].enableAllMetrics]",
"retentionPolicy": {
"enabled": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].enableAllMetricsRetentionPolicy]",
"days": "[parameters('applicationGatewaySettings').settings[copyIndex('appgwCopy')].allMetricsRetentionDays]"
}
}
]
}
}
]
}
],
"outputs": {}
}
Updated Parameter File:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"namingSettings": {
"value": {
"name": {
"app": "demo",
"cloud": "azu",
"region": "eus2",
}
}
},
"taggingSettings": {
"value": {
"tags": {
"AppID": "demo",
"Environment": "nonprod",
}
}
},
"applicationGatewaySettings": {
"value": {
"settings": [
{
"name": "appgw-pcs01",
"applicationGatewaySku": "WAF_Medium",
"applicationGatewayTier": "WAF",
"applicationGatewayInstanceCount": 2,
"policyType": "Predefined",
"policy": "AppGwSslPolicy20170401S",
"publicIP": null,
"firewallMode": "Prevention",
"diagname": "Demo-Appgw",
"ruleSetType": "OWASP",
"ruleSetVersion": "3.0",
"requestBodyCheck": true,
"maxReqBodySize": 10,
"enableHTTP2": false,
"enableApplicationGatewayAccessLog": true,
"applicationGatewayAccessLogRetentionDays": 30,
"enableApplicationGatewayAccessLogRetention": true,
"enableApplicationGatewayPerformanceLog": true,
"applicationGatewayPerformanceLogRetentionDays": 30,
"enableApplicationGatewayPerformanceLogRetention": true,
"enableApplicationGatewayFirewallLog": true,
"applicationGatewayFirewallLogRetentionDays": 30,
"enableApplicationGatewayFirewallLogRetention": true,
"enableAllMetrics": true,
"enableAllMetricsRetentionPolicy": true,
"allMetricsRetentionDays": 30,
"frontendPorts": [
{
"name": "feport-80",
"properties": {
"port": 80
}
},
{
"name": "feport-443",
"properties": {
"port": 443
}
}
],
"gatewayIPConfigurations": [
{
"name": "gwipconfig-pcs01",
"properties": {
"subnet": {
"vnetName": "demo-vnet",
"vnetRGName": "demo",
"subnetName": "demo-subgw"
}
}
}
],
"sslCertificates": [
{
"name": "appgwfesslcert",
"properties": {
"data": null,
"password": "password"
}
}
],
"authenticationCertificates": [
{
"name": "appgwbecert",
"properties": {
"data": null
}
}
],
"frontEndIPConfigurations": [
{
"name": "feipcfg-pcs01",
"properties": {
"subnet": {
"vnetName": "demo-vnet",
"vnetRGName": "demo",
"subnetName": "demo-subgw"
}
}
}
],
"httpListeners": [
{
"name": "httplistener-pcs01",
"properties": {
"frontendIPConfiguration": "feipcfg-pcs01",
"frontendPort": "feport-80",
"protocol": "Http",
"sslCertificate": {}
}
},
{
"name": "httpslistener-pcs01",
"properties": {
"frontendIPConfiguration": "feipcfg-pcs01",
"frontendPort": "feport-443",
"protocol": "Https",
"sslCertificate": {
"id": "/subscriptions/105dcee5-gy46-48e3-9046-265c7379e647/resourceGroups/demo/providers/Microsoft.Network/applicationGateways/azu-eus2-nonprod-appgw-pcs01/sslCertificates/appgwfesslcert"
}
}
}
],
"backendHttpSettingsCollection": [
{
"name": "httpsetcol-default",
"properties": {
"protocol": "Http",
"port": 80,
"authenticationCertificates": []
}
},
{
"name": "httpssetcol-default",
"properties": {
"protocol": "Https",
"port": 443,
"authenticationCertificates": [
{
"id": "/subscriptions/105dcee5-gy46-48e3-9046-265c7379e647/resourceGroups/demo/providers/Microsoft.Network/applicationGateways/azu-eus2-nonprod-appgw-pcs01/authenticationCertificates/appgwbecert"
}
]
}
}
],
"backendAddressPools": [
{
"name": "beap-pcs01"
}
],
"requestRoutingRules": [
{
"name": "httpreqrtrule-pcs01",
"properties": {
"httpListener": "httplistener-pcs01",
"backendAddressPool": "beap-pcs01",
"backendHttpSettings": "httpsetcol-default"
}
},
{
"name": "httpsreqrtrule-pcs01",
"properties": {
"httpListener": "httpslistener-pcs01",
"backendAddressPool": "beap-pcs01",
"backendHttpSettings": "httpssetcol-default"
}
}
]
}
]
}
},
"appgwfesslcertsecret": {
"value": {
"reference": {
"keyVault": {
"id": "/subscriptions/105dcee5-gy46-48e3-9046-265c7379e647/resourceGroups/demo/providers/Microsoft.KeyVault/vaults/demo-kv-new"
},
"secretName": "appgwfesslcert"
}
}
},
"appgwbecertsecret": {
"value": {
"reference": {
"keyVault": {
"id": "/subscriptions/105dcee5-gy46-48e3-9046-265c7379e647/resourceGroups/demo/providers/Microsoft.KeyVault/vaults/demo-kv-new"
},
"secretName": "appgwbecert"
}
}
}
}
}
you can only reference KV secrets in the parameters section of the template (or parameters file). You cannot use it in a random place in the template
https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/template-tutorial-use-key-vault
Related
cloudformation failed to create subnets
I was trying to run a code and i had this error but cant identify the problem. i got the error message The CIDR '10.0.1.0/24' conflicts with another subnet (Service: AmazonEC2; Status Code: 400; Error Code: InvalidSubnet.Conflict; Request ID: e0de23a8-d921-475f-aadd-84dac3109664; Proxy: null) { "AWSTemplateFormatVersion": "2010-09-09", "Description": "This is a network with one Vpc, 4 Subnet; 2 private, 2 public", "Metadata": {}, "Parameters": { "MyVpcCidr": { "Description": "This is the cidr for appVpc", "Type": "String", "Default": "10.0.0.0/16" }, "AZ1": { "Description": "AZ 1 for my network", "Type": "AWS::EC2::AvailabilityZone::Name" }, "Priv1Cidr": { "Description": "This is the cidr for my appPriv1Subnet", "Type": "String", "Default": "10.0.1.0/24" }, "Priv2Cidr": { "Description": "This is the cidr for my appPriv2Subnet", "Type": "String", "Default": "10.0.3.0/24" }, "AZ2": { "Description": "AZ 2 for my network", "Type": "AWS::EC2::AvailabilityZone::Name" }, "Pub1Cidr": { "Description": "Cidr for my appPubSN1", "Type": "String", "Default": "10.0.2.0/24" }, "Pub2Cidr": { "Description": "Cidr for appPubSN2", "Type": "String", "Default": "10.0.4.0/16" } }, "Mappings": {}, "Conditions": {}, "Resources": { "appVpc": { "Type": "AWS::EC2::VPC", "Properties": { "CidrBlock": { "Ref": "MyVpcCidr" }, "Tags": [ { "Key": "Name", "Value": "AppVpc" } ] } }, "appPriv1Subnet": { "Type": "AWS::EC2::Subnet", "Properties": { "AvailabilityZone": { "Ref": "AZ1" }, "VpcId": { "Ref": "appVpc" }, "CidrBlock": { "Ref": "Priv1Cidr" }, "Tags": [ { "Key": "Name", "Value": "Apppriv1subnet" } ] } }, "appPriv2Subnet": { "Type": "AWS::EC2::Subnet", "Properties": { "AvailabilityZone": { "Ref": "AZ2" }, "VpcId": { "Ref": "appVpc" }, "CidrBlock": { "Ref": "Priv2Cidr" }, "Tags": [ { "Key": "Name", "Value": "AppPriv2Subnet" } ] } }, "appPubSN1": { "Type": "AWS::EC2::Subnet", "Properties": { "AvailabilityZone": { "Ref": "AZ1" }, "VpcId": { "Ref": "appVpc" }, "CidrBlock": { "Ref": "Pub1Cidr" }, "Tags": [ { "Key": "Name", "Value": "AppPubsn1" } ] } }, "appPubSN2": { "Type": "AWS::EC2::Subnet", "Properties": { "AvailabilityZone": { "Ref": "AZ2" }, "VpcId": { "Ref": "appVpc" }, "CidrBlock": { "Ref": "Pub2Cidr" }, "Tags": [ { "Key": "Name", "Value": "AppPubsn2" } ] } }, "appIG": { "Type": "AWS::EC2::InternetGateway", "Properties": { "Tags": [ { "Key": "Name", "Value": "AppIG" } ] } }, "AttachGateway": { "Type": "AWS::EC2::VPCGatewayAttachment", "Properties": { "VpcId": { "Ref": "appVpc" }, "InternetGatewayId": { "Ref": "appIG" } } }, "appPrivRT": { "Type": "AWS::EC2::RouteTable", "Properties": { "VpcId": { "Ref": "appVpc" }, "Tags": [ { "Key": "Name", "Value": "AppPrivRt" } ] } }, "PrivRTA1": { "Type": "AWS::EC2::SubnetRouteTableAssociation", "Properties": { "SubnetId": { "Ref": "appPriv1Subnet" }, "RouteTableId": { "Ref": "appPrivRT" } } }, "PrivRTA2": { "Type": "AWS::EC2::SubnetRouteTableAssociation", "Properties": { "SubnetId": { "Ref": "appPriv2Subnet" }, "RouteTableId": { "Ref": "appPrivRT" } } }, "appEIP": { "Type": "AWS::EC2::EIP", "Properties": { "Domain": "vpc" } }, "appNatgw": { "Type": "AWS::EC2::NatGateway", "Properties": { "AllocationId": { "Fn::GetAtt": [ "appEIP", "AllocationId" ] }, "SubnetId": { "Ref": "appPubSN1" }, "Tags": [ { "Key": "Name", "Value": "Appnatgw" } ] } }, "appPrivRoute": { "Type": "AWS::EC2::Route", "Properties": { "RouteTableId": { "Ref": "appPrivRT" }, "DestinationCidrBlock": "0.0.0.0/0", "NatGatewayId": { "Ref": "appNatgw" } } }, "appPubRT": { "Type": "AWS::EC2::RouteTable", "Properties": { "VpcId": { "Ref": "appVpc" }, "Tags": [ { "Key": "Name", "Value": "AppPubRT" } ] } }, "PubRTA1": { "Type": "AWS::EC2::SubnetRouteTableAssociation", "Properties": { "SubnetId": { "Ref": "appPubSN1" }, "RouteTableId": { "Ref": "appPubRT" } } }, "PubRTA2": { "Type": "AWS::EC2::SubnetRouteTableAssociation", "Properties": { "SubnetId": { "Ref": "appPubSN2" }, "RouteTableId": { "Ref": "appPubRT" } } }, "appPubRoute": { "Type": "AWS::EC2::Route", "Properties": { "RouteTableId": { "Ref": "appPubRT" }, "DestinationCidrBlock": "0.0.0.0/0", "GatewayId": { "Ref": "appIG" } } }, "appSG": { "Type": "AWS::EC2::SecurityGroup", "Properties": { "GroupDescription": "Allow ssh port 22 and port 80", "SecurityGroupIngress": [ { "IpProtocol": "tcp", "FromPort": "22", "ToPort": "22", "CidrIp": "0.0.0.0/0" }, { "IpProtocol": "tcp", "FromPort": "80", "ToPort": "80", "CidrIp": "0.0.0.0/0" } ], "VpcId": { "Ref": "appVpc" }, "Tags": [ { "Key": "Name", "Value": "AppSG" } ] } }, "internalSG": { "DependsOn": "appSG", "Type": "AWS::EC2::SecurityGroup", "Properties": { "GroupDescription": "Allow traffic from appSG", "SecurityGroupIngress": [ { "IpProtocol": "-1", "SourceSecurityGroupId": { "Ref": "appSG" } } ], "VpcId": { "Ref": "appVpc" }, "Tags": [ { "Key": "Name", "Value": "appinternalSG" } ] } } }, "Outputs": { "appVpcId": { "Description": "Id for my vpc ", "Value": { "Ref": "appVpc" }, "Export": { "Name": "appVpcid" } }, "appPrivSN1Id": { "Description": "Id for my private SN1", "Value": { "Ref": "appPriv1Subnet" }, "Export": { "Name": "appPrivSNID1" } }, "appPrivSN2Id": { "Description": "Id for my subnet 2 private", "Value": { "Ref": "appPriv2Subnet" }, "Export": { "Name": "appPrivSNID2" } }, "appPubSN1Id": { "Description": "Id for Public subnet 1", "Value": { "Ref": "appPubSN1" }, "Export": { "Name": "appPubSNID1" } }, "appPubSN2Id": { "Description": "Id for Public subnet 2", "Value": { "Ref": "appPubSN2" }, "Export": { "Name": "appPubSNID2" } }, "externalSgid": { "Description": "Id for external security group", "Value": { "Ref": "appSG" }, "Export": { "Name": "appSGID" } }, "internalSGId": { "Description": "Id for internal security group", "Value": { "Ref": "internalSG" }, "Export": { "Name": "internalSGID" } } } }
I suspect 10.0.4.0/16 is a typo that was meant to be 10.0.4.0/24. The reason is that the cidr 10.0.4.0/16, which you have set for Pub2Cidr starts at 10.0.0.0 and ends at 10.0.255.255, which overlaps with 10.0.1.0/24 which starts at 10.0.1.0 and ends at 10.0.1.255.
Azure APIM deployment is getting failed. "code": "ResourceDeploymentFailure"
Deployment is getting failed for Azure APIM with the below error. { "status": "Failed", "error": { "code": "ResourceDeploymentFailure", "message": "The resource operation completed with terminal provisioning state 'Failed'." } } Below is the ARM template for APIM: { "type": "Microsoft.ApiManagement/service", "apiVersion": "2020-12-01", "name": "[variables('apim-name')]", "location": "[parameters('location')]", "sku": { "name": "[parameters('apim-sku')]", "capacity": "[parameters('apim-capacity')]" }, "identity": { "type": "UserAssigned", "userAssignedIdentities": { "[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('Identity-name'))]": {} } }, "dependsOn": [ "[resourceId('Microsoft.Network/virtualNetworks', variables('vnet-name'))]", "[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('Identity-name'))]", "[resourceId('Microsoft.Network/virtualNetworks/subnets',variables('vnet-name'),variables('apimsubnet'))]" ], "resources": [ { "type": "gateways", "apiVersion": "2020-12-01", "name": "my-gateway", "dependsOn": [ "[resourceId('Microsoft.ApiManagement/service', variables('apim-name'))]" ], "properties": { "locationData": { "name": "My internal location" }, "description": "Self hosted gateway bringing API Management to the edge" } }, { "type": "loggers", "apiVersion": "2020-12-01", "name": "AppInsightsLogger", "dependsOn": [ "[resourceId('Microsoft.ApiManagement/service', variables('apim-name'))]" ], "properties": { "loggerType": "applicationInsights", "resourceId": "[resourceId('Microsoft.Insights/components', variables('app-insights-name'))]", "credentials": { "instrumentationKey": "[reference(variables('app-insights-name')).InstrumentationKey]" } } }, { "type": "diagnostics", "apiVersion": "2020-12-01", "name": "applicationinsights", "dependsOn": [ "[resourceId('Microsoft.ApiManagement/service', variables('apim-name'))]", "[resourceId('Microsoft.ApiManagement/service/loggers', variables('apim-name'), 'AppInsightsLogger')]" ], "properties": { "alwaysLog": "allErrors", "httpCorrelationProtocol": "Legacy", "verbosity": "information", "logClientIp": true, "loggerId": "[resourceId('Microsoft.ApiManagement/service/loggers/', variables('apim-name'), 'AppInsightsLogger')]", "sampling": { "samplingType": "fixed", "percentage": 100 }, "frontend": { "request": { "body": { "bytes": 0 } }, "response": { "body": { "bytes": 0 } } }, "backend": { "request": { "body": { "bytes": 0 } }, "response": { "body": { "bytes": 0 } } } } } ], "properties": { "publisherName": "[variables('apim-publisher-name')]", "publisherEmail": "[parameters('apim-publisher-email')]", "virtualNetworkType": "Internal", "virtualNetworkConfiguration": { "subnetResourceId": "[resourceId('Microsoft.Network/virtualNetworks/subnets', variables('vnet-name'), variables('apimsubnet'))]" } } },
Azure Data Factory V2 Copy Activity with Rest API giving one row for nested JSON
I am trying to flatten a nested JSON returned from a Rest source. The pipeline code is as follows. The problem here is this pipeline returns only first object from JSON dataset and skips all the rest of the rows. Can you please guide me on how to iterate over nested objects. Thanks Sameet { "name": "STG_NCR2", "properties": { "activities": [ { "name": "Copy data1", "type": "Copy", "dependsOn": [], "policy": { "timeout": "7.00:00:00", "retry": 0, "retryIntervalInSeconds": 30, "secureOutput": false, "secureInput": false }, "userProperties": [], "typeProperties": { "source": { "type": "RestSource", "httpRequestTimeout": "00:01:40", "requestInterval": "00.00:00:00.010", "requestMethod": "GET", "additionalHeaders": { "OData-MaxVersion": "4.0", "OData-Version": "4.0", "Prefer": "odata.include-annotations=*" } }, "sink": { "type": "AzureSqlSink" }, "enableStaging": false, "translator": { "type": "TabularTranslator", "mappings": [ { "source": { "path": "$['value'][0]['tco_ncrid']" }, "sink": { "name": "NCRID" } }, { "source": { "path": "['tco_name']" }, "sink": { "name": "EquipmentSerialNumber" } } ], "collectionReference": "$['value'][0]['tco_ncr_tco_equipment']" } }, "inputs": [ { "referenceName": "Rest_PowerApps_NCR", "type": "DatasetReference" } ], "outputs": [ { "referenceName": "Prestaging_PowerApps_NCREquipments", "type": "DatasetReference" } ] } ], "annotations": [] } } The JSON is in the following format [ { "value":[ { "tco_ncrid":"abc-123", "tco_ncr_tco_equipment":[ { "tco_name":"abc" } ] }, { "tco_ncrid":"abc-456", "tco_ncr_tco_equipment":[ { "tco_name":"xyz" }, { "tco_name":"yzx" } } ] ] } ]
This can be resolved by amending the translator property as follows. "translator": { "type": "TabularTranslator", "mappings": [ { "source": { "path": "$.['value'][0].['tco_ncrid']" }, "sink": { "name": "NCRID", "type": "String" } }, { "source": { "path": "$.['value'][0].['tco_text_id']" }, "sink": { "name": "EquipmentDescription", "type": "String" } }, { "source": { "path": "['tco_name']" }, "sink": { "name": "EquipmentSerialNumber", "type": "String" } } ], "collectionReference": "$.['value'][*].['tco_ncr_tco_equipment']" } This code forces the pipeline to iterate over nested array but as you can see that the NCRID is hardcoded to first element of the value array. This is not exactly what I want as I am looking for all Equipment Serial Numbers against every NCRID. Still researching...
Kubernetes MySQL pod keeps crashing and shows Unable to lock ./ibdata1 error: 11
I have created new docker image and added in kubernetes statefulset yaml for mysql pod. When I scaled mysql pod to 1 , it's keeps crashing and throwing message unable to lock ./ibdata1 error: 11 I have googled lot about this error but none of them gave a solution. Appreciate if any one help me! Docker file: FROM mysql/mysql-server CMD [ "--max_connections=10000" ] And Created MySQL YAML like below: { "kind": "StatefulSet", "apiVersion": "apps/v1beta2", "metadata": { "name": "mysql-test", "namespace": "test", "selfLink": "/apis/apps/v1beta2/namespaces/test/statefulsets/mysql-test", "uid": "e7768a0b-faf9-11e9-a989-d8c497367e2a", "resourceVersion": "315823479", "generation": 33, "creationTimestamp": "2019-10-30T09:44:44Z", "labels": { "app": "mysql-test", "release": "mysql-test" } }, "spec": { "replicas": 1, "selector": { "matchLabels": { "app": "mysql-test" } }, "template": { "metadata": { "creationTimestamp": null, "labels": { "app": "mysql-test" } }, "spec": { "volumes": [ { "name": "data", "persistentVolumeClaim": { "claimName": "mysqldata" } }, { "name": "backup", "persistentVolumeClaim": { "claimName": "mysqlbackup" } } ], "containers": [ { "name": "latest", "image": "<Enterprise repository>/careercompass/mysql:latest", "ports": [ { "name": "mysql", "containerPort": 3306, "protocol": "TCP" } ], "env": [ { "name": "MYSQL_PASSWORD", "valueFrom": { "secretKeyRef": { "name": "mysql", "key": "mysql-password", "optional": true } } }, { "name": "MYSQL_ROOT_PASSWORD", "valueFrom": { "secretKeyRef": { "name": "mysql", "key": "mysql-root-password", "optional": true } } }, { "name": "MYSQL_USER", "value": "mysql" }, { "name": "MYSQL_DATABASE" } ], "resources": { "limits": { "cpu": "10", "memory": "10000Mi" }, "requests": { "cpu": "200m", "memory": "3000Mi" } }, "volumeMounts": [ { "name": "data", "mountPath": "/var/lib/mysql" }, { "name": "backup", "mountPath": "/var/lib/mysqlbackup" } ], "terminationMessagePath": "/var/lib/termination-log", "terminationMessagePolicy": "File", "imagePullPolicy": "IfNotPresent" } ], "restartPolicy": "Always", "terminationGracePeriodSeconds": 30, "dnsPolicy": "ClusterFirst", "securityContext": { "runAsUser": 999, "fsGroup": 999 }, "schedulerName": "default-scheduler" } }, "serviceName": "", "podManagementPolicy": "OrderedReady", "updateStrategy": { "type": "RollingUpdate", "rollingUpdate": { "partition": 0 } }, "revisionHistoryLimit": 10 }, "status": { "observedGeneration": 33, "replicas": 1, "currentReplicas": 1, "updatedReplicas": 1, "currentRevision": "mysql-test-68cb64885c", "updateRevision": "mysql-test-68cb64885c", "collisionCount": 0 } }
deploy cloudera on azure FAILED, error with JSON / ARM template function 'copyIndex'
I m trying to install cloudera on azure using the link below https://github.com/Azure/azure-quickstart-templates/tree/master/cloudera-on-centos It gave me error "message": "Deployment template validation failed: 'The template resource 'master-node' at line '493' and column '9' is not valid: The template function 'copyIndex' is not expected at this location. The function can only be used in a resource with copy specified. Please see https://aka.ms/arm-copy for usage details.. Please see https://aka.ms/arm-template-expressions for usage details The code looks like this "name": "master-node", "type": "Microsoft.Resources/deployments", "apiVersion": "2018-11-01", "dependsOn": [ "Microsoft.Resources/deployments/shared-resources" ], "properties": { "mode": "Incremental", "template": { "uri": "[concat(variables('scriptsUri'), '/master-node.json')]", "contentVersion": "1.0.0.1", "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json", "parameters": { "vnetID": { "type": "string" }, "resourceAPIVersion": { "type": "string" }, "dnsNamePrefix": { "type": "string", "defaultValue": "azuredevcluster" }, "scriptsUri": { "type": "string" }, "storageAccount": { "type": "object" }, "vmCount": { "type": "int" }, "vmSpec": { "type": "secureObject" }, "networkSpec": { "type": "object" }, "clusterSpec": { "type": "object" }, "location": { "type": "string", "defaultValue": "[resourceGroup().location]", "metadata": { "description": "Location for all resources." } } }, "variables": { "singleQuote": "'", "masterIP": "[parameters('networkSpec').masterIP]", "workerIP": "[parameters('networkSpec').workerIP]", "vmName": "[concat(parameters('dnsNamePrefix'), '-mn')]", "nicName": "[concat(variables('vmName'), '-nic')]", "publicIPAddressName": "[concat(variables('vmName'), '-publicIP')]", "storageAccountName": "[parameters('storageAccount').prefix]", "securityGroupName": "[concat(parameters('dnsNamePrefix'), '-mn-sg')]" }, "resources": [ { "type": "Microsoft.Network/networkSecurityGroups", "name": "[variables('securityGroupName')]", "location": "[parameters('location')]", "properties": { "securityRules": [ { "name": "SSH", "properties": { "description": "Allows SSH traffic", "protocol": "Tcp", "sourcePortRange": "*", "destinationPortRange": "22", "sourceAddressPrefix": "*", "destinationAddressPrefix": "*", "access": "Allow", "priority": 100, "direction": "Inbound" } } ] } }, { "type": "Microsoft.Network/publicIPAddresses", "name": "[concat(variables('publicIPAddressName'), copyIndex())]", "location": "[parameters('location')]", "copy": { "name": "publicIPLoop", "count": "[parameters('vmCount')]" }, "properties": { "publicIPAllocationMethod": "Dynamic", "dnsSettings": { "domainNameLabel": "[concat(parameters('dnsNamePrefix'), '-mn', copyIndex())]" } } }, { "type": "Microsoft.Network/networkInterfaces", "name": "[concat(variables('nicName'), copyIndex())]", "location": "[parameters('location')]", "dependsOn": [ "[concat('Microsoft.Network/publicIPAddresses/', variables('publicIPAddressName'), copyIndex())]", "[concat('Microsoft.Network/networkSecurityGroups/', variables('securityGroupName'))]" ], "copy": { "name": "masterNodeNicLoop", "count": "[parameters('vmCount')]" }, "properties": { "ipConfigurations": [ { "name": "ipconfig1", "properties": { "privateIPAllocationMethod": "Static", "privateIPAddress": "[concat(parameters('networkSpec').ipOctet01, add(parameters('networkSpec').ipOctet2, div(copyIndex(parameters('networkSpec').ipOctet3), 256)), '.', mod(copyIndex(parameters('networkSpec').ipOctet3), 256))]", "subnet": { "id": "[concat(parameters('vnetID'), '/subnets/', parameters('networkSpec').virtualNetworkSubnetName)]" }, "publicIPAddress": { "id": "[resourceId('Microsoft.Network/publicIPAddresses',concat(variables('publicIPAddressName'), copyIndex()))]" } } } ], "networkSecurityGroup": { "id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('securityGroupName'))]" } } }, { "type": "Microsoft.Compute/virtualMachines", "apiVersion": "2017-03-30", "name": "[concat(variables('vmName'), copyIndex())]", "plan": "[parameters('vmSpec').imageInfo.plan]", "location": "[parameters('location')]", "copy":{ "name": "masterNodeNicLoop", "count": "[parameters('vmCount')]" }, "dependsOn": [ "[concat('Microsoft.Network/networkInterfaces/', variables('nicName'), copyIndex())]" ], "properties": { "availabilitySet": { "id": "[resourceId('Microsoft.Compute/availabilitySets', parameters('vmSpec').masterNodeASName)]" }, "hardwareProfile": { "vmSize": "[parameters('vmSpec').vmSize]" }, "osProfile": { "computerName": "[reference(concat(variables('publicIPAddressName'), copyIndex())).dnsSettings.fqdn]", "adminUsername": "[parameters('vmSpec').adminUsername]", "adminPassword": "[parameters('vmSpec').adminPassword]", "linuxConfiguration": { "disablePasswordAuthentication": false } }, "storageProfile": { "imageReference": "[parameters('vmSpec').imageInfo.imageReference]", "osDisk": { "name":"[concat(variables('vmName'), copyIndex(),'_OSDisk')]", "caching": "ReadWrite", "createOption": "FromImage" }, "dataDisks": [ { "name": "[concat(variables('vmName'), copyIndex(),'_DataDisk1')]", "diskSizeGB": 512, "lun": 0, "createOption": "Empty" }, { "name": "[concat(variables('vmName'), copyIndex(),'_DataDisk2')]", "diskSizeGB": 512, "lun": 1, "createOption": "Empty" }, { "name": "[concat(variables('vmName'), copyIndex(),'_DataDisk3')]", "diskSizeGB": 512, "lun": 2, "createOption": "Empty" }, { "name": "[concat(variables('vmName'), copyIndex(),'_DataDisk4')]", "diskSizeGB": 512, "lun": 3, "createOption": "Empty" } ] }, "networkProfile": { "networkInterfaces": [ { "id": "[resourceId('Microsoft.Network/networkInterfaces',concat(variables('nicName'), copyIndex()))]" } ] } } }, { "type": "Microsoft.Compute/virtualMachines/extensions", "name": "[concat(variables('vmName'), copyIndex(), '/prepareDisks')]", "location": "[parameters('location')]", "copy": { "name": "nameNodeNicLoop", "count": "[parameters('vmCount')]" }, "dependsOn": [ "[concat('Microsoft.Compute/virtualMachines/', variables('vmName'), copyIndex())]" ], "properties": { "publisher": "Microsoft.Azure.Extensions", "type": "CustomScript", "typeHandlerVersion": "2.0", "autoUpgradeMinorVersion": true, "settings": { "fileUris": [ "[concat(parameters('scriptsUri'), '/scripts/prepare-masternode-disks.sh')]", "[concat(parameters('scriptsUri'), '/scripts/initialize-node.sh')]" ], "commandToExecute": "[concat('sh initialize-node.sh ', variables('singleQuote'), variables('masterIP'), variables('singleQuote'), ' ', variables('singleQuote'), variables('workerIP'), variables('singleQuote'), ' ', variables('singleQuote'), parameters('dnsNamePrefix'), variables('singleQuote'), ' ', variables('singleQuote'), reference(concat(variables('publicIPAddressName'), copyIndex())).dnsSettings.fqdn, variables('singleQuote'), ' ', variables('singleQuote'), parameters('clusterSpec').masterNodeCount, variables('singleQuote'), ' ', variables('singleQuote'), parameters('clusterSpec').dataNodeCount, variables('singleQuote'), ' ', variables('singleQuote'), parameters('vmSpec').adminUsername, variables('singleQuote'), ' ', 'masternode', ' >> /var/log/cloudera-azure-initialize.log 2>&1')]" } } } ], "outputs": { "fqdn": { "type": "string", "value": "[reference(concat(variables('publicIPAddressName'), '0')).dnsSettings.fqdn]" } } }, "parameters": { "vnetID": { "value": "[variables('VNetId')]" }, "resourceAPIVersion": { "value": "[variables('resourceAPIVersion')]" }, "dnsNamePrefix": { "value": "[parameters('dnsNamePrefix')]" }, "scriptsUri": { "value": "[variables('scriptsUri')]" }, "storageAccount": { "value": "[variables('masterStorageAccount')]" }, "vmCount": { "value": "[variables('clusterSpec').masterNodeCount]" }, "vmSpec": { "value": "[variables('vmSpec')]" }, "networkSpec": { "value": "[variables('networkSpec')]" }, "clusterSpec": { "value": "[variables('clusterSpec')]" } } } }, { "name": "data-node", "type": "Microsoft.Resources/deployments", "apiVersion": "2018-11-01", "dependsOn": [ "Microsoft.Resources/deployments/shared-resources" ], "properties": { "mode": "Incremental", "template": { "uri": "[variables('data-nodeUri')]", "contentVersion": "1.0.0.1", "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json", "parameters": { "vnetID": { "type": "string" }, "dnsNamePrefix": { "type": "string", "defaultValue": "azuredevcluster" }, "scriptsUri": { "type": "string" }, "vmCount": { "type": "int" }, "vmSpec": { "type": "secureObject" }, "networkSpec": { "type": "object" }, "clusterSpec": { "type": "object" }, "location": { "type": "string", "defaultValue": "[resourceGroup().location]", "metadata": { "description": "Location for all resources." } } }, "variables": { "singleQuote": "'", "masterIP": "[parameters('networkSpec').masterIP]", "workerIP": "[parameters('networkSpec').workerIP]", "vmName": "[concat(parameters('dnsNamePrefix'), '-dn')]", "nicName": "[concat(variables('vmName'), '-nic')]", "publicIPAddressName": "[concat(variables('vmName'), '-publicIP')]", "securityGroupName": "[concat(parameters('dnsNamePrefix'), '-dn-sg')]" }, "resources": [ { "type": "Microsoft.Network/networkSecurityGroups", "name": "[variables('securityGroupName')]", "location": "[parameters('location')]", "properties": { "securityRules": [ { "name": "SSH", "properties": { "description": "Allows SSH traffic", "protocol": "Tcp", "sourcePortRange": "*", "destinationPortRange": "22", "sourceAddressPrefix": "*", "destinationAddressPrefix": "*", "access": "Allow", "priority": 100, "direction": "Inbound" } } ] } }, { "type": "Microsoft.Network/publicIPAddresses", "name": "[concat(variables('publicIPAddressName'), copyIndex())]", "location": "[parameters('location')]", "copy": { "name": "publicIPLoop", "count": "[parameters('vmCount')]" }, "properties": { "publicIPAllocationMethod": "Dynamic", "dnsSettings": { "domainNameLabel": "[concat(parameters('dnsNamePrefix'), '-dn', copyIndex())]" } } }, { "type": "Microsoft.Network/networkInterfaces", "name": "[concat(variables('nicName'), copyIndex())]", "location": "[parameters('location')]", "dependsOn": [ "[concat('Microsoft.Network/publicIPAddresses/', variables('publicIPAddressName'), copyIndex())]", "[concat('Microsoft.Network/networkSecurityGroups/', variables('securityGroupName'))]" ], "copy": { "name": "dataNodeNicLoop", "count": "[parameters('vmCount')]" }, "properties": { "ipConfigurations": [ { "name": "ipconfig1", "properties": { "privateIPAllocationMethod": "Static", "privateIPAddress": "[concat(parameters('networkSpec').ipOctet01, add(parameters('networkSpec').ipOctet2, div(copyIndex(parameters('networkSpec').datanodeIpOctet3), 256)), '.', mod(copyIndex(parameters('networkSpec').datanodeIpOctet3), 256))]", "subnet": { "id": "[concat(parameters('vnetID'), '/subnets/', parameters('networkSpec').virtualNetworkSubnetName)]" }, "publicIPAddress": { "id": "[resourceId('Microsoft.Network/publicIPAddresses',concat(variables('publicIPAddressName'), copyIndex()))]" } } } ], "networkSecurityGroup": { "id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('securityGroupName'))]" } } }, { "type": "Microsoft.Compute/virtualMachines", "apiVersion": "2017-03-30", "name": "[concat(variables('vmName'), copyIndex())]", "plan": { "name": "cloudera-centos-6", "publisher": "cloudera", "product": "cloudera-centos-6" }, "location": "[parameters('location')]", "copy": { "name": "dataNodeNicLoop", "count": "[parameters('vmCount')]" }, "dependsOn": [ "[concat('Microsoft.Network/networkInterfaces/', variables('nicName'), copyIndex())]" ], "properties": { "availabilitySet": { "id": "[resourceId('Microsoft.Compute/availabilitySets', parameters('vmSpec').dataNodeASName)]" }, "hardwareProfile": { "vmSize": "[parameters('vmSpec').vmSize]" }, "osProfile": { "computerName": "[reference(concat(variables('publicIPAddressName'), copyIndex())).dnsSettings.fqdn]", "adminUsername": "[parameters('vmSpec').adminUsername]", "adminPassword": "[parameters('vmSpec').adminPassword]", "linuxConfiguration": { "disablePasswordAuthentication": false } }, "storageProfile": { "imageReference": { "publisher": "cloudera", "offer": "cloudera-centos-6", "sku": "cloudera-centos-6", "version": "latest" }, "osDisk": { "name": "[concat(variables('vmName'), copyIndex(),'_OSDisk')]", "caching": "ReadWrite", "createOption": "FromImage" }, "dataDisks": [ { "name": "[concat(variables('vmName'), copyIndex(),'_DataDisk1')]", "diskSizeGB": 512, "lun": 0, "createOption": "Empty" }, { "name": "[concat(variables('vmName'), copyIndex(),'_DataDisk2')]", "diskSizeGB": 1023, "lun": 1, "createOption": "Empty" }, { "name": "[concat(variables('vmName'), copyIndex(),'_DataDisk3')]", "diskSizeGB": 1023, "lun": 2, "createOption": "Empty" }, { "name": "[concat(variables('vmName'), copyIndex(),'_DataDisk4')]", "diskSizeGB": 1023, "lun": 3, "createOption": "Empty" }, { "name": "[concat(variables('vmName'), copyIndex(),'_DataDisk5')]", "diskSizeGB": 1023, "lun": 4, "createOption": "Empty" }, { "name": "[concat(variables('vmName'), copyIndex(),'_DataDisk6')]", "diskSizeGB": 1023, "lun": 5, "createOption": "Empty" } ] }, "networkProfile": { "networkInterfaces": [ { "id": "[resourceId('Microsoft.Network/networkInterfaces',concat(variables('nicName'), copyIndex()))]" } ] } } }, { "type": "Microsoft.Compute/virtualMachines/extensions", "name": "[concat(variables('vmName'), copyIndex(), '/prepareDisks')]", "location": "[parameters('location')]", "copy": { "name": "dataNodeNicLoop", "count": "[parameters('vmCount')]" }, "dependsOn": [ "[concat('Microsoft.Compute/virtualMachines/', variables('vmName'), copyIndex())]" ], "properties": { "publisher": "Microsoft.Azure.Extensions", "type": "CustomScript", "typeHandlerVersion": "2.0", "autoUpgradeMinorVersion": true, "settings": { "fileUris": [ "[concat(parameters('scriptsUri'), '/scripts/prepare-datanode-disks.sh')]", "[concat(parameters('scriptsUri'), '/scripts/initialize-node.sh')]" ], "commandToExecute": "[concat('sh initialize-node.sh ', variables('singleQuote'), variables('masterIP'), variables('singleQuote'), ' ', variables('singleQuote'), variables('workerIP'), variables('singleQuote'), ' ', variables('singleQuote'), parameters('dnsNamePrefix'), variables('singleQuote'), ' ', variables('singleQuote'), reference(concat(variables('publicIPAddressName'), copyIndex())).dnsSettings.fqdn, variables('singleQuote'), ' ', variables('singleQuote'), parameters('clusterSpec').masterNodeCount, variables('singleQuote'), ' ', variables('singleQuote'), parameters('clusterSpec').dataNodeCount, variables('singleQuote'), ' ', variables('singleQuote'), parameters('vmSpec').adminUsername, variables('singleQuote'), ' ', 'datanode', ' >> /var/log/cloudera-azure-initialize.log 2>&1')]" } } } ] }, "parameters": { "vnetID": { "value": "[variables('VNetId')]" }, "templateAPIVersion": { "value": "[variables('templateAPIVersion')]" }, "resourceAPIVersion": { "value": "[variables('resourceAPIVersion')]" }, "dnsNamePrefix": { "value": "[parameters('dnsNamePrefix')]" }, "scriptsUri": { "value": "[variables('scriptsUri')]" }, "storageAccount": { "value": "[variables('workerStorageAccount')]" }, "vmCount": { "value": "[variables('clusterSpec').dataNodeCount]" }, "vmSpec": { "value": "[variables('vmSpec')]" }, "networkSpec": { "value": "[variables('networkSpec')]" }, "clusterSpec": { "value": "[variables('clusterSpec')]" } } } }, { "name": "setup-cloudera", "type": "Microsoft.Resources/deployments", "apiVersion": "2018-11-01", "dependsOn": [ "Microsoft.Resources/deployments/data-node", "Microsoft.Resources/deployments/master-node" ], "properties": { "mode": "Incremental", "template": { "uri": "[concat(variables('scriptsUri'), '/setup-cloudera.json')]", "contentVersion": "1.0.0.1" }, "parameters": { "resourceAPIVersion": { "value": "[variables('resourceAPIVersion')]" }, "dnsNamePrefix": { "value": "[parameters('dnsNamePrefix')]" }, "scriptsUri": { "value": "[variables('scriptsUri')]" }, "vmSpec": { "value": "[variables('vmSpec')]" }, "networkSpec": { "value": "[variables('networkSpec')]" }, "clusterSpec": { "value": "[variables('clusterSpec')]" }, "fqdn": { "value": "[reference('master-node').outputs.fqdn.value]" }, "cmUsername": { "value": "[parameters('cmUsername')]" }, "cmPassword": { "value": "[parameters('cmPassword')]" }, "company": { "value": "[parameters('Company')]" }, "emailAddress": { "value": "[parameters('emailAddress')]" }, "businessPhone": { "value": "[parameters('businessPhone')]" }, "firstName": { "value": "[parameters('firstName')]" }, "lastName": { "value": "[parameters('lastName')]" }, "jobRole": { "value": "[parameters('jobRole')]" }, "jobFunction": { "value": "[parameters('jobFunction')]" }, "installCDH": { "value": "[variables('installCDH')]" } } } } ] }
yeah so, I've found the problem (but there will probably be more problems after that, sadly). problem comes from the fact you are using nested INLINE templates and how they work (or rather not) with the reference function: "[reference(concat(variables('publicIPAddressName'), copyIndex())).dnsSettings.fqdn]", this is the part thats breaking. so my advice would be to either make this a single template, without nesting those, or convert those to a full blown nested templates (using templateLink property, not template property: { "name": "name", "type": "Microsoft.Resources/deployments", "apiVersion": "2017-05-10", "properties": { "mode": "Incremental", "templateLink": { "uri": "url_goes_here" }, "parameters": { "prefix": { "value": "xxx" } } } }, I would suggest that you never use nested INLINE templates. they are just too edgy.