Passing the output from one arm deployment resource linked template to another - json

I am using a series of json ARM templates to deploy Azure VMs, and am having issues passing information from one resource deployment to another.
I deploy two resources using linked templates from blob storage, one which in and of itself deploys nothing, but returns an object populated with configuration settings, and a second which then passes that output of configuration settings to another template as a parameter:
"resources": [
{
"name": "[concat(deployment().name, '-config')]",
"type": "Microsoft.Resources/deployments",
"apiVersion": "2016-09-01",
"properties": {
"mode": "Incremental",
"templateLink": {
"uri": "[variables('configurationTemplate')]",
"contentVersion": "1.0.0.0"
},
"parameters": {
"subscriptionParameters": { "value": "[variables('subscriptionParameters')]" }
}
}
},
{
"name": "[concat(deployment().name, '-vm')]",
"type": "Microsoft.Resources/deployments",
"apiVersion": "2016-09-01",
"properties": {
"mode": "Incremental",
"templateLink": {
"uri": "[variables('vmTemplate')]",
"contentVersion": "1.0.0.0"
},
"parameters": {
"configuration": { "value": "[reference(concat(deployment().name, '-config').outputs.configuration.value)]" },
"vmName": { "value": "[parameters('vmName')]" },
"vmSize": { "value": "[parameters('vmSize')]" },
"os": { "value": "[parameters('os')]" },
"managedDiskTier": { "value": "[parameters('managedDiskTier')]" },
"dataDisksToProvision": { "value": "[parameters('dataDisksToProvision')]" },
"dataDiskSizeGB": { "value": "[parameters('dataDiskSizeGB')]" },
"domainJoined": { "value": "[parameters('domainJoined')]" },
"localAdminUsername": { "value": "[parameters('localAdminUsername')]" },
"localAdminPassword": { "value": "[parameters('localAdminPassword')]" },
"numberOfNics": { "value": "[parameters('numberOfNics')]" },
"subnetName": { "value": "[parameters('subnetName')]" },
"highlyAvailable": { "value": "[parameters('highlyAvailable')]" },
"availabilitySetName": { "value": "[parameters('availabilitySetName')]" },
"availabilitySetUpdateDomains": { "value": "[parameters('availabilitySetUpdateDomains')]" },
"availabilitySetFaultDomains": { "value": "[parameters('availabilitySetFaultDomains')]" }
}
}
}
],
"outputs": {
"configuration": {
"type": "object",
"value": "[reference(concat(deployment().name, '-config')).outputs.configuration.value]"
}
}
Deploying the first resource on it's own succeeds, and the output [reference(concat(deployment().name, '-config')).outputs.configuration.value] is correctly returned, and contains all the correct information and is well formed.
If I then add the second resource into the mix, then the deployment fails with
the following error:
08:57:41 - [ERROR] New-AzureRmResourceGroupDeployment : 08:57:41 - Error: Code=InvalidTemplate;
08:57:41 - [ERROR] Message=Deployment template validation failed: 'The template resource
08:57:41 - [ERROR] 'rcss.test.vm-0502-0757-rcss-vm' at line '317' and column '6' is not valid:
08:57:41 - [ERROR] The language expression property 'Microsoft.WindowsAzure.ResourceStack.Frontdoo
08:57:41 - [ERROR] r.Expression.Expressions.JTokenExpression' can't be evaluated.. Please see
08:57:41 - [ERROR] https://aka.ms/arm-template-expressions for usage details.'.
If I remove the "configuration" parameter from both this parameter set and from the referenced template (the referenced template has all contents commented out to ensure we are testing only the pass-through of the parameters), then the deployment succeeds, indicating that the issue is related to the parsing of the parameter string "[reference(concat(deployment().name, '-config').outputs.configuration.value)]".
Can anyone offer any insight as to whether I need to refer to output objects from deployment resources in a specific way in the context of a linked template parameter set?

So after examining this more closely, I found that the syntax I was using was incorrect, but not reported by the parser:
"[reference(concat(deployment().name, '-config').outputs.configuration.value)]"
Should have been:
"[reference(concat(deployment().name, '-config')).outputs.configuration.value]"
Schoolboy error.

Related

ARM template for Data Factory connector in Logic Apps with Managed Identity

I have a Logic App that uses the Azure Data Factory action "Create a pipeline run" that works perfectly.
This is how the Logic App looks like
The authentication method to Azure Data Factory that I use is "System assigned" managed identity.
After creating and testing the Logic App, I now want to create an ARM template to save it in the code repository for deployment, however I'm struggling to get the authentication part of the ARM template to work. I'm not sure how the syntax should be and I don't find anything in the Microsoft documentation.
In the Logic App resource I have added:
"identity": {
"type": "SystemAssigned"
}
This is how the connections part of the Logic app resource looks like:
"$connections": {
"value": {
"azuredatafactory": {
"connectionId": "[parameters('connections_azuredatafactory_externalid')]",
"connectionName": "[parameters('connections_azuredatafactory_name')]",
"connectionProperties": {
"authentication": {
"type": "ManagedServiceIdentity"
}
},
"id": "[concat('/subscriptions/', subscription().subscriptionId, '/providers/Microsoft.Web/locations/francecentral/managedApis/azuredatafactory')]"
}
}
}
And this is how the connector resource look like (I think I'm missing something here (?)):
{
"type": "Microsoft.Web/connections",
"apiVersion": "2016-06-01",
"name": "[parameters('connections_azuredatafactory_name')]",
"location": "francecentral",
"kind": "V1",
"properties": {
"displayName": "[parameters('connections_azuredatafactory_displayname')]",
"alternativeParameterValues": {},
"parameterValueSet": {
"name": "managedIdentityAuth",
"values": {}
},
"statuses": [
{
"status": "Ready"
}
],
"api": {
"id": "[concat('/subscriptions/', subscription().subscriptionId, '/providers/Microsoft.Web/locations/francecentral/managedApis/azuredatafactory')]"
}
}
}
The error message I get when trying to deploy this through Visual studio 2022 is:
Template deployment returned the following errors:
Resource Microsoft.Logic/workflows 'logic-d365-dwh-01-ip-dev-rxlse' failed with message '{
"error": {
"code": "WorkflowManagedIdentityConfigurationInvalid",
"message": "The workflow connection parameter 'azuredatafactory' is not valid. The API connection 'azuredatafactory' is not configured to support managed identity."
}
}'
Anyone who knows what the problem could be?
1)I have created azure logic App with 3 actions (http request, create ADF pipeline, response).
Here is the reference image:
2)Then to connect to ADF used system assigned managed identity & I have given access for logic App to create pipeline in ADF.
Here is the reference image:
Then I have tested in portal & it is succussed
Then I have exported ARM Template & downloaded.
Then in visual studio I have created new project of type Azure resource group then I have edited logicapp.json & logic app parameters file based on template.
Then I have deployed it and it is succussed.
ARM template code which I have used for reference:
{
"$schema": "[https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#"](https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#%22 "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#%22"),
"contentVersion": "1.0.0.0",
"parameters": {
"workflows_so1LP_name": {
"defaultValue": "so1LP",
"type": "String"
},
"connections_azuredatafactory_1_externalid": {
"defaultValue": "/subscriptions/<subscription-id>/resourceGroups/so1/providers/Microsoft.Web/connections/azuredatafactory-1",
"type": "String"
}
},
"variables": {},
"resources": [
{
"type": "Microsoft.Logic/workflows",
"apiVersion": "2017-07-01",
"name": "[parameters('workflows_so1LP_name')]",
"location": "centralus",
"identity": {
"type": "SystemAssigned"
},
"properties": {
"state": "Enabled",
"definition": {
"$schema": "[https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#"](https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#%22 "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#%22"),
"contentVersion": "1.0.0.0",
"parameters": {
"$connections": {
"defaultValue": {},
"type": "Object"
}
},
"triggers": {
"manual": {
"type": "Request",
"kind": "Http",
"inputs": {}
}
},
"actions": {
"Create_a_pipeline_run": {
"runAfter": {},
"type": "ApiConnection",
"inputs": {
"host": {
"connection": {
"name": "#parameters('$connections')['azuredatafactory_1']['connectionId']"
}
},
"method": "post",
"path": "/subscriptions/#{encodeURIComponent('<subscription id>')}/resourcegroups/#{encodeURIComponent('so1')}/providers/Microsoft.DataFactory/factories/#{encodeURIComponent('sodf1')}/pipelines/#{encodeURIComponent('sopipeline')}/CreateRun",
"queries": {
"x-ms-api-version": "2017-09-01-preview"
}
}
},
"Response": {
"runAfter": {
"Create_a_pipeline_run": [
"Succeeded"
]
},
"type": "Response",
"kind": "Http",
"inputs": {
"statusCode": 200
}
}
},
"outputs": {}
},
"parameters": {
"$connections": {
"value": {
"azuredatafactory_1": {
"connectionId": "[parameters('connections_azuredatafactory_1_externalid')]",
"connectionName": "azuredatafactory-1",
"connectionProperties": {
"authentication": {
"type": "ManagedServiceIdentity"
}
},
"id": "/subscriptions/<subscription-id>/<Subscriotion id>providers/Microsoft.Web/locations/centralus/managedApis/azuredatafactory"
}
}
}
}
}
}
],
"outputs": {}
}
Here is the reference image:
NOTE: I am using free subscription, so I don't have any restrictions but, in your case, maybe you have some restrictions that's why maybe your facing issue.
The second reasons may be your using system assigned access after creating logic app to give access to ADF & once check are you giving managed identity after creating ADF give access to logic app also. so maybe you are skipping one of managed identity that's why getting error in ARM template deployment. So, give access to both from ADF to logic app and logic app to ADF.
Here are some images for reference for logic app to ADF:
Go to "access control" of logic app.
Select owner as role.
Select managed identity as data factory.
Here are some images for reference for ADF to logic app:
Go to "access control" of data factory.
Select owner as role.
Select managed identity as logic app.
Did you try using "parameterValueType": "Alternative" instead of "parameterValueSet"?
{
"type": "Microsoft.Web/connections",
"apiVersion": "2016-06-01",
"name": "[parameters('connections_azuredatafactory_name')]",
"location": "francecentral",
"kind": "V1",
"properties": {
"displayName": "[parameters('connections_azuredatafactory_displayname')]",
"customParameterValues": {},
"parameterValueType": "Alternative"
"api": {
"id": "[concat('/subscriptions/', subscription().subscriptionId, '/providers/Microsoft.Web/locations/francecentral/managedApis/azuredatafactory')]"
}
}
}

How to pass multiple parameters into an Azure RM Template Custom Script Extension

I am trying to pass multiple parameters to a Custom Script Extension using an ARM template, here is a snippet of the ARM template that currently works without issue:
{
"name": "Microsoft.CustomScriptExtension-20210604105657",
"apiVersion": "2015-01-01",
"type": "Microsoft.Resources/deployments",
"properties": {
"mode": "incremental",
"templateLink": {
"uri": "https://catalogartifact.azureedge.net/publicartifactsmigration/Microsoft.CustomScriptExtension-arm.2.0.56/Artifacts/MainTemplate.json"
},
"parameters": {
"vmName": {
"value": "real-vm-name"
},
"location": {
"value": "uksouth"
},
"fileUris": {
"value": "https://realwebsite/script.ps1"
},
"arguments": {
"value": "[parameters('param1')]"
}
}
},
But whenever I add another parameter to the arguments section, the template validation fails. This is what I have tried to do:
"arguments": {
"value": "[parameters('param1')], [parameters('param2')]"
}
Please can somebody help?
Can you try to concat or format the parameters depending on what parameters you're using like this:
"value": "[concat(parameters('param1'), parameters('param2'))]"
"value": "[format(parameters('param1'), parameters('param2'))]"
The Azure doc I referred to:
ARM Deployment Scripts
Concat function for ARM templates
Format function for ARM templates

AWS Data Pipeline - Set Hive site values during EMR Creation

We are upgrading our Data pipeline version from 3.3.2 to 5.8, so those bootstrap actions on old AMI release have changed to be setup using configuration and specifying them under classification / property definition.
So my Json looks like below
{
"enableDebugging": "true",
"taskInstanceBidPrice": "1",
"terminateAfter": "2 Hours",
"name": "ExportCluster",
"taskInstanceType": "m1.xlarge",
"schedule": {
"ref": "Default"
},
"emrLogUri": "s3://emr-script-logs/",
"coreInstanceType": "m1.xlarge",
"coreInstanceCount": "1",
"taskInstanceCount": "4",
"masterInstanceType": "m3.xlarge",
"keyPair": "XXXX",
"applications": ["hadoop","hive", "tez"],
"subnetId": "XXXXX",
"logUri": "s3://pipelinedata/XXX",
"releaseLabel": "emr-5.8.0",
"type": "EmrCluster",
"id": "EmrClusterWithNewEMRVersion",
"configuration": [
{ "ref": "configureEmrHiveSite" }
]
},
{
"myComment": "This object configures hive-site xml.",
"name": "HiveSite Configuration",
"type": "HiveSiteConfiguration",
"id": "configureEmrHiveSite",
"classification": "hive-site",
"property": [
{"ref": "hive-exec-compress-output" }
]
},
{
"myComment": "This object sets a hive-site configuration
property value.",
"name":"hive-exec-compress-output",
"type": "Property",
"id": "hive-exec-compress-output",
"key": "hive.exec.compress.output",
"value": "true"
}
],
"parameters": []
With the above Json file it gets loaded into Data Pipeline but throws an error saying
Object:HiveSite Configuration
ERROR: 'HiveSiteConfiguration'
Object:ExportCluster
ERROR: 'configuration' values must be of type 'null'. Found values of type 'null'
I am not sure what this really means and could you please let me know if i am specifying this correctly which i think i am according to http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html
The below block should have the name as "EMR Configuration" only then its recognized correctly by the AWS Data pipeline and the Hive-site.xml is being set accordingly.
{
"myComment": "This object configures hive-site xml.",
"name": "EMR Configuration",
"type": "EmrConfiguration",
"id": "configureEmrHiveSite",
"classification": "hive-site",
"property": [
{"ref": "hive-exec-compress-output" }
]
},

ARM Template 'type' property required

I am trying to deploy a documentDb from the Azure-Cli (version 0.10.2) and so I exported the template directly from azure portal.
This resulted in an error of:
InvalidRequestContent : The request content was invalid and could not be deserialized: 'Could not find member 'defaultValue' on object of type 'DeploymentParameterDefinition'. Path 'properties.parameters.arm_document_db.defaultValue'
Changing those to "value" results in:
InvalidDeploymentParameterType : The type of deployment parameter 'arm_document_db' should not be specified. Please see https://aka.ms/arm-deploy/#parameter-file for details.
checking this page indicates that type is required, but is all in lowercase. This gave the same error.
removing "type" then gives me this error:
InvalidRequestContent : The request content was invalid and could not be deserialized: 'Required property 'type' not found in JSON. Path 'properties.template.parameters.arm_document_db'
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"arm_document_db": {
"defaultValue": null,
"type": "SecureString"
},
"arm_document_db01": {
"defaultValue": "arm-document-db01",
"type": "String"
}
},
"variables": {},
"resources": [
{
"type": "Microsoft.DocumentDB/databaseAccounts",
"kind": "DocumentDB",
"name": "[parameters('arm_document_db01')]",
"apiVersion": "2015-04-08",
"location": "Japan West",
"tags": {},
"properties": {
"databaseAccountOfferType": "Standard",
"name": "[parameters('arm_document_db')]"
},
"dependsOn": []
}
]
}
The aka.ms link shows that 'type' is required, however the error says that it isn't. Many thanks in advance!
The error is not for azuredeploy.json. It's for azuredeploy.parameters.json.
Remove the "type" field from your parameters file.

ARM Error: The Template Resource is not found using resource(), copyIndex()

I'm trying to conditionally provide resource property values through translation of runtime resource properties within a copyIndex loop..
Upon deploying the following ARM template, I receive the error:
Unable to process template language expressions for resource '/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.Compute/virtualMachines/{vm-name}/extensions/Microsoft.EnterpriseCloud.Monitoring' at line '30' and column '10'. 'The template resource '/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.Compute/virtualMachines/{vm-name}' is not found.' (Code: InvalidTemplate)
"type": "[variables('extensionType')[reference(concat('Microsoft.Compute/virtualMachines/', parameters('virtualMachines')[copyIndex()].name)).storageProfile.osDisk.osType]]",
However, the VM exists with the ID it provides, so it doesn't make sense that the engine cannot find it. If I hard-code the Extension Type, there are no errors and the Extension is installed on the VM with the same ID.
Unfortunately, I don't know if this is a bug within ARM or if I'm just doing something wrong..
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"workspaceResourceId": { "type": "string" },
"virtualMachines": { "type": "array" }
},
"variables": {
"extensionType": {
"Windows": "MicrosoftMonitoringAgent",
"Linux": "OmsAgentForLinux"
}
},
"resources": [
{
"copy": {
"name": "VMMonitoringExtensionsCopy",
"count": "[length(parameters('virtualMachines'))]"
},
"type": "Microsoft.Compute/virtualMachines/extensions",
"apiVersion": "2015-05-01-preview",
"location": "[parameters('virtualMachines')[copyIndex()].location]",
"name": "[concat(parameters('virtualMachines')[copyIndex()].name, '/Microsoft.EnterpriseCloud.Monitoring')]",
"properties": {
"publisher": "Microsoft.EnterpriseCloud.Monitoring",
"type": "[variables('extensionType')[reference(concat('Microsoft.Compute/virtualMachines/', parameters('virtualMachines')[copyIndex()].name)).storageProfile.osDisk.osType]]",
"typeHandlerVersion": "1.0",
"autoUpgradeMinorVersion": true,
"settings": {
"workspaceId": "[reference(parameters('workspaceResourceId'), '2015-11-01-preview').customerId]"
},
"protectedSettings": {
"workspaceKey": "[listKeys(parameters('workspaceResourceId'), '2015-11-01-preview').primarySharedKey]"
}
}
}
]
}
The object array being passed in for virtualMachines looks like this:
[
{ "name": "vm-name", "location": "azure-region" }
]
A couple things you can try:
1) Assuming the VM is not defined in the same template try using the "full" resourceId in the reference function. See the last example in this doc:
https://azure.microsoft.com/en-us/documentation/articles/resource-group-template-functions/#reference - it seems like the error already knows the full resourceId, but it's worth trying
2) the other thought is that the reference function is evaluated at runtime and the resource provider doesn't like the expression but that's a swag.
I will do some more poking and see if we can't nail this down.