ARM template for Data Factory connector in Logic Apps with Managed Identity - json

I have a Logic App that uses the Azure Data Factory action "Create a pipeline run" that works perfectly.
This is how the Logic App looks like
The authentication method to Azure Data Factory that I use is "System assigned" managed identity.
After creating and testing the Logic App, I now want to create an ARM template to save it in the code repository for deployment, however I'm struggling to get the authentication part of the ARM template to work. I'm not sure how the syntax should be and I don't find anything in the Microsoft documentation.
In the Logic App resource I have added:
"identity": {
"type": "SystemAssigned"
}
This is how the connections part of the Logic app resource looks like:
"$connections": {
"value": {
"azuredatafactory": {
"connectionId": "[parameters('connections_azuredatafactory_externalid')]",
"connectionName": "[parameters('connections_azuredatafactory_name')]",
"connectionProperties": {
"authentication": {
"type": "ManagedServiceIdentity"
}
},
"id": "[concat('/subscriptions/', subscription().subscriptionId, '/providers/Microsoft.Web/locations/francecentral/managedApis/azuredatafactory')]"
}
}
}
And this is how the connector resource look like (I think I'm missing something here (?)):
{
"type": "Microsoft.Web/connections",
"apiVersion": "2016-06-01",
"name": "[parameters('connections_azuredatafactory_name')]",
"location": "francecentral",
"kind": "V1",
"properties": {
"displayName": "[parameters('connections_azuredatafactory_displayname')]",
"alternativeParameterValues": {},
"parameterValueSet": {
"name": "managedIdentityAuth",
"values": {}
},
"statuses": [
{
"status": "Ready"
}
],
"api": {
"id": "[concat('/subscriptions/', subscription().subscriptionId, '/providers/Microsoft.Web/locations/francecentral/managedApis/azuredatafactory')]"
}
}
}
The error message I get when trying to deploy this through Visual studio 2022 is:
Template deployment returned the following errors:
Resource Microsoft.Logic/workflows 'logic-d365-dwh-01-ip-dev-rxlse' failed with message '{
"error": {
"code": "WorkflowManagedIdentityConfigurationInvalid",
"message": "The workflow connection parameter 'azuredatafactory' is not valid. The API connection 'azuredatafactory' is not configured to support managed identity."
}
}'
Anyone who knows what the problem could be?

1)I have created azure logic App with 3 actions (http request, create ADF pipeline, response).
Here is the reference image:
2)Then to connect to ADF used system assigned managed identity & I have given access for logic App to create pipeline in ADF.
Here is the reference image:
Then I have tested in portal & it is succussed
Then I have exported ARM Template & downloaded.
Then in visual studio I have created new project of type Azure resource group then I have edited logicapp.json & logic app parameters file based on template.
Then I have deployed it and it is succussed.
ARM template code which I have used for reference:
{
"$schema": "[https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#"](https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#%22 "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#%22"),
"contentVersion": "1.0.0.0",
"parameters": {
"workflows_so1LP_name": {
"defaultValue": "so1LP",
"type": "String"
},
"connections_azuredatafactory_1_externalid": {
"defaultValue": "/subscriptions/<subscription-id>/resourceGroups/so1/providers/Microsoft.Web/connections/azuredatafactory-1",
"type": "String"
}
},
"variables": {},
"resources": [
{
"type": "Microsoft.Logic/workflows",
"apiVersion": "2017-07-01",
"name": "[parameters('workflows_so1LP_name')]",
"location": "centralus",
"identity": {
"type": "SystemAssigned"
},
"properties": {
"state": "Enabled",
"definition": {
"$schema": "[https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#"](https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#%22 "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#%22"),
"contentVersion": "1.0.0.0",
"parameters": {
"$connections": {
"defaultValue": {},
"type": "Object"
}
},
"triggers": {
"manual": {
"type": "Request",
"kind": "Http",
"inputs": {}
}
},
"actions": {
"Create_a_pipeline_run": {
"runAfter": {},
"type": "ApiConnection",
"inputs": {
"host": {
"connection": {
"name": "#parameters('$connections')['azuredatafactory_1']['connectionId']"
}
},
"method": "post",
"path": "/subscriptions/#{encodeURIComponent('<subscription id>')}/resourcegroups/#{encodeURIComponent('so1')}/providers/Microsoft.DataFactory/factories/#{encodeURIComponent('sodf1')}/pipelines/#{encodeURIComponent('sopipeline')}/CreateRun",
"queries": {
"x-ms-api-version": "2017-09-01-preview"
}
}
},
"Response": {
"runAfter": {
"Create_a_pipeline_run": [
"Succeeded"
]
},
"type": "Response",
"kind": "Http",
"inputs": {
"statusCode": 200
}
}
},
"outputs": {}
},
"parameters": {
"$connections": {
"value": {
"azuredatafactory_1": {
"connectionId": "[parameters('connections_azuredatafactory_1_externalid')]",
"connectionName": "azuredatafactory-1",
"connectionProperties": {
"authentication": {
"type": "ManagedServiceIdentity"
}
},
"id": "/subscriptions/<subscription-id>/<Subscriotion id>providers/Microsoft.Web/locations/centralus/managedApis/azuredatafactory"
}
}
}
}
}
}
],
"outputs": {}
}
Here is the reference image:
NOTE: I am using free subscription, so I don't have any restrictions but, in your case, maybe you have some restrictions that's why maybe your facing issue.
The second reasons may be your using system assigned access after creating logic app to give access to ADF & once check are you giving managed identity after creating ADF give access to logic app also. so maybe you are skipping one of managed identity that's why getting error in ARM template deployment. So, give access to both from ADF to logic app and logic app to ADF.
Here are some images for reference for logic app to ADF:
Go to "access control" of logic app.
Select owner as role.
Select managed identity as data factory.
Here are some images for reference for ADF to logic app:
Go to "access control" of data factory.
Select owner as role.
Select managed identity as logic app.

Did you try using "parameterValueType": "Alternative" instead of "parameterValueSet"?
{
"type": "Microsoft.Web/connections",
"apiVersion": "2016-06-01",
"name": "[parameters('connections_azuredatafactory_name')]",
"location": "francecentral",
"kind": "V1",
"properties": {
"displayName": "[parameters('connections_azuredatafactory_displayname')]",
"customParameterValues": {},
"parameterValueType": "Alternative"
"api": {
"id": "[concat('/subscriptions/', subscription().subscriptionId, '/providers/Microsoft.Web/locations/francecentral/managedApis/azuredatafactory')]"
}
}
}

Related

Failed to retrieve function source code when deploying a cloud function from a repository on a different project

I am trying to deploy a Cloud Function from a Cloud Source Repository placed in a different project, but getting the following error: Failed to retrieve function source code (see full proto below).
Project-A contains the cloud function and service accounts listed below.
Project-B contains the source repository.
I have successfully deployed the function on Project-B.
I've tried giving the following service accounts the Source Repository Administrator role on the cloud source repository, but that did not help.
{project_A_number}#cloudservices.gserviceaccount.com
{project_A_number}-compute#developer.gserviceaccount.com
{project_A_number}#cloudbuild.gserviceaccount.com
Project-A#appspot.gserviceaccount.com
I have also tried disabling the Cloud Functions API on Project-A and turning it back on again.
I am not sure what is going wrong - if anyone has a clue as to where to further look, I would appreciate it - thanks in advance!
The deployment creates two entries in monitoring - a NOTICE followed by an ERROR:
The ERROR log:
{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"status": {
"code": 5,
"message": "Failed to retrieve function source code"
},
"authenticationInfo": {
"principalEmail": "***#***.**"
},
"serviceName": "cloudfunctions.googleapis.com",
"methodName": "google.cloud.functions.v1.CloudFunctionsService.UpdateFunction",
"resourceName": "projects/Project-A/locations/europe-west1/functions/pubsub-to-gcs"
},
"insertId": "-vmfbt4cd54",
"resource": {
"type": "cloud_function",
"labels": {
"function_name": "pubsub-to-gcs",
"region": "europe-west1",
"project_id": "Project-A"
}
},
"timestamp": "2021-10-20T12:21:45.352043Z",
"severity": "ERROR",
"logName": "projects/Project-A/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "operations/cm9ldHotbGlmZS1kYXRhLXRlc3QvZXVyb3BlLXdlc3QxL3B1YnN1Yi10by1nY3MvVEhFbUQtLTZITWM",
"producer": "cloudfunctions.googleapis.com",
"last": true
},
"receiveTimestamp": "2021-10-20T12:21:45.781856467Z"
}
The NOTICE log (logged right before the ERROR):
{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"authenticationInfo": {
"principalEmail": "***#****.**"
},
"requestMetadata": {
"callerIp": "35.205.252.75",
"callerSuppliedUserAgent": "google-cloud-sdk gcloud/360.0.0 command/gcloud.functions.deploy invocation-id/917d697431e84b91bfa2bd9f9cc4f302 environment/devshell environment-version/None interactive/True from-script/False python/3.7.3 term/screen (Linux 5.4.144+),gzip(gfe),gzip(gfe)",
"requestAttributes": {
"time": "2021-10-20T12:21:44.909430Z",
"auth": {}
},
"destinationAttributes": {}
},
"serviceName": "cloudfunctions.googleapis.com",
"methodName": "google.cloud.functions.v1.CloudFunctionsService.UpdateFunction",
"authorizationInfo": [
{
"resource": "projects/Project-A/locations/europe-west1/functions/pubsub-to-gcs",
"permission": "cloudfunctions.functions.update",
"granted": true,
"resourceAttributes": {}
}
],
"resourceName": "projects/Project-A/locations/europe-west1/functions/pubsub-to-gcs",
"request": {
"#type": "type.googleapis.com/google.cloud.functions.v1.UpdateFunctionRequest",
"function": {
"timeout": "60s",
"status": "UNKNOWN",
"serviceAccountEmail": "Project-A#appspot.gserviceaccount.com",
"availableMemoryMb": 256,
"name": "projects/Project-A/locations/europe-west1/functions/pubsub-to-gcs",
"runtime": "python39",
"labels": {
"deployment-tool": "cli-gcloud"
},
"entryPoint": "pubsub-to-gcs",
"updateTime": "2021-10-20T12:21:40.149Z",
"sourceRepository": {
"url": "https://source.developers.google.com/projects/Project-B/repos/my-repo/moveable-aliases/master/paths/my-folder"
},
"httpsTrigger": {},
"ingressSettings": "ALLOW_ALL",
"versionId": "1"
},
"updateMask": "eventTrigger,httpsTrigger,runtime,sourceRepository"
},
"resourceLocation": {
"currentLocations": [
"europe-west1"
]
}
},
"insertId": "1xdbim3e16pgu",
"resource": {
"type": "cloud_function",
"labels": {
"function_name": "pubsub-to-gcs",
"region": "europe-west1",
"project_id": "Project-A"
}
},
"timestamp": "2021-10-20T12:21:44.650257Z",
"severity": "NOTICE",
"logName": "projects/Project-A/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "operations/cm9ldHotbGlmZS1kYXRhLXRlc3QvZXVyb3BlLXdlc3QxL3B1YnN1Yi10by1nY3MvVEhFbUQtLTZITWM",
"producer": "cloudfunctions.googleapis.com",
"first": true
},
"receiveTimestamp": "2021-10-20T12:21:45.832588036Z"
}
Turns out it wasn't an IAM issue: I've tried deploying the function from the UI, but that's not possible when deploying from a source repo in a different project.
Deploying using gcloud function deploy solved the issue.

How to pass multiple parameters into an Azure RM Template Custom Script Extension

I am trying to pass multiple parameters to a Custom Script Extension using an ARM template, here is a snippet of the ARM template that currently works without issue:
{
"name": "Microsoft.CustomScriptExtension-20210604105657",
"apiVersion": "2015-01-01",
"type": "Microsoft.Resources/deployments",
"properties": {
"mode": "incremental",
"templateLink": {
"uri": "https://catalogartifact.azureedge.net/publicartifactsmigration/Microsoft.CustomScriptExtension-arm.2.0.56/Artifacts/MainTemplate.json"
},
"parameters": {
"vmName": {
"value": "real-vm-name"
},
"location": {
"value": "uksouth"
},
"fileUris": {
"value": "https://realwebsite/script.ps1"
},
"arguments": {
"value": "[parameters('param1')]"
}
}
},
But whenever I add another parameter to the arguments section, the template validation fails. This is what I have tried to do:
"arguments": {
"value": "[parameters('param1')], [parameters('param2')]"
}
Please can somebody help?
Can you try to concat or format the parameters depending on what parameters you're using like this:
"value": "[concat(parameters('param1'), parameters('param2'))]"
"value": "[format(parameters('param1'), parameters('param2'))]"
The Azure doc I referred to:
ARM Deployment Scripts
Concat function for ARM templates
Format function for ARM templates

When i deploy my Azure ARM Template it creates a storage, not an alert

I created a Azure Template for an alert, because i want to upload the script (.json) with the new microservice the same time. But if I deploy this .json file it creates a new storage, not an alert. I used the Powershell commands New-AzureRmResourceGroupDeployment -Name ExampleDeployment -ResourceGroupName ExampleResourceGroup -TemplateFile c:\MyTemplates\storage.json -storageAccountType Standard_GRS. In my template i need to define the parameter kind, which is only acceptable with a value of Storage or Blobstorage, but i want non of these two. So how can i create an alert by using a script .json file and does anybody have a template, because MS isn't providing the correct one.
EDIT: Here is the .json file:
{
"$schema":
"http://schema.management.azure.com/schemas/2015-01-
01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"resources": [
{
"name": "[concat('storage', uniqueString(resourceGroup().id))]",
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2016-01-01",
"sku": {
"name": "Standard_LRS"
},
"kind": "Storage",
"id":
"/subscriptions/subscriptionID/resourceGroups/resourceGroupName/providers/Microsoft.Storage/storageAccounts/storageName",
"location": "westeurope",
"properties": {
"name": "tryAgain",
"description": null,
"isEnabled": true,
"condition": {
"$type":
"Microsoft.WindowsAzure.Management.Monitoring.Alerts.Models.ThresholdRuleCondition, Microsoft.WindowsAzure.Management.Mon.Client",
"odata.type":
"Microsoft.Azure.Management.Insights.Models.ThresholdRuleCondition",
"dataSource": {
"$type":
"Microsoft.WindowsAzure.Management.Monitoring.Alerts.Models.RuleMetricDataSource, Microsoft.WindowsAzure.Management.Mon.Client",
"odata.type":
"Microsoft.Azure.Management.Insights.Models.RuleMetricDataSource",
"resourceUri":
"/subscriptions/subscriptionID/resourcegroups/resourceGroupName/providers/microsoft.web/sites/name",
"resourceLocation": null,
"metricNamespace": null,
"metricName": "AverageMemoryWorkingSet",
"legacyResourceId": null
},
"operator": "GreaterThanOrEqual",
"threshold": 120000000,
"windowSize": "PT10M",
"timeAggregation": "Average"
},
"actions": [
{
"$type":
"Microsoft.WindowsAzure.Management.Monitoring.Alerts.Models.RuleWebhookAction, Microsoft.WindowsAzure.Management.Mon.Client",
"odata.type":
"Microsoft.Azure.Management.Insights.Models.RuleWebhookAction",
"serviceUri":
"Logic-app URL",
"properties": {
"$type":
"Microsoft.WindowsAzure.Management.Common.Storage.CasePreservedDictionary`1[[System.String, mscorlib]], Microsoft.WindowsAzure.Management.Common.Storage",
"logicAppResourceId":
"/subscriptions/subscriptionID/resourceGroups/Default-Storage-WestEurope/providers/Microsoft.Logic/workflows/Microsoft-Teams-Notifier"
}
}
]
}
}
]
}
Please refer to the following reference for creating a metric alert via Azure Resource Manager template. If you want to create a single ARM template which creates a storage account and then a metric alert to monitor the created storage account, you should make sure you have a dependsOn so that the alert rule is only created after the storage account. The following document references the newer metric alerts, as opposed to the classic metric alerts.
https://learn.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-create-metric-alerts-with-templates

Passing the output from one arm deployment resource linked template to another

I am using a series of json ARM templates to deploy Azure VMs, and am having issues passing information from one resource deployment to another.
I deploy two resources using linked templates from blob storage, one which in and of itself deploys nothing, but returns an object populated with configuration settings, and a second which then passes that output of configuration settings to another template as a parameter:
"resources": [
{
"name": "[concat(deployment().name, '-config')]",
"type": "Microsoft.Resources/deployments",
"apiVersion": "2016-09-01",
"properties": {
"mode": "Incremental",
"templateLink": {
"uri": "[variables('configurationTemplate')]",
"contentVersion": "1.0.0.0"
},
"parameters": {
"subscriptionParameters": { "value": "[variables('subscriptionParameters')]" }
}
}
},
{
"name": "[concat(deployment().name, '-vm')]",
"type": "Microsoft.Resources/deployments",
"apiVersion": "2016-09-01",
"properties": {
"mode": "Incremental",
"templateLink": {
"uri": "[variables('vmTemplate')]",
"contentVersion": "1.0.0.0"
},
"parameters": {
"configuration": { "value": "[reference(concat(deployment().name, '-config').outputs.configuration.value)]" },
"vmName": { "value": "[parameters('vmName')]" },
"vmSize": { "value": "[parameters('vmSize')]" },
"os": { "value": "[parameters('os')]" },
"managedDiskTier": { "value": "[parameters('managedDiskTier')]" },
"dataDisksToProvision": { "value": "[parameters('dataDisksToProvision')]" },
"dataDiskSizeGB": { "value": "[parameters('dataDiskSizeGB')]" },
"domainJoined": { "value": "[parameters('domainJoined')]" },
"localAdminUsername": { "value": "[parameters('localAdminUsername')]" },
"localAdminPassword": { "value": "[parameters('localAdminPassword')]" },
"numberOfNics": { "value": "[parameters('numberOfNics')]" },
"subnetName": { "value": "[parameters('subnetName')]" },
"highlyAvailable": { "value": "[parameters('highlyAvailable')]" },
"availabilitySetName": { "value": "[parameters('availabilitySetName')]" },
"availabilitySetUpdateDomains": { "value": "[parameters('availabilitySetUpdateDomains')]" },
"availabilitySetFaultDomains": { "value": "[parameters('availabilitySetFaultDomains')]" }
}
}
}
],
"outputs": {
"configuration": {
"type": "object",
"value": "[reference(concat(deployment().name, '-config')).outputs.configuration.value]"
}
}
Deploying the first resource on it's own succeeds, and the output [reference(concat(deployment().name, '-config')).outputs.configuration.value] is correctly returned, and contains all the correct information and is well formed.
If I then add the second resource into the mix, then the deployment fails with
the following error:
08:57:41 - [ERROR] New-AzureRmResourceGroupDeployment : 08:57:41 - Error: Code=InvalidTemplate;
08:57:41 - [ERROR] Message=Deployment template validation failed: 'The template resource
08:57:41 - [ERROR] 'rcss.test.vm-0502-0757-rcss-vm' at line '317' and column '6' is not valid:
08:57:41 - [ERROR] The language expression property 'Microsoft.WindowsAzure.ResourceStack.Frontdoo
08:57:41 - [ERROR] r.Expression.Expressions.JTokenExpression' can't be evaluated.. Please see
08:57:41 - [ERROR] https://aka.ms/arm-template-expressions for usage details.'.
If I remove the "configuration" parameter from both this parameter set and from the referenced template (the referenced template has all contents commented out to ensure we are testing only the pass-through of the parameters), then the deployment succeeds, indicating that the issue is related to the parsing of the parameter string "[reference(concat(deployment().name, '-config').outputs.configuration.value)]".
Can anyone offer any insight as to whether I need to refer to output objects from deployment resources in a specific way in the context of a linked template parameter set?
So after examining this more closely, I found that the syntax I was using was incorrect, but not reported by the parser:
"[reference(concat(deployment().name, '-config').outputs.configuration.value)]"
Should have been:
"[reference(concat(deployment().name, '-config')).outputs.configuration.value]"
Schoolboy error.

ARM Error: The Template Resource is not found using resource(), copyIndex()

I'm trying to conditionally provide resource property values through translation of runtime resource properties within a copyIndex loop..
Upon deploying the following ARM template, I receive the error:
Unable to process template language expressions for resource '/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.Compute/virtualMachines/{vm-name}/extensions/Microsoft.EnterpriseCloud.Monitoring' at line '30' and column '10'. 'The template resource '/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.Compute/virtualMachines/{vm-name}' is not found.' (Code: InvalidTemplate)
"type": "[variables('extensionType')[reference(concat('Microsoft.Compute/virtualMachines/', parameters('virtualMachines')[copyIndex()].name)).storageProfile.osDisk.osType]]",
However, the VM exists with the ID it provides, so it doesn't make sense that the engine cannot find it. If I hard-code the Extension Type, there are no errors and the Extension is installed on the VM with the same ID.
Unfortunately, I don't know if this is a bug within ARM or if I'm just doing something wrong..
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"workspaceResourceId": { "type": "string" },
"virtualMachines": { "type": "array" }
},
"variables": {
"extensionType": {
"Windows": "MicrosoftMonitoringAgent",
"Linux": "OmsAgentForLinux"
}
},
"resources": [
{
"copy": {
"name": "VMMonitoringExtensionsCopy",
"count": "[length(parameters('virtualMachines'))]"
},
"type": "Microsoft.Compute/virtualMachines/extensions",
"apiVersion": "2015-05-01-preview",
"location": "[parameters('virtualMachines')[copyIndex()].location]",
"name": "[concat(parameters('virtualMachines')[copyIndex()].name, '/Microsoft.EnterpriseCloud.Monitoring')]",
"properties": {
"publisher": "Microsoft.EnterpriseCloud.Monitoring",
"type": "[variables('extensionType')[reference(concat('Microsoft.Compute/virtualMachines/', parameters('virtualMachines')[copyIndex()].name)).storageProfile.osDisk.osType]]",
"typeHandlerVersion": "1.0",
"autoUpgradeMinorVersion": true,
"settings": {
"workspaceId": "[reference(parameters('workspaceResourceId'), '2015-11-01-preview').customerId]"
},
"protectedSettings": {
"workspaceKey": "[listKeys(parameters('workspaceResourceId'), '2015-11-01-preview').primarySharedKey]"
}
}
}
]
}
The object array being passed in for virtualMachines looks like this:
[
{ "name": "vm-name", "location": "azure-region" }
]
A couple things you can try:
1) Assuming the VM is not defined in the same template try using the "full" resourceId in the reference function. See the last example in this doc:
https://azure.microsoft.com/en-us/documentation/articles/resource-group-template-functions/#reference - it seems like the error already knows the full resourceId, but it's worth trying
2) the other thought is that the reference function is evaluated at runtime and the resource provider doesn't like the expression but that's a swag.
I will do some more poking and see if we can't nail this down.