I have an Azure ARM template that successfully bootstraps a VM from a file directory within an Azure Storage Account. I would like to get this working in Terraform, but I am really struggling getting it to work correctly.
Here is a working Azure ARM template that creates the VM and bootstraps it with files in an Azure storage account. The bootstrapping occurs by using the customData parameter.
"variables": {
"uniqueId": "[uniqueString(resourceGroup().id)]",
"customData": "[concat('storage-account=', parameters('STORAGE_ACCOUNT'), ',access-key=', parameters('ACCESS_KEY'), ',file-share=', parameters('FILE_SHARE'), ',share-directory=', parameters('SHARE_DIRECTORY'))]"
},
"resources": [
{
"apiVersion": "2016-04-30-preview",
"type": "Microsoft.Compute/virtualMachines",
"name": "MY-VM",
"location": "[resourceGroup().location]",
"properties": {
"hardwareProfile": {
"vmSize": "Standard_DS3_v2"
},
"osProfile": {
"computerName": "My-Computer-Name",
"adminUsername": "[parameters('Username')]",
"adminPassword": "[parameters('Password')]",
"customData": "[base64(variables('customData'))]"
}
}
}
Here is my non-working Terraform script that does not work when I try to do the same type of Bootstrapping.
resource "azurerm_virtual_machine" "MY-VM" {
name = "${var.vm_name}"
location = "${var.location}"
resource_group_name = "${azurerm_resource_group.rg.name}"
vm_size = "${var.vm_size}"
primary_network_interface_id = "${azurerm_network_interface.nic0.id}"
os_profile {
computer_name = "${var.vm_name}"
admin_username = "${var.adminuser}"
admin_password = "${var.adminuserpassword}"
custom_data = "${base64encode(join("", list("storage-account=", var.STORAGE_ACCOUNT, ",access-key=", var.ACCESS_KEY, ",file-share=", var.FILE_SHARE, ",share-directory=None")))}"
}
}
This is the error that I receive when I run it. If I do not use the custom_data field, the machine launches fine, but is not bootstrapped. I am out of ideas here..
azurerm_virtual_machine.MY-VM:
compute.VirtualMachinesClient#CreateOrUpdate: Failure sending
request: StatusCode=0 -- Original Error: autorest/azure: Service
returned an error. Status=400 Code="InvalidRequestFormat"
Message="Cannot parse the request." Details=[]
i dont think join works for strings? for your case you can just do
"storage-account=${var.STORAGE_ACCCOUNT},access-key=${var.ACCESS_KEY},file-share=${var.FILE_SHARE},share-directory=None"
Related
This question already has answers here:
Azure Function - Python - ServiceBus Output Binding - Setting Custom Properties
(2 answers)
Closed 8 months ago.
I'm using an Azure Function (Python) to send a message to a Service Bus topic whenever a file lands in blob storage following a similar set up to that outlined here.
In particular, in order to send the message I have this in the JSON file:
{
"type": "serviceBus",
"direction": "out",
"connection": "AzureServiceBusConnectionString",
"name": "msg",
"queueName": "outqueue"
}
and in init.py file I have msg.set(input_msg) where input_msg is a JSON string, the output of doing json.dumps(list(reader)) on a CSV string.
When this message is picked up by the topic and subscriptions it has content type set to text/plain, whilst I'd like this to be application/json as mentioned here.
Is there a way to set this, for instance when I do msg.set, is there a way to specify the content type?
Full code:
init.py
def get_json_content_from_csv(csv_content: str) -> str:
reader = csv.DictReader(io.StringIO(csv_content))
json_content = json.dumps(list(reader))
return json_content
def main(event: func.EventGridEvent, msg: func.Out[str]):
data = event.get_json()
url = data["url"]
input_blob = BlobClient.from_blob_url(url, DefaultAzureCredential())
csv_content = input_blob.download_blob(encoding='UTF-8').readall()
json_content = get_json_content_from_csv(csv_content)
msg.set(json.dumps(json_content))
function.json
{
"bindings": [
{
"type": "eventGridTrigger",
"name": "event",
"direction": "in"
},
{
"type": "serviceBus",
"direction": "out",
"connection": "AzureServiceBus",
"name": "msg",
"topicName": "dev-iris-service-bus-topic"
}
]
}
According to this github issue for the Python SDK:
Cannot set Service Bus Message ContentType - Github Issue
The github issue response points to the docs here to set the contentType property on the message class
https://learn.microsoft.com/en-us/python/api/uamqp/uamqp.message.messageproperties?view=azure-python
I am about to refactor a couple of code for a business project. Among other tings, converting from JSON to YAML templates is necessary. I use terraform for infrastructure deployment.
I have this JSON template cf_sns.json.tpl file:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"SNSTopic": {
"Type": "AWS::SNS::Topic",
"Properties": {
"TopicName": "${sns_topic_name}",
"KmsMasterKeyId": "${kms_key_id}",
"DisplayName": "${sns_topic_name}",
"Subscription": [
"${sns_subscription_list}"
]
}
}
},
"Outputs" : {
"SNSTopicARN" : {
"Description": "The SNS Topic Arn",
"Value" : { "Ref" : "SNSTopic" }
}
}
}
This is a main.tf file using this template file:
data "template_file" "this" {
template = "${file("${path.module}/templates/cf_sns.json.tpl")}"
vars = {
kms_key_id = var.kms_key_id
sns_topic_name = var.sns_topic_name
sns_subscription_list = join(",", formatlist("{\"Endpoint\": \"%s\",\"Protocol\": \"%s\"}", var.sns_subscription_email_address_list, "email"))
}
}
I pass ["myemail", "myOtherEmail"] to var.sns_subscription_email_adress_list.
I had to use this approach with a cloudformation resource since Terraform does not support the email protocol for a sns subspription.
How can I refactor the cf_sns.json.tpl to a YAML file together with the data resource mentioned above in the main.tf file? Particularly, I have no clue how to properly pass the sns_subscription_list as YAML array.
That cf_sns.json.tpl is AWS CloudFormation code, if you are already using terraform just refactor that all the way, not just convert from JSON to YAML but completely get rid of that and use the proper terraform resources:
https://www.terraform.io/docs/providers/aws/r/sns_topic.html
https://www.terraform.io/docs/providers/aws/r/sns_topic_subscription.html
Here is some sample code:
resource "aws_sns_topic" "SNSTopic" {
name = var.sns_topic_name
kms_master_key_id = var.kms_key_id
display_name = var.sns_topic_name
}
output "SNSTopicARN" {
value = aws_sns_topic.SNSTopic.arn
}
I came across the Serilog.Sinks.Map addon today which will solve my challenge with routing specific log events to a specific sink interface. In my environment, I am writing to a log file as well as using the SQL interface. I only want certain logs to be written to the SQL Server though.
Reading the instructions on GitHub by the author, I can only see an example for implementing the LoggerConfiguration through C# in the Program.CS, but I am using the appsettings.json file and unsure what to change from the provided example to the required json format.
Example given by Serilog on GitHub:
Log.Logger = new LoggerConfiguration()
.WriteTo.Map("Name", "Other", (name, wt) => wt.File($"./logs/log-{name}.txt"))
.CreateLogger();
My current configuration: Note I haven't implemented the Sinks.Map in my code yet.
Program.CS File:
public static void Main(string[] args)
{
// Build a configuration system with the route of the app settings.json file.
// this is becuase we dont yet have dependancy injection available, that comes later.
var configuration = new ConfigurationBuilder()
.AddJsonFile("appsettings.json")
.Build();
Log.Logger = new LoggerConfiguration()
.ReadFrom.Configuration(configuration)
.CreateLogger();
var host = CreateHostBuilder(args).Build();
}
And here is my appsettings.json file. I want to be able configure sink name 'MSSqlServer' as the special route, then use the standard file appender sink for all the other general logging.
"AllowedHosts": "*",
"Serilog": {
"Using": [],
"MinumumLevel": {
"Default": "Information",
"Override": {
"Microsoft": "Warning",
"System": "Warning"
}
},
"Enrich": [ "FromLogContext", "WithMachineName", "WithProcessId", "WithThreadId" ],
"WriteTo": [
{ "Name": "Console" },
{
"Name": "File",
"Args": {
//"path": "C:\\NetCoreLogs\\log.txt", // Example path to Windows Drive.
"path": ".\\Logs\\logs.txt",
//"rollingInterval": "Day", // Not currently in use.
"rollOnFileSizeLimit": true,
//"retainedFileCountLimit": null, // Not currently in use.
"fileSizeLimitBytes": 10000000,
"outputTemplate": "{Timestamp:dd-MM-yyyy HH:mm:ss.fff G} {Message}{NewLine:1}{Exception:1}"
// *Template Notes*
// Timestamp 'G' means UTC Time
}
},
{
"Name": "MSSqlServer",
"Args": {
"connectionString": "DefaultConnection",
"schemaName": "EventLogging",
"tableName": "Logs",
"autoCreateSqlTable": true,
"restrictedToMinimumLevel": "Information",
"batchPostingLimit": 1000,
"period": "0.00:00:30"
}
}
//{
// "Name": "File",
// "Args": {
// "path": "C:\\NetCoreLogs\\log.json",
// "formatter": "Serilog.Formatting.Json.JsonFormatter, Serilog"
// }
//}
]
}
Lastly if i could squeeze in another quick question on the topic, when using the SQL sink interface, how do manage the automatic purging/deletion of the oldest events i.e. DB should only store max 1,000,000 events then automatically write over the oldest event first, thanks in advance
I believe it is currently impossible to configure the standard Map call in json, since it relies on a few types that have no serialization support right now, like Action<T1, T2>. I created an issue to discuss this in the repository itself:
Unable to configure default Map call in json? #22
However, there is a way to still get some functionality out of it in Json, by creating a custom extension method. In your particular case, it would be something like this:
public static class SerilogSinkConfigurationExtensions
{
public static LoggerConfiguration MapToFile(
this LoggerSinkConfiguration loggerSinkConfiguration,
string keyPropertyName,
string pathFormat,
string defaultKey)
{
return loggerSinkConfiguration.Map(
keyPropertyName,
defaultKey,
(key, config) => config.File(string.Format(pathFormat, key));
}
}
Then, on your json file, add a section like this:
"WriteTo": [
...
{
"Name": "MapToFile",
"Args": {
"KeyPropertyName": "Name",
"DefaultKey": "Other",
"PathFormat": "./logs/log-{0}.txt"
}
}
]
To have these customizations work properly, Serilog needs to understand that your assembly has these kinds of extensions, to load them during the parsing stage. As per the documentation, you either need to have these extensions on a *.Serilog.* assembly, or add the Using clause on the json:
// Assuming the extension method is inside the "Company.Domain.MyProject" dll
"Using": [ "Company.Domain.MyProject" ]
More information on these constraints here:
https://github.com/serilog/serilog-settings-configuration#using-section-and-auto-discovery-of-configuration-assemblies
According to https://learn.microsoft.com/en-gb/azure/virtual-machines/windows/extensions-dsc-template, the latest method for passing credentials from an ARM template to a DSC extension is by placing the whole credential within the configurationArguments of the protectedSettings section, as shown below:
"properties": {
"publisher": "Microsoft.Powershell",
"type": "DSC",
"typeHandlerVersion": "2.24",
"autoUpgradeMinorVersion": true,
"settings": {
"wmfVersion": "latest",
"configuration": {
"url": "[concat(parameters('_artifactsLocation'), '/', variables('artifactsProjectFolder'), '/', variables('dscArchiveFolder'), '/', variables('dscSitecoreInstallArchiveFileName'))]",
"script": "[variables('dscSitecoreInstallScriptName')]",
"function": "SitecoreInstall"
},
"configurationArguments": {
"nodeName": "[parameters('CMCD VMName')]",
"sitecorePackageUrl": "[concat(parameters('sitecorePackageLocation'), '/', parameters('sitecoreRelease'), '/', parameters('sitecorePackageFilename'))]",
"sitecorePackageUrlSasToken": "[parameters('sitecorePackageLocationSasToken')]",
"sitecoreLicense": "[concat(parameters('sitecorePackageLocation'), '/', parameters('sitecoreLicenseFilename'))]",
"domainName": "[parameters('domainName')]",
"joinOU": "[parameters('domainOrgUnit')]"
},
"configurationData": {
"url": "[concat(parameters('_artifactsLocation'), '/', variables('artifactsProjectFolder'), '/', variables('dscArchiveFolder'), '/', variables('dscSitecoreInstallConfigurationName'))]"
}
},
"protectedSettings": {
"configurationUrlSasToken": "[parameters('_artifactsLocationSasToken')]",
"configurationDataUrlSasToken": "[parameters('_artifactsLocationSasToken')]",
"configurationArguments": {
"domainJoinCredential": {
"userName": "[parameters('domainJoinUsername')]",
"password": "[parameters('domainJoinPassword')]"
}
}
}
}
Azure DSC is supposed to handle the encrypting/decrypting of the protectedSettings for me. This does appear to work, as I can see that the protectedSettings are encrypted within the settings file on the VM, however the operation ultimately fails with:
VM has reported a failure when processing extension 'dsc-sitecore-de
v-install'. Error message: "The DSC Extension received an incorrect input: Comp
ilation errors occurred while processing configuration 'SitecoreInstall'. Pleas
e review the errors reported in error stream and modify your configuration code
appropriately. System.InvalidOperationException error processing property 'Cre
dential' OF TYPE 'xComputer': Converting and storing encrypted passwords as pla
in text is not recommended. For more information on securing credentials in MOF
file, please refer to MSDN blog: http://go.microsoft.com/fwlink/?LinkId=393729
At C:\Packages\Plugins\Microsoft.Powershell.DSC\2.24.0.0\DSCWork\dsc-sitecore-d
ev-install.0\dsc-sitecore-dev-install.ps1:103 char:3
+ xComputer Converting and storing encrypted passwords as plain text is not r
ecommended. For more information on securing credentials in MOF file, please re
fer to MSDN blog: http://go.microsoft.com/fwlink/?LinkId=393729 Cannot find pat
h 'HKLM:\SOFTWARE\Microsoft\PowerShell\3\DSC' because it does not exist. Cannot
find path 'HKLM:\SOFTWARE\Microsoft\PowerShell\3\DSC' because it does not exis
t.
Another common error is to specify parameters of type PSCredential without an e
xplicit type. Please be sure to use a typed parameter in DSC Configuration, for
example:
configuration Example {
param([PSCredential] $UserAccount)
...
}.
Please correct the input and retry executing the extension.".
The only way that I can make it work is to add PsDscAllowPlainTextPassword = $true to my configurationData, but I thought I was using the protectedSettings section to avoid using plain text passwords...
Am I doing something wrong, or is it simply that my understanding is wrong?
Proper way of doing this:
"settings": {
"configuration": {
"url": "xxx",
"script": "xxx",
"function": "xx"
},
"configurationArguments": {
"param1": xxx,
"param2": xxx
etc...
}
},
"protectedSettings": {
"configurationArguments": {
"NameOfTheCredentialsParameter": {
"userName": "USERNAME",
"password": "PASSWORD!1"
}
}
}
this way you don't need PsDSCAllowPlainTextPassword = $true
Then you can receive the parameters in your Configuration with
Configuration MyConf
param (
[PSCredential] $NameOfTheCredentialsParameter
)
An use it in your resource
Registry DoNotOpenServerManagerAtLogon {
Ensure = "Present"
Key = "HKEY_CURRENT_USER\SOFTWARE\Microsoft\ServerManager"
ValueName = "DoNotOpenServerManagerAtLogon"
ValueData = 1
ValueType = REG_DWORD"
PsDscRunAsCredential = $NameOfTheCredentialsParameter
}
The fact that you still need to use the PsDSCAllowPlainTextPassword = $true is documented
Here is the quoted section:
However, currently you must tell PowerShell DSC it is okay for credentials to be outputted in plain text during node configuration MOF generation, because PowerShell DSC doesn’t know that Azure Automation will be encrypting the entire MOF file after its generation via a compilation job.
Based on the above, it seems that it is an order of operations issue. The MOF is generated and THEN encrypted.
Have been struggling with this..... Google Apps Script and the Big Query API are working well however when I try to use BigQuery.Tabledata.insertAll I keep getting an error saying 'no such field'.
When I try to run the same thing through the Google API explorer it works fine. The documentation says the command is :
BigQuery.TableData.insertAll(TableDataInsertAllRequest resource, String projectId, String datasetId, String tableId)
I have constructed the TableDataInsertAllRequest resource as per the documentation https://developers.google.com/bigquery/docs/reference/v2/tabledata/insertAll and it looks like this :
{
"kind": "bigquery#tableDataInsertAllRequest",
"rows":
[
{
"json":
{
"domain": "test",
"kind": "another test"
}
}
]
}
This matches my table schema.
When I run the command the error returned is :
{
"insertErrors": [
{
"index": 0,
"errors": [
{
"message": "no such field",
"reason": "invalid"
}
]
}
],
"kind": "bigquery#tableDataInsertAllResponse"
}
As I say the same TableDataInsertAllRequest resource works fine in the API explorer (clicking Try It on the documentation page above) it just does not work through Apps Script.
Any help gratefully received.
I've run into this too, and had somewhat better luck with this variation.
var rowObjects = [];
// Generally you'd do this next bit in a loop
var rowData = {};
rowData.domain = 'test';
rowData.kind = 'another test';
rowObjects.push(rowData);
// And at this point you'd have an array rowObjects with a bunch of objects
var response = BigQuery.Tabledata.insertAll({'rows': rowObjects}, projectId, datasetId, tableId);
Some things to note:
I don't indicate a kind -- it is implied by the call to insertAll()
I use dot notation (is that the right term?) rather than strings to stuff attributes into my "row objects"
I'm not sure which of these is the Secret Sauce. Anyways, in the end, the structure of the call looks about like this:
BigQuery.Tabledata.insertAll({'rows' : [
{
'domain' : 'test',
'kind' : 'another test'
}
]
},
projectId,
datasetId,
tableId);