Using a string in JSON format in Terraform variables - json

I am creating some resources using the for_each method on Terraform version 0.14.15. The resource has an attirbute, input_parameters that takes a string in JSON format as its value. I am defining this value in a map variable utilizing separate objects. The value I am specifying as a string in JSON format, and I am getting an error upon execution that I need to declare a string. Any insight on fixing this error would be helpful. Below is how I have my resource and variable declared.
Resource
resource "aws_config_config_rule" "managed_rules" {
for_each = var.managed_rules
name = each.value.name
description = each.value.description
input_parameters = each.value.input_parameters
source {
owner = each.value.owner
source_identifier = each.value.source_identifier
}
depends_on = [aws_config_configuration_recorder.config_recorder]
}
Variable
variable "managed_rules" {
type = map(object({
name = string
description = string
owner = string
source_identifier = string
# Is there a variable for strings in JSON format?
input_parameters = string
}))
default = {
"1" = {
name = "alb-http-to-https-redirection-check"
description = "Checks whether HTTP to HTTPS redirection is configured on all HTTP listeners of Application Load Balancers. The rule is NON_COMPLIANT if one or more HTTP listeners of Application Load Balancer do not have HTTP to HTTPS redirection configured."
owner = "AWS"
source_identifier = "ALB_HTTP_TO_HTTPS_REDIRECTION_CHECK"
input_parameters = {
"MaximumExecutionFrequency" : "TwentyFour_Hours",
}
}
Error
This default value is not compatible with the variable's type constraint:
element "2": attribute "input_parameters": string required.
After updating code with jsonencode function and changing input_parameters to any, this is the error:
This default value is not compatible with the variable's type constraint:
collection elements cannot be unified.

You have a couple things going on here:
The resource requires input_parameters to be a JSON-encoded string
You have the variable type as a string
You're passing an object type into the variable that only accepts a string type
So (2) and (3) are conflicting. At some point, you have to convert your object into a JSON string. You can either do that before passing it in as an input variable, or change your input variable to accept objects and convert the object to JSON when providing it to the resource.
I'd choose the second option because it's more intuitive to pass the object into the module instead of a string. So, try this:
resource "aws_config_config_rule" "managed_rules" {
for_each = var.managed_rules
name = each.value.name
description = each.value.description
input_parameters = jsonencode(each.value.input_parameters)
source {
owner = each.value.owner
source_identifier = each.value.source_identifier
}
depends_on = [aws_config_configuration_recorder.config_recorder]
}
variable "managed_rules" {
type = map(object({
name = string
description = string
owner = string
source_identifier = string
# Is there a variable for strings in JSON format?
input_parameters = any
}))
default = {
"1" = {
name = "alb-http-to-https-redirection-check"
description = "Checks whether HTTP to HTTPS redirection is configured on all HTTP listeners of Application Load Balancers. The rule is NON_COMPLIANT if one or more HTTP listeners of Application Load Balancer do not have HTTP to HTTPS redirection configured."
owner = "AWS"
source_identifier = "ALB_HTTP_TO_HTTPS_REDIRECTION_CHECK"
input_parameters = {
"MaximumExecutionFrequency" : "TwentyFour_Hours",
}
}
Note that I've used jsonencode in the resource's input_parameters and I've changed the variable type for that field to any (so it will accept an object of any structure).

You can create your json string as follows:
variable "managed_rules" {
type = map(object({
name = string
description = string
owner = string
source_identifier = string
# Is there a variable for strings in JSON format?
input_parameters = string
}))
default = {
"1" = {
name = "alb-http-to-https-redirection-check"
description = "Checks whether HTTP to HTTPS redirection is configured on all HTTP listeners of Application Load Balancers. The rule is NON_COMPLIANT if one or more HTTP listeners of Application Load Balancer do not have HTTP to HTTPS redirection configured."
owner = "AWS"
source_identifier = "ALB_HTTP_TO_HTTPS_REDIRECTION_CHECK"
input_parameters = <<EOL
{
"MaximumExecutionFrequency" : "TwentyFour_Hours",
}
EOL
}
}
}
But then you have to use jsondecode if you want to parse this string. You can't use functions in variables, so it must be done later.

To add to Jordan's answer. I Had a similar concern when trying to add a json policy to a module.
I used the any object type in place of the string object type.
Here's how I fixed it:
Module main.tf
resource "aws_ecr_repository_policy" "main" {
repository = var.repository_name
policy = var.repository_policy
}
Module variables.tf
variable "repository_name" {
type = string
description = "Name of the repository."
}
variable "repository_policy" {
type = any
description = "The policy document. This is a JSON formatted string."
}
Resource Creation main.tf
# ECR Repository for container images
module "ecr_repository_1" {
source = "../../../../modules/aws/ecr-repository"
ecr_repository_name = var.ecr_repository_name.1
image_tag_mutability = var.image_tag_mutability
image_scan_on_push = var.image_scan_on_push
tag_environment = local.tag_environment
tag_terraform = local.tag_terraform.true
}
# ECR Repository policies
module "ecr_repository_policy_1" {
source = "../../../../modules/aws/ecr-repository-policy"
repository_name = var.ecr_repository_name.1
repository_policy = var.repository_policy.1
}
Resource creation variables.tf
variable "ecr_repository_name" {
type = map(string)
description = "Name of the repository."
default = {
"1" = "my-backend-api"
}
}
variable "image_tag_mutability" {
type = string
description = "The tag mutability setting for the repository. Must be one of: MUTABLE or IMMUTABLE. Defaults to MUTABLE."
default = "MUTABLE"
}
variable "image_scan_on_push" {
type = bool
description = "Indicates whether images are scanned after being pushed to the repository (true) or not scanned (false)."
default = true
}
variable "repository_policy" {
type = any
description = "The policy document. This is a JSON formatted string."
default = {
"1" = <<EOF
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "new policy",
"Effect": "Allow",
"Principal": "*",
"Action": [
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability",
"ecr:PutImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:DescribeRepositories",
"ecr:GetRepositoryPolicy",
"ecr:ListImages",
"ecr:DeleteRepository",
"ecr:BatchDeleteImage",
"ecr:SetRepositoryPolicy",
"ecr:DeleteRepositoryPolicy"
]
}
]
}
EOF
}
}

input_parameters = {
"MaximumExecutionFrequency" : "TwentyFour_Hours",
}
This has to be a string instead of Object, Since you defined it as String

Related

how can print return object of data in Packer?

Please refer to below code
data "amazon-ami" "ubuntu" {
most_recent = true
filters = {
virtualization-type = "hvm"
name = "ubuntu/images/*ubuntu-focal-20.04-amd64-server-*"
root-device-type = "ebs"
}
owners = ["099720109477"]
}
source "null" "one" {
communicator = "none"
}
build {
name = "source-ouput"
sources = ["source.null.one"]
provisioner "shell-local" {
inline = [
"echo ${data.amazon-ami.ubuntu}", ##### <--- problem happens here
]
}
}
As you can see in the code, I want to print out the return value of data "amazon-ami" "ubuntu"
If I run this with packer build .. error occurs like below
Error: Failed preparing provisioner-block "shell-local" ""
on main.pkr.hcl line 20:
(source code not available)
main.pkr.hcl:22,15-37: Invalid template interpolation value; Cannot include the
given value in a string template: string required.
Is there any solution to print the object that returned by data?

Why does UserAuthExtensions.PopulateFromMap(session, jwtPayload) does not deserialize json values with escape correctly in ServiceStack.Auth?

We want to get the UserName from the ServiceStack session, but we find that the backslashes in the UserName are not deserialized as expected. The UserName has this format 'domainname\username' and serialized in a jwt token this looks like:
{
"typ": "JWT",
"alg": "HS256"
}.{
"iss": "ssjwt",
"iat": 1635952233,
"exp": 1635955833,
"name": "Robin Doe",
"preferred_username": "domainname\\robindoe"
}.[Signature]
After calling:
var sessionFromJwt = JwtAuthProviderReader.CreateSessionFromJwt(req);
userName = sessionFromJwt.UserName;
The userName variable contains the value 'domainname\\robindoe' instead of 'domainname\robindoe'.
After digging in the ServiceStack code, we pin this down to the PopulateFromMap() method in https://github.com/ServiceStack/ServiceStack/blob/36df74a8b1ba7bf06f85262c1155e1425c082906/src/ServiceStack/Auth/UserAuth.cs#L388.
To demonstrate this problem we have written a small program to prove the point:
class Program
{
static void Main(string[] args)
{
var jwtPayload = JsonObject.Parse(#"{
""iss"": ""ssjwt"",
""iat"": 1635952233,
""exp"": 1635955833,
""name"": ""John Doe"",
""preferred_username"": ""domainname\\username""
}");
var session = new AuthUserSession();
// The PopulateFromMap implementation does not deserialize the json values according to json standards
UserAuthExtensions.PopulateFromMap(session, jwtPayload);
// Notice that the session.UserName still has the escape character 'domainname\\username' instead of the expected 'domainname\username'
Console.WriteLine(session.UserName);
// The PopulateFromMap should deserialize also the values, like in test Can_dynamically_parse_JSON_with_escape_chars()
Can_dynamically_parse_JSON_with_escape_chars();
}
private const string JsonCentroid = #"{""place"":{ ""woeid"":12345, ""placeTypeName"":""St\\a\/te"" } }";
// Source: https://github.com/ServiceStack/ServiceStack.Text/blob/master/tests/ServiceStack.Text.Tests/JsonObjectTests.cs
public static void Can_dynamically_parse_JSON_with_escape_chars()
{
var placeTypeName = JsonObject.Parse(JsonCentroid).Object("place").Get("placeTypeName");
if (placeTypeName != "St\\a/te")
throw new InvalidCastException(placeTypeName + " != St\\a/te");
placeTypeName = JsonObject.Parse(JsonCentroid).Object("place").Get<string>("placeTypeName");
if (placeTypeName != "St\\a/te")
throw new InvalidCastException(placeTypeName + " != St\\a/te");
}
}
Why does UserAuthExtensions.PopulateFromMap(session, jwtPayload) does not deserialize json values with escape correctly in ServiceStack.Auth?
The issue is due to enumerating a JsonObject didn't return the same escaped string value as indexing it which has been resolved from this commit.
This change is available from v5.12.1+ that's now available on MyGet.

Downloading dwg to Forge

I'm in the process of learning the Forge platform. I'm currently using an example (Jigsawify) written by Kean Walmsley because it most accurately describes my goals. I'm running into an issue of getting my file to download from an Azure Storage Account to Forge. The error I receive is "The value for one of the HTTP headers is not in the correct format." My question is how does someone go about troubleshooting HTTP protocol when writing, in this case, a workitem in code? I can put in a breakpoint to view the workitem, but I'm not versed enough to understand where the flaw is in the HTTP header, or even where to find it. Is there a specific property of the workitem I should be looking at? If I could find the HTTP statement, I could test it, but I don't where I should find it.
Or am I just completely off base?
Anyway here's the code. It's a modified version of what Kean wrote:
static void SubmitWorkItem(Activity activity)
{
Console.WriteLine("Submitting workitem...");
CloudStorageAccount storageAccount =
CloudStorageAccount.Parse(Microsoft.Azure.CloudConfigurationManager.GetSetting("StorageConnectionString"));
StorageCredentials crd = storageAccount.Credentials;
CloudFileClient fileClient = storageAccount.CreateCloudFileClient();
CloudFileShare ShareRef = fileClient.GetShareReference("000scrub");
CloudFileDirectory rootDir = ShareRef.GetRootDirectoryReference();
CloudFile Fileshare = rootDir.GetFileReference("3359fort.dwg");
// Create a workitem
var wi = new WorkItem()
{
Id = "", // Must be set to empty
Arguments = new Arguments(),
ActivityId = activity.Id
};
if (Fileshare.Exists())
{
wi.Arguments.InputArguments.Add(new Argument()
{
Name = "HostDwg", // Must match the input parameter in activity
Resource = Fileshare.Uri.ToString(),
StorageProvider = StorageProvider.Generic // Generic HTTP download (vs A360)
});
}
wi.Arguments.OutputArguments.Add(new Argument()
{
Name = "Results", // Must match the output parameter in activity
StorageProvider = StorageProvider.Generic, // Generic HTTP upload (vs A360)
HttpVerb = HttpVerbType.POST, // Use HTTP POST when delivering result
Resource = null, // Use storage provided by AutoCAD.IO
ResourceKind = ResourceKind.ZipPackage // Upload as zip to output dir
});
container.AddToWorkItems(wi);
container.SaveChanges();
// Polling loop
do
{
Console.WriteLine("Sleeping for 2 sec...");
System.Threading.Thread.Sleep(2000);
container.LoadProperty(wi, "Status"); // HTTP request is made here
Console.WriteLine("WorkItem status: {0}", wi.Status);
}
while (
wi.Status == ExecutionStatus.Pending ||
wi.Status == ExecutionStatus.InProgress
);
// Re-query the service so that we can look at the details provided
// by the service
container.MergeOption =
Microsoft.OData.Client.MergeOption.OverwriteChanges;
wi = container.WorkItems.ByKey(wi.Id).GetValue();
// Resource property of the output argument "Results" will have
// the output url
var url =
wi.Arguments.OutputArguments.First(
a => a.Name == "Results"
).Resource;
if (url != null)
DownloadToDocs(url, "SGA.zip");
// Download the status report
url = wi.StatusDetails.Report;
if (url != null)
DownloadToDocs(url, "SGA-Report.txt");
}
Any help is appreciated,
Chuck
Azure requires that you specify the x-ms-blob-type header when you upload to a presigned URL. See https://github.com/Autodesk-Forge/design.automation-.net-input.output.sample/blob/master/Program.cs#L167
So, I was able to figure out how to download my file from Azure to Forge using Albert's suggestion of moving to a blob service. Here's the code:
static void SubmitWorkItem(Activity activity)
{
Console.WriteLine("Submitting workitem...");
CloudStorageAccount storageAccount =
CloudStorageAccount.Parse(Microsoft.Azure.CloudConfigurationManager.GetSetting("StorageConnectionString"));
CloudBlobClient BlobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer cloudBlobContainer = BlobClient.GetContainerReference("000scrub");
CloudBlockBlob blockBlob = cloudBlobContainer.GetBlockBlobReference("3359fort.dwg");
// Create a workitem
var wi = new WorkItem()
{
Id = "", // Must be set to empty
Arguments = new Arguments(),
ActivityId = activity.Id
};
if (blockBlob.Exists())
{
wi.Arguments.InputArguments.Add(new Argument()
{
Name = "HostDwg", // Must match the input parameter in activity
Resource = blockBlob.Uri.ToString(),
StorageProvider = StorageProvider.Generic, // Generic HTTP download (vs A360)
Headers = new System.Collections.ObjectModel.ObservableCollection<Header>()
{
new Header() { Name = "x-ms-blob-type", Value = "BlockBlob" } // This is required for Azure.
}
});
}
wi.Arguments.OutputArguments.Add(new Argument()
{
Name = "Results", // Must match the output parameter in activity
StorageProvider = StorageProvider.Generic, // Generic HTTP upload (vs A360)
HttpVerb = HttpVerbType.POST, // Use HTTP POST when delivering result
Resource = null, // Use storage provided by AutoCAD.IO
ResourceKind = ResourceKind.ZipPackage, // Upload as zip to output dir
});
container.AddToWorkItems(wi);
container.SaveChanges();
// Polling loop
do
{
Console.WriteLine("Sleeping for 2 sec...");
System.Threading.Thread.Sleep(2000);
container.LoadProperty(wi, "Status"); // HTTP request is made here
Console.WriteLine("WorkItem status: {0}", wi.Status);
}
while (
wi.Status == ExecutionStatus.Pending ||
wi.Status == ExecutionStatus.InProgress
);
// Re-query the service so that we can look at the details provided
// by the service
container.MergeOption =
Microsoft.OData.Client.MergeOption.OverwriteChanges;
wi = container.WorkItems.ByKey(wi.Id).GetValue();
// Resource property of the output argument "Results" will have
// the output url
var url =
wi.Arguments.OutputArguments.First(
a => a.Name == "Results"
).Resource;
if (url != null)
DownloadToDocs(url, "SGA.zip");
// Download the status report
url = wi.StatusDetails.Report;
if (url != null)
DownloadToDocs(url, "SGA-Report.txt");
}
What isn't complete is the result section. The ZIP has nothing in it, but hey, baby steps right?
Thanks Albert.
-Chuck

Combine JSON and String in a dictionary with Swifty

I'd like to create a JSON object in Swifty that has the form:
{
"store": {
"id": {
"test": "test"
},
"type": "retail",
"name": "store1"
}
}
Is there a way to combine types in a Dictionary to use with Swifty (String and JSON)? Quotes works, but when I try to assign a variable, it complains: Cannot assign value of type 'String' to type 'JSON?':
func jsonTest()->String {
var storeJson = [String: JSON]()
var someJson = JSON(["test":"test"])
storeJson["id"] = someJson
storeJson["type"] = "retail" // <-- works fine
var name = "store1"
storeJson["name"] = name // <-- Doesn't work
var store = JSON(storeJson)
return store.rawString()!
}
The reason
storeJson["type"] = "retail"
works differently than
storeJson["name"] = name
is because the first one follows a different path in the code. Specifically, it uses the init(stringLiteral value: StringLiteralType) method in the following extension (source).
extension JSON: Swift.StringLiteralConvertible {
public init(stringLiteral value: StringLiteralType) {
self.init(value)
}
public init(extendedGraphemeClusterLiteral value: StringLiteralType) {
self.init(value)
}
public init(unicodeScalarLiteral value: StringLiteralType) {
self.init(value)
}
}
I'll explain further after we talk about how to fix your specific problem.
Possible solution #1:
storeJson["name"]?.string = name
Output:
{
"id" : {
"test" : "test"
},
"type" : "retail"
}
The reason
storeJson["name"]?.string = name
doesn't work as we might think is because of the optional chaining. Right now, if we ran this through the debugger, we wouldn't see anything meaningful. In fact, we would see nothing. This is a bit concerning and likely means storeJson["name"] is nil, so the statement is not executing any further. Let's verify our hypothesis by making it blow up. We'll change the line to:
storeJson["name"]!.string = name
In this case, with your current code, you'll likely get
fatal error: unexpectedly found nil while unwrapping an Optional value
as you should because storeJson["name"] is in fact nil. Therefore, this solution doesn't work.
Possible solution #2:
As you correctly noted in your answer, if you add a storeJson["name"] = JSON(name), you'll get the desired behavior:
func jsonTest()->String {
var storeJson = [String: JSON]()
var someJson = JSON(["test":"test"])
storeJson["id"] = someJson
storeJson["type"] = "retail" // <-- works fine
var name = "store1"
storeJson["name"] = JSON(name) // <-- works!
var store = JSON(storeJson)
return store.rawString()!
}
Output:
{
"id" : {
"test" : "test"
},
"name" : "store1",
"type" : "retail"
}
Great! Therefore, this solution works! Now, later in your code you can alter it however you want using .string and the like.
Explanation
Back to why the string literal works. You'll notice in the init, it has
self.init(value)
which passes through the objects init, which then goes through the case statement
...
case let string as String:
_type = .String
self.rawString = string
...
When you call storeJson["name"] = JSON(name), you're skipping the StringLiteralType init and simply going into the switch.
Therefore, you could interchange
storeJson["type"] = "retail"
with
storeJson["type"] = JSON("retail")
It turns out it works to change:
storeJson["name"] = name
to
storeJson["name"] = JSON(name)

Antlr4 StringTemplate not compatible with Json.net dynamic items

I would like to read a dynamic object from a json file and then use this in a stringTemplate.
The following code works.
dynamic data = new { bcName = "Lixam B.V", periodName = "July 2013" };
var engine = new Template("<m.bcName> <m.periodName>");
engine.Add("m", data);
engine.Render().Should().Be("Lixam B.V July 2013");
The following code fails
var json = "{bcName : 'Lixam B.V', periodName : 'July 2013'}";
dynamic data = JsonConvert.DeserializeObject(json);
string name = (data.bcName);
name.Should().Be("Lixam B.V"); // this passes
var engine = new Template("<m.bcName> <m.periodName>");
engine.Add("m", data);
engine.Render().Should().Be("Lixam B.V July 2013"); //fails
Is there another way to configure JsonConverter to be compatible with StringTemplate
You need to create an IModelAdaptor for whatever the compiled type representing dynamic is, and register it using TemplateGroup.RegisterModelAdaptor.
Inspired on Mr. Harwell's answer, I've implemented an IModelAdaptor that enable the usage of Newtonsoft.Json parsed objects.
Here it goes:
internal class JTokenModelAdaptor : Antlr4.StringTemplate.IModelAdaptor
{
public object GetProperty(
Antlr4.StringTemplate.Interpreter interpreter,
Antlr4.StringTemplate.TemplateFrame frame,
object obj,
object property,
string propertyName)
{
var token = (obj as JToken)?.SelectToken(propertyName);
if (token == null)
return null;
if (token is JValue)
{
var jval = token as JValue;
return jval.Value;
}
return token;
}
}
You just need to register the adaptor in your template group, like this:
template.Group.RegisterModelAdaptor(typeof(JToken), new JTokenModelAdaptor());