How do I use s3 lifeycle rules in Terraform in a modular form, i.e. referenced in separate JSON? - json

Currently, I'm specifying lifecycle_rule under my s3 resource:
resource "aws_s3_bucket" "bucket-name" {
bucket = "bucket-name"
lifecycle_rule {
id = "Expiration Rule"
enabled = true
prefix = "reports/"
expiration {
days = 30
}
}
}
...but I imagine there must be a way to make this more modular, like putting the lifecycle rule into a separate JSON so I can reference it for multiple s3 buckets and reduce the need to edit each resource. I know how to do this in general and have done this with other resources as seen here:
resource "aws_iam_policy" "devops-admin-write" {
name = "devops-admin-s3"
description = "Devops-Admin group s3 policy."
policy = file("iam_policies/devops-admin-write.json")
}
...the difference is that "lifecycle_rule" is an argument and not an attribute - and it's not clear to me how to make it work. Google-Fu has not yielded any clear answers either.

You can use dynamic blocks that you execute with a generic local variable.
So you just need to change the local variable and changes will reflect in all places where this variable is used.
To make it more maintainable I would suggest building a module and reusing the module or using an exiting module.
But the locals + dynamic implementation could look like this:
locals {
lifecycle_rules = [
{
id = "Expiration Rule"
enabled = true
prefix = "reports/"
expiration = {
days = 30
}
}
]
}
resource "aws_s3_bucket" "bucket-name" {
bucket = "bucket-name"
dynamic "lifecycle_rule" {
for_each = local.lifecycle_rules
content {
id = lifecycle_rule.each.id
enabled = lifecycle_rule.each.enabled
prefix = lifecycle_rule.each.prefix
expiration {
days = lifecycle_rule.each.expiration.days
}
}
}
}
This does not check for errors and is not complete of course - it just implements your example.
See a more complete generic example in our terraform s3-bucket module: find code here

Related

In Terraform, how can I use a for_each and manipulate each resource that is created?

I would like to see if it's possible to manipulate each resource in a for_each without creating more resources by the reference of the resource.
For example, I'm creating 3 AWS S3 buckets, in my S3 module I have this resource:
resource "aws_s3_bucket" "s3-bucket" {
for_each = var.bucket_details
bucket = each.value.bucket
acl = each.value.acl
policy = each.value.policy
}
And this is my .tfvars:
bucket_details = {
bucket_1 = {
bucket = "stg-kikeman-bkt-1"
acl = "private"
policy = <<EOF
{ ... <= THIS IS A JSON S3 POLICY
}
EOF
},
bucket_2 = { ... }
bucket_3 = { ... }
After an apply, it suppose to create 3 buckets with the values of the bucket name, acl and its policy.
But I'm stuck with the error:
on modules/s3/s3.tf line 15, in resource "aws_s3_bucket" "s3-bucket":
│ 15: policy = each.value.policy
│ ├────────────────
│ │ each.value is map of string with 3 elements
│
│ This map does not have an element with the key "policy"
It looks like the value for the key policy which is a json policy is not being recognized as a "value", I've tried using the jsonencode({the_json_policy}) but I got the error of
Functions may not be called here
I'm looking on how can I use the json policy as a "value" form a map, if it's possible of course.
for_each is intended for situations where you want to systematically declare multiple objects based on data in a map. If you need to have differences between the instances then you will need to somehow represent those differences in the map you provide, so that the design is still systematic.
One way to do it would be to make your source collection instead be a mapping from names to objects containing policy documents and ACL strings:
locals {
buckets = {
stg-bkt-1 = {
acl = "private"
policy = {
Version = "2012-10-17"
Id = Test
Statement = [
{
Sid = "1"
Effect = "Deny"
Principal = "*"
Action = "*"
Resource = "arn:aws:s3:::stg-bkt-1/*"
Condition = {
Bool = {
"aws:SecureTransport": "false"
}
}
},
]
}
}
stg-bkt-2 = {
acl = "public-read"
policy = {
Version = "2012-10-17"
Id = Test
Statement = [
{
Sid = "1"
Effect = "Deny"
Principal = "*"
Action = "*"
Resource = "arn:aws:s3:::stg-bkt-2/*"
Condition = {
Bool = {
"aws:SecureTransport": "false"
}
}
},
{
Sid = "2"
Effect = "Allow"
Principal = "*"
Action = ["s3:GetObject"]
Resource = "arn:aws:s3:::stg-bkt-2/*"
},
]
}
}
}
}
resource "aws_s3_bucket" "s3-bucket" {
for_each = var.buckets
bucket = each.key
acl = each.value.acl
policy = jsonencode(each.value.policy)
}
In situations where the resource configuration ends up just entirely references to each.key and each.value I'd typically say that this isn't really systematic at all and so it'd be simpler and easier to just write out multiple resource "aws_s3_bucket" blocks. But if you can adjust this so that all of the buckets still have some characteristics in common -- for example, if the definition in the mapping only included additional IAM policy statements to be concatenated with the ones that are common to all buckets -- then it seems justified to use for_each to systematically declare them, so that those common parts will be forced to stay common as you change details in future.
Is it possible?
It is. Depending of what you want to change in your policy, you would have to conditionally set attributes. based on the bucket name which requires different settings.

Terraform 0.12 AWS resource containing JSON built from variable

To provision tag policies in an AWS organization, I need to build the JSON content from variables. Management of tag policies, scp, etc. shall be centralized, so changes can be applied everywhere: Renaming, adding, removing tags, etc.
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
}
}
}
provider "aws" {
profile = "default"
region = "us-west-1"
}
The problem at hand I am facing is: How would I build the JSON object?
Example variable/ tag map:
# tag_policies.tf
variable "resource_tags" {
description = "Central resource tags"
type = list( object( {
name = string
tags = map(string)
} ) )
default = [
{
name = "Environment"
tags = {
prod = "crn::env:prod"
lab = "crn::env:lab"
dev = "crn::env:dev"
}
}
]
}
What I have tried so far is to use HCL template tags, but I end up with one , comma too much when iterating through the map of tag names. This works fine for the join() with the sub-map of tag names, but does not workout if I try to wrap the template markup. Why did I try this? Because I ran out of ideas.
# vars.tf
resource "aws_organizations_policy" "root-tag-policy" {
name = "RootTagPolicy"
type = "TAG_POLICY"
content = <<CONTENT
{
"tags": {
%{ for tag in var.resource_tags_env ~}
"${tag.name}": {
"tag_key": {
"##assign": "${tag.name}",
"##operators_allowed_for_child_policies": [ "##none" ]
},
"tag_value": { "##assign": [ "${join( ", ", values( tag.tags ) )}" ] }
},
%{ endfor ~}
}
}
CONTENT
}
The solution actually was quite simple: Iterate of the tags using a for expression and enclose it with curly braces { … } to return an object (=> returns tuples).
Finally jsonencode() cares about converting the HCL key = value syntax to proper JSON.
resource "aws_organizations_policy" "root-tag-policy" {
name = "RootTagPolicy"
type = "TAG_POLICY"
content = jsonencode( [ for key, tag in var.resource_tags: {
"${tag.name}" = {
"tag_key" = {
"##assign" = tag.name,
"##operators_allowed_for_child_policies" = [ "##none" ]
},
"tag_value" = { "##assign" = [ join( ", ", values( tag.tags ) ) ] }
}
} ] )
}
EDIT This still does not work, as I forgot that the whole JSON object needs to get wrapped inside a tags: {}.
kaiser's answer shows a good general approach: build a suitable data structure and then pass it to jsonencode to get a valid JSON string from it.
Here's an example that I think matches what the string template in the original question would've produced:
content = jsonencode({
tags = {
for tag in var.resource_tags_env : tag.name => {
tag_key = {
"##assign" = tag.name
"##operators_allowed_for_child_policies" = ["##none"]
}
tag_value = {
"##assign" = values(tag.tags)
}
}
}
})
I'm not familiar with the aws_organizations_policy resource type so I'm sorry if I got some details wrong here, but hopefully you can adapt the above example to generate the JSON data structure you need.
After reading #martin-atkins answer, I finally understood how the for works for objects and maps. The var before the => arrow actually is part of the resulting object. (This highly confused me as I compared it to other languages arrow functions and arguments.)
The first part of the process is to build a map of maps. The main reason is that I don't want to have a convention of a name key in a map of variables. This might lead to handling of conventions later on, what should be avoided at all costs as it is a possible trap if one does not pay close attention or is aware of it. So the key actually is the name now.
Data Structure
variable "resource_tags" {
description = "Central resource tags"
type = map(
map(string)
)
default = {
Environment = {
common = "grn::env:common"
prod = "grn::env:prod"
stage = "grn::env:stage"
dev = "grn::env:dev"
demo = "grn::env:demo"
lab = "grn::env:lab"
},
Foo = {
bar = "baz"
}
}
}
The content as JSON
After understanding that the key in { "tags": { … } } is just the part before the =>, I could reduce the final resource to the following block.
resource "aws_organizations_policy" "root-tag-policy" {
name = "RootTagPolicy"
description = "Tag policies, assigned to the root org."
type = "TAG_POLICY"
content = jsonencode({
tags = {
for key, tags in var.resource_tags : key => {
tag_key = {
"##assign" = key
"##operators_allowed_for_child_policies" = ["##none"]
}
tag_value = {
"##assign" = values( tags )
}
}
}
})
}
Quick test:
Add the following output statement after the resource block:
output "debug" {
value = aws_organizations_policy.tp_root-tag-policy.content
}
Now apply (or plan or refresh) just this resource. It's faster this way. Then output the built debug from the apply or refresh run.
$ terraform apply -target=aws_organizations_policy.root-tag-policy
…things happening…
$ terraform output debug | json_pp
ProTips:
Pipe the output of the output directly into json_pp or jq so you can read it.
Use jq . if you want validation on top. If you see the output, it means it's valid. Else you should receive 0 as response.

How to solve circlular dependencies when deploying cloud endpoints service using cloud run in Terraform

I am currently trying to set up Google Cloud Endpoints on Cloud Run to be able to have an OpenApi Documentation for my Cloud Functions. I followed the instructions in here for a PoC and it worked fine.
Now I have tried to set it up using terraform 0.12.24
Service Endpoint
data "template_file" "openapi_spec" {
template = file("../../cloud_functions/openapi_spec.yaml")
vars = {
endpoint_service = local.service_name
feedback_post_target = google_cloudfunctions_function.feedbackPOST.https_trigger_url
}
}
resource "google_endpoints_service" "api-gateway" {
service_name = local.service_name
project = var.project_id
openapi_config = data.template_file.openapi_spec.rendered
depends_on = [
google_project_service.endpoints,
google_project_service.service-usage,
]
}
Cloud RUN
locals {
service_name = "${var.endpoint_service_name}.endpoints.${var.project_id}.cloud.goog"
}
resource "google_cloud_run_service" "api-management" {
name = "api-gateway-1233"
location = "europe-west1"
template {
spec {
containers {
image = "gcr.io/endpoints-release/endpoints-runtime-serverless:2"
env {
name = "ENDPOINTS_SERVICE_NAME"
value = local.service_name
}
}
}
}
traffic {
percent = 100
latest_revision = true
}
depends_on = [google_project_service.run]
}
If I try to execute my function from the Endpoints portal now, I get the following error
ENOTFOUND: Error resolving domain "https://function-api-gateway.endpoints.PROJECT_ID.cloud.goog"
which makes total sense, as my endpoints service should use the host url of the cloud run service which is given by
google_cloud_run_service.api-management.status[0].url
which means, that I have to use this in the Service endpoints definition above as service name and host-environmental variable in the openApi definition.
Only when this is set, I can again apply my cloud run service with the env variable being its url itself.
This is a circular dependency which I do not know how to solve.
Any help is highly appreciated!

Create Terraform resources out of JSON values

I am looking for a way to generate Terraform code based on JSON values.
Imagine I have a JSON file with the following structure:
{
"settings": [
{
"conf": [
{
"setting": "DeploymentPolicy",
"namespace": "aws:elasticbeanstalk:command",
"value": "AllAtOnce"
},
{
"setting": "BatchSize",
"namespace": "aws:elasticbeanstalk:command",
"value": "30"
},
{
"setting": "BatchSizeType",
"namespace": "aws:elasticbeanstalk:command",
"value": "Percentage"
}
]
}
]
}
What I want to do is the following:
Creating a working Terraform resource based on the JSON file values, e.g. a beanstalk environment like this:
resource "aws_elastic_beanstalk_environment" "app_prod" {
name = "${aws_elastic_beanstalk_application_version.app.name}-prod"
application = aws_elastic_beanstalk_application.app.name
solution_stack_name = data.aws_elastic_beanstalk_solution_stack.latest_linux_java.name
wait_for_ready_timeout = "10m"
version_label = aws_elastic_beanstalk_application_version.app.name
# Elastic beanstalk configuration
setting {
name = "DeploymentPolicy"
namespace = "aws:elasticbeanstalk:command"
value = "AllAtOnce"
}
setting {
name = "BatchSize"
namespace = "aws:elasticbeanstalk:command"
value = "30"
}
...
}
Therefore I have to create the settings block in HCL (Terraform configuration) based on the JSON values.
This means the JSON file above should result in:
setting {
name = "DeploymentPolicy"
namespace = "aws:elasticbeanstalk:command"
value = "AllAtOnce"
}
setting {
name = "BatchSize"
namespace = "aws:elasticbeanstalk:command"
value = "30"
}
setting {
name = "BatchSizeType"
namespace = "aws:elasticbeanstalk:command"
value = "Percentage"
}
As you can see, the structure of JSON and HCL is very similar, but not identical. See e.g. settings, conf, or setting instead of name in the JSON.
A possible approach would be to read the JSON values and store them in an array or a map. But I have no idea how I could generate valid HCL and inject it in the desired part of the resource. Furthermore I tried to use a template but Terraform does not support the looping functionality that I need to iterate over the settings.
To sum up:
Input is a JSON file that must be read
JSON contains settings (besides other information)
The number of settings can differ
Somehow I have to generate a settings block
Somehow I have to inject this settings blok in the resource
Does anyone have an idea how to do that? Any other approaches?
Thanks a lot!
Assuming that your JSON object were in a file called settings.json inside your module directory, you could do something like this:
locals {
environment_settings = jsondecode(file("${path.module}/settings.json")).settings[0].conf[0]
}
resource "aws_elastic_beanstalk_environment" "app_prod" {
name = "${aws_elastic_beanstalk_application_version.app.name}-prod"
application = aws_elastic_beanstalk_application.app.name
solution_stack_name = data.aws_elastic_beanstalk_solution_stack.latest_linux_java.name
wait_for_ready_timeout = "10m"
version_label = aws_elastic_beanstalk_application_version.app.name
dynamic "setting" {
for_each = local.environment_settings
content {
namespace = setting.value.namespace
name = setting.value.setting
value = setting.value.value
}
}
}
This special dynamic block is a sort of macro to create repeated setting blocks, each one correlating with one element of the collection given in for_each.
You can do whatever transformations of the input you need using Terraform's expression language in the locals block to ensure that the local.environment_settings value contains one element for each setting block you will generate, and then in the content nested block tell Terraform how to populate the setting arguments based on those element values.

defining a custom logger in grails 3 using logback

I want to define a different logger from the default to log things in a different file. I've try to define a custom logger. The file is created but grails never logs anything.
appender("APPENDER", FileAppender) {
file = "logs/info.log"
append = true
encoder(PatternLayoutEncoder) {
pattern = "%level - %msg%n"
}
}
logger("logger", INFO, ['APPENDER'], false)
Then in a controller I want to call it using: log.info "something" but nothing is written.
This contains daily rolling policy and max size of the log file. The catch is /var/logs should have a write access.
def currentDay = timestamp("yyyyMMdd")
appender(FILE, RollingFileAppender) {
file = "/var/logs/info_${currentDay}.log"
rollingPolicy(FixedWindowRollingPolicy) {
fileNamePattern = "/var/logs/info_${currentDay}.%i.log"
minIndex = 1
maxIndex = 9
}
triggeringPolicy(SizeBasedTriggeringPolicy) {
maxFileSize = "50MB"
}
encoder(PatternLayoutEncoder) {
pattern = "%level %date %logger - %msg%n"
}
append = true
}
Note: your user should have a write access to /var/logs/
Add: import static ch.qos.logback.classic.Level.INFO
Where: conf/logback.groovy file
Thanks "nayan kakati" I've figured out earlier!