Parsing json in terraform locals - json

I have another question about JSON parsing. Imagine I have this JSON:
{
"all_dogs" :[
{
"name": "foo",
"groups": ["morning", "evening"]
},
{
"name": "bar",
"groups": ["evening", "saturday"]
},
{
"name": "feet",
"groups": ["afternoon"]
}
]
}
I can extract all the groups like this:
locals {
all_dogs = jsondecode(file("${path.module}/dogs.json"))
all_groups = toset(flatten(local.all_dogs.all_dogs[*].groups))
}
Now, I’m trying to create a MAP. Each group is a key of the map and values of the maps are the different dogs in that group.
So I would like to create a map like this:
afternoon = [feet]
evening= [foo, bar]
morning= [foo]
saturday= [bar]
I'm trying with something like this and I tried several options... But I can't make it work.
output "ex" {
value = flatten([
for group in local.all_groups: [
for dog in local.all_dogs : {
group = group
dog = dog
}
]
]
)
}
Later on, I would like to use that map to provision some resources. Is this eventually possible?

locals {
all_dogs = jsondecode(file("${path.module}/dogs.json"))
groups = flatten([for d in local.all_dogs.all_dogs : [for g in d.groups : { key : g, value : d.name }]])
}
The value for local groups will be a list of tuples and it will look something like this:
groups = [
{
"key" = "morning"
"value" = "foo"
},
{
"key" = "evening"
"value" = "foo"
},
{
"key" = "evening"
"value" = "bar"
},
{
"key" = "saturday"
"value" = "bar"
},
{
"key" = "afternoon"
"value" = "feet"
},
]
Now we have to create a map from this list:
output "my_map" {
value = {
for g in local.groups : g.key => g.value...
}
}
This will produce the following output:
my_map= {
"afternoon" = [
"feet",
]
"evening" = [
"foo",
"bar",
]
"morning" = [
"foo",
]
"saturday" = [
"bar",
]

Related

Accessing the first object in a tuple using terraform

I am trying to access the first key on a given tuple. The key's name is dynamic and may change so it cannot be accessed using a static value and has to be done either through a for loop or through the use of terraform fuctions.
I've created a small local resource that outputs the following section
skip_empty_mails = { for key, value in local.j.settings.tasks : key => value.email_notifications if value.email_notifications != {} }
The output sent back is
{
"3" = {
"on_start" = [
"foo1#aligntech.com",
"foo2#aligntech.com",
"foo3#aligntech.com",
"foo4#aligntech.com",
]
}
"4" = {
"no_alert_for_skipped_runs" = false
"on_start" = [
"foo21#aligntech.com",
"foo22#aligntech.com",
"foo23#aligntech.com",
"foo24#aligntech.com",
]
]
"on_start" = [
"foo21#aligntech.com",
"foo22#aligntech.com",
"foo23#aligntech.com",
"foo24#aligntech.com",
]
"on_success" = [
"foo21#aligntech.com",
"foo22#aligntech.com",
"foo23#aligntech.com",
"foo24#aligntech.com",
]
}
}
As seen above the key that holds all the values needs to be accessed in a way that will give me the ability to attach it to a string and use string.on_start to pull its values.
The issue is that our key's name is dynamic and may vary.
I've tried following the terraform function documentation But haven't found anything that might be of use in this case.
You may be able to replicate using the following code
locals {
json = {
"3" = {
"on_start" = [
"foo1#aligntech.com",
"foo2#aligntech.com",
"foo3#aligntech.com",
"foo4#aligntech.com",
]
},
"4" = {
"no_alert_for_skipped_runs" = false
"on_failure" = [
"foo21#aligntech.com",
"foo22#aligntech.com",
"foo23#aligntech.com",
"foo24#aligntech.com",
]
"on_start" = [
"foo21#aligntech.com",
"foo22#aligntech.com",
"foo23#aligntech.com",
"foo24#aligntech.com",
]
"on_success" = [
"foo1#foo.com",
"foo2#foo.com",
"foo3#foo.com",
"foo4#foo.com",
]
}
}
}
You can try with a combination of values, element or flatten see the documentation:
https://developer.hashicorp.com/terraform/language/functions/values
https://developer.hashicorp.com/terraform/language/functions/element
https://developer.hashicorp.com/terraform/language/functions/flatten
Below are samples:
First key extraction
locals {
json = {
"4" = {
"no_alert_for_skipped_runs" = false
"on_failure" = [
"foo1#foo.com",
"foo2#foo.com",
]
"on_start" = [
"foo1#foo.com",
"foo2#foo.com",
]
}
}
}
output "data" {
value = element(values(local.json), 1).on_start
}
the Terraform plan will be:
Changes to Outputs:
+ data = [
+ "foo1#foo.com",
+ "foo2#foo.com",
]
Extract and combine on_start item from all
locals {
json = {
"3" = {
"on_start" = [
"foo1#aligntech.com",
"foo2#aligntech.com",
]
},
"4" = {
"on_start" = [
"foo1#foo.com",
"foo2#foo.com",
]
}
}
}
output "data" {
value = flatten(values(local.json)[*].on_start)
}
the Terraform plan will be:
Changes to Outputs:
+ data = [
+ "foo1#aligntech.com",
+ "foo2#aligntech.com",
+ "foo1#foo.com",
+ "foo2#foo.com",
]

How to get the All index values in Groovy JSON xpath

Please find the attached Groovy code which I am using to get the particular filed from the response body.
Query 1 :
It is retrieving the results when the I am using the correct Index value like if the data.RenewalDetails[o], will give output as Value 1 and if the data.RenewalDetails[1], output as Value 2.
But in my real case, I will never know about number of blocks in the response, so I want to get all the values that are satisficing the condition, I tried data.RenewalDetails[*] but it is not working. Can you please help ?
Query 2:
Apart from the above condition, I want to add one more filter, where "FamilyCode": "PREMIUM" in the Itemdetails, Can you help on the same ?
def BoundId = new groovy.json.JsonSlurper().parseText('{"data":{"RenewalDetails":[{"ExpiryDetails":{"duration":"xxxxx","destination":"LHR","from":"AUH","value":2,"segments":[{"valudeid":"xxx-xx6262-xxxyyy-1111-11-11-1111"}]},"Itemdetails":[{"BoundId":"Value1","isexpired":true,"FamilyCode":"PREMIUM","availabilityDetails":[{"travelID":"AAA-AB1234-AAABBB-2022-11-10-1111","quota":"X","scale":"XXX","class":"X"}]}]},{"ExpiryDetails":{"duration":"xxxxx","destination":"LHR","from":"AUH","value":2,"segments":[{"valudeid":"xxx-xx6262-xxxyyy-1111-11-11-1111"}]},"Itemdetails":[{"BoundId":"Value2","isexpired":true,"FamilyCode":"PREMIUM","availabilityDetails":[{"travelID":"AAA-AB1234-AAABBB-2022-11-10-1111","quota":"X","scale":"XXX","class":"X"}]}]}]},"warnings":[{"code":"xxxx","detail":"xxxxxxxx","title":"xxxxxxxx"}]}')
.data.RenewalDetails[0].Itemdetails.find { itemDetail ->
itemDetail.availabilityDetails[0].travelID.length() == 33
}?.BoundId
println "Hello " + BoundId
Something like this:
def txt = '''\
{
"data": {
"RenewalDetails": [
{
"ExpiryDetails": {
"duration": "xxxxx",
"destination": "LHR",
"from": "AUH",
"value": 2,
"segments": [
{
"valudeid": "xxx-xx6262-xxxyyy-1111-11-11-1111"
}
]
},
"Itemdetails": [
{
"BoundId": "Value1",
"isexpired": true,
"FamilyCode": "PREMIUM",
"availabilityDetails": [
{
"travelID": "AAA-AB1234-AAABBB-2022-11-10-1111",
"quota": "X",
"scale": "XXX",
"class": "X"
}
]
}
]
},
{
"ExpiryDetails": {
"duration": "xxxxx",
"destination": "LHR",
"from": "AUH",
"value": 2,
"segments": [
{
"valudeid": "xxx-xx6262-xxxyyy-1111-11-11-1111"
}
]
},
"Itemdetails": [
{
"BoundId": "Value2",
"isexpired": true,
"FamilyCode": "PREMIUM",
"availabilityDetails": [
{
"travelID": "AAA-AB1234-AAABBB-2022-11-10-1111",
"quota": "X",
"scale": "XXX",
"class": "X"
}
]
}
]
}
]
},
"warnings": [
{
"code": "xxxx",
"detail": "xxxxxxxx",
"title": "xxxxxxxx"
}
]
}'''
def json = new groovy.json.JsonSlurper().parseText txt
List<String> BoundIds = json.data.RenewalDetails.Itemdetails*.find { itemDetail ->
itemDetail.availabilityDetails[0].travelID.size() == 33 && itemDetail.FamilyCode == 'PREMIUM'
}?.BoundId
assert BoundIds.toString() == '[Value1, Value2]'
Note, that you will get the BoundIds as a List
If you amend your code like this:
def json = new groovy.json.JsonSlurper().parse(prev.getResponseData()
you would be able to access the number of returned items as:
def size = json.data.RenewalDetails.size()
as RenewalDetails represents a List
Just add as many queries you want using Groovy's && operator:
find { itemDetail ->
itemDetail.availabilityDetails[0].travelID.length() == 33 &&
itemDetail.FamilyCode.equals('PREMIUM')
}
More information:
Apache Groovy - Parsing and producing JSON
Apache Groovy: What Is Groovy Used For?

Modifying JSON in Groovy (or JOLT)

I've a simple JSON look like:
{
"account_login" : "google#gmail.com",
"view_id" : 1868715,
"join_id" : "utm_campaign=toyota&utm_content=multiformat_sites&utm_medium=cpc&utm_source=facebook",
"start_date" : "2020-02-03",
"end_date" : "2020-08-30"
}
With following Groovy script (from this answer):
def content = """
{
"account_login" : "google#gmail.com",
"view_id" : 1868715,
"join_id" : "utm_campaign=toyota&utm_content=multiformat_sites&utm_medium=cpc&utm_source=facebook",
"start_date" : "2020-02-03",
"end_date" : "2020-08-30"
}
"""
def slurped = new JsonSlurper().parseText(content)
def builder = new JsonBuilder(slurped)
builder.content.join_id = builder.content.join_id.split("\\s*&\\s*") //# to array
.collectEntries{
//# convert each item to map entry
String[] utmMarks = it.trim().split("\\s*=\\s*")
utmMarks[0] = [
"utm_medium" : "ga:medium",
"utm_campaign" : "ga:campaign",
"utm_source" : "ga:source",
"utm_content" : "ga:adContent",
"utm_term" : "ga:keyword",
].get( utmMarks[0] )
utmMarks
}
.findAll{
k,v-> k && v!=null //# filter out empty/null keys
}
//builder.content.filters = ...
println(builder.toPrettyString())
I'll get:
{
"account_login": "google#gmail.com",
"view_id": 1868715,
"join_id": {
"ga:campaign": "toyota",
"ga:adContent": "multiformat_sites",
"ga:medium": "cpc",
"ga:source": "facebook"
},
"start_date": "2020-02-03",
"end_date": "2020-08-30"
}
I want to update this script (or write new) and add new property: array filters to modified json above. Expected output:
{
"account_login":"google#gmail.com",
"view_id":1868715,
"join_id":{
"ga:campaign":"toyota",
"ga:adContent":"multiformat_sites",
"ga:medium":"cpc",
"ga:source":"facebook"
},
"start_date":"2020-02-03",
"end_date":"2020-08-30",
"converted_utm_marks":"ga:campaign=toyota&ga:adContent=multiformat_sites&ga:medium=cpc&ga:source=facebook",
"filters":[
{
"dimensionName":"ga:medium",
"operator":"EXACT",
"expressions":[
"cpc"
]
},
{
"dimensionName":"ga:adContent",
"operator":"EXACT",
"expressions":[
"multiformat_sites"
]
},
{
"dimensionName":"ga:campaign",
"operator":"EXACT",
"expressions":[
"toyota"
]
},
{
"dimensionName":"ga:source",
"operator":"EXACT",
"expressions":[
"facebook"
]
}
]
}
But the problem is that the set of filters for each JSON will be different. This set depends directly on the join_id set. If JSON join_id will contain:
"join_id": {
"ga:campaign": "toyota",
"ga:keyword": "car"
}
filters array should be:
[
{
"dimensionName":"ga:campaign",
"operator":"EXACT",
"expressions":[
"toyota"
]
},
{
"dimensionName":"ga:keyword",
"operator":"EXACT",
"expressions":[
"car"
]
}
]
operator is always equals EXACT. Property dimensionName - is a join_id.propety name. Expressions is a join_id.property value. So, property filters based on join_id and I need to loop through join_id property and build filters array with described structure. How to achieve expected output? JOLT configuration appreciated also.
I can't even simple iterate through join_id map:
slurped.join_id.each { println "Key: $it.key = Value: $it.value" }
I got the error:
/home/jdoodle.groovy: 24: illegal colon after argument expression;
solution: a complex label expression before a colon must be parenthesized # line 24, column 28.
.collect { [it.ga:campaign] }
UPDATE
I found out how to build this array:
def array =
[
filters: slurped.join_id.collect {key, value ->
[
dimensionName: key,
operator: "EXACT",
expressions: [
value
]
]
}
]
Seems like i got it:
def slurped = new JsonSlurper().parseText(content)
def builder = new JsonBuilder(slurped)
builder.content.filters = builder.content.join_id.collect {key, value ->
[
dimensionName: key,
operator: "EXACT",
expressions: [
value
]
]
}
Are there any better solutions?
def slurped = new JsonSlurper().parseText(content)
def builder = new JsonBuilder(slurped)
builder.content.filters = builder.content.join_id.collect {key, value ->
[
dimensionName: key,
operator: "EXACT",
expressions: [
value
]
]
}

Terraform: JSON Path Query doesn't work in terraform

I have data like below in json file site24x7IPs.json, and new filter it in terraform:
{
"LocationDetails": [
{
"IPv6_Address_External": "2803:eb80:4000:d::0/64",
"City": "Buenos Aires",
"Place": "Argentina",
"external_ip": "170.78.75.88"
},
{
"IPv6_Address_External": "",
"City": "Buenos Aires",
"Place": "Argentina",
"external_ip": "170.78.75.87"
},
{
"IPv6_Address_External": "",
"City": "Melbourne",
"Place": "Australia",
"external_ip": "103.91.166.0/24"
}
]
}
And terraform code:
locals {
site24x7IPs = jsondecode(file("${path.module}/site24x7IPs.json"))
}
output "site24x7IPs" {
#value = local.site24x7IPs.LocationDetails[*].external_ip # This works
# I'd like to filter the IP from Australia,
value = local.site24x7IPs.LocationDetails[?(#.Place == "Australia")].external_ip
}
Expecting Result:
"103.91.166.0/24"
Output:
value = local.site24x7IPs.LocationDetails[?(#.Place == "Australia")].external_ip
This character is not used within the language.
.LocationDetails[?(#.Place == "Australia")].external_ip is the JSON query syntax, but it doesn't work in Terraform.
Is there a similar way to achieve the filtering goal in Terraform?
Thanks,
This should give you the result:
output "australia_ip_with_quotes" {
value = format("%q",element([for i in local.site24x7IPs.LocationDetails: i.external_ip if i.Place == "Australia"],0))
}
output "australia_ip_without_quotes" {
value = element([for i in local.site24x7IPs.LocationDetails: i.external_ip if i.Place == "Australia"],0)
}
output "list" {
value = [for i in local.site24x7IPs.LocationDetails: i.external_ip if i.Place == "Australia"]
}
Outputs:
australia_ip_with_quotes = "103.91.166.0/24"
australia_ip_without_quotes = 103.91.166.0/24
list = [
"103.91.166.0/24",
]

merge lists of dictionaries in terraform v0.12

I would like to do the following using terraform:
I have 2 JSONs:
1.json:
[
{
"description": "description1",
"url": "url1",
"data": "data1"
},
{
"description": "description2",
"url": "url2",
"data": "data2",
"action": "action2"
},
{
"description": "description3",
"url": "url3",
"data": "data3"
}
]
2.json:
[
{
"description": "description1",
"url": "url1",
"data": "data1"
},
{
"description": "description2_new",
"url": "url2",
"data": "data2_new"
},
{
"description": "description4",
"url": "url4",
"data": "data4"
}
]
and I want to merge them into one. Dictionaries from the second JSON should override dictionaries from the first one if url key is the same. I.e. combined JSON should look like:
[
{
"description": "description1",
"url": "url1",
"data": "data1"
},
{
"description": "description2_new",
"url": "url2",
"data": "data2_new"
},
{
"description": "description3",
"url": "url3",
"data": "data3"
},
{
"description": "description4",
"url": "url4",
"data": "data4"
}
]
Using python I can easily do it:
import json
with open('1.json') as f:
json1 = json.load(f)
with open('2.json') as f:
json2 = json.load(f)
def list_to_dict(json_list):
res_dict = {}
for d in json_list:
res_dict[d['url']] = d
return res_dict
def merge_json(json1, json2):
j1 = list_to_dict(json1)
j2 = list_to_dict(json2)
j1.update(j2)
res_list = []
for key in j1.keys():
res_list.append(j1[key])
return res_list
print(json.dumps(merge_json(json1, json2), indent=4))
How can I do that using terraform?
Using terraform 0.12.x
$ cat main.tf
locals {
# read from files and turn into json
list1 = jsondecode(file("1.json"))
list2 = jsondecode(file("2.json"))
# iterate over lists and turn url into a unique key
dict1 = { for item in local.list1 : item.url => item }
dict2 = { for item in local.list2 : item.url => item }
# combine both dictionaries so values converge
# only take its values
merged = values(merge(local.dict1, local.dict2))
}
output "this" {
value = local.merged
}
$ terraform apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
this = [
{
"data" = "data1"
"description" = "description1"
"url" = "url1"
},
{
"data" = "data2_new"
"description" = "description2_new"
"url" = "url2"
},
{
"data" = "data3"
"description" = "description3"
"url" = "url3"
},
{
"data" = "data4"
"description" = "description4"
"url" = "url4"
},
]
Terraform supports expanding a list into function parameters using the ... operator. This will allow an arbitrary number of documents to be read.
(I'm not sure, but I believe this feature was added in v0.15)
For this example, I added a new file 3.json with the contents:
[
{
"description": "description4_new",
"url": "url4",
"data": "data4_new"
}
]
For main.tf, I'm using the same logic as #someguyonacomputer's answer:
$ cat main.tf
locals {
jsondocs = [
for filename in fileset(path.module, "*.json") : jsondecode(file(filename))
]
as_dicts = [
for arr in local.jsondocs : {
for obj in arr : obj.url => obj
}
]
# This is where the '...' operator is used
merged = merge(local.as_dicts...)
}
output "as_list" {
value = values(local.merged)
}
Result:
Changes to Outputs:
+ as_list = [
+ {
+ data = "data1"
+ description = "description1"
+ url = "url1"
},
+ {
+ data = "data2_new"
+ description = "description2_new"
+ url = "url2"
},
+ {
+ data = "data3"
+ description = "description3"
+ url = "url3"
},
+ {
+ data = "data4_new"
+ description = "description4_new"
+ url = "url4"
},
]
References:
Terraform Docs -- Function Calls # Expanding Function Arguments