merge lists of dictionaries in terraform v0.12 - json

Question

I would like to do the following using terraform:
I have 2 JSONs:
1.json:
[
{
"description": "description1",
"url": "url1",
"data": "data1"
},
{
"description": "description2",
"url": "url2",
"data": "data2",
"action": "action2"
},
{
"description": "description3",
"url": "url3",
"data": "data3"
}
]
2.json:
[
{
"description": "description1",
"url": "url1",
"data": "data1"
},
{
"description": "description2_new",
"url": "url2",
"data": "data2_new"
},
{
"description": "description4",
"url": "url4",
"data": "data4"
}
]
and I want to merge them into one. Dictionaries from the second JSON should override dictionaries from the first one if url key is the same. I.e. combined JSON should look like:
[
{
"description": "description1",
"url": "url1",
"data": "data1"
},
{
"description": "description2_new",
"url": "url2",
"data": "data2_new"
},
{
"description": "description3",
"url": "url3",
"data": "data3"
},
{
"description": "description4",
"url": "url4",
"data": "data4"
}
]
Using python I can easily do it:
import json
with open('1.json') as f:
json1 = json.load(f)
with open('2.json') as f:
json2 = json.load(f)
def list_to_dict(json_list):
res_dict = {}
for d in json_list:
res_dict[d['url']] = d
return res_dict
def merge_json(json1, json2):
j1 = list_to_dict(json1)
j2 = list_to_dict(json2)
j1.update(j2)
res_list = []
for key in j1.keys():
res_list.append(j1[key])
return res_list
print(json.dumps(merge_json(json1, json2), indent=4))
How can I do that using terraform?

Answer 1

Using terraform 0.12.x
$ cat main.tf
locals {
# read from files and turn into json
list1 = jsondecode(file("1.json"))
list2 = jsondecode(file("2.json"))
# iterate over lists and turn url into a unique key
dict1 = { for item in local.list1 : item.url => item }
dict2 = { for item in local.list2 : item.url => item }
# combine both dictionaries so values converge
# only take its values
merged = values(merge(local.dict1, local.dict2))
}
output "this" {
value = local.merged
}
$ terraform apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
this = [
{
"data" = "data1"
"description" = "description1"
"url" = "url1"
},
{
"data" = "data2_new"
"description" = "description2_new"
"url" = "url2"
},
{
"data" = "data3"
"description" = "description3"
"url" = "url3"
},
{
"data" = "data4"
"description" = "description4"
"url" = "url4"
},
]

Answer 2

Terraform supports expanding a list into function parameters using the ... operator. This will allow an arbitrary number of documents to be read.
(I'm not sure, but I believe this feature was added in v0.15)
For this example, I added a new file 3.json with the contents:
[
{
"description": "description4_new",
"url": "url4",
"data": "data4_new"
}
]
For main.tf, I'm using the same logic as #someguyonacomputer's answer:
$ cat main.tf
locals {
jsondocs = [
for filename in fileset(path.module, "*.json") : jsondecode(file(filename))
]
as_dicts = [
for arr in local.jsondocs : {
for obj in arr : obj.url => obj
}
]
# This is where the '...' operator is used
merged = merge(local.as_dicts...)
}
output "as_list" {
value = values(local.merged)
}
Result:
Changes to Outputs:
+ as_list = [
+ {
+ data = "data1"
+ description = "description1"
+ url = "url1"
},
+ {
+ data = "data2_new"
+ description = "description2_new"
+ url = "url2"
},
+ {
+ data = "data3"
+ description = "description3"
+ url = "url3"
},
+ {
+ data = "data4_new"
+ description = "description4_new"
+ url = "url4"
},
]
References:
Terraform Docs -- Function Calls # Expanding Function Arguments