Generate multiple JQ output documents from a single input, modifying each result - json

Question

I want to do 2 operations in my JSON file . I tried to do it with JQ and SHELL .
First one : I want to tranform the parents elements to an pure text value
Second one : I want to remove one specific level in the JSON tree
Input :
{
"template_first": {
"order": 0,
"index_patterns": [
"first"
],
"settings": {
"index": {
"codec": "best_compression",
"refresh_interval": "30s",
"analysis": {
"normalizer": {
"norm_case_insensitive": {
"filter": "lowercase",
"type": "custom"
}
}
},
"number_of_shards": "1",
"number_of_replicas": "1"
}
},
"mappings": {
"_doc": {
"dynamic": true,
"dynamic_templates": [
{
"strings": {
"mapping": {
"type": "keyword"
},
"match_mapping_type": "string"
}
}
],
"properties": {
"log.id": {
"type": "keyword"
},
"host.indexer.hostname": {
"type": "keyword"
},
"ts_indexer": {
"format": "strict_date_optional_time||epoch_millis",
"type": "date"
}
}
}
}
},
"template_second": {
"order": 0,
"index_patterns": [
"second"
],
"settings": {
"index": {
"codec": "best_compression",
"refresh_interval": "30s",
"analysis": {
"normalizer": {
"norm_case_insensitive": {
"filter": "lowercase",
"type": "custom"
}
}
},
"number_of_shards": "1",
"number_of_replicas": "1"
}
},
"mappings": {
"_doc": {
"dynamic": true,
"dynamic_templates": [
{
"strings": {
"mapping": {
"type": "keyword"
},
"match_mapping_type": "string"
}
}
],
"properties": {
"log.id": {
"type": "keyword"
},
"host.indexer.hostname": {
"type": "keyword"
},
"ts_indexer": {
"format": "strict_date_optional_time||epoch_millis",
"type": "date"
}
}
}
}
}
}
You see there two JSON object in the file
{
"template_first" : { ...},
"template_second" : { ... }
}
The first modification comes from the appearance of this command
PUT _template/template_number
instead of the key of the first JSON object.
So the expected result
PUT _template/template_first
{...}
PUT _template/template_second
{...}
The second change comes with the removal of _doc level
Before :
"mappings": {
"_doc": {
"dynamic": true,
"dynamic_templates": [
{
"strings": {
"mapping": {
"type": "keyword"
},
"match_mapping_type": "string"
}
}
],
"properties": {
"log.id": {
"type": "keyword"
},
"host.indexer.hostname": {
"type": "keyword"
},
"ts_indexer": {
"format": "strict_date_optional_time||epoch_millis",
"type": "date"
}
}
}
}
Expected result
"mappings": {
"dynamic": true,
"dynamic_templates": [
{
"strings": {
"mapping": {
"type": "keyword"
},
"match_mapping_type": "string"
}
}
],
"properties": {
"log.id": {
"type": "keyword"
},
"host.indexer.hostname": {
"type": "keyword"
},
"ts_indexer": {
"format": "strict_date_optional_time||epoch_millis",
"type": "date"
}
}
}
So the actual result look like this
PUT _template/template_first
{
"order": 0,
"index_patterns": [
"first"
],
"settings": {
"index": {
"codec": "best_compression",
"refresh_interval": "30s",
"analysis": {
"normalizer": {
"norm_case_insensitive": {
"filter": "lowercase",
"type": "custom"
}
}
},
"number_of_shards": "1",
"number_of_replicas": "1"
}
},
"mappings": {
"dynamic": true,
"dynamic_templates": [
{
"strings": {
"mapping": {
"type": "keyword"
},
"match_mapping_type": "string"
}
}
],
"properties": {
"log.id": {
"type": "keyword"
},
"host.indexer.hostname": {
"type": "keyword"
},
"ts_indexer": {
"format": "strict_date_optional_time||epoch_millis",
"type": "date"
}
}
}
}
PUT _template/template_second
{
"order": 0,
"index_patterns": [
"second"
],
"settings": {
"index": {
"codec": "best_compression",
"refresh_interval": "30s",
"analysis": {
"normalizer": {
"norm_case_insensitive": {
"filter": "lowercase",
"type": "custom"
}
}
},
"number_of_shards": "1",
"number_of_replicas": "1"
}
},
"mappings": {
"dynamic": true,
"dynamic_templates": [
{
"strings": {
"mapping": {
"type": "keyword"
},
"match_mapping_type": "string"
}
}
],
"properties": {
"log.id": {
"type": "keyword"
},
"host.indexer.hostname": {
"type": "keyword"
},
"ts_indexer": {
"format": "strict_date_optional_time||epoch_millis",
"type": "date"
}
}
}
}
I achieved to do the second change : delete one level of the JSON array by using the command
jq 'keys[] as $k | map( .mappings =.mappings._doc )' template.json
But i don't know how to do the first change and the 2nd change in the same time .
I tried to loop into the array like this , without success
for row in $(jq 'keys[] as $k | "\($k)"' template.json); do
_jq() {
echo ${row}
}
echo $(_jq '.name')
done

Answer 1

Calling jq just once, and having it write a NUL-delimited list of template-name / modified-template-content pairs (which a bash while read loop can then iterate over):
while IFS= read -r -d '' template_name && IFS= read -r -d '' template_content; do
echo "We want to do PUT the following to _template/$template_name"
printf '%s\n' "$template_content"
done < <(
jq -j '
to_entries[] |
.key as $template_name |
.value as $template_content |
($template_name, "\u0000",
($template_content | (.mappings = .mappings._doc) | tojson), "\u0000")
' <infile.json
)

Answer 2

I had some trouble with the done < <( that caused syntax error in my shell (don't know why ) .
So I modified your script like this :
jq -j 'to_entries[] | .key as $template_name | .value as $template_content | ($template_name, "\u0000", ($template_content | (.mappings = .mappings._doc) | tojson), "\u0000")' < infile.json |
while IFS= read -r -d '' template_name && IFS= read -r -d '' template_content; do
echo "PUT _template/$template_name"
printf '%s\n' "$template_content"
done
Which perfectly does the job !
Thanks Charles