fluentd nested json parsing - json

Question

I have logs like the following:
{
"log": {
"header": {
"key": "value",
"nested": "{\"key1\":\"value\",\"key2\":\"value\"}",
"dateTime": "2019-05-08T20:58:06+00:00"
},
"body": {
"path": "/request/path/",
"method": "POST",
"ua": "curl/7.54.0",
"resp": 200
}
}
}
I'm trying to aggregate logs using fluentd and I want the entire record to be JSON. The specific problem is the "$.log.header.nested" field, which is a JSON string. How can I parse and replace that string with its contents?
For clarity, I'd like the logs output by fluentd to look like this:
{
"log": {
"header": {
"key": "value",
"nested": {
"key1": "value",
"key2": "value"
},
"dateTime": "2019-05-08T20:58:06+00:00"
},
"body": {
"path": "/request/path/",
"method": "POST",
"ua": "curl/7.54.0",
"resp": 200
}
}
}
I've found a way to parse the nested field as JSON, but storing to back to the same key it was parsed from isn't clear. It doesn't seem like hash_value_field supports storing to a nested key. Is there some other way to accomplish this?

Answer 1

The following config seems to accomplish what I want. However, I'm not sure if this is the best way. I assume using ruby is far less performant. Any improvements to this are welcome.
<filter logs>
#type parser
key_name "$.log.header.nested"
hash_value_field "parsed_nested"
reserve_data true
remove_key_name_field true
<parse>
#type json
</parse>
</filter>
<filter logs>
#type record_transformer
enable_ruby true
<record>
parsed_nested ${record["log"]["header"]["nested"] = record["parsed_nested"]}
</record>
remove_keys parsed_nested
</filter>