Logstash is not converting correctly in JSON - json

Question

Following is my json log file
[
{
"error_message": " Failed to get line from input file (end of file?).",
"type": "ERROR",
"line_no": "2625",
"file": "GTFplainText.c",
"time": "17:40:02",
"date": "01/07/16",
"error_code": "GTF-00014"
},
{
"error_message": " Bad GTF plain text file header or footer line. ",
"type": "ERROR",
"line_no": "2669",
"file": "GTFplainText.c",
"time": "17:40:02",
"date": "01/07/16",
"error_code": "GTF-00004"
},
{
"error_message": " '???' ",
"type": "ERROR",
"line_no": "2670",
"file": "GTFplainText.c",
"time": "17:40:02",
"date": "01/07/16",
"error_code": "GTF-00005"
},
{
"error_message": " Failed to find 'event source'/'product detail' records for event source '3025188506' host event type 1 valid",
"type": "ERROR",
"line_no": "0671",
"file": "RGUIDE.cc",
"time": "15:43:48",
"date": "06/07/16",
"error_code": "RGUIDE-00033"
}
]
According to my understanding As the log is already in json, We do not need filter section in logstash configuration. Following is my logstash config
input {
file{
path => "/home/ishan/sf_shared/log_json.json"
start_position => "beginning"
codec => "json"
}
}
and the output configuration is
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
}
stdout { codec => rubydebug }
}
But it seems like the data is not going into ES, as I am not able to see the data when I query the index. What am I missing?

Answer 1

I think the problem is that the json codec expects a full json message on one line and won't work with a message on multiple lines.
A possible work around would be to use the multiline codex and use the json filter.
The configuration for the multiline codec would be:
multiline {
pattern => "]"
negate => "true"
what => "next"
}
All the lines that do not begin with ] will be regrouped with the next line, so you'll have one full json document to give to the json filter.