I have an ELK stack that receives from filebeat structured JSON logs like these:
{"what": "Connected to proxy service", "who": "proxy.service", "when": "03.02.2016 13:29:51", "severity": "DEBUG", "more": {"host": "127.0.0.1", "port": 2004}}
{"what": "Service registered with discovery", "who": "proxy.discovery", "when": "03.02.2016 13:29:51", "severity": "DEBUG", "more": {"ctx": {"node": "igz0", "ip": "127.0.0.1:5301", "irn": "proxy"}, "irn": "igz0.proxy.827378e7-3b67-49ef-853c-242de033e645"}}
{"what": "Exception raised while setting service value", "who": "proxy.discovery", "when": "03.02.2016 13:46:34", "severity": "WARNING", "more": {"exc": "ConnectionRefusedError('Connection refused',)", "service": "igz0.proxy.827378e7-3b67-49ef-853c-242de033e645"}}
The "more" field which is a nested JSON is broken down (not sure by what part of the stack) to different fields ("more.host", "more.ctx" and such) in kibana.
This is my beats input:
input {
beats {
port => 5044
}
}
filter {
if [type] == "node" {
json {
source => "message"
add_field => {
"who" => "%{name}"
"what" => "%{msg}"
"severity" => "%{level}"
"when" => "%{time}"
}
}
} else {
json {
source => "message"
}
}
date {
match => [ "when" , "dd.MM.yyyy HH:mm:ss", "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"]
}
}
And this is my output:
output {
elasticsearch {
hosts => ["localhost"]
sniffing => true
manage_template => false
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
}
stdout { codec => rubydebug }
}
Is there any way of making a field which will contain the entire "more" field without breaking it apart?
You should be able to use a ruby filter to take the hash and convert it back into a string.
filter {
ruby {
code => "event['more'] = event['more'].to_s"
}
}
You'd probably want to surround it with an if to make sure that the field exists first.
Related
I have a json log like this being streamed into ELK
{
"event": "Events Report",
"level": "info",
"logger": "XXXXX",
"method": "YYYYY",
"report_duration": {
"duration": "5 days, 12:43:16",
"end": "2021-12-13 03:43:16",
"start": "2021-12-07 15:00:00"
},
"request_type": "GET",
"rid": "xyz-123-yzfs",
"field_id": "arefer-e3-adfe93439",
"timestamp": "12/13/2021 03:43:53 AM",
"user": "8f444233ed4-91b8-4839-a57d-ande2534"
}
I would like to further split duration value i.e "5 days, 12:43:16" as some thing like "days": "5"
I have tried using below logstash filter and still its not working
filter {
if "report_duration" in [reports]{
mutate {
split => { "duration" => " " }
add_field => { "days" => "%{[duration][0]}" }
convert => {
"days" => "integer"
}
}
}
}
I think I have config that fits what you want:
# Since I wasn't sure of what you wanted, I changed the conditional here to check if the duration nested field is present
if [report_duration][duration]{
mutate {
# Since duration is nested under report_duration, it has to be accessed this way:
split => { "[report_duration][duration]" => " " }
# The split option replace the text field with an array, so it's still nested
add_field => { "days" => "%{[report_duration][duration][0]}" }
}
# the convert option is executed before the split option, so it has to be moved in its own plugin call
mutate {
convert => {
"days" => "integer"
}
}
}
Some references: accessing nested fields, mutate filter process order
We want to implement service request trace using http plugin of logstash in JSON Array format.
We are getting the following error when trying to parse the JSON array:
error:
:message=>"gsub mutation is only applicable for Strings, skipping", :field=>"message", :value=>nil, :level=>:debug, :file=>"logstash/filters/mutate.rb", :line=>"322", :method=>"gsub"}
:message=>"Exception in filterworker", "exception"=>#<LogStash::ConfigurationError: Only String and Array types are splittable. field:message is of type = NilClass>
My json array is :
{
"data": [
{
"appName": "DemoApp",
"appVersion": "1.1",
"deviceId": "1234567",
"deviceName": "moto e",
"deviceOSVersion": "5.1",
"packageName": "com.DemoApp",
"message": "testing null pointer exception",
"errorLog": "null pointer exception"
},
{
"appName": "DemoApp",
"appVersion": "1.1",
"deviceId": "1234567",
"deviceName": "moto e",
"deviceOSVersion": "5.1",
"packageName": "com.DemoApp",
"message": "testing illegal state exception",
"errorLog": "illegal state exception"
}
]
}
my logstash config is :
input {
http {
codec => "plain"
}
}
filter{
json {
source => "message"
}
mutate { gsub => [ "message", "},", "shr" ] }
split {
terminator => "shr"
field => "data"
}
}
}
output {
stdout { codec => "json" }
gelf{
host => localhost
facility => "%{type}"
level =>["%{SeverityLevel}", "INFO"]
codec => "json"
}
file{
path => "/chroot/result.log"
}
}
Any help would be appreciated.
Logstash has a default metadata field named message. So your json message field is overlapping that. Consider changing json field name message to another.
The other option maybe using target setting and referencing the target field like:
json { source => "message" target => "data"}
mutate { gsub => [ "[data][message]", "\}\,\r\n\r\n\{", "\}shr\{" ] }
I hope this helps.
Hi I am trying to parse a json file. I have tried troubleshooting with suggestions from stackoverflow (links at bottom)but none have worked for me. I am hoping someone has some insight on probably a silly mistake I am making.
I have tried using only the json codec, only the json filter, as well as both. For some reason I am still getting this _jsonparsefailure. What can I do to get this to work?
Thanks in advance!
My json file:
{
"log": {
"version": "1.2",
"creator": {
"name": "WebInspector",
"version": "537.36"
},
"pages": [
{
"startedDateTime": "2015-10-13T20:28:46.081Z",
"id": "page_1",
"title": "https://demo.com",
"pageTimings": {
"onContentLoad": 377.8560000064317,
"onLoad": 377.66200001351535
}
},
{
"startedDateTime": "2015-10-13T20:29:01.734Z",
"id": "page_2",
"title": "https://demo.com",
"pageTimings": {
"onContentLoad": 1444.0670000039972,
"onLoad": 2279.20100002666
}
},
{
"startedDateTime": "2015-10-13T20:29:04.014Z",
"id": "page_3",
"title": "https://demo.com",
"pageTimings": {
"onContentLoad": 1802.0240000041667,
"onLoad": 2242.4060000048485
}
},
{
"startedDateTime": "2015-10-13T20:29:09.224Z",
"id": "page_4",
"title": "https://demo.com",
"pageTimings": {
"onContentLoad": 274.82699998654425,
"onLoad": 1453.034000005573
}
}
]
}
}
My logstash conf:
input {
file {
type => "json"
path => "/Users/anonymous/Documents/demo.json"
start_position => beginning
}
}
filter{
json{
source => "message"
}
}
output {
elasticsearch { host => localhost protocol => "http" port => "9200" }
stdout { codec => rubydebug }
}
Output I am getting from logstash hopefully with clues:
Trouble parsing json {:source=>"message", :raw=>" \"startedDateTime\": \"2015-10-19T18:05:37.887Z\",", :exception=>#<TypeError: can't convert String into Hash>, :level=>:warn}
{
"message" => " {",
"#version" => "1",
"#timestamp" => "2015-10-26T20:05:53.096Z",
"host" => "15mbp-09796.local",
"path" => "/Users/anonymous/Documents/demo.json",
"type" => "json",
"tags" => [
[0] "_jsonparsefailure"
]
}
Decompose Logstash json message into fields
How to use logstash's json filter?
I test my JSON here JSONLint. Perhaps this will solve your problem. The error I am getting is that it is expecting string.
It seems that you have an unnecessary comma(',') at the end. Either remove it or add another JSON variable after that.
I've got a question regarding JSON in Logstash.
I have got a JSON Input that looks something like this:
{
"2": {
"name": "name2",
"state": "state2"
},
"1": {
"name": "name1",
"state": "state1"
},
"0": {
"name": "name0",
"state": "state0"
}
}
Now, let's say I want to add a field in the logstash config
json{
source => "message"
add_field => {
"NAME" => "%{ What to write here ?}"
"STATE" => "%{ What to write here ?}"
}
}
Is there a way to access the JSON Input such that I get a field Name with value name1, another field with name 2 and a third field with name 3. The first key in the JSON is changing, that means there can only be one or many more parts. So I don't want to hardcode it like
%{[0][name]}
Thanks for your help.
If you remove all new lines in your input you can simply use the json filter. You don't need any add_field action.
Working config without new lines:
filter {
json { source => message }
}
If you can't remove the new lines in your input you need to merge the lines with the multiline codec.
Working config with new lines:
input {
file {
path => ["/path/to/your/file"] # I suppose your input is a file.
start_position => "beginning"
sincedb_path => "/dev/null" # just for testing
codec => multiline {
pattern => "^}"
what => "previous"
negate => "true"
}
}
}
filter {
mutate { replace => { "message" => "%{message}}" } }
json { source => message }
}
I suppose that you use the file input. In case you don't, just change it.
Output (for both):
"2" => {
"name" => "name2",
"state" => "state2"
},
"1" => {
"name" => "name1",
"state" => "state1"
},
"0" => {
"name" => "name0",
"state" => "state0"
}
I am using the latest ELK (Elasticsearch 1.5.2 , Logstash 1.5.0, Kibana 4.0.2)
I have a question that
sample .json
{ "field1": "This is value1", "field2": "This is value2" }
longstash.conf
input {
stdin{ }
}
filter {
json {
source => "message"
add_field =>
{
"field1" => "%{field1}"
"field2" => "%{field2}"
}
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
host => "localhost"
index => "scan"
}
}
Output:
{
"message" => "{ \"field1\": \"This is value1\", \"field2\": \"This is value2\" }",
"#version" => "1",
"#timestamp" => "2015-05-07T06:02:56.088Z",
"host" => "myhost",
"field1" => [
[0] "This is value1",
[1] "This is value1"
],
"field2" => [
[0] "This is value2",
[1] "This is value2"
]
}
My question is 1) why the field result appear double in the result? 2) If there is nested array , how is it should reference in the logstash configure?
Thanks a lot!
..Petera
I think you have misunderstood what the json filter does. When you process a field through the json filter it will look for field names and corresponding values.
In your example, you have done that with this part:
filter {
json {
source => "message"
Then you have added a field called "field1" with the content of field "field1", since the field already exists you have just added the same information to the field that was already there, it has now become an array:
add_field =>
{
"field1" => "%{field1}"
"field2" => "%{field2}"
}
}
}
If you simplify your code to the following you should be fine:
filter {
json {
source => "message"
}
}
I suspect your question about arrays becomes moot at this point, as you probably don't need the nested array, and therefore, won't need to address it, but in case you do, I believe you can do this like so:
[field1][0]
[field1][1]