Logstash Multiline filter - multiline

We have some files that are written out to our web servers whenever we have php errors. Each error has it's own file, but there are always multiple lines in each file. The files always start with the text "Excepton:". Is there a way to easily just say, "take the whole file as a log event?" See example below:
Exception: ABC_Exception_Domain -- Message: There is no valid performance dimension for the given nodeRootId.
Error Date and Time:
Date: September 25, 2014
Time: 10:38:15
Timestamp: 1411659495
PersonId: 3947680
ProcessId: 18055
Memory Usage: 18194784
Machine Id:...
and here is my configuration file that I am using to test output:
input {
file {
type => "stack_trace"
path => [ "/root/20[1-9][0-9]*" ]
start_position => "beginning"
}
}
filter {
multiline {
type => "stack_trace"
pattern => "^Exception.*$"
negate => true
what => "previous"
}
}
output {
stdout {
codec => rubydebug
}
}
I have also tried this to use the last line as the log delimiter:
input {
file {
type => "stack_trace"
path => [ "/root/20[1-9][0-9]*" ]
start_position => "beginning"
}
}
filter {
multiline {
type => "stack_trace"
pattern => "^#[0-9]{1,3} \{main\}.*$"
negate => true
what => "previous"
}
}
output {
stdout {
codec => rubydebug
}
}
The only way I ever get any results is if I update the files manually and add another Exception or main line.
Thanks in advance for any help, and let me know if I can provide anymore information.
Thanks,
Chris.

Related

Need help in identifying TYPE of document ingested by Elasticsearch (through Logstash)

I used Logstash to ingest csv files from https://www.kaggle.com/wcukierski/the-simpsons-by-the-data and saved it to Elasticsearch. For starters, I ingested simpsons_characters.csv using the following conf:
input {
file {
path => "/Users/xyz/Downloads/the-simpsons-by-the-data/simpsons_characters.csv"
start_position => beginning
sincedb_path => "/dev/null"
}
}
filter {
csv {
columns => ["id", "name", "normalized_name", "gender"]
separator => ","
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => "localhost"
action => "index"
index => "simpsons"
}
}
However, when I query like so: http://localhost:9200/simpsons/name/Lou
where
simpsons = index
name = type (I think ... not sure)
I get the following response back:
{
"_index": "simpsons",
"_type": "name",
"_id": "Lou",
"found": false
}
So, the question is, why am I not getting the correct response. Further, when you do bulk ingestion through csv, what is the type of the document?
Thanks!
The default type in Logstash Elasticsearch output is logs. So, no matter how you define your IDs (either take it from the csv - document_id => "%{id}" or let ES define its own), you can get those documents as http://localhost:9200/simpsons/logs/THE_ID.
If you don't know the id and want to simply check if something is there: http://localhost:9200/simpsons/logs/_search?pretty.
If you want to see what is the mapping of your index, for example to find out the _type of the index: http://localhost:9200/simpsons/_mapping?pretty.
To change the default _type:
elasticsearch {
hosts => "localhost"
action => "index"
index => "simpsons"
document_type => "characters"
document_id => "%{id}"
}
Here you haven't specified id field in your logstash output. In this case elasticsearch would asign a random id to your documents and you are searching for a document with id=Lou.
Adding document_id => "%{id}" would solve your problem.
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => "localhost"
action => "index"
index => "simpsons"
document_id => "%{id}"
}
}

Logstash Produces No Output

I followed the tutorial found here to set up the ELK stack, and eventually I managed to get everything working. When I tried to modify the system to read in CSV files, however, it stopped working entirely. The conf file looks like this:
input {
file {
path => "/home/user/remaining/path/*.csv"
type => "transaction"
start_position => "beginning"
}
}
filter {
if [type] == "transaction" {
csv {
columns => ["#timestamp", "ip address", "domain", "user", "demo", "id", "activity", "detail"]
separator => ","
}
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["localhost:9200"]
action => "index"
index => "logstash-test"
workers => 1
}
}
I've tried following a number of tutorials and guides for setting this up, and as far as I can tell logstash is still connected to elasticsearch. I just don't see any output. My suspicion is that it isn't reading the files at all. Part of the problem is that I'm not sure how to test each component of logstash. Is there something I'm likely to have missed?
Edit:
csv files look generally like this:
2016-02-29T22:26:39.319700,22.111.11.11,place.domain.ca,bob,DEMO,95081299250aa8,TI_START,"{'field': 'data', 'field2': 'moredata', 'anotherfield': 'evenmoredata', 'continuedfield': 'habbo', 'size': '16'}"
2016-02-29T22:27:00.098426,24.111.11.11,otherplace.domain.ca,bob,DEMO,390s8clearlyfake,TI_END,"{'field': 'data', 'field2': 'moredata', 'anotherfield': 'evenmoredata', 'continuedfield': 'habbo', 'size': '16'}"
I've also noticed that when I go to localhost:9200/logstash-test/ I'm getting a 404. I'm not sure if this is because there's no data transmitting, or if it's a different connection issue.
As part of the discussion here: https://discuss.elastic.co/t/logstash-not-showing-any-output-solved/28636/16
My files were more than 24 hours old, which is the default expiry time for a file input. It was fixed by changing the following where ignore_older was 86400 as a default:
input {
file {
ignore_older => 864000
path => "/home/sean/cost-logs/transaction/*.csv"
type => "transaction"
start_position => "beginning"
}
}

Grok match json field and value

I'm using koajs with bunyan to save error logs to my server then I use filebeat to have them shipped to my logstash application.
My error logs are being forwarded correctly however I would now like to create a filter which will add a tag to specific logs.
{"name":"myapp","hostname":"sensu-node-dev","pid":227,"level":50,"err":{"message":"Cannot find module 'lol'","name":"Error","stack":"Error: Cannot find module 'lol'\n at Function.Module._resolveFilename (module.js:339:15)\n at Function.Module._load (module.js:290:25)\n at Module.require (module.js:367:17)\n at require (internal/module.js:16:19)\n at Object.<anonymous> (/srv/www/dev.site/app.js:27:6)\n at next (native)\n at Object.<anonymous> (/srv/www/dev.site/node_modules/koa-compose/index.js:29:5)\n at next (native)\n at onFulfilled (/srv/www/dev.site/node_modules/co/index.js:65:19)\n at /srv/www/dev.site/node_modules/co/index.js:54:5","code":"MODULE_NOT_FOUND"},"msg":"Cannot find module 'lol'","time":"2016-02-24T22:04:26.492Z","v":0}
Now the interesting part in that specific log is "err":{...} and the "name":"Error" bits. For simplicity reasons I would just like to create a filter which detects "name":"Error" in the log (if it exists) and then apply a tag add_tag => ["error"] to the log.
Here is my /etc/logstash/conf.d/logstash.conf file:
input {
beats {
port => 5044
type => "logs"
}
}
filter {
grok {
type => "log"
pattern => "???" // <--- have no idea what to do here
add_tag => ["error"]
}
}
output {
elasticsearch {
hosts => "localhost:9200"
sniffing => true
manage_template => false
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
}
http {
http_method => "post"
url => "<MY_URL>"
format => "message"
message => "{"text":"dis is workinz, you has error"}"
tags => ["error"]
}
}
I tried the following:
pattern => ""name":"Error""
But got the following error:
Error: Expected one of #, {, } at line 9, column 31 (byte 107) after filter {
grok {
match => { "message" => ""
You may be interested in the '--configtest' flag which you can
use to validate logstash's configuration before you choose
to restart a running system.
There is no simple example of this specific type of matching anywhere.
Bonus: Also how does one escape in logstash, I couldn't find anything on the subject?
If you only want to see if a string exists in your message, try this:
if [message] =~ /"name":"Error"/ {
mutate {
add_tag { ... }
}
}
If you really want to grok the input into fields, check out the json codec or filter instead.

Use Logstash CSV filter doesn't work

I was trying to use CSV filter on Logstash but it can upload values of my file.
I'm using Ubuntu Server 14.04, kibana 4, logstash 1.4.2 and elasticsearch 1.4.4.
Next I show my CSV file and filter I wrote. Am I doing something wrong?
CSV File:
Joao,21,555
Miguel,24,1000
Rodrigo,43,443
Maria,54,2343
Antonia,67,213
Logstash CSV filter:
#Este e filtro que le o ficheiro e permite alocar os dados num index do Elasticsearch
input
{
file
{
path => ["/opt/logstash/bin/testeFile_lite.csv"]
start_position => "beginning"
# sincedb_path => "NIL"
}
}
filter
{
csv
{
columns => ["nome", "idade", "salario"]
separator => ","
}
}
output
{
elasticsearch
{
action => "index"
host => "localhost"
index => "logstash-%{+YYYY.MM.dd}"
}
stdout
{
codec => rubydebug
}
}
When I execute filter, appears: using milestone 2 input plugin 'file'... and using milestone 2 input plugin 'csv'... and the OK message doesn't appear.
Someone can help me?
I solved the problem adding the field sincedb_path in the input file.
Here's the Logstash CSV filter:
input
{
file
{
path => "/opt/logstash/bin/testeFile_lite.csv"
type => "testeFile_lite"
start_position => "beginning"
sincedb_path => "/opt/logstash/bin/dbteste"
}
}
filter
{
csv
{
columns => ['nome', 'idade', 'salario']
separator => ","
}
}
output
{
elasticsearch
{
action => "index"
host => "localhost"
index => "xpto"
cluster => "SIC_UTAD"
}
stdout
{
codec => rubydebug
}
}

Multiline filter with logstash not working as expected

I was playing around with logstash and wanted to try the multiline filter. I'm trying to parse a single file which can has multiline content in it. I'm using the multiline filter but it does not work the way it should. I have the following content in the file.
2014-10-11 10:10:10 xxxx yyyy
2013-09-12 11:11:11 aaaa bbbb
2012-01-01 10:10:10 cccc dddd
2011-10-12 01:01:01 mmmm Nan
Grok-pattern used
CUSTOMTIME %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}
I'm using the following config file in logstash
input {
file{
path => "/Users/akshayanilkapoor/Kumo/logs/akshay"
codec => multiline {
pattern => "^.*"
what => "previous"
negate => true
}
}
stdin{
codec => multiline {
pattern => "^%{CUSTOMTIME}"
what => "previous"
negate => true
}
}
}
filter {
grok {
patterns_dir => "./patterns"
match => ["message", "%{CUSTOMTIME:date1} %{GREEDYDATA:lumber-type} %{GREEDYDATA:lumber-desc}"]
}
}
output {
stdout {codec => rubydebug}
}
When i copy paste the data shown above from the stdin, it works as expected i.e. it displays the output with the message containing all the the events in the file. When i pass the same with the file it outputs a different message for every log entry i.e. I get 4 different log events (which is not what i desire)
Note: I have tried using the following options along with the multiline,
(?m) multi-line mode in grok and that too does not help in any way.
Also used mutate filter to replace "\n" with " " and parse it with grok.
I think there is a very silly mistake that I have been making or i'm understanding the filter incorrectly to be implemented. Any help would be much appreciated!
I misunderstood the negate option of the multiline codec incorrectly as i expected :p This is the config file I used for the the config to work incase it helps someone.
input {
file{
path => "/Users/akshayanilkapoor/Kumo/logs/akshay"
codec => multiline {
pattern => "^[0-9]"
what => "previous"
#negate => true
}
}
stdin{
codec => multiline {
pattern => "^[a-z]"
what => "previous"
negate => "true"
}
}
}
filter {
grok {
match => ["message", "(?m)%{RSMROLLBACKTIME:date1} %{GREEDYDATA:lumber-type} %{GREEDYDATA:lumber-desc}"]
}
}
output {
stdout {codec => rubydebug}
}