I'm using logstash to push log data from csv files into elastic search. This is my configuration file to push the csv files into elasticsearch.
input
{
file {
path => "C:\Projects\Tools\ELK\data\CI_Report\*.csv"
start_position =>"beginning"
}
}
filter {
csv {
columns =>["LOG_LEVEL", "TOOL_ID", "ID_ANALYSIS", "ARCH_COMP", "Files",
"Component Analysed","Date","Tool Version"]
separator =>","
}
}
output {
elasticsearch {
hosts => ["localhost"]
index => "cilog_all"
}
stdout {
codec => json
}
}
This is my command to start the logstash, logstash -f logcsv.conf
Note : It works If I use the actual file path to a single csv file. It never works if I use *.csv to load log data from all csv files located in the specified folder.
Is Glob pattern supported by logstash input plugin ? as mentioned here
https://www.elastic.co/guide/en/logstash/current/glob-support.html
Everything looks ok save for your path.
Being windows you need to change the path to this as \ is an escape character
path => "C:/Projects/Tools/ELK/data/CI_Report/*.csv"
Related
I have test results that are being stored in json files. I then have logstash locate the file and attempt to send all of the lines to elasticsearch. Only about half of the lines are being sent and can't figure out why certain lines are being left out. For example, there will be 34 lines, but only 14 are sent.
input {
file {
path => "/data/*.json"
start_position => "beginning"
}
}
# ----------------------------------------------------------------------
filter {
# Parse fields out of JSON message, then remove the raw JSON.
json {
source => "message"
}
}
# ----------------------------------------------------------------------
output {
elasticsearch {
hosts => ["host:9200", "localhost:9200"]
index => "ct-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
I'm not sure if there is something within the json itself that causes logstash to just skip over it, or if there's something wrong with my logstash.conf file that I posted above.
Logstash computes files from different types to send it to elasticsearch in Json format. In your case, a Filebeat agent with an elasticsearch output would be enough to send a json file to ES and to index it.
It would look like this using Filebeat 6.x :
#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
# Paths to the logs
paths:
- "/your/path/to/your/logs/file.json"
# tags to identify the logs source, .gz files excluded from the prospector
tags: ["beats","yourtag"]
exclude_files: ['\.gz$']
#================================ Outputs =====================================
#----------------------------- Elasticsearch output --------------------------------
output.elasticsearch:
# The ES host & index name
hosts: ["yourEShost:9200"]
index: "ct-%{+YYYY.MM.dd}"
I am trying to configure Logstash which should read all json files which are located in specific directory.
JSON files update every time by python script and I need to read all files from scratch every time (not just checking the last lines).
I have tested two variants of configs:
1 config:
input {
file {
path => "/tmp/*.json"
start_position => "beginning"
}
}
output{
stdout{codec => rubydebug}
}
2 config:
input {
file {
path => "/tmp/*.json"
start_position => "beginning"
sincedb_path => "/dev/null"
ignore_older => 0
}
}
output{
stdout{codec => rubydebug}
}
Result in both cases:
Logstash can`t detect the changes when python updates the files (File timestamp is changing every time)
Even if I change something inside of the file logstash detects it only first time and print full file but after that it can`t detect changes anymore.
Final result:
At the end Logstash just need to get all files (new files should be detected automatically) in specific directory, read the content and push it somewhere. Can you please help to do it?
I have a CSV file, and I want to import it in my Elastic Search.
I am on Windows 10 and I also have a Kibana be able to browse data once imported. I use Logstash to try to make this import.
All of my services (Kibana, ES and Logstash) are running on my localhost.
I tried with a the following Logstash configuration file (my csv file is in the correct path):
input {
file {
path => ["E:\Programmes\Logstash\logstash-2.2.0\data\*.csv"]
start_position => "beginning"
}
}
filter {
csv {
columns => ["billing_entity","invoice","company","username","reference","line_number","recipient_number","zone","network","date","time","country","duration","cost","currency","call_origin","billing_type"]
separator => ";"
}
#grok {
# match => { "call" => "%{WORD:billing_entity} %{WORD:invoice} %{WORD:company} %{WORD:username} %{WORD:reference} %{NUMBER:line_number} %{NUMBER:recipient_number} %{WORD:zone} %{WORD:network} %{DATE:date} %{TIME:time} %{WORD:country} %{WORD:duration} %{NUMBER:cost} %{WORD:currency} %{WORD:call_origin} %{WORD:billing_type}" }
#}
}
output {
elasticsearch {
action => "index"
index => "call_samples"
#index => "call-%{+YYYY.MM.dd}"
hosts => "localhost"
workers => 1
}
}
As you can see I tried to use 'csv' or 'grok' filter.
Then I launched in verbose mode logstash with this configuration file :
logstash.bat -f ..\conf\logstash.conf -v > logfile.txt
EDIT : after each try, I delete the generated sincedb files to simulate changes. But anyway I noticed they are empty
But in the logs I see nothing relevant :
message=>"Using mapping template from"
message=>"Attempting to install template"
message=>"New Elasticsearch output"
message=>"Registering file input"
message=>"No sincedb_path set,generating o....
message=>"Using mapping template from ...
message=>"Attempting to install template"
message=>"New Elasticsearch output"
message=>"Starting pipeline"
message=>"Pipeline started"
Quite alike my file is ignored .... I also tried several indexes , etc ... it will never import data.
To check if data is present I make query on localhost:9200 or I browse Kibana search bar "Index name or pattern" with the index "call_samples".
Can anyone help me on this please ? I'm stuck at this point ... Thanks
EDIT 2 :
Ok I'm dumb on this one, I just wanted to redirect the logs streams to a file when I was launching logstash :
logstash.bat -f ..\conf\logstash.conf -v > logfile.txt
But it was probably breaking the input file from being imported. So I just removed the part where I redirect on file :
logstash.bat -f ..\conf\logstash.conf -v
Now, my index is correctly created, but no data is being imported ...
It was an encoding issue, and even in verbose mode, it never told me it was failing or something ... not even a little clue.
So I tested with a new test file encoded in UTF-8 and it worked well ...
My config file is shown below:
input {
file {
codec => "json"
path => "/home/data/*"
sincedb_path => "/dev/null"
}
}
output {
elasticsearch{
protocol => "http"
host => "localhost"
index => "data"
}
}
When I download a .json file to the data directory, logstash will not receive the data or output to elasticsearch unless I first open the file and save it in gedit.
Running logstash with the -vvv flag shows no errors all I get when a file is put in that directory is
_discover_file: /home/data/*: new: /home/data/prediction1.json (exclude is []) {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"126"}
_open_file: /home/data/prediction1.json: opening {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"98"}
/home/data/prediction1.json: staying at position 0, no sincedb {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"143"}
writing sincedb (delta since last write = 377) {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"177"}
/home/data/prediction1.json: file grew, old size 0, new size 3952 {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"81"}
This led me to believe that there may be a character encoding problem as the json file is created using json.dumps in python before reaching elasticsearch. I tried converting the data to yaml:
outfile.write(yaml.dump(json.dumps(data)))
which logstash accepts as plain text as it is now in the form:
'data:{ properties:
{ location:
{ here }}}'
after using sed and tr commands to convert this BACK to valid json logstash no longer accepts it unless saved through gedit!!!
So, I have a web platform that prints a JSON file per request containing some log data about that request. I can configure several rules about when should it log stuff, only at certain levels, etc...
Now, I've been toying with the Logstash + Elasticsearch + Kibana3 stack, and I'd love to find a way to see those logs in Kibana. My question is, is there a way to make Logstash import these kind of files, or would I have to write a custom input plugin for it? I've searched around and for what I've seen, plugins are written in Ruby, a language I don't have experience with.
Logstash is a very good tool for processing dynamic files.
Here is the way to import your json file into elasticsearch using logstash:
configuration file:
input
{
file
{
path => ["/path/to/json/file"]
start_position => "beginning"
sincedb_path => "/dev/null"
exclude => "*.gz"
}
}
filter
{
mutate
{
replace => [ "message", "%{message}" ]
gsub => [ 'message','\n','']
}
if [message] =~ /^{.*}$/
{
json { source => message }
}
}
output
{
elasticsearch {
protocol => "http"
codec => json
host => "localhost"
index => "json"
embedded => true
}
stdout { codec => rubydebug }
}
example of json file:
{"foo":"bar", "bar": "foo"}
{"hello":"world", "goodnight": "moon"}
Note the json need to be in one line. if you want to parse a multiline json file, replace relevant fields in your configuration file:
input
{
file
{
codec => multiline
{
pattern => '^\{'
negate => true
what => previous
}
path => ["/opt/mount/ELK/json/*.json"]
start_position => "beginning"
sincedb_path => "/dev/null"
exclude => "*.gz"
}
}
filter
{
mutate
{
replace => [ "message", "%{message}}" ]
gsub => [ 'message','\n','']
}
if [message] =~ /^{.*}$/
{
json { source => message }
}
}
Logstash is just a tool for converting various kinds of syslog files into JSON and loading them into elasticsearch (or graphite, or... ).
Since your files are already in JSON, you don't need logstash. You can upload them directly into elasticsearch using curl.
See Import/Index a JSON file into Elasticsearch
However, in order to work well with Kibana, your JSON files need to be at a minimum.
Flat - Kibana does not grok nested JSON structs. You need a simple hash of key/value pairs.
Have a identifiable timestamp.
What I would suggest is looking the JSON files logstash outputs and seeing if you can massage your JSON files to match that structure. You can do this in any language you
like that supports JSON. The program jq is very handy for filtering json from one format to another.
Logstash format - https://gist.github.com/jordansissel/2996677
jq - http://stedolan.github.io/jq/
Logstash can import different formats and sources as it provides a lot of plugins. There are also other log collector and forwarder tools that can send logs to logstash such as nxlog, rsyslog, syslog-ng, flume, kafka, fluentd, etc. From what I've heard most people use nxlog on windows (though it works on linux equally well) in combination with the ELK stack because of its low resource footprint. (Disclaimer: I'm affiliated with the project)