Use Logstash CSV filter doesn't work - csv

I was trying to use CSV filter on Logstash but it can upload values of my file.
I'm using Ubuntu Server 14.04, kibana 4, logstash 1.4.2 and elasticsearch 1.4.4.
Next I show my CSV file and filter I wrote. Am I doing something wrong?
CSV File:
Joao,21,555
Miguel,24,1000
Rodrigo,43,443
Maria,54,2343
Antonia,67,213
Logstash CSV filter:
#Este e filtro que le o ficheiro e permite alocar os dados num index do Elasticsearch
input
{
file
{
path => ["/opt/logstash/bin/testeFile_lite.csv"]
start_position => "beginning"
# sincedb_path => "NIL"
}
}
filter
{
csv
{
columns => ["nome", "idade", "salario"]
separator => ","
}
}
output
{
elasticsearch
{
action => "index"
host => "localhost"
index => "logstash-%{+YYYY.MM.dd}"
}
stdout
{
codec => rubydebug
}
}
When I execute filter, appears: using milestone 2 input plugin 'file'... and using milestone 2 input plugin 'csv'... and the OK message doesn't appear.
Someone can help me?

I solved the problem adding the field sincedb_path in the input file.
Here's the Logstash CSV filter:
input
{
file
{
path => "/opt/logstash/bin/testeFile_lite.csv"
type => "testeFile_lite"
start_position => "beginning"
sincedb_path => "/opt/logstash/bin/dbteste"
}
}
filter
{
csv
{
columns => ['nome', 'idade', 'salario']
separator => ","
}
}
output
{
elasticsearch
{
action => "index"
host => "localhost"
index => "xpto"
cluster => "SIC_UTAD"
}
stdout
{
codec => rubydebug
}
}

Related

CSV & Lgstash & Elasticsearch

I need help !
I am new in Elasticsearch ... I just imported data in CSV format from Kaggle and I integrated them into elasticsearch via logstash by specifying the configuration file, but on elasticsearch my data is no longer readable (it is in this format: u0000E \ u0000l )
Did I need to add any converts to the config file?
Thank you.
I tried: codec => plain { charset=> "UTF-8" } and ruby { code => 'event.set("decoded", Base64.decode64(event.get("message")))' } but but it didn't work
this is what I get: {
"host" => "HIND",
"id" => "\u0000G\u0000u\u0000e\u0000s\u0000t\u0000s\u0000 \u0000c\u0000a\u0000n\u0000 \u0000r\u0000e\u0000l\u0000a\u0000x\u0000 \u0000i\u0000n\u0000 \u0000t\u0000h\u0000e\u0000 \u0000g\u0000a\u0000r\u0000d\u0000e\u0000n\u0000 \u0000a\u0000t\u0000 \u0000t\u0000h\u0000e\u0000 \u0000p\u0000r\u0000o\u0000p\u0000e\u0000r\u0000t\u0000y\u0000.\u0000",
"message" => "\u0000G\u0000u\u0000e\u0000s\u0000t\u0000s\u0000 \u0000c\u0000a\u0000n\u0000 \u0000r\u0000e\u0000l\u0000a\u0000x\u0000 \u0000i\u0000n\u0000 \u0000t\u0000h\u0000e\u0000 \u0000g\u0000a\u0000r\u0000d\u0000e\u0000n\u0000 \u0000a\u0000t\u0000 \u0000t\u0000h\u0000e\u0000 \u0000p\u0000r\u0000o\u0000p\u0000e\u0000r\u0000t\u0000y\u0000.\u0000",
"decoded" => "\x1A\xE7\xAC\xB6\xC7\x1A\x9E\xB7\xA5k\x18\xA7\xB6\x17\xA0j\xB7^\x9D\xABm\x85\xEAk\xA2\x97\xAB\xB7",
"path" => "C:/elastic_stack/data/hotelsb_df_es.csv",
"#version" => "1",
"#timestamp" => 2021-08-04T18:04:46.170Z
}

How to migrate Mysql data to elasticsearch using logstash

I need a brief explanation of how can I convert MySQL data to Elastic Search using logstash.
can anyone explain the step by step process about this
This is a broad question, I don't know how much you familiar with MySQL and ES. Let's say you have a table user. you may just simply dump it as csv and load it at your ES will be good. but if you have a dynamic data, like the MySQL just like a pipeline, you need to write a Script to do those stuff. anyway you can check the below link to build your basic knowledge before you ask How.
How to dump mysql?
How to load data to ES
Also, since you probably want to know how to covert your CSV to json file, which is the best suite for ES to understand.
How to covert CSV to JSON
You can do it using the jdbc input plugin for logstash.
Here is a config example.
Let me provide you with a high level instruction set.
Install Logstash, and Elasticsearch.
In Logstash bin folder copy jar ojdbc7.jar.
For logstash, create a config file ex: config.yml
#
input {
# Get the data from database, configure fields to get data incrementally
jdbc {
jdbc_driver_library => "./ojdbc7.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
jdbc_connection_string => "jdbc:oracle:thin:#db:1521:instance"
jdbc_user => "user"
jdbc_password => "pwd"
id => "some_id"
jdbc_validate_connection => true
jdbc_validation_timeout => 1800
connection_retry_attempts => 10
connection_retry_attempts_wait_time => 10
#fetch the db logs using logid
statement => "select * from customer.table where logid > :sql_last_value order by logid asc"
#limit how many results are pre-fetched at a time from the cursor into the client’s cache before retrieving more results from the result-set
jdbc_fetch_size => 500
jdbc_default_timezone => "America/New_York"
use_column_value => true
tracking_column => "logid"
tracking_column_type => "numeric"
record_last_run => true
schedule => "*/2 * * * *"
type => "log.customer.table"
add_field => {"source" => "customer.table"}
add_field => {"tags" => "customer.table" }
add_field => {"logLevel" => "ERROR" }
last_run_metadata_path => "last_run_metadata_path_table.txt"
}
}
# Massage the data to store in index
filter {
if [type] == 'log.customer.table' {
#assign values from db column to custom fields of index
ruby{
code => "event.set( 'errorid', event.get('ssoerrorid') );
event.set( 'msg', event.get('errormessage') );
event.set( 'logTimeStamp', event.get('date_created'));
event.set( '#timestamp', event.get('date_created'));
"
}
#remove the db columns that were mapped to custom fields of index
mutate {
remove_field => ["ssoerrorid","errormessage","date_created" ]
}
}#end of [type] == 'log.customer.table'
} #end of filter
# Insert into index
output {
if [type] == 'log.customer.table' {
amazon_es {
hosts => ["vpc-xxx-es-yyyyyyyyyyyy.us-east-1.es.amazonaws.com"]
region => "us-east-1"
aws_access_key_id => '<access key>'
aws_secret_access_key => '<secret password>'
index => "production-logs-table-%{+YYYY.MM.dd}"
}
}
}
Go to bin, Run as
logstash -f config.yml

Need help in identifying TYPE of document ingested by Elasticsearch (through Logstash)

I used Logstash to ingest csv files from https://www.kaggle.com/wcukierski/the-simpsons-by-the-data and saved it to Elasticsearch. For starters, I ingested simpsons_characters.csv using the following conf:
input {
file {
path => "/Users/xyz/Downloads/the-simpsons-by-the-data/simpsons_characters.csv"
start_position => beginning
sincedb_path => "/dev/null"
}
}
filter {
csv {
columns => ["id", "name", "normalized_name", "gender"]
separator => ","
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => "localhost"
action => "index"
index => "simpsons"
}
}
However, when I query like so: http://localhost:9200/simpsons/name/Lou
where
simpsons = index
name = type (I think ... not sure)
I get the following response back:
{
"_index": "simpsons",
"_type": "name",
"_id": "Lou",
"found": false
}
So, the question is, why am I not getting the correct response. Further, when you do bulk ingestion through csv, what is the type of the document?
Thanks!
The default type in Logstash Elasticsearch output is logs. So, no matter how you define your IDs (either take it from the csv - document_id => "%{id}" or let ES define its own), you can get those documents as http://localhost:9200/simpsons/logs/THE_ID.
If you don't know the id and want to simply check if something is there: http://localhost:9200/simpsons/logs/_search?pretty.
If you want to see what is the mapping of your index, for example to find out the _type of the index: http://localhost:9200/simpsons/_mapping?pretty.
To change the default _type:
elasticsearch {
hosts => "localhost"
action => "index"
index => "simpsons"
document_type => "characters"
document_id => "%{id}"
}
Here you haven't specified id field in your logstash output. In this case elasticsearch would asign a random id to your documents and you are searching for a document with id=Lou.
Adding document_id => "%{id}" would solve your problem.
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => "localhost"
action => "index"
index => "simpsons"
document_id => "%{id}"
}
}

Logstash Produces No Output

I followed the tutorial found here to set up the ELK stack, and eventually I managed to get everything working. When I tried to modify the system to read in CSV files, however, it stopped working entirely. The conf file looks like this:
input {
file {
path => "/home/user/remaining/path/*.csv"
type => "transaction"
start_position => "beginning"
}
}
filter {
if [type] == "transaction" {
csv {
columns => ["#timestamp", "ip address", "domain", "user", "demo", "id", "activity", "detail"]
separator => ","
}
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["localhost:9200"]
action => "index"
index => "logstash-test"
workers => 1
}
}
I've tried following a number of tutorials and guides for setting this up, and as far as I can tell logstash is still connected to elasticsearch. I just don't see any output. My suspicion is that it isn't reading the files at all. Part of the problem is that I'm not sure how to test each component of logstash. Is there something I'm likely to have missed?
Edit:
csv files look generally like this:
2016-02-29T22:26:39.319700,22.111.11.11,place.domain.ca,bob,DEMO,95081299250aa8,TI_START,"{'field': 'data', 'field2': 'moredata', 'anotherfield': 'evenmoredata', 'continuedfield': 'habbo', 'size': '16'}"
2016-02-29T22:27:00.098426,24.111.11.11,otherplace.domain.ca,bob,DEMO,390s8clearlyfake,TI_END,"{'field': 'data', 'field2': 'moredata', 'anotherfield': 'evenmoredata', 'continuedfield': 'habbo', 'size': '16'}"
I've also noticed that when I go to localhost:9200/logstash-test/ I'm getting a 404. I'm not sure if this is because there's no data transmitting, or if it's a different connection issue.
As part of the discussion here: https://discuss.elastic.co/t/logstash-not-showing-any-output-solved/28636/16
My files were more than 24 hours old, which is the default expiry time for a file input. It was fixed by changing the following where ignore_older was 86400 as a default:
input {
file {
ignore_older => 864000
path => "/home/sean/cost-logs/transaction/*.csv"
type => "transaction"
start_position => "beginning"
}
}

Logstash Multiline filter

We have some files that are written out to our web servers whenever we have php errors. Each error has it's own file, but there are always multiple lines in each file. The files always start with the text "Excepton:". Is there a way to easily just say, "take the whole file as a log event?" See example below:
Exception: ABC_Exception_Domain -- Message: There is no valid performance dimension for the given nodeRootId.
Error Date and Time:
Date: September 25, 2014
Time: 10:38:15
Timestamp: 1411659495
PersonId: 3947680
ProcessId: 18055
Memory Usage: 18194784
Machine Id:...
and here is my configuration file that I am using to test output:
input {
file {
type => "stack_trace"
path => [ "/root/20[1-9][0-9]*" ]
start_position => "beginning"
}
}
filter {
multiline {
type => "stack_trace"
pattern => "^Exception.*$"
negate => true
what => "previous"
}
}
output {
stdout {
codec => rubydebug
}
}
I have also tried this to use the last line as the log delimiter:
input {
file {
type => "stack_trace"
path => [ "/root/20[1-9][0-9]*" ]
start_position => "beginning"
}
}
filter {
multiline {
type => "stack_trace"
pattern => "^#[0-9]{1,3} \{main\}.*$"
negate => true
what => "previous"
}
}
output {
stdout {
codec => rubydebug
}
}
The only way I ever get any results is if I update the files manually and add another Exception or main line.
Thanks in advance for any help, and let me know if I can provide anymore information.
Thanks,
Chris.