Logstash - create terms from relational database group by - mysql

Question

I have a table in MySQL that I want to import into Elasticsearch
As an example the data looks like this
team buyer
==== ======
one Q76876
one Q66567
one T99898
two Q45456
two S77676
I want to import this into elasticsearch using logstash and create an index that looks like this
{
"id": "one",
"team": one,
"buyers": ["Q76876", "Q66567", "T99898"]
},
{
"id": "two",
"team": "two",
"buyers": ["Q45456", "S77676"]
}
How would I write my .conf script to achieve this?

Answer 1

Logstash put the events in the index as they come to it unless you apply some filter. You case looks pretty straight forward. If you format your sql query to return the data in the format your require, then you dont need to apply any filter and can just hook up the database and sql query to run in logstash config and the output as elastic search index.
For example:
MySql query would look something like: (i'm not good at mysql, below is just to give an idea - please verify it works)
SELECT team as id,
team,
GROUP_CONCAT(DISTINCT buyer SEPARATOR ', ') as buyers
FROM tablename GROUP BY team
This would return something like:
+-----+------+------------------------+
| id | team | buyers |
+-----+------+------------------------+
| one | one | Q76876, Q66567, T99898 |
| two | two | Q45456, S77676 |
+-----+------+------------------------+
And the logstash config would simply look like :
input {
jdbc {
jdbc_driver_library => "${DATABASE_DRIVER_PATH}"
jdbc_driver_class => "${DATABASE_DRIVER_PATH}"
jdbc_connection_string => "{CONNECTIONSTRING}"
jdbc_user => "${DATABASE_USERNAME}"
jdbc_password => "${DATABASE_PASSWORD}"
statement_filepath => "${LOGSTASH_SQL_FILEPATH}" #this will be the sql written above
}
}
filter {
}
output {
elasticsearch {
action => "index"
hosts => ["${ELASTICSEARCH_HOST}"]
user => "${ELASTICSEARCH_USER}"
password => "${ELASTICSEARCH_PASSWORD}"
index => "${INDEX_NAME}"
document_type => "doc"
document_id => "%{id}"
}
stdout { codec => rubydebug }
stdout { codec => dots }
}