I am new in ELK. I need to visualize data from a PostgreSQL in Kibana. I ran into a little problem and need a help.
I use:
Elasticsearch 6.4.1
Kibana 6.4.1
Logstash 6.4.1
When I run next logstash.conf file it don't send me correct data to elasticsearch server. What I need to change in my configuration file?
logstash.conf:
input
{
jbdc_connection_string => "path_to_database"
jdbc_user => "postgres"
jdbc_password => "postgres"
jdbc_driver_library => "/path_to/postgresql-42.2.5.jar"
jdbc_driver_class => "org.postgresql.Driver"
statement => "SELECT * from documents"
}
output
{
elasticsearch
{
hosts => ["localhost:9200"]
index => "documents"
}
}
Only when in output I use next configuration I see correct data in terminal:
strout
{
codes => json_lines
}
Related
I have a MySQL DB hosted on AWS RDS. I am running ElasticSearch locally and using Logstash to retrieve data from MYSQL server that is on AWS to then push the data to my ElasticSearch DB.
The problem is that my logstash file isn't settup correctly I guess:
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://aws.ffffffffff.us-east-1.rds.amazonaws.com:3306/dbName?user=userName&password=pword"
jdbc_user => "user"
jdbc_password => "pword"
schedule => "* * * * *"
jdbc_validate_connection => true
jdbc_driver_library => "C:\Program Files (x86)\MySQL\Connector J 8.0\mysql-connector-java-8.0.19.jar"
jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
statement => "SELECT * from data-5"
type => "data-5"
tags => ["data-5"]
}
jdbc {
jdbc_connection_string => "jdbc:mysql://aws.ffffffffff.us-east-1.rds.amazonaws.com:3306/dbName?user=userName&password=pword"
jdbc_user => "user"
jdbc_password => "pword"
schedule => "* * * * *"
jdbc_validate_connection => true
jdbc_driver_library => "C:\Program Files (x86)\MySQL\Connector J 8.0\mysql-connector-java-8.0.19.jar"
jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
statement => "SELECT * from data-4"
type => "data-4"
tags => ["data-4"]
}
jdbc {
jdbc_connection_string => "jdbc:mysql://aws.ffffffffff.us-east-1.rds.amazonaws.com:3306/dbName?user=userName&password=pword"
jdbc_user => "user"
jdbc_password => "pword"
schedule => "* * * * *"
jdbc_validate_connection => true
jdbc_driver_library => "C:\Program Files (x86)\MySQL\Connector J 8.0\mysql-connector-java-8.0.19.jar"
jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
statement => "SELECT * from data-3"
type => "data-3"
tags => ["data-3"]
}
}
output {
stdout { codec => json_lines }
if "data-5" in [tags] {
elasticsearch {
hosts => ["http://127.0.0.1:9200/"]
index => "data-5"
document_type => "data-%{+YYYY.MM.dd}"
}
}
if "data-4" in [tags] {
elasticsearch {
hosts => ["http://127.0.0.1:9200/"]
index => "data-4"
document_type => "data-%{+YYYY.MM.dd}"
}
}
if "data-3" in [tags] {
elasticsearch {
hosts => ["http://127.0.0.1:9200/"]
index => "data-3"
document_type => "data-%{+YYYY.MM.dd}"
}
}
}
This is the fun part of programming right?
Anyway, locally I am on windows as you may be able to tell from the file path to the jdbc driver library. My jdbc connection to the AWS RDS is copied and pasted from AWS Console, so no typos were involved.
I am told that I only need to append jdbc:mysql:// to the url. But is there anything I'm missing to do in the AWS console? Do I need to modify my RDS instance?
The error by the way is:
Unable to connect to database. Tried 1 times
{:error_message=>"Java::ComMysqlCjJdbcExceptions::CommunicationsException:
Communications link failure\n\n
The last packet sent successfully to the server was 0 milliseconds ago.
The driver has not received any packets from the server."
I had a similar issue, and SSL was not enabled for communication. So, AWS RDS was not allowing to connect.
I specified an additional query param useSSL=false to JDBC connection string, and problem was solved.
So, in your case, the jdbc_connection_string would be as follows:
"jdbc:mysql://aws.ffffffffff.us-east-1.rds.amazonaws.com:3306/dbName?useSSL=false&user=userName&password=pword"
I'm new to the elastic search concept to make connection with mySQL.
I followed multiple tutorials to install but I'm getting these errors:
Unknown setting '"index"'and '"host'" for elasticsearch
The output of
sudo -Hu root /usr/share/logstash/bin/logstash --path.settings /etc/logstash/
returns:
> Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
> [2019-04-20T17:48:47,293][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.0.0"}
> [2019-04-20T17:48:53,873][ERROR][logstash.outputs.elasticsearch] Unknown setting '"document_type"' for elasticsearch
> [2019-04-20T17:48:53,878][ERROR][logstash.outputs.elasticsearch] Unknown setting '"hosts"' for elasticsearch > [2019-04-20T17:48:53,878][ERROR][logstash.outputs.elasticsearch] Unknown setting '"index"' for elasticsearch
> [2019-04-20T17:48:53,891][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Something is wrong with your configuration.", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:86:inconfig_init'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:60:in initialize'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:232:ininitialize'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:48:in initialize'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:30:ininitialize'", "org/logstash/plugins/PluginFactoryExt.java:239:in plugin'", "org/logstash/plugins/PluginFactoryExt.java:137:inbuildOutput'", "org/logstash/execution/JavaBasePipelineExt.java:50:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:23:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:inblock in converge_state'"]}
> [2019-04-20T17:48:54,190][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
> [2019-04-20T17:48:59,066][INFO ][logstash.runner ] Logstash shut down.
Here is the content of the logstash.conf file:
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://localhost:3306/archief"
# The user we wish to execute our statement as
jdbc_user => "root"
jdbc_password => "pswxxx"
# The path to our downloaded jdbc driver
jdbc_driver_library => "/usr/share/java/mysql-connector-java-8.0.15.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
# our query
statement => "SELECT * FROM archief"
}
}
output {
stdout { codec => json_lines }
elasticsearch {
"hosts" => ["localhost:9200"]
"index" => "archief"
}
}
There is no double quotes in the options name.
output {
stdout { codec => json_lines }
elasticsearch {
hosts => ["localhost:9200"]
index => "archief"
}
}
I had the same issue, and solved it by changing my logstash version from 7.4.2 to 6.3.2.
Logstash 6.3.2 link
I am trying to make connection with MySQL using logstash and write into elastic search below in my code in conf file
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://192.168.2.24:3306/test"
# The user we wish to execute our statement as
jdbc_user => "uname"
jdbc_password => "pass"
# The path to our downloaded jdbc driver
jdbc_driver_library => "/usr/local/Cellar/logstash/6.2.4/mysql-connector-java-8.0.11.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
# our query
statement => "SELECT * FROM report_table"
}
}
output {
elasticsearch {
action => "index"
hosts => "localhost:9200"
index => "mysqlsample"
document_type => "record"
}
}
on running the above getting the below error :
Error: com.mysql.jdbc.Driver not loaded. Are you sure you've included
the correct jdbc driver in :jdbc_driver_library? Exception:
LogStash::ConfigurationError Stack:
/usr/local/Cellar/logstash/6.2.4/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/plugin_mixins/jdbc.rb:162:in
open_jdbc_connection'
/usr/local/Cellar/logstash/6.2.4/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/plugin_mixins/jdbc.rb:220:in
execute_statement'
/usr/local/Cellar/logstash/6.2.4/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/inputs/jdbc.rb:264:in
execute_query'
/usr/local/Cellar/logstash/6.2.4/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/inputs/jdbc.rb:250:in
run'
/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:514:in
inputworker'
/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:507:in
block in start_input'
Sounds like it's an issue with jdbc_driver_library => "/usr/local/Cellar/logstash/6.2.4/mysql-connector-java-8.0.11.jar".
Are you sure it's a valid path? Is that the correct connector? Maybe try to use the one that the documentation mentions: mysql-connector-java-5.1.36-bin.jar
I am new to ELK stack. I am working with data migration from MySQL TO elasticsearch. I am following this tutorial:
https://qbox.io/blog/migrating-mysql-data-into-elasticsearch-using-logstash
and I have installed and configured MySQL and ElasticSearch. I could not configure Logstash.
I dont know where to find logstash.conf, so i created a file named logstash.conf in my conf.d file of logstash folder. I wrote the following in logstash.conf:
input {
jdbc {
jdbc_driver_library => "usr/share/java/mysql-connector-java.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost:3306/books"
jdbc_user => "root"
jdbc_password => "root"
statement => "SELECT * FROM authors"
}
}
output {
stdout { codec => json_lines }
elasticsearch {
"hosts" => "localhost:9200"
"index" => "my-authors"
"document_type" => "data"
}
}
But when I run my command bin/logstash -f logstash.conf by going into /etc/logstash/conf.d folder from ubuntu terminal, It gives an error stating that bin/logstash does not exist.
Please help me with the issue.
I using ejabberd server for chat communication. I'd like be able dynamicly search my archive messages. Now I'm using elasticsearch and logstash, but it working only on mysql db. It's my logstash config
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://localhost:3306/ejabberd"
jdbc_user => "ejabber"
jdbc_password => "password"
jdbc_driver_library => "mysql-connector-java-5.1.39-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
statement => "SELECT * FROM ejabberd.archive"
}
}
output {
# stdout { codec => json_lines }
elasticsearch {
index => "muc_room"
hosts => ["localhost:9200"]
}
}
I need use mnesia DB, its default base for ejabber. How can connect mnesia DB with logstash, or it is possible use another way to include search engione to mnesia DB. Thank you
I would send the data directly to elasticsearch from ejabberd. That way, you don't need to have two separate things that need to be updated if you change storage engines. There's an Erlang package to talk to Elasticsearch. The documentation on it isn't great, but it's a pretty simple interface anyway.