We plan to migrate from Jboss 5 to Wildfly 8.2. On the database side, we have a three-node Galera cluster with MariaDB 10.
On Jboss 5, we had the following setting in the ds.xml file:
...
<connection-url>jdbc:mysql:loadbalance://ip-node1,ip-node2,ip-node3/DBname</connection-url>
...
Everything worked well on Jboss 5. But I can't achieve the same on Wildfly 8.2. From the management console I was able to add a non-clustered datasource without problems and it works.
Example URL: jdbc:mysql://ip-node1/DBname
But when I try to add the clustered URL as above, I get the following error:
Unexpected HTTP response: 500
Request
{
"address" => [
("subsystem" => "datasources"),
("xa-data-source" => "dsName")
],
"operation" => "test-connection-in-pool"
}
Response
Internal Server Error
{
"outcome" => "failed",
"failure-description" => "JBAS010440: failed to invoke operation: JBAS010447: Connection is not valid",
"rolled-back" => true
}
How could I connect Wildfly to a clustered datasource? I know that it is possible to plug an external load balancer like HAProxy but I would prefer to keep the architecture as simple as possible.
You have to mention the datasource in standalone.xml file inside wildfly-8.2.0.Final_1\standalone\configuration\ as shown below.
<datasource jndi-name="java:/jdbc/DB1" pool-name="PostgresDS" enabled="true" use-java-context="true">
<connection-url>jdbc:postgresql://localhost:5432/DB1</connection-url>
<driver>postgres</driver>
<security>
<user-name>DB1</user-name>
<password>DB1</password>
</security>
</datasource>
<datasource jndi-name="java:/jdbc/DB2" pool-name="PostgresDS1" enabled="true" use-java-context="true">
<connection-url>jdbc:postgresql://localhost:5432/DB2</connection-url>
<driver>postgres</driver>
<security>
<user-name>DB2</user-name>
<password>DB2</password>
</security>
</datasource>
Related
For some reason Logstash with the Elastic Stack X-Pack is breaking. I believe it's to do with the recent renaming of the mySQL connector, which hasn't been updated in the various config files. However, I can't find where the error file is originating in this error log. Additionally, if anyone knows how to rename the actual mySQL connector from com.mysql.cj.jdbc.Driver to com.mysql.jdbc.Driver this should fix everything.
Error Log:
C:\Program Files\logstash-6.3.2\bin>logstash -f sql.conf
Sending Logstash's logs to C:/Program Files/logstash-6.3.2/logs which is now configured via log4j2.properties
[2018-08-31T15:01:52,502][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-08-31T15:01:53,031][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.3.2"}
[2018-08-31T15:01:55,217][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-08-31T15:02:06,342][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2018-08-31T15:02:06,342][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-08-31T15:02:06,539][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-08-31T15:02:06,575][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-08-31T15:02:06,592][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-08-31T15:02:06,609][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2018-08-31T15:02:06,625][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-08-31T15:02:06,659][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-08-31T15:02:06,909][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x598b8674 sleep>"}
[2018-08-31T15:02:06,996][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-08-31T15:02:07,359][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-08-31T15:02:07,895][ERROR][logstash.inputs.jdbc ] Unable to connect to database. Tried 1 times {:error_message=>"Java::JavaSql::SQLNonTransientConnectionException: Cannot load connection class because of underlying exception: com.mysql.cj.exceptions.WrongArgumentException: Failed to parse the host:port pair 'localhost:3306;user=test;password=test123;databaseName=test;integratedSecurity=true;'."}
[2018-08-31T15:02:07,916][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Jdbc jdbc_connection_string=>"jdbc:mysql://localhost:3306;user=test;password=test123;databaseName=test;integratedSecurity=true;", jdbc_driver_class=>"com.mysql.cj.jdbc.Driver", jdbc_user=>"doesntmatterwithauthentication", statement=>"SELECT * FROM phones", id=>"de31f73d4505e1de7e76bce4917c48b412909473f3872288edd51acccf0e0be6", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_ad3ad9d5-a3e8-492a-9231-1e729e8c4190", enable_metric=>true, charset=>"UTF-8">, jdbc_paging_enabled=>false, jdbc_page_size=>100000, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdbc_pool_timeout=>5, sql_log_level=>"info", connection_retry_attempts=>1, connection_retry_attempts_wait_time=>0.5, parameters=>{"sql_last_value"=>1970-01-01 01:00:00 +0100}, last_run_metadata_path=>"C:\\Users\\ross.massie/.logstash_jdbc_last_run", use_column_value=>false, tracking_column_type=>"numeric", clean_run=>false, record_last_run=>true, lowercase_column_names=>true>
Error: Java::JavaSql::SQLNonTransientConnectionException: Cannot load connection class because of underlying exception: com.mysql.cj.exceptions.WrongArgumentException: Failed to parse the host:port pair 'localhost:3306;user=test;password=test123;databaseName=test;integratedSecurity=true;'.
Exception: Sequel::DatabaseConnectionError
Stack: com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(com/mysql/cj/jdbc/exceptions/SQLError.java:110)
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(com/mysql/cj/jdbc/exceptions/SQLError.java:97)
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(com/mysql/cj/jdbc/exceptions/SQLError.java:89)
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(com/mysql/cj/jdbc/exceptions/SQLError.java:63)
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(com/mysql/cj/jdbc/exceptions/SQLError.java:73)
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(com/mysql/cj/jdbc/exceptions/SQLExceptionsMapping.java:79)
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(com/mysql/cj/jdbc/exceptions/SQLExceptionsMapping.java:131)
com.mysql.cj.jdbc.NonRegisteringDriver.connect(com/mysql/cj/jdbc/NonRegisteringDriver.java:227)
java.lang.reflect.Method.invoke(java/lang/reflect/Method)
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:423)
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:290)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_10_dot_0.lib.sequel.adapters.jdbc.connect(C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/sequel-5.10.0/lib/sequel/adapters/jdbc.rb:215)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_10_dot_0.lib.sequel.adapters.jdbc.RUBY$method$connect$0$__VARARGS__(C_3a_/Program_20_Files/logstash_minus_6_dot_3_dot_2/vendor/bundle/jruby/$2_dot_3_dot_0/gems/sequel_minus_5_dot_10_dot_0/lib/sequel/adapters/C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/sequel-5.10.0/lib/sequel/adapters/jdbc.rb)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_10_dot_0.lib.sequel.connection_pool.make_new(C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/sequel-5.10.0/lib/sequel/connection_pool.rb:127)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_10_dot_0.lib.sequel.connection_pool.RUBY$method$make_new$0$__VARARGS__(C_3a_/Program_20_Files/logstash_minus_6_dot_3_dot_2/vendor/bundle/jruby/$2_dot_3_dot_0/gems/sequel_minus_5_dot_10_dot_0/lib/sequel/C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/sequel-5.10.0/lib/sequel/connection_pool.rb)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_10_dot_0.lib.sequel.connection_pool.threaded.assign_connection(C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/sequel-5.10.0/lib/sequel/connection_pool/threaded.rb:206)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_10_dot_0.lib.sequel.connection_pool.threaded.RUBY$method$assign_connection$0$__VARARGS__(C_3a_/Program_20_Files/logstash_minus_6_dot_3_dot_2/vendor/bundle/jruby/$2_dot_3_dot_0/gems/sequel_minus_5_dot_10_dot_0/lib/sequel/connection_pool/C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/sequel-5.10.0/lib/sequel/connection_pool/threaded.rb)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_10_dot_0.lib.sequel.connection_pool.threaded.acquire(C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/sequel-5.10.0/lib/sequel/connection_pool/threaded.rb:138)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_10_dot_0.lib.sequel.connection_pool.threaded.RUBY$method$acquire$0$__VARARGS__(C_3a_/Program_20_Files/logstash_minus_6_dot_3_dot_2/vendor/bundle/jruby/$2_dot_3_dot_0/gems/sequel_minus_5_dot_10_dot_0/lib/sequel/connection_pool/C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/sequel-5.10.0/lib/sequel/connection_pool/threaded.rb)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_10_dot_0.lib.sequel.connection_pool.threaded.hold(C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/sequel-5.10.0/lib/sequel/connection_pool/threaded.rb:90)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_10_dot_0.lib.sequel.database.connecting.synchronize(C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/sequel-5.10.0/lib/sequel/database/connecting.rb:270)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_10_dot_0.lib.sequel.database.connecting.test_connection(C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/sequel-5.10.0/lib/sequel/database/connecting.rb:279)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_10_dot_0.lib.sequel.database.connecting.connect(C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/sequel-5.10.0/lib/sequel/database/connecting.rb:58)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_10_dot_0.lib.sequel.core.connect(C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/sequel-5.10.0/lib/sequel/core.rb:116)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_9.lib.logstash.plugin_mixins.jdbc.block in jdbc_connect(C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/plugin_mixins/jdbc.rb:114)
org.jruby.RubyKernel.loop(org/jruby/RubyKernel.java:1292)
org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(org/jruby/RubyKernel$INVOKER$s$0$0$loop.gen)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_9.lib.logstash.plugin_mixins.jdbc.jdbc_connect(C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/plugin_mixins/jdbc.rb:111)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_9.lib.logstash.plugin_mixins.jdbc.RUBY$method$jdbc_connect$0$__VARARGS__(C_3a_/Program_20_Files/logstash_minus_6_dot_3_dot_2/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_9/lib/logstash/plugin_mixins/C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/plugin_mixins/jdbc.rb)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_9.lib.logstash.plugin_mixins.jdbc.open_jdbc_connection(C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/plugin_mixins/jdbc.rb:164)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_9.lib.logstash.plugin_mixins.jdbc.RUBY$method$open_jdbc_connection$0$__VARARGS__(C_3a_/Program_20_Files/logstash_minus_6_dot_3_dot_2/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_9/lib/logstash/plugin_mixins/C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/plugin_mixins/jdbc.rb)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_9.lib.logstash.plugin_mixins.jdbc.execute_statement(C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/plugin_mixins/jdbc.rb:220)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_9.lib.logstash.plugin_mixins.jdbc.RUBY$method$execute_statement$0$__VARARGS__(C_3a_/Program_20_Files/logstash_minus_6_dot_3_dot_2/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_9/lib/logstash/plugin_mixins/C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/plugin_mixins/jdbc.rb)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_9.lib.logstash.inputs.jdbc.execute_query(C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/inputs/jdbc.rb:264)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_9.lib.logstash.inputs.jdbc.RUBY$method$execute_query$0$__VARARGS__(C_3a_/Program_20_Files/logstash_minus_6_dot_3_dot_2/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_9/lib/logstash/inputs/C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/inputs/jdbc.rb)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_9.lib.logstash.inputs.jdbc.run(C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/inputs/jdbc.rb:250)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_9.lib.logstash.inputs.jdbc.RUBY$method$run$0$__VARARGS__(C_3a_/Program_20_Files/logstash_minus_6_dot_3_dot_2/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_9/lib/logstash/inputs/C:/Program Files/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/inputs/jdbc.rb)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.logstash_minus_core.lib.logstash.pipeline.inputworker(C:/Program Files/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:512)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.logstash_minus_core.lib.logstash.pipeline.RUBY$method$inputworker$0$__VARARGS__(C_3a_/Program_20_Files/logstash_minus_6_dot_3_dot_2/logstash_minus_core/lib/logstash/C:/Program Files/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb)
C_3a_.Program_20_Files.logstash_minus_6_dot_3_dot_2.logstash_minus_core.lib.logstash.pipeline.block in start_input(C:/Program Files/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:505)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)
sql.conf:
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://localhost:3306/test?useSSL=false&serverTimezone=GMT&DatabaseName=test"
jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
jdbc_driver_library => "C:\Program Files (x86)\MySQL\Connector J 8.0\mysql-connector-java-8.0.12.jar"
jdbc_user => "test"
jdbc_password => "test123"
statement => "SELECT * FROM phones"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "phones"
}
}
Rewriting the whole connection string worked after a few tweaks. At the time of the issue a few drivers references I found within x-pack were referencing the wrong driver if memory serves and needed changing.
When I first created Logstash jdbc conf file to import my MySQL data to Elasticsearch, it was working good. But, suddenly, the same files which worked okay, are not working any more and giving an error "Error registering plugin".
Here is my sms-logstash.conf file
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://localhost:3306/sms"
# The user we wish to execute our statement as
jdbc_user => "root"
jdbc_password => ""
# The path to our downloaded jdbc driver
jdbc_driver_library => "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/bin/mysql-connector-java-5.1.45-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
# our query
statement => "SELECT * FROM salon_reg"
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
"hosts" => "localhost:9200"
"index" => "sms"
"document_type" => "salon_reg"
}
}
When I run this command as bin/logstash -f sms-logstash.conf
It gives the following error
C:\Users\robesh\Downloads\logstash-6.2.3\logstash-6.2.3\bin>logstash -f sms-logstash.conf
Sending Logstash's logs to C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/logs which is now configured via log4j2.properties
[2018-04-15T15:05:46,900][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/modules/fb_apache/configuration"}
[2018-04-15T15:05:47,028][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/modules/netflow/configuration"}
[2018-04-15T15:05:47,665][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-04-15T15:05:49,635][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.3"}
[2018-04-15T15:05:51,303][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-04-15T15:06:04,935][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch hosts=>[//localhost:9200], index=>"sms", document_type=>"salon_reg", id=>"7eecf64f77b050d7ebba1e645e2de1d988a4f3d4b88814c75044d6e6c4606a2b", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_cbc6f6b2-287c-44bf-8771-3a951d7ceabf", enable_metric=>true, charset=>"UTF-8">, workers=>1, manage_template=>true, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, action=>"index", ssl_certificate_verification=>true, sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[2018-04-15T15:06:05,141][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-04-15T15:06:06,405][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2018-04-15T15:06:06,426][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-04-15T15:06:07,079][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-04-15T15:06:07,280][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-04-15T15:06:07,289][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-04-15T15:06:07,336][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-04-15T15:06:07,424][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-04-15T15:06:07,533][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2018-04-15T15:06:08,863][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::Jdbc jdbc_connection_string=>\"jdbc:mysql://localhost:3306/sms\", jdbc_user=>\"root\", jdbc_driver_library=>\"C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/bin/mysql-connector-java-5.1.46-bin.jar\", jdbc_driver_class=>\"com.mysql.jdbc.Driver\", statement=>\"SELECT * FROM salon_reg\", id=>\"0c99246377cb88117db974a51d7bdcb982e8fe882ab825575c8ebdc3c890fb5a\", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>\"plain_d687f831-1ac5-4480-b23d-e7fc976f5e9a\", enable_metric=>true, charset=>\"UTF-8\">, jdbc_paging_enabled=>false, jdbc_page_size=>100000, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdbc_pool_timeout=>5, sql_log_level=>\"info\", connection_retry_attempts=>1, connection_retry_attempts_wait_time=>0.5, last_run_metadata_path=>\"C:\\\\Users\\\\robesh/.logstash_jdbc_last_run\", use_column_value=>false, tracking_column_type=>\"numeric\", clean_run=>false, record_last_run=>true, lowercase_column_names=>true>", :error=>"(<unknown>): 'reader' unacceptable code point '\u0000' (0x0) special characters are not allowed\nin \"'reader'\", position 0 at line 0 column 0", :thread=>"#<Thread:0x1ad70417 run>"}
[2018-04-15T15:06:09,386][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Psych::SyntaxError: (<unknown>): 'reader' unacceptable code point ' ' (0x0) special characters are not allowed
in "'reader'", position 0 at line 0 column 0>, :backtrace=>["org/jruby/ext/psych/PsychParser.java:231:in `parse'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/psych.rb:377:in `parse_stream'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/psych.rb:325:in `parse'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/psych.rb:252:in `load'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.5/lib/logstash/plugin_mixins/value_tracking.rb:102:in `read'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.5/lib/logstash/plugin_mixins/value_tracking.rb:78:in `get_initial'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.5/lib/logstash/plugin_mixins/value_tracking.rb:36:in `initialize'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.5/lib/logstash/plugin_mixins/value_tracking.rb:29:in `build_last_value_tracker'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.5/lib/logstash/inputs/jdbc.rb:216:in `register'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/logstash-core/lib/logstash/pipeline.rb:341:in `register_plugin'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/logstash-core/lib/logstash/pipeline.rb:352:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/logstash-core/lib/logstash/pipeline.rb:352:in `register_plugins'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/logstash-core/lib/logstash/pipeline.rb:502:in `start_inputs'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/logstash-core/lib/logstash/pipeline.rb:393:in `start_workers'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/logstash-core/lib/logstash/pipeline.rb:289:in `run'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/logstash-core/lib/logstash/pipeline.rb:249:in `block in start'"], :thread=>"#<Thread:0x1ad70417 run>"}
[2018-04-15T15:06:09,506][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: LogStash::PipelineAction::Create/pipeline_id:main, action_result: false", :backtrace=>nil}
C:\Users\robesh\Downloads\logstash-6.2.3\logstash-6.2.3\bin>
The interesting thing here is that this same file was working fine previously and it indexed my data nicely to Elasticsearch, but suddenly now its giving an error.
It may be an encoding problem, ensure that the logstash file has UTF-8 encoding.
Make sure your config is correct :
bin/logstash --config.test_and_exit -f <path_to_config_file>
If your config is valid, then looks at $USER_HOME/.logstash_jdbc_last_run file, probably that file exists but isn't valid YAML. Fix what's broken about the file, or just delete it.
I have a problem on connecting to a mysql datasource from JBoss EAP 6.3 with Teiid 8.10.1 copied over it. I need to mention that I've copied mysql connector driver as a module for using mysql datasource.
Also, when I try with JBoss EAP 6.3 without Teiid, the connection works.
Anybody that confronted the same problem?
This is the error message I receive in administration console of JBoss:
"Unknown error
Unexpected HTTP response: 500
Request {
"address" => [
("subsystem" => "datasources"),
("data-source" => "database")
],
"operation" => "test-connection-in-pool" }
Response
Internal Server Error {
"outcome" => "failed",
"failure-description" => "JBAS010440: failed to invoke operation: JBAS010447: Connection is not valid",
"rolled-back" => true }"
Need to mention that in logs I don't receive a more verbose message for the error, with all logs enabled(Trace, Debug, Error, etc).
So I have followed this tutorial: https://zorq.net/b/2011/07/12/adding-a-mysql-datasource-to-jboss-as-7/
I have done everything as described in the tutorial; yet I keep getting this error on start up:
14:42:19,442 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014613: Operation ("add") failed - address: ([
("subsystem" => "datasources"),
("data-source" => "ExpensesDataSourcePool")
]) - failure description: {"JBAS014771: Services with missing/unavailable dependencies" => [
"jboss.data-source.java:/ExpensesDataSource is missing [jboss.jdbc-driver.mysql]",
"jboss.driver-demander.java:/ExpensesDataSource is missing [jboss.jdbc-driver.mysql]"
]}
14:42:19,450 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014613: Operation ("add") failed - address: ([
("subsystem" => "datasources"),
("data-source" => "ExpensesDataSourcePool")
]) - failure description: {
"JBAS014771: Services with missing/unavailable dependencies" => [
"jboss.data-source.java:/ExpensesDataSource is missing [jboss.jdbc-driver.mysql]",
"jboss.driver-demander.java:/ExpensesDataSource is missing [jboss.jdbc-driver.mysql]"
],
"JBAS014879: One or more services were unable to start due to one or more indirect dependencies not being available." => {
"Services that were unable to start:" => [
"jboss.data-source.reference-factory.ExpensesDataSourcePool",
"jboss.naming.context.java.ExpensesDataSource"
],
"Services that may be the cause:" => ["jboss.jdbc-driver.mysql"]
}
}
14:42:19,489 INFO [org.jboss.as.server] (ServerService Thread Pool -- 28) JBAS018559: Deployed "mysql-connector-java-5.1.34.jar" (runtime-name : "mysql-connector-java-5.1.34.jar")
14:42:19,491 INFO [org.jboss.as.controller] (Controller Boot Thread) JBAS014774: Service status report
JBAS014775: New missing/unsatisfied dependencies:
service jboss.jdbc-driver.mysql (missing) dependents: [service jboss.data-source.java:/ExpensesDataSource, service jboss.driver-demander.java:/ExpensesDataSource]
This is the structure of my modules dir in the Jboss folder: modules/com/mysql/main with two files module.xml and mysql-connector-java-5.1.34.jar
This is my module.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<module xmlns="urn:jboss:module:1.0" name="com.mysql">
<resources>
<resource-root path="mysql-connector-java-5.1.34.jar"/>
</resources>
<dependencies>
<module name="javax.api"/>
</dependencies>
</module>
I have also added the mysql jar in standalone/deployments
This is the configuration of my datasource and driver in my standalone.xml under standalone/configuration
<datasource jndi-name="java:/ExpensesDataSource" pool-name="ExpensesDataSourcePool" enabled="true" use-java-context="true">
<connection-url>jdbc:mysql://localhost:3306/expenses</connection-url>
<driver>mysql</driver>
<security>
<user-name>root</user-name>
<password>root</password>
</security>
</datasource>
<driver name="mysql" module="com.mysql"/>
I found my error, I had to put my module.xml and jar under system modules instead of creating a new com folder
I have been trying to configure mysql as a datasource in wildfly. I am not sure what i am missing out, i get an error on startup .
I have the mysql-connector-java-5.0.8-bin.jar and the module.xml in the folder:
"/wildfly-8.1.0.Final/modules/system/layers/base/com/mysql/main"
below are the files
module.xml
<module xmlns="urn:jboss:module:1.1"
name="com.mysql">
<resources>
<resource-root path="mysql-connector-java-5.0.8.jar"/>
<!-- Insert resources here -->
</resources>
<dependencies>
<module name="javax.api"/>
<module name="javax.transaction.api"/>
<module name="javax.servlet.api" optional="true"/>
</dependencies>
</module>
standalone.xml
<datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">
<connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>
<driver>h2</driver>
<security>
<user-name>sa</user-name>
<password>sa</password>
</security>
</datasource>
<datasource jta="true" jndi-name="java:jboss/datasources/proj" pool-name="proj" enabled="true" use-java-context="true" use-ccm="true">
<connection-url>jdbc:mysql://localhost:3306</connection-url>
<driver>mysql</driver>
<security>
<user-name>root</user-name>
<password>admin123</password>
</security>
<statement>
<prepared-statement-cache-size>32</prepared-statement-cache-size>
<share-prepared-statements>true</share-prepared-statements>
</statement>
</datasource>
<drivers>
<driver name="mysql" module="com.mysql">
<xa-datasource-class>com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</xa-datasource-class>
</driver>
<driver name="h2" module="com.h2database.h2">
<xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
</driver>
</drivers>
</datasources>
I have tested the jdbc connectivity with a standalone program in eclipse and it worked
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
public class ConnectSql {
public static void main(String []args){
String userName = "root";
String pass = "admin123";
String url = "jdbc:mysql://localhost/";
String driver ="com.mysql.jdbc.Driver";
String db = "proj";
try{
//registering the driver.
Class.forName(driver);
Connection con = DriverManager.getConnection(url+db,userName,pass);
Statement st = con.createStatement();
ResultSet rs = st.executeQuery("select f_name from t_users");
while(rs.next()){
System.out.println("name :"+rs.getString(1));
}
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
catch(ClassNotFoundException cnf){
cnf.printStackTrace();
}
}
}
Referred to the below link and modified the module.xml
Wildfly 8.0.0 mysql problems with datasource
Here is a the error log i get on start up
02:45:17,169 ERROR [org.jboss.as.controller.management-operation]
(Controller Boot Thread) JBAS014613: Operation ("add") failed -
address: ([
("subsystem" => "datasources"),
("data-source" => "proj") ]) - failure description: {"JBAS014771: Services with missing/unavailable dependencies" => [
"jboss.data-source.java:jboss/datasources/proj is missing [jboss.jdbc-driver.mysql]",
"jboss.driver-demander.java:jboss/datasources/proj is missing [jboss.jdbc-driver.mysql]" ]} 02:45:17,175 ERROR
[org.jboss.as.controller.management-operation] (Controller Boot
Thread) JBAS014613: Operation ("add") failed - address: ([
("subsystem" => "datasources"),
("data-source" => "proj") ]) - failure description: {
"JBAS014771: Services with missing/unavailable dependencies" => [
"jboss.data-source.java:jboss/datasources/proj is missing [jboss.jdbc-driver.mysql]",
"jboss.driver-demander.java:jboss/datasources/proj is missing [jboss.jdbc-driver.mysql]"
],
"JBAS014879: One or more services were unable to start due to one or more indirect dependencies not being available." => {
"Services that were unable to start:" => [
"jboss.data-source.reference-factory.proj",
"jboss.naming.context.java.jboss.datasources.proj"
],
"Services that may be the cause:" => ["jboss.jdbc-driver.mysql"]
} }
To resolve the mysql datasource configuration issue on Wildfly i used the admin console to add the datasource and test it.
If you login to the web console and find the datasource you tried configuring disable
that a remove it . The standalone.xml and the module.xml get reset to the orginal.
Steps for accessing web console
Configure a new datasource :
name :mysql
JNDI :java:jboss/datasources/proj
Click next and enter the url (i used the below) and click enable and then click test
url : jdbc:mysql://localhost/proj
The test should show success
The changes to the standalone.xml and the module.xml are automatically made .
If you restart the server now it should start without any errors and you should be able to access the database from your web project
I have working configuration of mysql driver on wildfly.
<datasources>
// ...
<drivers>
// ...
<driver name="mysql" module="com.mysql.jdbc">
// !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
<driver-class>com.mysql.jdbc.Driver</driver-class>
// !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
<xa-datasource-class>com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</xa-datasource-class>
</driver>
</drivers>
</datasources>
The little difference seems to be presence of "driver-class" tag.
I hope it will help.
(Deploying mysql driver as a deployment also works and it's recommended way. https://docs.jboss.org/author/display/WFLY8/DataSource+configuration )
[edit]
I have mysql module under wildfly/modules/com/mysql/jdbc/main/ . I've noticed just now you module path doesn't correspond to module name. And I haven't find any doc's just yet, but I don't think you should mess with modules/system directory.
module.xml
<?xml version="1.0" ?>
<module xmlns="urn:jboss:module:1.1" name="com.mysql.jdbc">
<resources>
<resource-root path="mysql-connector-java-5.1.31-bin.jar"/>
</resources>
<dependencies>
<module name="javax.api"/>
<module name="javax.transaction.api"/>
</dependencies>
</module>
Two things: I just got mine working, and have the driver and xml located in $JBOSS_HOME/modules/com/mysql/jdbc/main - I'm fairly new with this as well so I'm not sure if that matters. Another thing: I don't know if it is just a typo, but if it was just copied and pasted it looks like your JAR name is incorrect
<resource-root path="mysql-connector-java-5.0.8.jar"/>
when it should be
<resource-root path="mysql-connector-java-5.0.8-bin.jar"/>
I forgot the .jar in mine, and that ended up fixing it. Hope that helps!
This may happen when you download the mysql connector zip and uploading the zip as a deployment. But the correct way is to unzip your download and point to the contained jar.
There are three ways using which you can simply create datasource into wildfly
Using admin console
Manually adding into standalone.xml
Creating datasource file that is xml file.
Go to WildFly directory/standalone/deployments
Simplest way to create datasource xml with following content
<datasources xmlns="http://www.jboss.org/ironjacamar/schema">
<datasource jndi-name="APP_DS" pool-name="APP_DS" enabled="true" use-ccm="false">
<connection-url>jdbc:mysql://localhost:3306/DB_NAME</connection-url>
<driver-class>com.mysql.jdbc.Driver</driver-class>
<driver>mysql</driver>
<!-- sql to call when connection is created -->
<new-connection-sql>SELECT 1</new-connection-sql>
<pool>
<min-pool-size>5</min-pool-size>
<max-pool-size>50</max-pool-size>
</pool>
<security>
<user-name>username</user-name>
<password>password</password>
</security>
<!-- sql to call on an existing pooled connection when it is obtained from pool -->
<validation>
<check-valid-connection-sql>SELECT 1</check-valid-connection-sql>
</validation>
<timeout>
<blocking-timeout-millis>300000</blocking-timeout-millis>
<idle-timeout-minutes>5</idle-timeout-minutes>
</timeout>
<statement>
<track-statements>true</track-statements>
</statement>
</datasource>
</datasources>