So I set up Elasticsearch, I have the server running. When I connect to http://localhost:9200/ I get the status message. It is version number 1.4.3.
Now I download Kibana 3.1.2. As instruced in the readme, I add the following lines to elasticsearch.yml, the config file:
# making kibana local work
http.cors.enabled: true
http.cors.allow-origin: 127.0.0.1
I also add the following lines to Kibana's config.js:
elasticsearch: 'http://localhost:9200',
When I navigate to Kibana's index.html in my up-to-date Firefox, I get a page stating that:
Connection Failed
Possibility #1: Your elasticsearch server is down or unreachable.......
Now I've tried some other variables for the config files, but it seems like it should work. Both are present on my local Ubuntu 14.04/64bit install.
When I check elasticsearch.log, I find the following:
[2015-02-18 13:50:12,697][INFO ][node ] [Puma] version[1.4.3], pid[14592], build[36a29a7/2015-02-11T14:23:15Z]
[2015-02-18 13:50:12,698][INFO ][node ] [Puma] initializing ...
[2015-02-18 13:50:12,706][INFO ][plugins ] [Puma] loaded [], sites []
[2015-02-18 13:50:16,540][INFO ][node ] [Puma] initialized
[2015-02-18 13:50:16,540][INFO ][node ] [Puma] starting ...
[2015-02-18 13:50:16,735][INFO ][transport ] [Puma] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.192.25:9300]}
[2015-02-18 13:50:16,761][INFO ][discovery ] [Puma] elasticsearch/zEjlD0HdR2OtpjykOo1e7g
[2015-02-18 13:50:20,542][INFO ][cluster.service ] [Puma] new_master [Puma][zEjlD0HdR2OtpjykOo1e7g][zirconcode][inet[/192.168.192.25:9300]], reason: zen-disco-join (elected_as_master)
[2015-02-18 13:50:20,569][INFO ][http ] [Puma] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.192.25:9200]}
[2015-02-18 13:50:20,570][INFO ][node ] [Puma] started
[2015-02-18 13:50:20,632][INFO ][gateway ] [Puma] recovered [0] indices into cluster_state
[2015-02-18 13:50:29,195][WARN ][transport.netty ] [Puma] exception caught on transport layer [[id: 0x8e5c4505, /127.0.0.1:53332 => /127.0.0.1:9300]], closing connection
java.io.StreamCorruptedException: invalid internal transport message format, got (47,45,54,20)
at org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:47)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2015-02-18 13:50:29,202][WARN ][transport.netty ] [Puma] exception caught on transport layer [[id: 0x8e5c4505, /127.0.0.1:53332 :> /127.0.0.1:9300]], closing connection
java.io.StreamCorruptedException: invalid internal transport message format, got (47,45,54,20)
at org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:47)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:482)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:365)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireChannelDisconnected(Channels.java:396)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:81)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.close(Channels.java:812)
at org.elasticsearch.common.netty.channel.AbstractChannel.close(AbstractChannel.java:197)
at org.elasticsearch.transport.netty.NettyTransport.exceptionCaught(NettyTransport.java:622)
at org.elasticsearch.transport.netty.MessageChannelHandler.exceptionCaught(MessageChannelHandler.java:234)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
at org.elasticsearch.common.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:566)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I've only been getting this error once in multiple attempts though.
Instead of
http.cors.allow-origin: 127.0.0.1
in your ES 1.4.3 elasticsearch.yml try
http.cors.allow-origin: "http://127.0.0.1"
That assumes you're running kibana on port 80
The above will work if you access kibana from http://127.0.0.1/kibana. If you use the url http://localhost/kibana instead, replace 127.0.0.1 with localhost in your yml file
Follow below steps to solve above problem:
Open file elasticsearch<Version>/config/elasticsearch.yml
Add one line http.cors.allow-origin: "http://127.0.0.1:5601"
Now start elasticserch and then start Kibana. It will now connect.
Related
Started the docker containers (cp-kafka-connect-base:7.0.1) and installed the self-managed connector (debezium-connector-mysql:latest).
Docker containers works good.
When I am trying to configure the connector configuration as below
{
"name": "MySqlConnectorConnector_0",
"config": {
"database.history.consumer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule",
"database.history.kafka.topic": "pageviews",
"database.history.consumer.security.protocol": "SASL_SSL",
"database.history.consumer.ssl.endpoint.identification.algorithm": "https",
"schema.history.internal.kafka.topic": "PLAIN",
"database.whitelist": "database-test",
"database.history.producer.sasl.mechanism": "PLAIN",
"database.history.producer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule",
"database.history.producer.ssl.endpoint.identification.algorithm": "https",
"database.history.producer.security.protocol": "SASL_SSL",
"database.history.kafka.bootstrap.servers": "localhost:9092",
"database.server.name": "database-test:3306",
"schema.history.internal.kafka.bootstrap.servers": "localhost:9092",
"database.history.consumer.sasl.mechanism": "PLAIN",
"name": "MySqlConnectorConnector_0",
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"tasks.max": "1",
"errors.log.enable": "true",
"errors.log.include.messages": "true",
"topic.prefix": "test2",
"database.hostname": "database-test",
"database.port": "3306",
"database.user": "admin",
"database.password": "**********",
"database.server.id": "11",
"database.ssl.mode": "disabled",
"connect.keep.alive": "true",
"include.schema.changes": "true",
"inconsistent.schema.handling.mode": "skip"
}
}
the connector creates but connection task failed
Connector log
[2023-01-24 17:25:44,827] INFO [Consumer clientId=test3-schemahistory, groupId=test3-schemahistory] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
[2023-01-24 17:25:44,827] WARN [Consumer clientId=test3-schemahistory, groupId=test3-schemahistory] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2023-01-24 17:25:44,827] WARN [Consumer clientId=test3-schemahistory, groupId=test3-schemahistory] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2023-01-24 17:25:44,831] INFO WorkerSourceTask{id=MySqlConnectorConnector_01-0} Either no records were produced by the task since the last offset commit, or every record has been filtered out by a transformation or dropped due to transformation or conversion errors. (org.apache.kafka.connect.runtime.WorkerSourceTask)
[2023-01-24 17:25:44,831] WARN Couldn't commit processed log positions with the source database due to a concurrent connector shutdown or restart (io.debezium.connector.common.BaseSourceTask)
[2023-01-24 17:25:44,873] INFO [Consumer clientId=test3-schemahistory, groupId=test3-schemahistory] Resetting generation due to: consumer pro-actively leaving the group (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2023-01-24 17:25:44,873] INFO [Consumer clientId=test3-schemahistory, groupId=test3-schemahistory] Request joining group due to: consumer pro-actively leaving the group (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2023-01-24 17:25:44,874] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
[2023-01-24 17:25:44,874] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
[2023-01-24 17:25:44,874] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
[2023-01-24 17:25:44,875] INFO App info kafka.consumer for test3-schemahistory unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2023-01-24 17:25:44,875] ERROR WorkerSourceTask{id=MySqlConnectorConnector_01-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
[2023-01-24 17:33:57,334] INFO [Consumer clientId=test3-schemahistory, groupId=test3-schemahistory] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
[2023-01-24 17:33:57,334] WARN [Consumer clientId=test3-schemahistory, groupId=test3-schemahistory] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Trying to add some extra config parameters, but documentation is not clear for me. Can not feagure out how to fix this and how should connector config looks like.
As the error says, schema.history.internal.kafka.bootstrap.servers needs to be a host:port list to Kafka brokers, not a Java class name.
Similarly, schema.history.internal.kafka.topic should be plain string, not necessarily a Java class.
plz someone help me resolve this errors. im trying to build an application which will achive cdc for5 this i have used debezium connector
2022-01-25 12:52:02.806 INFO 108 --- [server-snapshot] i.d.connector.mysql.SnapshotReader : including 'world.countrylanguage' among known tables
2022-01-25 12:52:02.806 INFO 108 --- [server-snapshot] i.d.connector.mysql.SnapshotReader : 'world.countrylanguage' is filtered out of capturing
2022-01-25 12:52:02.806 WARN 108 --- [server-snapshot] io.debezium.config.Configuration : Using configuration property "table.whitelist" is deprecated and will be removed in future versions. Please use "table.include.list" instead.
2022-01-25 12:52:02.807 INFO 108 --- [server-snapshot] i.d.connector.mysql.SnapshotReader : snapshot continuing with database(s): []
2022-01-25 12:52:02.808 INFO 108 --- [server-snapshot] i.d.connector.mysql.SnapshotReader : Step 6: generating DROP and CREATE statements to reflect current database schemas:
2022-01-25 12:52:04.879 INFO 108 --- [server-snapshot] i.d.connector.mysql.SnapshotReader : Step 7: committing transaction
2022-01-25 12:52:05.033 INFO 108 --- [server-snapshot] i.d.connector.mysql.SnapshotReader : Step 8: releasing global read lock to enable MySQL writes
2022-01-25 12:52:05.051 INFO 108 --- [server-snapshot] i.d.connector.mysql.SnapshotReader : Writes to MySQL tables prevented for a total of 00:00:04.843
2022-01-25 12:52:05.119 ERROR 108 --- [server-snapshot] i.d.connector.mysql.SnapshotReader : Failed due to error: Aborting snapshot due to error when last running 'UNLOCK TABLES': tried to access field com.mysql.cj.CharsetMapping.COLLATION_INDEX_TO_COLLATION_NAME from class io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser
org.apache.kafka.connect.errors.ConnectException: tried to access field com.mysql.cj.CharsetMapping.COLLATION_INDEX_TO_COLLATION_NAME from class io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser
at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:241) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:218) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:846) [debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_281]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_281]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_281]
Caused by: java.lang.IllegalAccessError: tried to access field com.mysql.cj.CharsetMapping.COLLATION_INDEX_TO_COLLATION_NAME from class io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser
at io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.extractCharset(MySqlAntlrDdlParser.java:403) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.antlr.listener.CreateAndAlterDatabaseParserListener.enterCreateDatabaseOption(CreateAndAlterDatabaseParserListener.java:49) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.ddl.parser.mysql.generated.MySqlParser$CreateDatabaseOptionContext.enterRule(MySqlParser.java:5754) ~[debezium-ddl-parser-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.antlr.ProxyParseTreeListenerUtil.delegateEnterRule(ProxyParseTreeListenerUtil.java:46) ~[debezium-ddl-parser-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.antlr.listener.MySqlAntlrDdlParserListener.enterEveryRule(MySqlAntlrDdlParserListener.java:89) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at org.antlr.v4.runtime.tree.ParseTreeWalker.enterRule(ParseTreeWalker.java:41) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:25) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at io.debezium.antlr.AntlrDdlParser.parse(AntlrDdlParser.java:85) ~[debezium-ddl-parser-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.MySqlSchema.applyDdl(MySqlSchema.java:326) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:527) [debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
... 3 common frames omitted
2022-01-25 12:52:05.123 INFO 108 --- [pool-2-thread-1] io.debezium.jdbc.JdbcConnection : Connection gracefully closed
2022-01-25 12:52:05.361 INFO 108 --- [pool-1-thread-1] i.d.connector.common.BaseSourceTask : Stopping down connector
2022-01-25 12:52:05.362 INFO 108 --- [pool-1-thread-1] i.d.connector.mysql.MySqlConnectorTask : Stopping MySQL connector task
2022-01-25 12:52:05.362 INFO 108 --- [pool-1-thread-1] i.d.connector.mysql.ChainedReader : ChainedReader: Stopping the snapshot reader
2022-01-25 12:52:05.362 INFO 108 --- [pool-1-thread-1] i.d.connector.mysql.SnapshotReader : Discarding 0 unsent record(s) due to the connector shutting down
2022-01-25 12:52:05.362 INFO 108 --- [pool-1-thread-1] i.d.connector.mysql.SnapshotReader : Discarding 0 unsent record(s) due to the connector shutting down
2022-01-25 12:52:05.363 INFO 108 --- [pool-1-thread-1] i.d.connector.mysql.MySqlConnectorTask : Connector task finished all work and is now shutdown
2022-01-25 12:52:05.364 INFO 108 --- [pool-1-thread-1] o.a.k.c.storage.FileOffsetBackingStore : Stopped FileOffsetBackingStore
2022-01-25 12:52:05.367 ERROR 108 --- [pool-1-thread-1] io.debezium.embedded.EmbeddedEngine : Error while trying to run connector class 'io.debezium.connector.mysql.MySqlConnector'
org.apache.kafka.connect.errors.ConnectException: tried to access field com.mysql.cj.CharsetMapping.COLLATION_INDEX_TO_COLLATION_NAME from class io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser
at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:241) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:218) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:846) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_281]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_281]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_281]
Caused by: java.lang.IllegalAccessError: tried to access field com.mysql.cj.CharsetMapping.COLLATION_INDEX_TO_COLLATION_NAME from class io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser
at io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.extractCharset(MySqlAntlrDdlParser.java:403) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.antlr.listener.CreateAndAlterDatabaseParserListener.enterCreateDatabaseOption(CreateAndAlterDatabaseParserListener.java:49) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.ddl.parser.mysql.generated.MySqlParser$CreateDatabaseOptionContext.enterRule(MySqlParser.java:5754) ~[debezium-ddl-parser-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.antlr.ProxyParseTreeListenerUtil.delegateEnterRule(ProxyParseTreeListenerUtil.java:46) ~[debezium-ddl-parser-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.antlr.listener.MySqlAntlrDdlParserListener.enterEveryRule(MySqlAntlrDdlParserListener.java:89) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at org.antlr.v4.runtime.tree.ParseTreeWalker.enterRule(ParseTreeWalker.java:41) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:25) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at io.debezium.antlr.AntlrDdlParser.parse(AntlrDdlParser.java:85) ~[debezium-ddl-parser-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.MySqlSchema.applyDdl(MySqlSchema.java:326) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:527) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
... 3 common frames omitted
As per the error looks like the you don't have enough permissions to have this running. What privileges do you have for the debezium user that connects to your mysql database ? Does it have replication client/slave and super privileges ?
d.connector.mysql.SnapshotReader : Writes to MySQL tables prevented for a total of 00:00:04.843 2022-01-25 12:52:05.119 ERROR 108 --- [server-snapshot] i.d.connector.mysql.SnapshotReader : Failed due to error: Aborting snapshot due to error when last running 'UNLOCK TABLES': tried to access field
When I first created Logstash jdbc conf file to import my MySQL data to Elasticsearch, it was working good. But, suddenly, the same files which worked okay, are not working any more and giving an error "Error registering plugin".
Here is my sms-logstash.conf file
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://localhost:3306/sms"
# The user we wish to execute our statement as
jdbc_user => "root"
jdbc_password => ""
# The path to our downloaded jdbc driver
jdbc_driver_library => "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/bin/mysql-connector-java-5.1.45-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
# our query
statement => "SELECT * FROM salon_reg"
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
"hosts" => "localhost:9200"
"index" => "sms"
"document_type" => "salon_reg"
}
}
When I run this command as bin/logstash -f sms-logstash.conf
It gives the following error
C:\Users\robesh\Downloads\logstash-6.2.3\logstash-6.2.3\bin>logstash -f sms-logstash.conf
Sending Logstash's logs to C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/logs which is now configured via log4j2.properties
[2018-04-15T15:05:46,900][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/modules/fb_apache/configuration"}
[2018-04-15T15:05:47,028][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/modules/netflow/configuration"}
[2018-04-15T15:05:47,665][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-04-15T15:05:49,635][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.3"}
[2018-04-15T15:05:51,303][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-04-15T15:06:04,935][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch hosts=>[//localhost:9200], index=>"sms", document_type=>"salon_reg", id=>"7eecf64f77b050d7ebba1e645e2de1d988a4f3d4b88814c75044d6e6c4606a2b", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_cbc6f6b2-287c-44bf-8771-3a951d7ceabf", enable_metric=>true, charset=>"UTF-8">, workers=>1, manage_template=>true, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, action=>"index", ssl_certificate_verification=>true, sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[2018-04-15T15:06:05,141][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-04-15T15:06:06,405][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2018-04-15T15:06:06,426][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-04-15T15:06:07,079][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-04-15T15:06:07,280][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-04-15T15:06:07,289][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-04-15T15:06:07,336][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-04-15T15:06:07,424][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-04-15T15:06:07,533][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2018-04-15T15:06:08,863][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::Jdbc jdbc_connection_string=>\"jdbc:mysql://localhost:3306/sms\", jdbc_user=>\"root\", jdbc_driver_library=>\"C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/bin/mysql-connector-java-5.1.46-bin.jar\", jdbc_driver_class=>\"com.mysql.jdbc.Driver\", statement=>\"SELECT * FROM salon_reg\", id=>\"0c99246377cb88117db974a51d7bdcb982e8fe882ab825575c8ebdc3c890fb5a\", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>\"plain_d687f831-1ac5-4480-b23d-e7fc976f5e9a\", enable_metric=>true, charset=>\"UTF-8\">, jdbc_paging_enabled=>false, jdbc_page_size=>100000, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdbc_pool_timeout=>5, sql_log_level=>\"info\", connection_retry_attempts=>1, connection_retry_attempts_wait_time=>0.5, last_run_metadata_path=>\"C:\\\\Users\\\\robesh/.logstash_jdbc_last_run\", use_column_value=>false, tracking_column_type=>\"numeric\", clean_run=>false, record_last_run=>true, lowercase_column_names=>true>", :error=>"(<unknown>): 'reader' unacceptable code point '\u0000' (0x0) special characters are not allowed\nin \"'reader'\", position 0 at line 0 column 0", :thread=>"#<Thread:0x1ad70417 run>"}
[2018-04-15T15:06:09,386][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Psych::SyntaxError: (<unknown>): 'reader' unacceptable code point ' ' (0x0) special characters are not allowed
in "'reader'", position 0 at line 0 column 0>, :backtrace=>["org/jruby/ext/psych/PsychParser.java:231:in `parse'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/psych.rb:377:in `parse_stream'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/psych.rb:325:in `parse'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/psych.rb:252:in `load'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.5/lib/logstash/plugin_mixins/value_tracking.rb:102:in `read'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.5/lib/logstash/plugin_mixins/value_tracking.rb:78:in `get_initial'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.5/lib/logstash/plugin_mixins/value_tracking.rb:36:in `initialize'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.5/lib/logstash/plugin_mixins/value_tracking.rb:29:in `build_last_value_tracker'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.5/lib/logstash/inputs/jdbc.rb:216:in `register'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/logstash-core/lib/logstash/pipeline.rb:341:in `register_plugin'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/logstash-core/lib/logstash/pipeline.rb:352:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/logstash-core/lib/logstash/pipeline.rb:352:in `register_plugins'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/logstash-core/lib/logstash/pipeline.rb:502:in `start_inputs'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/logstash-core/lib/logstash/pipeline.rb:393:in `start_workers'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/logstash-core/lib/logstash/pipeline.rb:289:in `run'", "C:/Users/robesh/Downloads/logstash-6.2.3/logstash-6.2.3/logstash-core/lib/logstash/pipeline.rb:249:in `block in start'"], :thread=>"#<Thread:0x1ad70417 run>"}
[2018-04-15T15:06:09,506][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: LogStash::PipelineAction::Create/pipeline_id:main, action_result: false", :backtrace=>nil}
C:\Users\robesh\Downloads\logstash-6.2.3\logstash-6.2.3\bin>
The interesting thing here is that this same file was working fine previously and it indexed my data nicely to Elasticsearch, but suddenly now its giving an error.
It may be an encoding problem, ensure that the logstash file has UTF-8 encoding.
Make sure your config is correct :
bin/logstash --config.test_and_exit -f <path_to_config_file>
If your config is valid, then looks at $USER_HOME/.logstash_jdbc_last_run file, probably that file exists but isn't valid YAML. Fix what's broken about the file, or just delete it.
I have a JSON file and want to load using Apache Pig.
I am using the built-in JSONLOADER to load json data, Below is the sample json data.
cat jsondata1.json
{ "response": { "id": 10123, "thread": "Sloths", "comments": ["Sloths are adorable So chill"] }, "response_time": 0.425 }
{ "response": { "id": 13828, "thread": "Bigfoot", "comments": ["hello world"] } , "response_time": 0.517 }
Here I loading json data using builtin Json loader. While loading there is no error, but while dumping the data it gives the following error.
grunt> a = load '/home/cloudera/jsondata1.json' using JsonLoader('response:tuple (id:int, thread:chararray, comments:bag {tuple(comment:chararray)}), response_time:double');
grunt> dump a;
2016-04-17 01:11:13,286 [pool-4-thread-1] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed file:/home/cloudera/jsondata1.json:0+229
2016-04-17 01:11:13,287 [pool-4-thread-1] WARN org.apache.hadoop.conf.Configuration - dfs.https.address is deprecated. Instead, use dfs.namenode.https-address
2016-04-17 01:11:13,311 [pool-4-thread-1] WARN org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized
2016-04-17 01:11:13,321 [pool-4-thread-1] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map - Aliases being processed per job phase (AliasName[line,offset]): M: a[5,4] C: R:
2016-04-17 01:11:13,349 [Thread-16] INFO org.apache.hadoop.mapred.LocalJobRunner - Map task executor complete.
2016-04-17 01:11:13,351 [Thread-16] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local801054416_0004
java.lang.Exception: org.codehaus.jackson.JsonParseException: Current token (FIELD_NAME) not numeric, can not use numeric value accessors
at [Source: java.io.ByteArrayInputStream#2484de3c; line: 1, column: 120]
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:406)
Caused by: org.codehaus.jackson.JsonParseException: Current token (FIELD_NAME) not numeric, can not use numeric value accessors
at [Source: java.io.ByteArrayInputStream#2484de3c; line: 1, column: 120]
at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1291)
at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
at org.codehaus.jackson.impl.JsonNumericParserBase._parseNumericValue(JsonNumericParserBase.java:399)
at org.codehaus.jackson.impl.JsonNumericParserBase.getDoubleValue(JsonNumericParserBase.java:311)
at org.apache.pig.builtin.JsonLoader.readField(JsonLoader.java:203)
at org.apache.pig.builtin.JsonLoader.getNext(JsonLoader.java:157)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:211)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:483)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:268)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2016-04-17 01:11:13,548 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local801054416_0004
2016-04-17 01:11:13,548 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases a
2016-04-17 01:11:13,548 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: a[5,4] C: R:
2016-04-17 01:11:18,059 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
2016-04-17 01:11:18,059 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_local801054416_0004 has failed! Stop running all dependent jobs
2016-04-17 01:11:18,059 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2016-04-17 01:11:18,059 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2016-04-17 01:11:18,060 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats reported below may be incomplete
2016-04-17 01:11:18,060 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
2.0.0-cdh4.7.0 0.11.0-cdh4.7.0 cloudera 2016-04-17 01:11:12 2016-04-17 01:11:18 UNKNOWN
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
job_local801054416_0004 a MAP_ONLY Message: Job failed! file:/tmp/temp-1766116741/tmp1151698221,
Input(s):
Failed to read data from "/home/cloudera/jsondata1.json"
Output(s):
Failed to produce result in "file:/tmp/temp-1766116741/tmp1151698221"
Job DAG:
job_local801054416_0004
2016-04-17 01:11:18,060 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2016-04-17 01:11:18,061 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias a
Details at logfile: /home/cloudera/pig_1460877001124.log
I could not able to find the issue. Can I know how to define the correct schema for the above json data?.
Try this:
comments:{(chararray)}
because this version:
comments:bag {tuple(comment:chararray)}
fits this JSON schema:
"comments": [{comment:"hello world"}]
and you have simple string values, not another nested documents:
"comments": ["hello world"]
I've setup hadoop on a two node cluster. The first node "namenode" runs the following daemons:
hadoop#namenode:~$ jps
2916 SecondaryNameNode
2692 NameNode
3159 NodeManager
5834 Jps
2771 DataNode
3076 ResourceManager
The seconds node "datanode" runs the following daemons:
hadoop#datanode:~$ jps
2559 Jps
2087 DataNode
2198 NodeManager
In the /etc/hosts file I added on BOTH machines:
10.240.40.246 namenode
10.240.172.201 datanode
which are the corresponding ips and I check that I can ssh to any other machine from each machine. Now, I wanted to test my hadoop installation by performing a sample map reduce benchmark job:
hadoop#namenode:~$ hadoop jar /opt/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar TestDFSIO -write -nrFiles 20 -fileSize 10
However this job fails:
14/02/17 22:22:53 INFO fs.TestDFSIO: TestDFSIO.1.7
14/02/17 22:22:53 INFO fs.TestDFSIO: nrFiles = 20
14/02/17 22:22:53 INFO fs.TestDFSIO: nrBytes (MB) = 10.0
14/02/17 22:22:53 INFO fs.TestDFSIO: bufferSize = 1000000
14/02/17 22:22:53 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
14/02/17 22:22:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/02/17 22:22:55 INFO fs.TestDFSIO: creating control file: 10485760 bytes, 20 files
14/02/17 22:22:56 INFO fs.TestDFSIO: created control files for: 20 files
14/02/17 22:22:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/02/17 22:22:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/02/17 22:22:57 INFO mapred.FileInputFormat: Total input paths to process : 20
14/02/17 22:22:57 INFO mapreduce.JobSubmitter: number of splits:20
14/02/17 22:22:57 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/02/17 22:22:57 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/02/17 22:22:57 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/02/17 22:22:57 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/02/17 22:22:57 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/02/17 22:22:57 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/02/17 22:22:57 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/02/17 22:22:57 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/02/17 22:22:57 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/02/17 22:22:57 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/02/17 22:22:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1392675199090_0001
14/02/17 22:22:59 INFO impl.YarnClientImpl: Submitted application application_1392675199090_0001 to ResourceManager at /0.0.0.0:8032
14/02/17 22:22:59 INFO mapreduce.Job: The url to track the job: http://namenode.c.forward-camera-473.internal:8088/proxy/application_1392675199090_0001/
14/02/17 22:22:59 INFO mapreduce.Job: Running job: job_1392675199090_0001
14/02/17 22:23:10 INFO mapreduce.Job: Job job_1392675199090_0001 running in uber mode : false
14/02/17 22:23:10 INFO mapreduce.Job: map 0% reduce 0%
14/02/17 22:23:42 INFO mapreduce.Job: map 20% reduce 0%
14/02/17 22:23:43 INFO mapreduce.Job: map 30% reduce 0%
14/02/17 22:24:14 INFO mapreduce.Job: map 60% reduce 0%
14/02/17 22:24:41 INFO mapreduce.Job: map 60% reduce 20%
14/02/17 22:24:45 INFO mapreduce.Job: map 85% reduce 20%
14/02/17 22:24:48 INFO mapreduce.Job: map 85% reduce 28%
14/02/17 22:24:59 INFO mapreduce.Job: map 90% reduce 28%
14/02/17 22:25:00 INFO mapreduce.Job: map 90% reduce 30%
14/02/17 22:25:02 INFO mapreduce.Job: map 100% reduce 30%
14/02/17 22:25:03 INFO mapreduce.Job: map 100% reduce 100%
14/02/17 22:25:16 INFO mapreduce.Job: map 0% reduce 0%
14/02/17 22:25:16 INFO mapreduce.Job: Job job_1392675199090_0001 failed with state FAILED due to: Application application_1392675199090_0001 failed 2 times due to AM Container for appattempt_1392675199090_0001_000002 exited with exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
.Failing this attempt.. Failing the application.
14/02/17 22:25:16 INFO mapreduce.Job: Counters: 0
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836)
at org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:443)
at org.apache.hadoop.fs.TestDFSIO.writeTest(TestDFSIO.java:425)
at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:755)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:650)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:115)
at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Having a look into the log file I find on the machine datanode in that:
hadoop#datanode:/opt/hadoop-2.2.0/logs$ cat yarn-hadoop-nodemanager-datanode.log
...
2014-02-17 22:29:33,432 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
On my namenode I did:
hadoop#namenode:/opt/hadoop-2.2.0/logs$ cat yarn-hadoop-*log
2014-02-17 22:13:20,833 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: STARTUP_MSG:
...
2014-02-17 22:13:25,240 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class class org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config.
...
2014-02-17 22:13:25,505 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: NodeManager configured with 8 G physical memory allocated to containers, which is more than 80% of the total physical memory available (3.6 G). Thrashing might happen.
...
2014-02-17 22:24:48,779 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Event EventType: KILL_CONTAINER sent to absent container container_1392675199090_0001_01_000023
2014-02-17 22:24:48,779 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Event EventType: KILL_CONTAINER sent to absent container container_1392675199090_0001_01_000024
...
2014-02-17 22:25:15,733 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1392675199090_0001_02_000001 is : 1
2014-02-17 22:25:15,734 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1392675199090_0001_02_000001 and exit code: 1
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
...
2014-02-17 22:25:15,736 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 1
...
2014-02-17 22:25:15,751 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hadoop OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1392675199090_0001 CONTAINERID=container_1392675199090_0001_02_000001
...
2014-02-17 22:13:19,150 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: STARTUP_MSG:
...
2014-02-17 22:25:15,837 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1392675199090_0001 failed 2 times due to AM Container for appattempt_1392675199090_0001_000002 exited with exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
.Failing this attempt.. Failing the application. APPID=application_1392675199090_0001
However, I checked on machine namenode that port 8031 is listening. I get:
hadoop#namenode:~$ netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 namenode.c.forwar:36975 metadata.google.in:http TIME_WAIT
tcp 0 0 namenode.c.forwar:36969 metadata.google.in:http TIME_WAIT
tcp 0 0 namenode.c.forwar:40616 namenode.c.forwar:10001 TIME_WAIT
tcp 0 0 namenode.c.forwar:36974 metadata.google.in:http ESTABLISHED
tcp 0 0 namenode.c.forward:8031 namenode.c.forwar:41229 ESTABLISHED
tcp 0 352 namenode.c.forward-:ssh e178064245.adsl.a:64305 ESTABLISHED
tcp 0 0 namenode.c.forwar:41229 namenode.c.forward:8031 ESTABLISHED
tcp 0 0 namenode.c.forwar:40365 namenode.c.forwar:10001 ESTABLISHED
tcp 0 0 namenode.c.forwar:10001 namenode.c.forwar:40365 ESTABLISHED
tcp 0 0 namenode.c.forwar:10001 datanode:48786 ESTABLISHED
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags Type State I-Node Path
unix 10 [ ] DGRAM 4604 /dev/log
unix 2 [ ] STREAM CONNECTED 10490
unix 2 [ ] STREAM CONNECTED 10488
unix 2 [ ] STREAM CONNECTED 10452
unix 2 [ ] STREAM CONNECTED 8452
unix 2 [ ] STREAM CONNECTED 7800
unix 2 [ ] STREAM CONNECTED 7797
unix 2 [ ] STREAM CONNECTED 6762
unix 2 [ ] STREAM CONNECTED 6702
unix 2 [ ] STREAM CONNECTED 6698
unix 2 [ ] STREAM CONNECTED 6208
unix 2 [ ] DGRAM 5750
unix 2 [ ] DGRAM 5737
unix 2 [ ] DGRAM 5734
unix 3 [ ] STREAM CONNECTED 5643
unix 3 [ ] STREAM CONNECTED 5642
unix 2 [ ] DGRAM 5640
unix 2 [ ] DGRAM 5192
unix 2 [ ] DGRAM 5171
unix 2 [ ] DGRAM 4889
unix 2 [ ] DGRAM 4723
unix 2 [ ] DGRAM 4663
unix 3 [ ] DGRAM 3132
unix 3 [ ] DGRAM 3131
So, what could be the problem here. In my opinion everything is setup fine. Why is my job failing then?
The log on the datanode says
Retrying connect to server: 0.0.0.0/0.0.0.0:8031
So it trys to connect to this port on the local machine which is datanode. However, the service runs on namenode. Therefore one has to add the following config lines to yarn-site.xml
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>namenode:8031</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>namenode:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>namenode:8030</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>namenode:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>namenode:8088</value>
</property>
where namenode is an alias in /etc/hosts for the machine that runs the resource manager daemon.
Also add the same properties in the yarn-site.xml file on the namenode to ensure that these services connect to the same ports.