Errors while setting up a Kafka connector pipeline
Enviroment overview:
Docker containers
docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=mypw -d mysql:latest
docker run -d --name Elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" Elasticsearch: 7.9.2
Operating System
WSL2 ON - Windows-10 Version 1909 (OS build 18363.1139)
Kafka Version
confluent-6.0.0
Hello
I’m reading Kafka: The Definitive Guide by Neha Narkhede, Gwen Shapira, and Todd Palino
I’ve reached the Connector Example: MySQL to Elasticsearch chapter (p.146) and i'm
following the instructions to create a pipeline from a MySql source to a Elasticsearch sink.
I have made some diversions from the instructions
I created the mysql and Elasticsearch connectors using mvn package instead of mvn build
I placed the said connectors in a folder named C:\Users\ROY\confluent-6.0.0\share\kafka
tougher with some other connectors I’ve downloaded. I set the plugin.path vaiable in connect-distributed.properties to:
plugin.path=C://Users//ROY//confluent-6.0.0//share//kafka,/mnt/c/Users/ROY/confluent-6.0.0/share/kafka
I'm using mysql and Elasticsearch as docker containers.
The mysql connector works fine and reads data into a topic,
but when I try to create the Elasticsearch connector I get the following error:
...
(io.confluent.connect._Elasticsearch_._Elasticsearch_SinkConnectorConfig:354)
[2020-10-16 12:22:27,170] ERROR WorkerSinkTask{id=elastic-login-connector-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:187)
java.lang.NoClassDefFoundError: io/searchbox/action/Action
at io.confluent.connect._Elasticsearch_._Elasticsearch_SinkTask.start(_Elasticsearch_SinkTask.java:74)
at io.confluent.connect._Elasticsearch_._Elasticsearch_SinkTask.start(_Elasticsearch_SinkTask.java:48)
at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:302)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:193)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.ClassNotFoundException: io.searchbox.action.Action
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:104)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 11 more
...
Can someone assist resolving the issue and give and explanation to the error?
Thank you
Roy
Ok I found the problem. After I ran 'mvn package' on the elastic search connector source code I should have copied a whole directory into Kafka's /share dir, instead I only took one .jar file.
Related
I'm new to podman, and I just trying to run containers on it.
(podman version 3.4.0, installed by brew, intel Core MAC)
However, when I trying to run "podman run {image-name}", below errors were thrown.
$ podman run -ti -d --name web httpd 125
Error: error configuring network namespace for container b0e70d672cb66005833c0a300c8661b88eab49e942c240d69d17587e0b75c47b: error adding pod web_2_web_2 to CNI network "podman": unexpected end of JSON input
$ podman run centos:7
Error: error preparing container a6d0bc1ad217cd8207935561dc8ff7bd33672da3fa513917f9965cb39520c449 for attach: error configuring network namespace for container a6d0bc1ad217cd8207935561dc8ff7bd33672da3fa513917f9965cb39520c449: error adding pod quirky_snyder_quirky_snyder to CNI network "podman": unexpected end of JSON input
By reading https://issueexplorer.com/issue/containers/podman/11452, I removed ~/.docker/, but the solution doesn't work in my case.
Of course, the error message says there was "unexpected end of JSON input", but I don't know how to fix it. Could anyone guess why podman didn't work even running these base images, or how to debug it?
Thanks in advance.
on macos, current machine version 3.3.1 has this problem. I had this problem on server version 3.3.1 and I do not encounter it on server version 3.4.0. You can check server version with podman version.
Try removing current machine and installing a newer one
podman machine stop
podman machine rm
podman machine init --image-path next
podman machine start
Check server version again with podman version.
Try running your image again.
I tried to Creating and Deploying Oracle Cloud Functions by following the official documentation instructions. I can create and deploy using java runtime but when I deploy go runtime always return error.
I tried to init Go function using this command in Oracle Cloud Shell:
fn init --runtime go hello-go
then I tried to deploy it
fn -v deploy --app test
but it returned error like below:
Deploying hello-go to app: test
Bumped to version 0.0.7
Building image bom.ocir.io/bmptwl2psusa/repo/hello-go:0.0.7
FN_REGISTRY: bom.ocir.io/bmptwl2psusa/repo
Current Context: ap-mumbai-1
Sending build context to Docker daemon 5.632kB
Step 1/10 : FROM fnproject/go:dev as build-stage
---> 96c8fb94a8e1
Step 2/10 : WORKDIR /function
---> Using cache
---> 8961dd299ec1
Step 3/10 : WORKDIR /go/src/func/
---> Using cache
---> 5a4c2c6e13f1
Step 4/10 : ENV GO111MODULE=on
---> Using cache
---> 22022ff2fcf8
Step 5/10 : COPY . .
---> 714622a6ff03
Step 6/10 : RUN cd /go/src/func/ && go build -o func
---> Running in 39fedbc476f4
build func: cannot find module for path github.com/fnproject/fdk-go
The command '/bin/sh -c cd /go/src/func/ && go build -o func' returned a non-zero code: 1
Fn: error running docker build: exit status 1
When I'm using java runtime with fn init --runtime java hello-java command, it's successfully deployed, Why always fail when using go?
I tried to run go build -o func in hello-go directory but it's returned:
go: finding module for package github.com/fnproject/fdk-go
go: writing stat cache: mkdir /usr/share/gocode/pkg: permission denied
go: downloading github.com/fnproject/fdk-go v0.0.3
func.go:10:2: mkdir /usr/share/gocode/pkg: permission denied
I know it happened because /usr/share/gocode/ directory is under root user, but I dont know how to change the permission on that folder because Oracle Cloud Shell can not use root user or sudo. (based on this answer)
Maybe I can do it if I use real VM shell or local shell/terminal, but I want to use Oracle Cloud Shell because I just followed official instructions that suggested me using Oracle Cloud Shell, so how to deploy Oracle Cloud Functions with Go runtime using Oracle Cloud Shell?
Mostly the official documentations only give the examples using Java runtime, that make me paranoid when using go.
This is a bug in cloudshell that we are figuring out the best way to solve.
As a short-term workaround you can do this once:
mkdir ${HOME}/gopath
Then set this in your terminal:
export GOPATH=${HOME}/gopath
You should probably edit your ~/.bashrc to set the GOPATH variable automatically so you don't forget
I am trying to perform an sqoop export on HDP sandbox 2.1 via Oozie. When I run the Oozie job I get the following java runtime exception.
'>>> Invoking Sqoop command line now >>>
7598 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR
has not been set in the environment. Cannot check for additional
configuration.
7714 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version:
1.4.4.2.1.1.0-385
7760 [main] WARN org.apache.sqoop.SqoopOptions - Character argument
'\t' has multiple characters; only the first will be used.
7791 [main] WARN org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has
not been set in the environment. Cannot check for additional
configuration.
7904 [main] INFO org.apache.sqoop.manager.MySQLManager - Preparing
to use a MySQL streaming resultset.
7905 [main] INFO org.apache.sqoop.tool.CodeGenTool - Beginning code
generation
7946 [main] ERROR org.apache.sqoop.Sqoop - Got exception running
Sqoop: java.lang.RuntimeException: Could not load db driver class:
com.mysql.jdbc.Driver Intercepting System.exit(1)
I have copied jdbc driver file "mysql-connector-java.jar" to Oozie's shared library folder which I believe is "/usr/lib/oozie/share/lib/sqoop/". I have restarted my sandbox and tried to perform the export with Oozie again and I still get the same error.
The export works perfectly fine when I try performing it only via sqoop, so I presume Oozie needs its own set of drivers.
My question is, which Oozie directory am I suppose to copy my jdbc drivers to?
If you guys think I'm doing something wrong or you need further information, please let me know.
Thank you for your time.
Normally for Oozie the sharelib directory is /user/oozie/share/lib/ on HDFS where "oozie" would be the name of the user which is used to start the Oozie Server. I don't know what that is in case of HDP sandbox 2.1 , but you can use ps command to figure that out.
And for jars needed for sqoop action, I think you should copy the jar to /user/oozie/share/lib/sqoop/ folder.
I'm trying to run my jruby application on trinidad.
Following the readme, I included the trinidad gem in the gemfile and ran
rails server -e production -p 80 -d trinidad
When I access my server with a browser, it gives me an error message about how it can't find the correct class for my database driver, which is in lib/odbc6.jar and works fine if I run the project using WEBrick. The error message looks like this:
org.jruby.rack.RackInitializationException:
load error: /home/bdares/project/config/environment
java.sql.SQLException: No suitable driver found for jdbc:oracle:thin:#(my_jdbc_connection_string)
What am I doing wrong?
Plz try to setup $CLASSPATH global var.
Here are few more details: https://stackoverflow.com/a/11130246/255633
I would like to use springMVC and JPA (using hibernate) on my tomcat 7 server (running locally on my Mac).
I was able to set up everything successfully with an embedded H2 database.
Now I switched to mysql and am getting the following error
java.lang.ClassNotFoundException: "com.mysql.jdbc.Driver"
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1711)
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1556)
This suggests tomcat is having trouble finding the mysql-connector for java.
There are a multitude of tutorials on how to add the connector to $CATALINA_HOME/lib.
After trying to use a maven dependency for my project, I followed the advice and copied the .jar file into the lib directory:
$ ls $CATALINA_HOME/lib/mysql*.jar
/Users/david/Applications/tomcat/lib/mysql-connector-java-5.1.20-bin.jar
I have read and execute permissions on the directory and file.
At the moment I can't figure out how to make tomcat aware of the jar. The folder is included in
$CATALINA_HOME/conf/catalina.properties
and I have restarted the server multiple times.
Thanks for you help.
Normally, the output of a CNFE is as follows:
java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
whereas your error message shows
java.lang.ClassNotFoundException: "com.mysql.jdbc.Driver"
I think you've still got quotes around the driver class name in your config.