My Topic Schema looks like this,
{
"schema": {
"type": "string",
"optional": false
},
"payload": "{...}"
}
I am trying to sink this into an Influx DB instance with this configuration,
curl -X PUT \
-H "Content-Type: application/json" \
--data '{
"connector.class":"io.confluent.influxdb.InfluxDBSinkConnector",
"tasks.max": "1",
"topics" : "mongo-sink-json.tractor.job",
"influxdb.url": "http://10.100.87.169:8086",
"influxdb.db": "mongo-sink",
"infuxdb.password": "password",
"influxdb.username": "user",
"measurement.name.format": "${topic}",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "true"
}' \
But I get this error in Kafka Connect logs,
ERROR
WorkerSinkTask{id=influx_sink_json_job-0} Task threw an uncaught and
unrecoverable exception. Task is being killed and will not recover
until manually restarted. Error: java.util.ArrayList cannot be cast to
java.util.Map (org.apache.kafka.connect.runtime.WorkerSinkTask)
java.lang.ClassCastException: java.util.ArrayList cannot be cast to
java.util.Map at
io.confluent.influxdb.sink.writer.InfluxDBWriter.extractPointTags(InfluxDBWriter.java:288)
at
io.confluent.influxdb.sink.writer.InfluxDBWriter.getPointTags(InfluxDBWriter.java:266)
at
io.confluent.influxdb.sink.writer.InfluxDBWriter.getKey(InfluxDBWriter.java:329)
at
io.confluent.influxdb.sink.writer.InfluxDBWriter.getBatch(InfluxDBWriter.java:149)
at
io.confluent.influxdb.sink.writer.InfluxDBWriter.write(InfluxDBWriter.java:126)
at
io.confluent.influxdb.sink.InfluxDBSinkTask.put(InfluxDBSinkTask.java:40)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:546)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:326)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:228)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:196)
at
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)
at
org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) [2020-09-25 20:46:53,519]
ERROR WorkerSinkTask{id=influx_sink_json_job-0} Task threw an uncaught
and unrecoverable exception
(org.apache.kafka.connect.runtime.WorkerTask)
org.apache.kafka.connect.errors.ConnectException: Exiting
WorkerSinkTask due to unrecoverable exception. at
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:568)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:326)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:228)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:196)
at
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)
at
org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by:
java.lang.ClassCastException: java.util.ArrayList cannot be cast to
java.util.Map at
io.confluent.influxdb.sink.writer.InfluxDBWriter.extractPointTags(InfluxDBWriter.java:288)
at
io.confluent.influxdb.sink.writer.InfluxDBWriter.getPointTags(InfluxDBWriter.java:266)
at
io.confluent.influxdb.sink.writer.InfluxDBWriter.getKey(InfluxDBWriter.java:329)
at
io.confluent.influxdb.sink.writer.InfluxDBWriter.getBatch(InfluxDBWriter.java:149)
at
io.confluent.influxdb.sink.writer.InfluxDBWriter.write(InfluxDBWriter.java:126)
at
io.confluent.influxdb.sink.InfluxDBSinkTask.put(InfluxDBSinkTask.java:40)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:546)
Anyone has any ideas?
I used the similar config to sink the same topic in MongoDB and it works perfectly.
Should I use any other influxdb sink connector or fall onto using avro serialization or this is unrelated?
Related
I was trying to configure kafka sink connector to mysql DB. Kafka topic has value in AVRO format, and i want to dump data to mysql. I was getting error saying table not found (Table 'airflow.mytopic' doesn't exist). I was expecting table to be created in 'myschema.mytopic', but it was looking for table in airflow. I had enabled "auto.create": "true" expecting the table to be created wherever it wants.
I am using Confluent Kafka 5.4.1 and started it manually
Configuration:
"topics": "mytopic",
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"connection.url": "jdbc:mysql://<mysqlDB>:3306/myschema",
"connection.user": "db_user",
"connection.password": "db_pwd",
"tasks.max": "1",
"auto.evolve": "true",
"auto.create": "true",
"transforms": "routeRecords",
"transforms.routeRecords.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.routeRecords.replacement": "$1",
"transforms.routeRecords.regex": "(.*)",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://localhost:8084",
"connection.attempts": "1",
"dialect.name": "MySqlDatabaseDialect",
"table.name.format": "myschema.mytopic"
Error stack:
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:561)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:322)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.connect.errors.ConnectException: java.sql.SQLException: Exception chain:
java.sql.SQLSyntaxErrorException: Table 'airflow.mytopic' doesn't exist
at io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:122)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:539)
... 10 more
Caused by: java.sql.SQLException: Exception chain:
java.sql.SQLSyntaxErrorException: Table 'airflow.mytopic' doesn't exist
at io.confluent.connect.jdbc.sink.JdbcSinkTask.getAllMessagesException(JdbcSinkTask.java:150)
at io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:102)
... 11 more
Any clue what could be the reason for error?
Issue got resolved by downgrading the mysql driver (mysql-connector-java-5.1.17.jar), below are the configurations
"topics": "mytopic",
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"connection.url": "jdbc:mysql://<mysqlDB>:3306/myschema",
"connection.user": "db_user",
"connection.password": "db_pwd",
"tasks.max": "1",
"insert.mode": "insert",
"auto.evolve": "true",
"auto.create": "true",
"transforms": "unwrap",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
"transforms.unwrap.drop.tombstones": "false",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://localhost:8084",
"connection.attempts": "1",
"dialect.name": "MySqlDatabaseDialect",
"table.name.format": "myschema.mytopic"
I'm trying to access my MySQL container from another Spring Boot application container, which are both in the same network.
I've created a network and linked both containers to it :
docker network create --driver bridge dmw_network
Details of my network :
[
{
"Name": "dmw_network",
"Id": "2e204b7b48efafc5db0daccb801a3e38a0f1b99a6a745dd9b58a74172e966da3",
"Created": "2018-09-25T17:28:35.530261782+02:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.19.0.0/16",
"Gateway": "172.19.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"28b1368d9256f0b323b528507ea9133ba410c691fbd01029ee3913eacfa94660": {
"Name": "dmwapp",
"EndpointID": "f59f82bf0e62fb97fa7b1b4e0c3750adffaf570a93de633938de32a199ba03ab",
"MacAddress": "02:42:ac:13:00:03",
"IPv4Address": "172.19.0.3/16",
"IPv6Address": ""
},
"c6ed1cbdd376714c90c5e67024cc576030b030e29a9a8d766bb4bb637025dd08": {
"Name": "mysql5",
"EndpointID": "0ea9594ebb4251d5f13c05f407e660c2ed1b5e4ccaed2d71cedc610984b9b076",
"MacAddress": "02:42:ac:13:00:02",
"IPv4Address": "172.19.0.2/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]
Here is my application.yml file :
spring:
application:
name : ${appName:tasky}
datasource:
url: jdbc:mysql://mysql5:3406/dmw_db?autoReconnect=true&useSSL=false
username: user
password: pass
driver-class-name: com.mysql.jdbc.Driver
tomcat:
max-active: 50
max-idle: 50
And this is how I run both my containers :
docker run --name=mysql5 --network=dmw_network -p 3406:3306 -v $(pwd)/mysql:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=pass -d dmw_mysql5
docker run --name dmwapp --network=dmw_network -d -p 8888:8888 -p 9999:9999 -v $(pwd)/conf:/home/conf/config/prod dmwimg
Yet I'm still unable to access my database by hostname from my other spring boot container in the same network.
Here is the error :
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.Util.getInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:919)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:898)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:887)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:861)
at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2096)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2021)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:776)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:319)
at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:212)
at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:736)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:668)
at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:483)
at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:154)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:118)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:107)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at net.bull.javamelody.JdbcWrapper$3.invoke(JdbcWrapper.java:781)
at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:294)
at com.sun.proxy.$Proxy139.getConnection(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216)
... 117 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:342)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2188)
at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2037)
... 148 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:301)
... 150 more
When I'm switching to the IP address of the server hosting the docker containers then it works. For example :
url: jdbc:mysql://184.4.x.x:3406/dmw_db?autoReconnect=true&useSSL=false
What am I doing wrong ?
Change port form 3406 to proper one.
url: jdbc:mysql://mysql5:3306/dmw_db?autoReconnect=true&useSSL=false
If you are connecting using docker name you should use internal port not exposed one.
dmwapp doesn't know anything about an alias mysql5. You may link them during running:
docker run --name dmwapp --network=dmw_network -d -p 8888:8888 -p 9999:9999 -v $(pwd)/conf:/home/conf/config/prod --link mysql5:mysql5 dmwimg
I use Cepheus GE for my use case and I enabled it in the multi tenant mode and uploaded my config.json file to it, But when I start to send updates to the Cepheus broker in order to forward them to the CEP, the Cepheus broker just receives the updates but doesn't forward them to the Cepheus cap as it couldn't recognize the service and service-path that are set in the config.json. And when I tried to send my updates directly to the Cepheus-CEP, it accepted them and processed them successfully. So I wonder why the Cepheus broker cannot recognize the Fiware-service when it's enabled in the multi tenant mode.
the config file service definition is as follows
"brokers":[
{
"url":"http://XXX.XX.XX.XX:1026",
"serviceName": "f",
"servicePath": "/f",
"authToken": "XXX"
}
]
this is the Logs of Cepheus broker.
2017-09-02 08:55:32,546 [/O dispatcher 1] WARN c.o.c.b.c.NgsiController - NotifyContext failed for http://localhost:8080/ngsi10/notifyContext$
2017-09-02 09:05:33,358 [nio-8081-exec-1] WARN c.o.c.b.c.NgsiController - UpdateContext failed for http://localhost:8082: Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_72-internal]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_72-internal]
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:173) ~[httpcore-nio-4.4.1.$
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:147) ~[httpcore-nio-4.4.1$
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:350) ~[httpcore-nio-4.4.1.j$
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191) ~[httpasync$
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.jar!$
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_72-internal]
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_72-internal]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_72-internal]
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:173) ~[httpcore-nio-4.4.1.$
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:147) ~[httpcore-nio-4.4.1$
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:350) ~[httpcore-nio-4.4.1.j$
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191) ~[httpasync$
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.jar!$
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_72-internal]
2017-09-02 09:33:12,316 [pool-2-thread-1] WARN c.o.c.b.c.NgsiController - UpdateContext failed for http://localhost:8082: Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_72-internal]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_72-internal]
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:173) [httpcore-nio-4.4.1.j$
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:147) [httpcore-nio-4.4.1.$
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:350) ~[httpcore-nio-4.4.1.j$
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191) ~[httpasync$
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.jar!$
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_72-internal]
And the script file that sends the updates is as follows
(curl XXX.XXX.XXX.XXX:8081/v1/updateContext/ -s -S --header 'Content-Type: application/json' --header "Fiware-Service: f" --header "Fiware-ServicePath:/f " --header 'Accept: application/json' -d #- | python -mjson.tool ) <<EOF
{ "contextElements": [
{
"type": "Lab",
"isPattern": "false",
"id": "Lab111",
"attributes": [
{
"name": "priority",
"type": "double",
"value": "1"
},
{
"name": "controller",
"type": "string",
"value": "Controller111"
}
]
}
],
"updateAction": "UPDATE"
}
EOF
Now please I want to know where the problem could be?
As of today, the Fiware-Cepheus broker does not support multi-tenant requests (using Fiware-Service and Fiware-ServicePath headers), only the CEP can handle multi-tenancy. More generally, the broker has very few features compared to a full-fledge broker like Orion.
If you need a multi-tenant broker, use the Orion Context Broker: https://fiware-orion.readthedocs.io/en/master/user/multitenancy/index.html
I'm working with an Orion context broker version 1.2.0. I have subscribed in it two different cygnus (0.11 and 0.13) using NGSIv2, as follows:
(curl 172.21.0.23:1026/v2/subscriptions -s -S --header 'Fiware-Service: prueba_015_adapter' --header 'Fiware-ServicePath: /Prueba/Planta_3' --header 'Content-Type: application/json' -d #- ) <<EOF
{
"description": "Cygnus subscription",
"subject": {
"entities": [
{
"idPattern": ".*",
"type": "density_algorithm"
}
],
"condition": {
"attrs": []
}
},
"notification": {
"http": {
"url": "http://172.21.0.33:5050/notify"
},
"attrs": []
}
}
EOF
But when the context broker sends a notification to any of these cygnus modules, the next error appears in the log:
15 jun 2016 12:46:48,641 INFO [1469152682#qtp-857344131-3153] (com.telefonica.iot.cygnus.handlers.OrionRestHandler.getEvents:150) - Starting transaction (1463998603-759-0001644173) 15 jun 2016 12:46:48,641 INFO [1469152682#qtp-857344131-3153] (com.telefonica.iot.cygnus.handlers.OrionRestHandler.getEvents:232) - Received data ({"subscriptionId":"57612ed9efa20b5b23e71bd5","data":[{"id":"C-A2","type":"density_algorithm","densityPlan":{"type":"string","value":"C-A2","metadata":{}},"devices":{"type":"string","value":"43","metadata":{}},"timestamp":{"type":"string","value":"2016-06-15T12:53:26.294+02:00","metadata":{}}}]}) 15 jun 2016 12:46:48,641 INFO [1469152682#qtp-857344131-3153] (com.telefonica.iot.cygnus.handlers.OrionRestHandler.getEvents:255) - Event put in the channel (id=957931298, ttl=-1) 15 jun 2016 12:46:48,642 WARN [1469152682#qtp-857344131-3153] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.intercept:289)
- No context responses within the notified entity, nothing is done 15 jun 2016 12:46:48,642 WARN [1469152682#qtp-857344131-3153] (org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost:203)
- Error appending event to channel. Channel might be full. Consider increasing the channel capacity or make sure the sinks perform faster. org.apache.flume.ChannelException: Unable to put batch on required channel: org.apache.flume.channel.MemoryChannel{name: mongo-channel}
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
at org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:201)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.lang.IllegalArgumentException: put() called with null event!
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:89)
at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
... 16 more
If I use NGSIv1 instead to register both subscriptions, everything goes fine: no log error is shown and the data is persisted into both cygnus modules.
(curl 172.21.0.23:1026/v1/subscribeContext -s -S --header 'Fiware-Service: prueba_015_adapter' --header 'Fiware-ServicePath: /Prueba/Planta_3' --header 'Content-Type: application/json' --header 'Accept: application/json' -d #- ) <<EOF
{
"entities": [
{
"type": "density_algorithm",
"isPattern": "true",
"id": ".*"
}
],
"attributes": [],
"reference": "http://172.21.0.33:5050/notify",
"duration": "P1M",
"notifyConditions": [
{
"type": "ONCHANGE",
"condValues": []
}
]
}
EOF
I'm sending the entities to the context broker using NGSIv1. Can the problem be due to an incompatibility between NGSIv1 and NGSIv2?
Thanks in advance
For the time being, NGSIv2 notifications are not supported in Cygnus. It is expected to be implemented, but it has not been scheduled yet.
However, you can use attrFormat (inside nofitication field) equal to legacy to use NGSIv1 notification format (have a look to more detailed information here). NGSIv1 notification format is fully supported by Cygnus, so that should work.
Hi i want to write a river for my MSSQL 2008 R2 Server. I have copied the MSSQL Driver and the jdbc jar into the bin/plugins/jdbc directory. I have tested the MSSQL driver in Solr and it works fine. So the connection string is correct.
When i'm starting the ES server i get this exceprions:
org.elasticsearch.common.settings.NoClassSettingsException: Failed to load class with value [jdbc]
at org.elasticsearch.river.RiverModule.loadTypeModule(RiverModule.java:87)
at org.elasticsearch.river.RiverModule.spawnModules(RiverModule.java:58)
at org.elasticsearch.common.inject.ModulesBuilder.add(ModulesBuilder.java:44)
at org.elasticsearch.river.RiversService.createRiver(RiversService.java:135)
at org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:270)
at org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:264)
at org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run(TransportAction.java:93)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.ClassNotFoundException: jdbc
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.elasticsearch.river.RiverModule.loadTypeModule(RiverModule.java:73)
... 9 more
My curl looks like that:
curl -XPUT http://localhost:9200/_river/documents/_meta -d '
{
"type" : "jdbc",
"jdbc" : {
"driver" : "com.microsoft.sqlserver.jdbc.SQLServerDriver",
"url" : "jdbc:sqlserver://CPCDKD3;databaseName=Sampledocs",
"user" : "XXXX",
"password" : "XXXX",
"sql" : "SELECT * from Docs"
},
"index" : {
"index" : "documents",
"type" : "FieldId",
"type" : "FieldTitle",
"type" : "FieldCOntent"
}
}'
ok SOLVED. The plugin must be placed into $ES_Home\plugins not into $ES_Home\bin\plugins