I am following this tutorial to setup spring boot project with docker-compose. Here is what I did:
My application.properties:
spring.datasource.url = jdbc:mysql://mysql-demo-container:3306/demo_db?useSSL=false
spring.datasource.username = root
spring.datasource.password =root
spring.datasource.tomcat.testWhileIdle = true
spring.datasource.tomcat.timeBetweenEvictionRunsMillis = 60000
spring.datasource.tomcat.validationQuery = SELECT 1
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect
# Hibernate ddl auto (create, create-drop, validate, update)
spring.jpa.hibernate.ddl-auto = update
My Dockerfile:
`FROM java:8
LABEL maintainer=“foo.bar#gmail.com”
VOLUME /tmp
EXPOSE 8080
ADD target/spring-boot-data-jpa-example-0.0.1-SNAPSHOT.jar spring-boot-data-jpa-example-0.0.1-SNAPSHOT.jar
ENTRYPOINT ["java","-jar","spring-boot-data-jpa-example-0.0.1-SNAPSHOT.jar"]`
I have this docker-compose.yml:
version: '3'
services:
mysql-demo-container:
image: mysql:latest
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_DATABASE=demo_db
- MYSQL_PASSWORD=myPass
ports:
- 2012:3306
volumes:
- /data/mysql
spring-boot-jpa-app:
image: spring-boot-jpa-image
build:
context: ./
dockerfile: Dockerfile
depends_on:
- mysql-demo-container
ports:
- 8087:8080
volumes:
- /data/spring-boot-app
I firstly mvn clean install to build the project jar file, then I run docker-compose up to build images & hopefully run containers.
The logs:
$ docker-compose up
Creating network "spring-boot-data-jpa-example-master_default" with the default driver
Creating spring-boot-data-jpa-example-master_mysql-demo-container_1 ... done
Creating spring-boot-data-jpa-example-master_spring-boot-jpa-app_1 ... done
Attaching to spring-boot-data-jpa-example-master_mysql-demo-container_1, spring-boot-data-jpa-example-master_spring-boot-jpa-app_1
mysql-demo-container_1 | Initializing database
mysql-demo-container_1 | 2019-07-02T13:03:05.097872Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
mysql-demo-container_1 | 2019-07-02T13:03:05.098037Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.16) initializing of server in progress as process 29
mysql-demo-container_1 | 2019-07-02T13:03:09.171833Z 5 [Warning] [MY-010453] [Server] root#localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
spring-boot-jpa-app_1 |
spring-boot-jpa-app_1 | . ____ _ __ _ _
spring-boot-jpa-app_1 | /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
spring-boot-jpa-app_1 | ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
spring-boot-jpa-app_1 | \\/ ___)| |_)| | | | | || (_| | ) ) ) )
spring-boot-jpa-app_1 | ' |____| .__|_| |_|_| |_\__, | / / / /
spring-boot-jpa-app_1 | =========|_|==============|___/=/_/_/_/
spring-boot-jpa-app_1 | :: Spring Boot :: (v1.5.9.RELEASE)
spring-boot-jpa-app_1 |
spring-boot-jpa-app_1 | 2019-07-02 13:03:11.516 INFO 1 --- [ main] .s.e.SpringBootDataJpaExampleApplication : Starting SpringBootDataJpaExampleApplication v0.0.1-SNAPSHOT on 3c0d94b76fd6 with PID 1 (/spring-boot-data-jpa-example-0.0.1-SNAPSHOT.jar started by root in /)
spring-boot-jpa-app_1 | 2019-07-02 13:03:11.556 INFO 1 --- [ main] .s.e.SpringBootDataJpaExampleApplication : No active profile set, falling back to default profiles: default
spring-boot-jpa-app_1 | 2019-07-02 13:03:11.979 INFO 1 --- [ main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#6d5380c2: startup date [Tue Jul 02 13:03:11 UTC 2019]; root of context hierarchy
mysql-demo-container_1 | 2019-07-02T13:03:13.525348Z 0 [System] [MY-013170] [Server] /usr/sbin/mysqld (mysqld 8.0.16) initializing of server has completed
mysql-demo-container_1 | Database initialized
mysql-demo-container_1 | MySQL init process in progress...
mysql-demo-container_1 | MySQL init process in progress...
mysql-demo-container_1 | MySQL init process in progress...
mysql-demo-container_1 | 2019-07-02T13:03:15.636380Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
mysql-demo-container_1 | 2019-07-02T13:03:15.636539Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.16) starting as process 80
mysql-demo-container_1 | 2019-07-02T13:03:18.070695Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
mysql-demo-container_1 | 2019-07-02T13:03:18.081045Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
mysql-demo-container_1 | 2019-07-02T13:03:18.152808Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.16' socket: '/var/run/mysqld/mysqld.sock' port: 0 MySQL Community Server - GPL.
mysql-demo-container_1 | 2019-07-02T13:03:18.311442Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/run/mysqld/mysqlx.sock'
spring-boot-jpa-app_1 | 2019-07-02 13:03:20.039 INFO 1 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat initialized with port(s): 8080 (http)
spring-boot-jpa-app_1 | 2019-07-02 13:03:20.221 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
spring-boot-jpa-app_1 | 2019-07-02 13:03:20.232 INFO 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet Engine: Apache Tomcat/8.5.23
spring-boot-jpa-app_1 | 2019-07-02 13:03:21.210 INFO 1 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
spring-boot-jpa-app_1 | 2019-07-02 13:03:21.213 INFO 1 --- [ost-startStop-1] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 9251 ms
spring-boot-jpa-app_1 | 2019-07-02 13:03:22.153 INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.ServletRegistrationBean : Mapping servlet: 'dispatcherServlet' to [/]
spring-boot-jpa-app_1 | 2019-07-02 13:03:22.169 INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'characterEncodingFilter' to: [/*]
spring-boot-jpa-app_1 | 2019-07-02 13:03:22.178 INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
spring-boot-jpa-app_1 | 2019-07-02 13:03:22.178 INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'httpPutFormContentFilter' to: [/*]
spring-boot-jpa-app_1 | 2019-07-02 13:03:22.181 INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'requestContextFilter' to: [/*]
mysql-demo-container_1 | Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
mysql-demo-container_1 | Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
spring-boot-jpa-app_1 | 2019-07-02 13:03:25.161 ERROR 1 --- [ main] o.a.tomcat.jdbc.pool.ConnectionPool : Unable to create initial connections of pool.
spring-boot-jpa-app_1 |
spring-boot-jpa-app_1 | com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
spring-boot-jpa-app_1 |
spring-boot-jpa-app_1 | The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
spring-boot-jpa-app_1 | at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_111]
spring-boot-jpa-app_1 | at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_111]
spring-boot-jpa-app_1 | at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_111]
spring-boot-jpa-app_1 | at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_111]
spring-boot-jpa-app_1 | at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) ~[mysql-connector-java-5.1.44.jar!/:5.1.44]
spring-boot-jpa-app_1 | at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989) ~[mysql-connector-java-5.1.44.jar!/:5.1.44]
spring-boot-jpa-app_1 | at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:341) ~[mysql-connector-java-5.1.44.jar!/:5.1.44]
spring-boot-jpa-app_1 | at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2189) ~[mysql-connector-java-5.1.44.jar!/:5.1.44]
spring-boot-jpa-app_1 | at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2222) ~[mysql-connector-java-5.1.44.jar!/:5.1.44]
spring-boot-jpa-app_1 | at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2017) ~[mysql-connector-java-5.1.44.jar!/:5.1.44]
spring-boot-jpa-app_1 | at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779) ~[mysql-connector-java-5.1.44.jar!/:5.1.44]
spring-boot-jpa-app_1 | at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) ~[mysql-connector-java-5.1.44.jar!/:5.1.44]
spring-boot-jpa-app_1 | at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_111]
spring-boot-jpa-app_1 | at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_111]
spring-boot-jpa-app_1 | at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_111]
spring-boot-jpa-app_1 | at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_111]
spring-boot-jpa-app_1 | at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) ~[mysql-connector-java-5.1.44.jar!/:5.1.44]
spring-boot-jpa-app_1 | at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389) ~[mysql-connector-java-5.1.44.jar!/:5.1.44]
spring-boot-jpa-app_1 | at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330) ~[mysql-connector-java-5.1.44.jar!/:5.1.44]
spring-boot-jpa-app_1 | at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:310) ~[tomcat-jdbc-8.5.23.jar!/:na]
spring-boot-jpa-app_1 | at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:203) ~[tomcat-jdbc-8.5.23.jar!/:na]
spring-boot-jpa-app_1 | at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:735) [tomcat-jdbc-8.5.23.jar!/:na]
spring-boot-jpa-app_1 | at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:667) [tomcat-jdbc-8.5.23.jar!/:na]
spring-boot-jpa-app_1 | at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:482) [tomcat-jdbc-8.5.23.jar!/:na]
spring-boot-jpa-app_1 | at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:154) [tomcat-jdbc-8.5.23.jar!/:na]
spring-boot-jpa-app_1 | at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:118) [tomcat-jdbc-8.5.23.jar!/:na]
spring-boot-jpa-app_1 | at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:107) [tomcat-jdbc-8.5.23.jar!/:na]
spring-boot-jpa-app_1 | at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131) [tomcat-jdbc-8.5.23.jar!/:na]
spring-boot-jpa-app_1 | at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
spring-boot-jpa-app_1 | at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
spring-boot-jpa-app_1 | at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:326) [spring-jdbc-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
spring-boot-jpa-app_1 | at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:366) [spring-jdbc-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
spring-boot-jpa-app_1 | at org.springframework.boot.autoconfigure.orm.jpa.DatabaseLookup.getDatabase(DatabaseLookup.java:72) [spring-boot-autoconfigure-1.5.9.RELEASE.jar!/:1.5.9.RELEASE]
spring-boot-jpa-app_1 | at org.springframework.boot.autoconfigure.orm.jpa.JpaProperties.determineDatabase(JpaProperties.java:139) [spring-boot-autoconfigure-1.5.9.
As you can see from the beginning of the log, the containers seem created successfully.
Run docker ps shows me:
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8fa64acb363d mysql:latest "docker-entrypoint.s…" 30 minutes ago Up 30 minutes 33060/tcp, 0.0.0.0:2012->3306/tcp spring-boot-data-jpa-example-master_mysql-demo-container_1
What could be the reason for the error?
You should be aware of this very important footnote in the reference documentation of Docker Compose:
depends_on does not wait for db and redis to be “ready” before starting web - only until they have been started. If you need to wait for a service to be ready, see Controlling startup order for more on this problem and strategies for solving it.
In your case, your Spring boot container starts right after your MySQL container is being started. There's no guarantee though that MySQL will be able to take connections when Spring boot is setting up its connection pool.
This means that your Spring boot connection will fail to startup, because MySQL isn't ready yet. This results in your application container to be killed.
The solution to this problem is mentioned within the documentation I quoted. Namely, you have to use some kind of polling mechanism to see if you can connect to your database or not.
If you don't want to use tools like wait-for-it or wait-for, you could write a simple Shell script that does something like this:
while ! exec 6<>/dev/tcp/mysql-demo-container/3306; do
echo "Trying to connect to MySQL..."
sleep 10
done
exec java -jar spring-boot-data-jpa-example-0.0.1-SNAPSHOT.jar
This also means you have to change the ENTRYPOINT within your Dockerfile to refer to such shell script rather than directly running your JAR file.
I've updated cygnus from version 0.13 to 1.7.0 by installing NGSI following this tutorial:
https://github.com/telefonicaid/fiware-cygnus/tree/master/cygnus-ngsi
Error the subscription
[
{
"id": "59d38a92dbaa1e477aef9c00",
"description": "A subscription to get info about pruebas",
"status": "failed",
"subject": {
"entities": [
{
"id": "pruebas",
"type": "pruebas"
}
],
"condition": {
"attrs": [
"pressure"
]
}
},
"notification": {
"timesSent": 2,
"lastNotification": "2017-10-03T13:03:43.00Z",
"attrs": [
"temperature",
"pressure"
],
"attrsFormat": "legacy",
"http": {
"url": "http://localhost:5050/notify"
},
"lastFailure": "2017-10-03T13:03:43.00Z"
}
}
]
viewing the contextBroker log gives the following:
$pp[328]:notificationError | msg=Raising alarm NotificationError http://localhost:5050/notify: (curl_easy_perform failed: Couldn't connect to server)
I have contextBroker on the same machine as cygnus so I have already tried to change the notify ip for the server and for localhost and it does not work for any of it.
with version 0.13 if it works with localhost.
What could be the problem?
It does not even come to cygnus configuration files because it can not access from the contextBroker.
Greetings and thank you.
EDIT1:
I am tested with the fiwareLab machines and removing cygnus 0.13 that comes pre installed with YUM REMOVE CYGNUS. Then I installed 1.7 with YUM INSTALL CYGNUS-NGSI and installed two packages ngsi and common.
Restarting the service with service cygnus restart indicates the following:
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
cygnus-ngsi x86_64 1.7.1-0.g9df0d4d fiware 74 M
Installing for dependencies:
cygnus-common x86_64 1.7.1-0.g9df0d4d fiware 128 M
Transaction Summary
================================================================================
Install 2 Package(s)
Total size: 202 M
Installed size: 223 M
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
[INFO] Creating cygnus user
Installing : cygnus-common-1.7.1-0.g9df0d4d.x86_64 1/2
[INFO] Creating log directory
Done
Installing : cygnus-ngsi-1.7.1-0.g9df0d4d.x86_64 2/2
Verifying : cygnus-common-1.7.1-0.g9df0d4d.x86_64 1/2
Verifying : cygnus-ngsi-1.7.1-0.g9df0d4d.x86_64 2/2
Installed:
cygnus-ngsi.x86_64 0:1.7.1-0.g9df0d4d
Dependency Installed:
cygnus-common.x86_64 0:1.7.1-0.g9df0d4d
Complete!
[centos#centos6 cygnus]$ sudo service cygnus restart
There aren't any instance of Cygnus running [ OK ]
Starting Cygnus 1... [ OK ]
When I try on my server I do the same steps but when doing the service cygnus restart has two cygnus the 1 and 2 not as in vuesta machine that only has one and therefore indicates that the port 8081 is already in use.
Dependencias resueltas
============================================================================================================================================================================
Paquete Arquitectura Versión Repositorio Tamaño
============================================================================================================================================================================
Instalando:
cygnus-ngsi x86_64 1.7.1-0.g9df0d4d fiware 74 M
Instalando para las dependencias:
cygnus-common x86_64 1.7.1-0.g9df0d4d fiware 128 M
Resumen de la transacción
============================================================================================================================================================================
Instalar 2 Paquete(s)
Tamaño total de la descarga: 202 M
Tamaño instalado: 223 M
Está de acuerdo [s/N]:s
Descargando paquetes:
(1/2): cygnus-common_hadoopcore_1.2.1-1.7.1-0.g9df0d4d.x86_64.rpm | 128 MB 00:14
(2/2): cygnus-ngsi-1.7.1-0.g9df0d4d.x86_64.rpm | 74 MB 00:07
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 8.9 MB/s | 202 MB 00:22
Ejecutando el rpm_check_debug
Ejecutando prueba de transacción
La prueba de transacción ha sido exitosa
Ejecutando transacción
[INFO] Creating cygnus user
Instalando : cygnus-common-1.7.1-0.g9df0d4d.x86_64 1/2
[INFO] Creating log directory
Done
Instalando : cygnus-ngsi-1.7.1-0.g9df0d4d.x86_64 2/2
Verifying : cygnus-common-1.7.1-0.g9df0d4d.x86_64 1/2
Verifying : cygnus-ngsi-1.7.1-0.g9df0d4d.x86_64 2/2
Instalado:
cygnus-ngsi.x86_64 0:1.7.1-0.g9df0d4d
Dependencia(s) instalada(s):
cygnus-common.x86_64 0:1.7.1-0.g9df0d4d
¡Listo!
[root#UAL-IoF2020 conf]# ls
agent_1.conf agent_ngsi.conf.template cygnus_instance_2.conf grouping_rules_2.conf krb5_login.conf README-cygnus-common.md
agent_3.conf cartodb_keys.conf.template cygnus_instance.conf.template grouping_rules.conf.template log4j.properties README-cygnus-ngsi.md
agent.conf.template cygnus_instance_1.conf flume-env.sh.template krb5.conf.template name_mappings.conf.template
[root#UAL-IoF2020 conf]# service cygnus restart
There aren't any instance of Cygnus running [ OK ]
Starting Cygnus 1... [ OK ]
Starting Cygnus 2... [ OK ]
[root#UAL-IoF2020 conf]#
Is it possible that this is the problem and that is not recognizing my NGSI and this occupying the 8081 the common? or is this normal?
Log cygnus :
time=2017-10-03T21:51:09.326Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnusagent | op=main | msg=com.telefonica.iot.cygnus.nodes.CygnusApplication[301] : Starting a Jetty server listening on 0.0.0.0:8081 (Management Interface)
time=2017-10-03T21:51:09.381Z | lvl=WARN | corr= | trans= | srv= | subsrv= | comp=cygnusagent | op=warn | msg=org.mortbay.log.Slf4jLog[76] : failed SelectChannelConnector#0.0.0.0:8081: java.net.BindException: La dirección ya se está usando
time=2017-10-03T21:51:09.381Z | lvl=WARN | corr= | trans= | srv= | subsrv= | comp=cygnusagent | op=warn | msg=org.mortbay.log.Slf4jLog[76] : failed Server#52992ace: java.net.BindException: La dirección ya se está usando
time=2017-10-03T21:51:09.381Z | lvl=FATAL | corr= | trans= | srv= | subsrv= | comp=cygnusagent | op=run | msg=com.telefonica.iot.cygnus.http.JettyServer[90] : Fatal error running the Management Interface. Details=La dirección ya se está usando
EDIT2
I have already solved the problem of two cygnus, had two agent_1 and agent_2 created. I have deleted one of them and already performing service cygnus restart appears only one cygnus. We are getting better.
But I still have the same problem with the subscriptions:
The contextBroker log indicates:
msg = Raising alarm NotificationError http: // localhost: 5050 / notify: (curl_easy_perform failed: could not connect to server)
When I try:
[root # UAL-IoF2020 conf] # netstat -np | grep 5050
I do not think anything.
When I launch this:
[root # UAL-IoF2020 conf] # netstat -np | grep 1026
tcp 0 0 150.XXX.XXX.XXX:1026 XXX.XXX.XXX.XXX:50348 ESTABLISHED 5169 / contextBroker
I am trying to launch a Test of your page.
./notification-json-simple.sh http: // localhost: 5050 / notify myservice myservicepath
and gives me the following error:
[root # UAL-IoF2020 ngsi-examples] # ./notification-json-simple.sh http: // localhost: 5050 / notify myservice myservicepath
* About to connect () to localhost port 5050 (# 0)
* Trying :: 1 ... Connection refused
* Trying 127.0.0.1 ... Connection refused
* could not connect to host
* Closing connection # 0
curl: (7) could not connect to host
It gives the impression that in the 5050 I have nothing listening.
Any clue what that might be?
cygnus-ngsi.sources = http-source
cygnus-ngsi.sinks = mysql-sink
cygnus-ngsi.channels = mysql-channel
#=============================================
# source configuration
# channel name where to write the notification events
cygnus-ngsi.sources.http-source.channels = mysql-channel
# source class, must not be changed
cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming notifications
cygnus-ngsi.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
# URL target
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
# default service (service semantic depends on the persistence sink)
cygnus-ngsi.sources.http-source.handler.default_service = default
# default service path (service path semantic depends on the persistence sink)
cygnus-ngsi.sources.http-source.handler.default_service_path = /
# source interceptors, do not change
cygnus-ngsi.sources.http-source.interceptors = ts gi
# TimestampInterceptor, do not change
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
# GroupingInterceptor, do not change
cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
# Grouping rules for the GroupingInterceptor, put the right absolute path to the file if necessary
# see the doc/design/interceptors document for more details
cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
Do I have to install cygnus-common manual?
Reading the documentation (https://github.com/telefonicaid/fiware-cygnus/tree/master/cygnus-ngsi) it's wrote:
Cygnus NGSI is based on Apache Flume, which is used through cygnus-common and which Cygnus NGSI depends on.
I think you need to install cygnus-common.
I have already solved the problem.
It was in the configuration file of cygnus_instance_1.conf, you had to rename the cygnusagent agent by cygnus-ngsi.
For installation, I have simply followed these steps.
Install ContextBroker.
Install MongoDB (for contextBroker to
work).
Install cygnus-ngsi, this in turn installed automatically
cygnus-common.
Copy the agent_ngsi.conf.template and rename it with
agent_1.conf
Copy the cygnus_instance.conf.template to cygnus_instance_1.conf
Rename the agent from cygnus_instance_1.conf to cygnus-ngsi and the configuration file created above (agent_1.conf)
All this has been with Yum Install, with RPM, I have not had to install apache flume or anything, this way it does everything automatically.
I hope this helps and thanks.
The last error log you have posted is the key: there is another running process listening on TCP/5050 port. Most probably, a previous run of Cygnus not stopped/killed properly.
I just finished testing the entire thing on my virtual machine environment and now I am trying to launch it on the dedicated server. And now I ran into a completely new issue. First I confirmed that I have both context Broker and centos running (on 1026 and 5050 respectively):
[root#centos conf]# netstat -ntlpd
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:1026 0.0.0.0:* LISTEN 1321/contextBroker
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 1282/mongod
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 1791/mysqld
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1260/sshd
tcp 0 0 :::1026 :::* LISTEN 1321/contextBroker
tcp 0 0 :::8081 :::* LISTEN 2481/java
tcp 0 0 :::22 :::* LISTEN 1260/sshd
tcp 0 0 :::5050 :::* LISTEN 2481/java
[root#centos conf]# service cygnus status
Cygnus 1 status...
cygnus-flume-ng (pid 2481) is running...
Then I confirmed that I have data on contextBroker because this command gave me an appropriate response:
(curl localhost:1026/v1/queryContext -s -S --header 'Content-Type: application/json' \
--header 'Accept: application/json' -d #- | python -mjson.tool) <<EOF
{
"entities": [
{
"type": "Room",
"isPattern": "false",
"id": "Room1"
}
]
}
EOF
Following the workaround to an issue with root user and logging I fixed the log4j.properties and changed the follwing:
flume.log.dir=/var/log/cygnus
I then started cygnus and got the following log:
Starting an ordered shutdown of Cygnus
Stopping sources
Stopping http-source (lyfecycle state=START)
All the channels are empty
Stopping channels
Stopping mysql-channel (lyfecycle state=START)
Stopping sinks
Stopping mysql-sink (lyfecycle state=START)
Info: Sourcing environment configuration script /usr/cygnus/conf/flume-env.sh
Warning: JAVA_HOME is not set!
+ exec /usr/bin/java -Xmx20m -Dflume.log.file=cygnus.log -cp '/usr/cygnus/conf:/usr/cygnus/lib/*:/usr/cygnus/plugins.d/cygnus/lib/*:/usr/cygnus/plugins.d/cygnus/libext/*' -Djava.library.path= com.telefonica.iot.cygnus.nodes.CygnusApplic$
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/cygnus/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/cygnus/plugins.d/cygnus/lib/cygnus-0.11.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
17 Dec 2015 13:35:37,684 INFO [main] (com.telefonica.iot.cygnus.nodes.CygnusApplication.main:235) - Starting a Jetty server listening on port 8081 (Management Interface)
17 Dec 2015 13:35:37,700 INFO [main] (org.mortbay.log.Slf4jLog.info:67) - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
17 Dec 2015 13:35:37,700 INFO [main] (com.telefonica.iot.cygnus.nodes.CygnusApplication.main:238) - Starting Cygnus application
17 Dec 2015 13:35:37,700 INFO [Thread-1] (org.mortbay.log.Slf4jLog.info:67) - jetty-6.1.26
17 Dec 2015 13:35:37,713 INFO [lifecycleSupervisor-1-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start:61) - Configuration provider starting
17 Dec 2015 13:35:37,715 INFO [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133) - Reloading configuration file:/usr/cygnus/conf/agent_1.conf
17 Dec 2015 13:35:37,725 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
17 Dec 2015 13:35:37,725 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
17 Dec 2015 13:35:37,725 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
17 Dec 2015 13:35:37,755 INFO [Thread-1] (org.mortbay.log.Slf4jLog.info:67) - Started SocketConnector#0.0.0.0:8081
17 Dec 2015 13:35:37,764 WARN [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid:319) - Agent configuration for 'cygunsagent' does not contain any channels. Marking it as invalid.
17 Dec 2015 13:35:37,765 WARN [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:127) - Agent configuration invalid for agent 'cygunsagent'. It will be removed.
17 Dec 2015 13:35:37,766 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140) - Post-validation flume configuration contains configuration for agents: [cygnusagent]
17 Dec 2015 13:35:37,766 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:150) - Creating channels
17 Dec 2015 13:35:37,778 INFO [conf-file-poller-0] (org.apache.flume.channel.DefaultChannelFactory.create:40) - Creating instance of channel mysql-channel type memory
17 Dec 2015 13:35:37,782 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:205) - Created channel mysql-channel
17 Dec 2015 13:35:37,783 INFO [conf-file-poller-0] (org.apache.flume.source.DefaultSourceFactory.create:39) - Creating instance of source http-source, type org.apache.flume.source.http.HTTPSource
17 Dec 2015 13:35:37,791 INFO [conf-file-poller-0] (com.telefonica.iot.cygnus.handlers.OrionRestHandler.<init>:75) - Cygnus version (0.11.0.2a9c87fb7fd6156225e2eed7fbc9792f1d9c5dfe)
17 Dec 2015 13:35:37,807 INFO [conf-file-poller-0] (com.telefonica.iot.cygnus.handlers.OrionRestHandler.configure:141) - Startup completed
17 Dec 2015 13:35:37,826 INFO [conf-file-poller-0] (org.apache.flume.sink.DefaultSinkFactory.create:40) - Creating instance of sink: mysql-sink, type: com.telefonica.iot.cygnus.sinks.OrionMySQLSink
17 Dec 2015 13:35:37,839 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:119) - Channel mysql-channel connected to [http-source, mysql-sink]
17 Dec 2015 13:35:37,843 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138) - Starting new configuration:{ sourceRunners:{http-source=EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTT$
17 Dec 2015 13:35:37,844 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:145) - Starting Channel mysql-channel
17 Dec 2015 13:35:37,910 INFO [lifecycleSupervisor-1-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:110) - Monitoried counter group for type: CHANNEL, name: mysql-channel, registered successfully.
17 Dec 2015 13:35:37,910 INFO [lifecycleSupervisor-1-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:94) - Component type: CHANNEL, name: mysql-channel started
17 Dec 2015 13:35:37,911 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:173) - Starting Sink mysql-sink
17 Dec 2015 13:35:37,913 INFO [lifecycleSupervisor-1-1] (com.telefonica.iot.cygnus.sinks.OrionMySQLSink.start:153) - [mysql-sink] Startup completed
17 Dec 2015 13:35:37,915 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:184) - Starting Source http-source
17 Dec 2015 13:35:37,916 INFO [lifecycleSupervisor-1-2] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.initialize:92) - Grouping rules read:
17 Dec 2015 13:35:37,916 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:101) - Shutting down configuration: { sourceRunners:{http-source=EventDrivenSourceRunner: { source:org.apache.flume.source.http.HT$
17 Dec 2015 13:35:37,917 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:105) - Stopping Source http-source
17 Dec 2015 13:35:37,920 ERROR [lifecycleSupervisor-1-2] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.parseGroupingRules:165) - Error while parsing the Json-based grouping rules file. Details=null
17 Dec 2015 13:35:37,921 WARN [lifecycleSupervisor-1-2] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.initialize:98) - Grouping rules syntax has errors
17 Dec 2015 13:35:37,948 INFO [lifecycleSupervisor-1-2] (org.mortbay.log.Slf4jLog.info:67) - jetty-6.1.26
17 Dec 2015 13:35:37,973 INFO [lifecycleSupervisor-1-2] (org.mortbay.log.Slf4jLog.info:67) - Started SocketConnector#0.0.0.0:5050
17 Dec 2015 13:35:37,974 INFO [lifecycleSupervisor-1-2] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:110) - Monitoried counter group for type: SOURCE, name: http-source, registered successfully.
17 Dec 2015 13:35:37,974 INFO [lifecycleSupervisor-1-2] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:94) - Component type: SOURCE, name: http-source started
17 Dec 2015 13:35:37,974 INFO [conf-file-poller-0] (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:171) - Stopping component: EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTTPSource{name:http-source,state$
17 Dec 2015 13:35:37,974 INFO [conf-file-poller-0] (org.mortbay.log.Slf4jLog.info:67) - Stopped SocketConnector#0.0.0.0:5050
17 Dec 2015 13:35:37,975 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:139) - Component type: SOURCE, name: http-source stopped
17 Dec 2015 13:35:37,976 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:145) - Shutdown Metric for type: SOURCE, name: http-source. source.start.time == 1450355737974
17 Dec 2015 13:35:37,976 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:151) - Shutdown Metric for type: SOURCE, name: http-source. source.stop.time == 1450355737975
17 Dec 2015 13:35:37,976 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.append-batch.accepted == 0
17 Dec 2015 13:35:37,976 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.append-batch.received == 0
17 Dec 2015 13:35:37,976 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.append.accepted == 0
17 Dec 2015 13:35:37,976 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.append.received == 0
17 Dec 2015 13:35:37,977 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.events.accepted == 0
17 Dec 2015 13:35:37,977 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.events.received == 0
17 Dec 2015 13:35:37,977 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.open-connection.count == 0
17 Dec 2015 13:35:37,977 INFO [conf-file-poller-0] (org.apache.flume.source.http.HTTPSource.stop:172) - Http source http-source stopped. Metrics: SOURCE:http-source{src.events.accepted=0, src.open-connection.count=0, src.append.receiv$
17 Dec 2015 13:35:37,977 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:115) - Stopping Sink mysql-sink
17 Dec 2015 13:35:37,977 INFO [conf-file-poller-0] (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:171) - Stopping component: SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor#76da521f counterGroup:{ name:nul$
17 Dec 2015 13:35:37,987 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:125) - Stopping Channel mysql-channel
17 Dec 2015 13:35:37,987 INFO [conf-file-poller-0] (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:171) - Stopping component: org.apache.flume.channel.MemoryChannel{name: mysql-channel}
17 Dec 2015 13:35:37,987 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:139) - Component type: CHANNEL, name: mysql-channel stopped
17 Dec 2015 13:35:37,987 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:145) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.start.time == 1450355737910
17 Dec 2015 13:35:37,988 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:151) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.stop.time == 1450355737987
17 Dec 2015 13:35:37,988 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.capacity == 1000
17 Dec 2015 13:35:37,988 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.current.size == 0
17 Dec 2015 13:35:37,988 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.event.put.attempt == 0
17 Dec 2015 13:35:37,988 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.event.put.success == 0
17 Dec 2015 13:35:37,988 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.event.take.attempt == 1
17 Dec 2015 13:35:37,989 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.event.take.success == 0
17 Dec 2015 13:35:37,989 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138) - Starting new configuration:{ sourceRunners:{http-source=EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTT$
17 Dec 2015 13:35:37,989 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:145) - Starting Channel mysql-channel
17 Dec 2015 13:35:37,989 INFO [lifecycleSupervisor-1-3] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:94) - Component type: CHANNEL, name: mysql-channel started
17 Dec 2015 13:35:37,992 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:173) - Starting Sink mysql-sink
17 Dec 2015 13:35:37,993 INFO [lifecycleSupervisor-1-8] (com.telefonica.iot.cygnus.sinks.OrionMySQLSink.start:153) - [mysql-sink] Startup completed
17 Dec 2015 13:35:37,993 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:184) - Starting Source http-source
17 Dec 2015 13:35:37,993 INFO [lifecycleSupervisor-1-4] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.initialize:92) - Grouping rules read:
17 Dec 2015 13:35:37,994 ERROR [lifecycleSupervisor-1-4] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.parseGroupingRules:165) - Error while parsing the Json-based grouping rules file. Details=null
17 Dec 2015 13:35:37,994 WARN [lifecycleSupervisor-1-4] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.initialize:98) - Grouping rules syntax has errors
17 Dec 2015 13:35:37,994 INFO [lifecycleSupervisor-1-4] (org.mortbay.log.Slf4jLog.info:67) - jetty-6.1.26
17 Dec 2015 13:35:37,996 INFO [lifecycleSupervisor-1-4] (org.mortbay.log.Slf4jLog.info:67) - Started SocketConnector#0.0.0.0:5050
17 Dec 2015 13:35:37,996 INFO [lifecycleSupervisor-1-4] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:94) - Component type: SOURCE, name: http-source started
Then I tried to subscribe the previously mentioned data to cygnus:
(curl localhost:1026/v1/subscribeContext -s -S --header 'Content-Type: application/json' \
--header 'Accept: application/json' -d #- | python -mjson.tool) <<EOF
{
"entities": [
{
"type": "Room",
"isPattern": "false",
"id": "Room1"
}
],
"attributes": [
"pressure"
"temperature"
],
"reference": "http://localhost:5050/notify",
"duration": "P1M",
"notifyConditions": [
{
"type": "ONCHANGE",
"condValues": [
"pressure",
"temperature"
]
}
],
"throttling": "PT1S"
}
EOF
Even after I updated the information on context Broker thinking it would trigger an event:
(curl localhost:1026/v1/updateContext -s -S --header 'Content-Type: application/json' \
--header 'Accept: application/json' -d #- | python -mjson.tool) <<EOF
{
"contextElements": [
{
"type": "Room",
"isPattern": "false",
"id": "Room1",
"attributes": [
{
"name": "temperature",
"type": "float",
"value": "333"
},
{
"name": "pressure",
"type": "integer",
"value": "555"
}
]
}
],
"updateAction": "APPEND"
}
EOF
But the cygnus log remainged exactly the same and its like nothing even got through to it. Which is odd considering my agent_1.conf:
# Copyright 2014 Telefónica Investigación y Desarrollo, S.A.U
#
# This file is part of fiware-cygnus (FI-WARE project).
#
# fiware-cygnus is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General
# Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any
# later version.
# fiware-cygnus is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along with fiware-cygnus. If not, see
# http://www.gnu.org/licenses/.
#
# For those usages not covered by the GNU Affero General Public License please contact with iot_support at tid dot es
#=============================================
# To be put in APACHE_FLUME_HOME/conf/agent.conf
#
# General configuration template explaining how to setup a sink of each of the available types (HDFS, CKAN, MySQL).
#=============================================
# The next tree fields set the sources, sinks and channels used by Cygnus. You could use different names than the
# ones suggested below, but in that case make sure you keep coherence in properties names along the configuration file.
# Regarding sinks, you can use multiple types at the same time; the only requirement is to provide a channel for each
# one of them (this example shows how to configure 3 sink types at the same time). Even, you can define more than one
# sink of the same type and sharing the channel in order to improve the performance (this is like having
# multi-threading).
cygnusagent.sources = http-source
cygnusagent.sinks = mysql-sink
cygnusagent.channels = mysql-channel
#=============================================
# source configuration
# channel name where to write the notification events
cygnusagent.sources.http-source.channels = mysql-channel
# source class, must not be changed
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming notifications
cygnusagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
# URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
# Default service (service semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service = Trace_Data
# Default service path (service path semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service_path = Sensor
# Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
cygnusagent.sources.http-source.handler.events_ttl = 10
# Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts gi
# TimestampInterceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
# GroupinInterceptor, do not change
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
# Grouping rules for the GroupingInterceptor, put the right absolute path to the file if necessary
# See the doc/design/interceptors document for more details
cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
# ============================================
# OrionMySQLSink configuration
# channel name from where to read notification events
cygnusagent.sinks.mysql-sink.channel = mysql-channel
# sink class, must not be changed
cygnusagent.sinks.mysql-sink.type = com.telefonica.iot.cygnus.sinks.OrionMySQLSink
# true if the grouping feature is enabled for this sink, false otherwise
cygnusagent.sinks.mysql-sink.enable_grouping = false
# the FQDN/IP address where the MySQL server runs
cygnusagent.sinks.mysql-sink.mysql_host = 127.0.0.1
# the port where the MySQL server listes for incomming connections
cygnusagent.sinks.mysql-sink.mysql_port = 3306
# a valid user in the MySQL server
cygnusagent.sinks.mysql-sink.mysql_username = root
# password for the user above
cygnusagent.sinks.mysql-sink.mysql_password = klasika
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.mysql-sink.attr_persistence = column
# select the table type from table-by-destination and table-by-service-path
cygnusagent.sinks.mysql-sink.table_type = table-by-destination
# number of notifications to be included within a processing batch
cygnusagent.sinks.mysql-sink.batch_size = 1
# timeout for batch accumulation
cygunsagent.sinks.mysql-sink.batch_timeout = 30
#=============================================
# mysql-channel configuration
# channel type (must not be changed)
cygnusagent.channels.mysql-channel.type = memory
# capacity of the channel
cygnusagent.channels.mysql-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnusagent.channels.mysql-channel.transactionCapacity = 100
#============================================
It has 5050 and notify as the reference address. I double checked the cygnus_instance_1.conf as well and it is pointing at agent_1.conf
#####
#
# Configuration file for apache-flume
#
#####
# Copyright 2014 Telefonica Investigación y Desarrollo, S.A.U
#
# This file is part of fiware-cygnus (FI-WARE project).
#
# fiware-cygnus is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General
# Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any
# later version.
# fiware-cygnus is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along with fiware-cygnus. If not, see
# http://www.gnu.org/licenses/.
#
# For those usages not covered by the GNU Affero General Public License please contact with iot_support at tid dot es
# Who to run cygnus as. Note that you may need to use root if you want
# to run cygnus in a privileged port (<1024)
CYGNUS_USER=cygnus
# Where is the config folder
CONFIG_FOLDER=/usr/cygnus/conf
# Which is the config file
CONFIG_FILE=/usr/cygnus/conf/agent_1.conf
# Name of the agent. The name of the agent is not trivial, since it is the base for the Flume parameters
# naming conventions, e.g. it appears in .sources.http-source.channels=...
AGENT_NAME=cygnusagent
# Name of the logfile located at /var/log/cygnus. It is important to put the extension '.log' in order to the log rotation works properly
LOGFILE_NAME=cygnus.log
# Administration port. Must be unique per instance
ADMIN_PORT=8081
# Polling interval (seconds) for the configuration reloading
POLLING_INTERVAL=30
This is the content of my config folder in /usr/cygnus/conf:
[root#centos conf]# ls
agent_1.conf cygnus_instance_1.conf flume-env.sh grouping_rules.conf krb5.conf krb5_login.conf log4j.properties.template
agent.conf.template cygnus_instance.conf.template flume-env.sh.template grouping_rules.conf.template krb5.conf.template log4j.properties README.md
I noticed that there is an exacted mirror of this in /etc/cygnus/conf but I didn't touch anything because the installation only intructs me to use the /usr/ folder.
Here is my Mysql create statement. In this table I am expecting to receive the context broker data but I get nothing of course since the log didnt register anything.
CREATE TABLE sensor_room1_room (
sensorID INT NOT NULL AUTO_INCREMENT,
recvTime mediumtext,
fiwareservicepath text,
entityId text,
entityType text,
pressure text,
pressure_md text,
temperature text,
temperature_md text,
PRIMARY KEY (sensorID));
Edit 1:
Here is the listener
[root#centos conf]# nc -l 5050
But when I tried subscribing or updating context, nothing was received on the listener side. I am not taking into account the client side of nc: nc 127.0.0.1 5050 because it successfully sends everything i type (even gibberish).
I also tried the test: /usr/cygnus/bin/cygnus-flume-ng agent --conf /usr/cygnus/conf/ -f /usr/cygnus/conf/agent_1.conf -n cygnusagent -Dflume.root.logger=DEBUG,console. I tried both 5050 and 8081 ports to subscribe to and then update context but nothing is read on the console.
Since I seriously have no idea why that subscription didn't work, but thanks to #fgalan I did manage to read the logs so I am posting the subscription that did trigger the event to cygnus:
(curl localhost:1026/v1/subscribeContext -s -S --header 'Content-Type: application/json' \
--header 'Accept: application/json' -d #- | python -mjson.tool) <<EOF
{
"entities": [
{
"type": "Room",
"isPattern": "false",
"id": "Room1"
}
],
"attributes": [
"temperature"
],
"reference": "http://localhost:5050/notify",
"duration": "P1M",
"notifyConditions": [
{
"type": "ONCHANGE",
"condValues": [
"pressure"
]
}
],
"throttling": "PT5S"
}
EOF
Thank you #fgalan one more time!
I want to subscribe Orion to send notifications to Cygnus. Then cygnus will save all data in mysql database. I use this script to subscribe the speed attribute of car1.
(curl 130.206.118.44:1026/v1/subscribeContext -s -S --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'Fiware-Service: vehicles' --header 'Fiware-ServicePath: /4wheels' -d #- | python -mjson.tool) <<EOF
{
"entities": [
{
"type": "car",
"isPattern": "false",
"id": "car1"
}
],
"attributes": [
"speed",
"oil_level"
],
"reference": "http://192.168.1.49:5050/notify",
"duration": "P1M",
"notifyConditions": [
{
"type": "ONCHANGE",
"condValues": [
"speed"
]
}
],
"throttling": "PT1S"
}
EOF
But when I update the speed attribute of car 1, cygnus doesn't update the database.
Databases available:
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
+--------------------+
3 rows in set (0.00 sec)
Some information about my cygnus service and my cygnus configuration (systemctl status cygnus):
cygnus.service - SYSV: cygnus
Loaded: loaded (/etc/rc.d/init.d/cygnus)
Active: active (exited) since Wed 2015-10-21 17:54:07 UTC; 8min ago
Process: 31566 ExecStop=/etc/rc.d/init.d/cygnus stop (code=exited, status=0/SUCCESS)
Process: 31588 ExecStart=/etc/rc.d/init.d/cygnus start (code=exited, status=0/SUCCESS)
Oct 21 17:54:05 cygnus systemd[1]: Starting SYSV: cygnus...
Oct 21 17:54:05 cygnus su[31593]: (to cygnus) root on none
Oct 21 17:54:07 cygnus cygnus[31588]: Starting Cygnus mysql... [ OK ]
Oct 21 17:54:07 cygnus systemd[1]: Started SYSV: cygnus.
agent_mysql.conf:
# main configuration
cygnusagent.sources = http-source
cygnusagent.sinks = mysql-sink
cygnusagent.channels = mysql-channel
# source configuration
cygnusagent.sources.http-source.channels = mysql-channel
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnusagent.sources.http-source.port = 5050
cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
# url target
cygnusagent.sources.http-source.handler.notification_target = /notify
cygnusagent.sources.http-source.handler.default_service = def_serv
cygnusagent.sources.http-source.handler.default_service_path = def_servpath
cygnusagent.sources.http-source.handler.events_ttl = 10
cygnusagent.sources.http-source.interceptors = ts gi
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
#Orion MysqlSink Configuration
cygnusagent.sinks.mysql-sink.channel = mysql-channel
cygnusagent.sinks.mysql-sink.type = com.telefonica.iot.cygnus.sinks.OrionMySQLSink
cygnusagent.sinks.mysql-sink.enable_grouping = false
# mysqldb ip
cygnusagent.sinks.mysql-sink.mysql_host = 127.0.0.1
# mysqldb port
cygnusagent.sinks.mysql-sink.mysql_port = 3306
cygnusagent.sinks.mysql-sink.mysql_username = root
cygnusagent.sinks.mysql-sink.mysql_password = 12345
cygnusagent.sinks.mysql-sink.attr_persistence = column
cygnusagent.sinks.mysql-sink.table_type = table-by-destination
# configuracao do canal mysql
cygnusagent.channels.mysql-channel.type = memory
cygnusagent.channels.mysql-channel.capacity = 1000
cygnusagent.channels.mysql-channel.transactionCapacity = 100
After read this question, I changed my agent_mysql.conf in this line:
cygnusagent.sinks.mysql-sink.attr_persistence = column to cygnusagent.sinks.mysql-sink.attr_persistence = row and restarted the service. Then I updated orion entity and I queried database and nothing happened.
Cygnus log file: http://pastebin.com/B2FNKcVf
Note: My JAVA_HOME is set.
As you can see in the logs you posted, there is a problem with the Cygnus log file:
java.io.FileNotFoundException: ./logs/cygnus.log (No such file or directory)
After that, Cygnus stops. You must check your configuration regarding log4j, everything is at /usr/cygnus/conf/log4j.properties (it should exists, it is created by the RPM... if not existing -because you italled from sources instead of the RPM-, it must be created from the available template). In addition, can you post your instance configuration file? Anyway, which version are you running?
EDIT 1
Recently, we have found another user dealing with the same error, and the problem was the content of the /usr/cygnus/conf/log4j.properties file was:
flume.root.logger=INFO,LOGFILE
flume.log.dir=./log
flume.log.file=flume.log
Instead of what the template contains:
flume.root.logger=INFO,LOGFILE
flume.log.dir=/var/log/cygnus/
flume.log.file=flume.log
Once changed it worked because the RPM creates /var/log/cygnus but not ./log.