Fiware - Cygnus: Issue with collection's names - fiware

Question

I have encountered a strange behaviour on the Cygnus module. I’m using Context Broker version 0.28 and Cygnus version 0.13. Let’s suppose that I have loaded on the CB (Context Broker) some entities similars to this one:
(curl 172.21.0.33:1026/v1/updateContext -s -S
--header 'Fiware-Service: cb33_1003_06'
--header 'Fiware-ServicePath: /Lugar'
--header 'Content-Type: application/json'
--header 'Accept: application/json' -d #- | python -mjson.tool) <<EOF
{
"contextElements": [
{
"type": "Acceso_Wifi",
"isPattern": "false",
"id": "AP_1.3",
"attributes": [
{
"name": "temperature",
"type": "float",
"value": "25"
},
{
"name": "pressure",
"type": "integer",
"value": "1"
},
{
"name": "position",
"type": "coords",
"value": "13.322326, -1.983824",
"metadatas": [
{
"name": "location",
"type": "string",
"value": "WGS84"
}
]
}
]
}
],
"updateAction": "APPEND"
}
EOF
And now I subscribe the Cygnus:
(curl 172.21.0.33:1026/v1/subscribeContext -s -S
--header 'Fiware-Service: cb33_1003_06'
--header 'Fiware-ServicePath: /Lugar'
--header 'Content-Type: application/json'
--header 'Accept: application/json' -d #- | python -mjson.tool) <<EOF
{
"entities": [
{
"type": "Acceso_Wifi",
"isPattern": "true",
"id": "AP_.*"
}
],
"attributes": [
"temperature",
"pressure"
],
"reference": "http://172.21.0.33:5050/notify",
"duration": "P1M",
"notifyConditions": [
{
"type": "ONCHANGE",
"condValues": []
}
],
"throttling": "PT1S"
}
EOF
The CB will send one notification to the Cygnus for each matching entity loaded previously on the CB database. Let’s assume that there are 3 entities that meet the subscription’s criteria. In that case, one collection will be created on the Cygnus for each one of these entities (I’m using the default dm-by-entity data model). The problem is that the collection's name are not well formed. The Fiware-ServicePath is concatenated once per each entity, on each collection name:
cygnus2_/Lugar__Lugar__Lugar_AP_1.1_Acceso_Wifi
cygnus2_/Lugar__Lugar__Lugar_AP_1.2_Acceso_Wifi
cygnus2_/Lugar__Lugar__Lugar_AP_1.3_Acceso_Wifi
If we update one of these entities, a new collection will be created with a correct name:
cygnus2_/Lugar_AP_1.1_Acceso_Wifi
cygnus2_/Lugar__Lugar__Lugar_AP_1.1_Acceso_Wifi
cygnus2_/Lugar__Lugar__Lugar_AP_1.2_Acceso_Wifi
cygnus2_/Lugar__Lugar__Lugar_AP_1.3_Acceso_Wifi
On this example there were just 3 entities, but as the amount of matching entities increases, the longer will be the collection's name. And when it arrives to 50 characters, a warning will be raised and no collection will be created.
org.apache.flume.source.http.HTTPBadRequestException: 'fiware-servicePath' header length greater than 50)
at com.telefonica.iot.cygnus.handlers.OrionRestHandler.getEvents(OrionRestHandler.java:209)
at org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:184)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
¿It’s that behaviour due to an incorrect configuration, or it’s Cygnus’s problem?
Here is the OrionMongoSink's configuration:
# OrionMongoSink configuration
# sink class, must not be changed
cygnusagent.sinks.mongo-sink.type = com.telefonica.iot.cygnus.sinks.OrionMongoSink
# channel name from where to read notification events
cygnusagent.sinks.mongo-sink.channel = mongo-channel
# true if the grouping feature is enabled for this sink, false otherwise
cygnusagent.sinks.mongo-sink.enable_grouping = false
# true if lower case is wanted to forced in all the element names, false otherwise
cygnusagent.sinks.mongo-sink.enable_lowercase = false
# FQDN/IP:port where the MongoDB server runs (standalone case) or comma-separated list of FQDN/IP:port pairs where the MongoDB replica set members run
cygnusagent.sinks.mongo-sink.mongo_hosts = mongodb1.spider.net:27017,mongodb2.spider.net:27017,mongodb3.spider.net:27017
###cygnusagent.sinks.mongo-sink.mongo_hosts = 172.21.0.25:27017,172.21.0.26:27017,172.21.0.28:27017
# a valid user in the MongoDB server (or empty if authentication is not enabled in MongoDB)
cygnusagent.sinks.mongo-sink.mongo_username =
# password for the user above (or empty if authentication is not enabled in MongoDB)
cygnusagent.sinks.mongo-sink.mongo_password =
# prefix for the MongoDB databases
cygnusagent.sinks.mongo-sink.db_prefix = cygnus2_
# prefix pro the MongoDB collections
cygnusagent.sinks.mongo-sink.collection_prefix = cygnus2_
# true is collection names are based on a hash, false for human redable collections
cygnusagent.sinks.mongo-sink.should_hash = false
# Must be dm-by-service-path or dm-by-entity
cygnusagent.sinks.mongo-sink.data_model = dm-by-entity
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.mongo-sink.attr_persistence = column
# number of notifications to be included within a processing batch
cygnusagent.sinks.mongo-sink.batch_size = 1
# timeout for batch accumulation
cygnusagent.sinks.mongo-sink.batch_timeout = 30
# number of retries upon persistence error
cygnusagent.sinks.mongo-sink.batch_ttl = 10
# Collections will be removed if older than the value specified in seconds. Set to 0 if not wanting this policy.
cygnusagent.sinks.mongo-sink.data_expiration = 0
# The oldest data (according to insertion time) will be removed if the size of the data collection gets bigger than the value specified in bytes. Minimum value (different than 0) is 4096 bytes.
cygnusagent.sinks.mongo-sink.collection_size = 0
# The oldest data (according to insertion time) will be removed if the number of documents in the data collections goes beyond the specified value. Set to 0 if not wanting this policy
cygnusagent.sinks.mongo-sink.max_documents = 0
And another agent's configuration:
# source configuration
# channel name where to write the notification events
#####cygnusagent.sources.http-source.channels = hdfs-channel mysql-channel postgresql-channel ckan-channel mongo-channel sth-channel kafka-channel
cygnusagent.sources.http-source.channels = mongo-channel
# source class, must not be changed
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming notifications
cygnusagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
# URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
# Default service (service semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service = def_serv
# Default service path (service path semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service_path = def_serv_path
# Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
cygnusagent.sources.http-source.handler.events_ttl = -1
# Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts gi
# TimestampInterceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
# GroupinInterceptor, do not change
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
# Grouping rules for the GroupingInterceptor, put the right absolute path to the file if necessary
# See the doc/design/interceptors document for more details
#####cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
Channel's conf:
# mongo-channel configuration
# channel type (must not be changed)
cygnusagent.channels.mongo-channel.type = memory
# capacity of the channel
cygnusagent.channels.mongo-channel.capacity = 10000
# amount of bytes that can be sent per transaction
cygnusagent.channels.mongo-channel.transactionCapacity = 100

Answer 1

After talking with an Orion expert, this seems to be a problem with the initial notification, i.e. when the subscription is created and a first notification is sent; in that case, a multi-valued fiware-servicePath is sent. Nevertheless, after that, next notifications should contain a single Context Element and, thus, a single-valued fiware-servicePath header.
This needs a fix in the code (either to ignore the first notification, which is useless btw, either by supporting multi-valued service path headers). In the meantime, a workaround would be to make the subscriptions with Cygnus stopped, and once made all the subscriptions, start Cygnus.
EDIT 1:
I've create this issue at Cygnus Github: https://github.com/telefonicaid/fiware-cygnus/issues/923