Using perseo cep, I set a rule to it. I get this message in the Perseo fron End's Log: http://perseo-core:8080/perseo-core/rules returns {"code":200,"body":{}}. Then when I modify an attribute, in the Perseo Core's Log I see this message: msg=Firing Rule: MapEventBean. So far, so good, but then again in the Perseo FE's log I get the message: msg=missing subservice header msg=missing service header, and it is not updating my entity.
This is the log:
time=2019-05-11T14:30:48.774Z | lvl=INFO | corr=n/a | trans=n/a | op=checkRequest | comp=perseo-fe | srv=n/a | subsrv=n/a | msg=missing subservice header
time=2019-05-11T14:30:48.774Z | lvl=INFO | corr=n/a | trans=n/a | op=checkRequest | comp=perseo-fe | srv=n/a | subsrv=n/a | msg=missing service header
time=2019-05-11T14:30:48.774Z | lvl=INFO | corr=5229c32c-362a-43f6-acda-11d6d8198261; perseocep=21 | trans=5229c32c-362a-43f6-acda-11d6d8198261 | op=/version | path=/version | comp=perseo-fe | srv=unknownt | subsrv=/ | from=::1 | msg=incoming request {"method":"GET","url":"/version","headers":{"host":"localhost:9090","user-agent":"curl/7.52.1","accept":"/"},"body":{},"subservice":"/","service":"unknownt"}
time=2019-05-11T14:30:48.774Z | lvl=INFO | corr=5229c32c-362a-43f6-acda-11d6d8198261; perseocep=21 | trans=5229c32c-362a-43f6-acda-11d6d8198261 | op=/version | path=/version | comp=perseo-fe | srv=unknownt | subsrv=/ | from=::1 | msg=sending
Mi Rule:
{
"name": "pruebaUpdate",
"text": "select ev.temperature? as temperature, ev.id? as id, \"pruebaUpdate\" as ruleName from pattern [every ev=iotEvent(cast(cast(temperature?,String),float) > 45 and type=\"AirQualityObserved\")]",
"action": [{
"type": "update",
"parameters": {
"id":"${id}",
"type":"AirQualityObserved",
"version": "2",
"attributes":[
{
"name": "abnormal",
"type": "boolean",
"value": "true"
}
]
}
}]
}
I want to update one attribute when other attribute is higher than a value, that attribute is in some entities but not in other of the same type. I am not getting to update that attribute.
msg=missing subservice header, msg=missing service header typically mean that the incoming request doesn't have the fiware-service and fiware-servicepath HTTP headers in which multitenancy is based. The FIWARE multitenancy model is described here.
Placing the rule in the correct place I didn't get the error of "missing service subservice". I registered the rule with http://perseo-fe-ip:9090/rules and I fixed the problems.
Related
I have been able to send JSON format measures via MQTT publisher to the mosquitto docker container ( that is a broker ), also i have been able to pass that data to the IOT-Agent, these are curl commands that i used to create a service-path and to register a device on the IOT-Agent:
curl -iX POST 'http://localhost:4041/iot/services' -H 'Content-Type: application/json' -H 'Fiware-Service: myClassRoom' -H 'Fiware-ServicePath: /' -d '{ "services": [ { "apikey": "12345", "cbroker": "http://localhost:1026", "entity_type": "Device", "resource": /iot/d" } ] }
This is how i registered the device in the IOT-Agent:
curl -iX POST 'http://localhost:4041/iot/devices' -H 'Content-Type: application/json' -H 'Fiware-Service: myClassRoom' -H 'Fiware-ServicePath: /' -d '{ "devices": [ { "device_id": "SensTemp", "entity_name": "urn:ngsi-ld:temperature-sensor:001", "entity_type": "Device", "protocol": "PDI-IoTA-UltraLight", "transport": "MQTT", "timezone": "Europe/Madrid", "attributes":[ { "object_id": "temperature", "name": "temperature", "type": "Integer"} ] } ] }'
And this is the error that is giving me in the Agent, can someone help me to solve this issue?
fiware-agent | time=2022-07-25T08:19:11.562Z | lvl=FATAL | corr=n/a | trans=n/a | op=IoTAgentNGSI.Global | from=n/a | srv=n/a | subsrv=n/a | msg=An unexpected exception has been raised. Ignoring: TypeError: parsedMessage.reduce is not a function
And here is the evidence of connecting, Service Path, information about the device etc...
fiware-agent | time=2022-07-25T08:20:11.931Z | lvl=DEBUG | corr=785bb065-6c0a-41ba-96d2-182f0040ea93 | trans=785bb065-6c0a-41ba-96d2-182f0040ea93 | op=IOTAUL.IoTUtils | from=n/a | srv=n/a | subsrv=n/a | msg=deviceData after merge with conf: {"_id":"62d931f861815b54e26fa2b2","active":[{"object_id":"temperature","name":"temperature","type":"Integer"}],"commands":[],"staticAttributes":[],"subscriptions":[],"creationDate":"2022-07-21T11:01:12.968Z","id":"SensTemp","type":"Device","name":"urn:ngsi-ld:temperature-sensor:001","service":"myclassroom","subservice":"/","internalId":null,"protocol":"PDI-IoTA-UltraLight","transport":"MQTT","explicitAttrs":false,"lazy":null,"internalAttributes":null} | comp=IoTAgent
And here is what i dont understand, why is giving me that error, parsed message error?
fiware-agent | time=2022-07-25T08:20:11.932Z | lvl=DEBUG | corr=785bb065-6c0a-41ba-96d2-182f0040ea93 | trans=785bb065-6c0a-41ba-96d2-182f0040ea93 | op=IOTAUL.Common.Binding | from=n/a | srv=n/a | subsrv=n/a | msg=Processing multiple measures for device [SensTemp] with apiKey [12345] | comp=IoTAgent
fiware-agent | time=2022-07-25T08:20:11.932Z | lvl=DEBUG | corr=785bb065-6c0a-41ba-96d2-182f0040ea93 | trans=785bb065-6c0a-41ba-96d2-182f0040ea93 | op=IOTAUL.Common.Binding | from=n/a | srv=n/a | subsrv=n/a | msg=Parse error parsing incoming message [%]. Forcing to hex string | comp=IoTAgent {"temperature": 47}
fiware-agent | time=2022-07-25T08:20:11.932Z | lvl=DEBUG | corr=785bb065-6c0a-41ba-96d2-182f0040ea93 | trans=785bb065-6c0a-41ba-96d2-182f0040ea93 | op=IOTAUL.Common.Binding | from=n/a | srv=n/a | subsrv=n/a | msg=stringMessage: [7b2274656d7065726174757265223a2034377d] parsedMessage: [7b2274656d7065726174757265223a2034377d] | comp=IoTAgent
And Why cant i see the Temperature change in the MongoDB depository?
Thank you !!!
Problem solved! The error was in the Agent, I was using Ultralight IOT Agent. The solution is simple, when i used IOT JSON agent the problem was solved!
Now the Agent tells me that the measures are updated successfully!
I am trying to connect Orion with AWS DocumentDB but it's not getting connected. However I tried two other FIWARE components IoTAgent and Sth-Comet with DocumentDB and both are working fine.
Same hostname and credential are working for IoTAgent and Sth-Comet. I also checked for the connectivity, which is fine, as IoTAgent and Sth-Comet are in same network. I also checked from a different mongo host in same network and this also worked. Below is the error that I am getting for Orion.
time=2021-02-18T07:03:46.293Z | lvl=ERROR | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=mongoConnectionPool.cpp[180]:mongoConnect | msg=Database Startup Error (cannot connect to mongo - doing 100 retries with a 1000 millisecond interval)
Is there any possibility that Orion is not compatible with AWS DocumentDB?
Update1:
bash-4.2$ ps ax | grep contextBroker
1 ? Ss 0:00 /usr/bin/contextBroker -fg -multiservice -ngsiv1Autocast -disableFileLog -dbhost xxxxxxxxxxxxxxxxxx.docdb.amazonaws.com -db admin -dbuser test -dbpwd xxxxxxxxxx
Update2:
Earlier, I was using Orion docker images by pulling directly from dockerhub and that was not working. So this time, I build two docker images by building source code of version 2.4.2 and 2.5.2. Now, I was able to connect with AWS DocuemntDB with these docker images but getting a different error as below.
time=2021-02-23T06:10:41.982Z | lvl=ERROR | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=safeMongo.cpp[360]:getField | msg=Runtime Error (field '_id' is missing in BSONObj <{ ok: 0.0, code: 303, errmsg: "Legacy opcodes are not supported" }> from caller mongoSubCacheItemInsert:83)
time=2021-02-23T06:10:41.982Z | lvl=ERROR | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=AlarmManager.cpp[211]:dbError | msg=Raising alarm DatabaseError: error retrieving _id field in doc: '{ ok: 0.0, code: 303, errmsg: "Legacy opcodes are not supported" }'
Below is the Orion version
contextBroker --version
2.5.0-next (git version: 3984f9fc30e90fa04682131ca4516b4d277eb27e)
curl -X GET 'http://localhost:1026/version'
{
"orion" : {
"version" : "2.5.0-next",
"uptime" : "0 d, 0 h, 4 m, 56 s",
"git_hash" : "3984f9fc30e90fa04682131ca4516b4d277eb27e",
"compile_time" : "Mon Feb 22 17:39:30 UTC 2021",
"compiled_by" : "root",
"compiled_in" : "4c7575c7c27f",
"release_date" : "Mon Feb 22 17:39:30 UTC 2021",
"doc" : "https://fiware-orion.rtfd.io/",
"libversions": {
"boost": "1_53",
"libcurl": "libcurl/7.29.0 NSS/3.53.1 zlib/1.2.7 libidn/1.28 libssh2/1.8.0",
"libmicrohttpd": "0.9.70",
"openssl": "1.0.2k",
"rapidjson": "1.1.0",
"mongodriver": "legacy-1.1.2"
}
}
}
I am also able to connect to DocumentDB from Orion Pod using Mongo Shell.
mongo --host xxxxxxxxxxxxxxxxxx.docdb.amazonaws.com:27017 --username xxxx --password xxxx
rs0:PRIMARY> show dbs;
rs0:PRIMARY>
I am also able to create entries using below command and it creates a DB and collection in DocumentDB:
curl localhost:1026/v2/entities -s -S --header 'Content-Type: application/json' \
> -X POST -d #- <<EOF
> {
> "id": "Room2",
> "type": "Room",
> "temperature": {
> "value": 23,
> "type": "Number"
> },
> "pressure": {
> "value": 720,
> "type": "Number"
> }
> }
> EOF
rs0:PRIMARY> show dbs;
orion 0.000GB
But I am not able to get that data using orion API and after executing this command it getting exited from container with a empty response. I have checked the same with Orion version 2.4.2 and 2.5.2 with DocumentDB 4.0 and 3.6.
[root#orion-docdb-7748fd9c85-gbjz7 /]# curl localhost:1026/v2/entities/Room2 -s -S --header 'Accept: application/json' | python -mjson.tool
curl: (52) Empty reply from server
command terminated with exit code 137
At the end, still getting same error in logs.
time=2021-02-23T06:16:04.564Z | lvl=ERROR | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=safeMongo.cpp[360]:getField | msg=Runtime Error (field '_id' is missing in BSONObj <{ ok: 0.0, code: 303, errmsg: "Legacy opcodes are not supported" }> from caller mongoSubCacheItemInsert:83)
time=2021-02-23T06:16:04.564Z | lvl=ERROR | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=AlarmManager.cpp[211]:dbError | msg=Raising alarm DatabaseError: error retrieving _id field in doc: '{ ok: 0.0, code: 303, errmsg: "Legacy opcodes are not supported" }'
Update3:
I have added -noCache and deployed again. Below are the commands output and logs for your reference.
Process check:
#ps ax | grep contextBroker
1 ? Ssl 0:00 /usr/bin/contextBroker -fg -multiservice -ngsiv1Autocast -disableFileLog -dbhost xxxxxxxxxxxxxxxxxx.docdb.amazonaws.com -dbuser xxxxxxxx -dbpwd xxxxxxxx -logLevel DEBUG -noCache
Entries in DB:
rs0:PRIMARY> show dbs
orion 0.000GB
rs0:PRIMARY> use orion
switched to db orion
rs0:PRIMARY> show collections
entities
rs0:PRIMARY> db.entities.find()
{ "_id" : { "id" : "Room2", "type" : "Room", "servicePath" : "/" }, "attrNames" : [ "temperature", "pressure" ], "attrs" : { "temperature" : { "type" : "Number", "creDate" : 1614323032.671698, "modDate" : 1614323032.671698, "value" : 23, "mdNames" : [ ] }, "pressure" : { "type" : "Number", "creDate" : 1614323032.671698, "modDate" : 1614323032.671698, "value" : 720, "mdNames" : [ ] } }, "creDate" : 1614323032.671698, "modDate" : 1614323032.671698, "lastCorrelator" : "c8a73f40-7800-11eb-bd9b-bea9c419835d" }
Orion Pod Logs:
time=2021-02-26T06:46:33.966Z | lvl=INFO | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=contextBroker.cpp[1008]:main | msg=start command line </usr/bin/contextBroker -fg -multiservice -ngsiv1Autocast -disableFileLog -dbhost -dbhost xxxxxxxxxxxxxxxxxx.docdb.amazonaws.com -dbuser xxxxxxxx -dbpwd xxxxxxxx -logLevel DEBUG -noCache>
time=2021-02-26T06:46:33.966Z | lvl=INFO | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=contextBroker.cpp[1076]:main | msg=Orion Context Broker is running
time=2021-02-26T06:46:34.280Z | lvl=INFO | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=MongoGlobal.cpp[243]:mongoInit | msg=Connected to mongo at xxxxxxxxxxxxxxxxxx.docdb.amazonaws.com/orion, as user 'xxxxxxx' (poolsize: 10)
time=2021-02-26T06:46:34.282Z | lvl=INFO | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=contextBroker.cpp[1202]:main | msg=Startup completed
time=2021-02-26T07:03:24.546Z | lvl=INFO | corr=b7e44e5a-7800-11eb-9531-bea9c419835d | trans=1614321993-966-00000000001 | from=127.0.0.1 | srv=<none> | subsrv=<none> | comp=Orion | op=logTracing.cpp[79]:logInfoRequestWithoutPayload | msg=Request received: GET /version, response code: 200
time=2021-02-26T07:03:52.672Z | lvl=ERROR | corr=c8a73f40-7800-11eb-bd9b-bea9c419835d | trans=1614321993-966-00000000002 | from=127.0.0.1 | srv=<none> | subsrv=<none> | comp=Orion | op=safeMongo.cpp[360]:getField | msg=Runtime Error (field '_id' is missing in BSONObj <{ ok: 0.0, code: 303, errmsg: "Legacy opcodes are not supported", operationTime: Timestamp 1614323032|1 }> from caller processContextElement:3493)
time=2021-02-26T07:03:52.672Z | lvl=ERROR | corr=c8a73f40-7800-11eb-bd9b-bea9c419835d | trans=1614321993-966-00000000002 | from=127.0.0.1 | srv=<none> | subsrv=<none> | comp=Orion | op=AlarmManager.cpp[211]:dbError | msg=Raising alarm DatabaseError: error retrieving _id field in doc: '{ ok: 0.0, code: 303, errmsg: "Legacy opcodes are not supported", operationTime: Timestamp 1614323032|1 }'
time=2021-02-26T07:03:52.782Z | lvl=ERROR | corr=c8a73f40-7800-11eb-bd9b-bea9c419835d | trans=1614321993-966-00000000002 | from=127.0.0.1 | srv=<none> | subsrv=<none> | comp=Orion | op=AlarmManager.cpp[235]:dbErrorReset | msg=Releasing alarm DatabaseError
time=2021-02-26T07:03:52.790Z | lvl=ERROR | corr=c8a73f40-7800-11eb-bd9b-bea9c419835d | trans=1614321993-966-00000000002 | from=127.0.0.1 | srv=<none> | subsrv=<none> | comp=Orion | op=safeMongo.cpp[360]:getField | msg=Runtime Error (field '_id' is missing in BSONObj <{ ok: 0.0, code: 303, errmsg: "Legacy opcodes are not supported", operationTime: Timestamp 1614323032|1 }> from caller addTriggeredSubscriptions_noCache:1408)
time=2021-02-26T07:03:52.790Z | lvl=ERROR | corr=c8a73f40-7800-11eb-bd9b-bea9c419835d | trans=1614321993-966-00000000002 | from=127.0.0.1 | srv=<none> | subsrv=<none> | comp=Orion | op=AlarmManager.cpp[211]:dbError | msg=Raising alarm DatabaseError: error retrieving _id field in doc: '{ ok: 0.0, code: 303, errmsg: "Legacy opcodes are not supported", operationTime: Timestamp 1614323032|1 }'
time=2021-02-26T07:03:52.791Z | lvl=INFO | corr=c8a73f40-7800-11eb-bd9b-bea9c419835d | trans=1614321993-966-00000000002 | from=127.0.0.1 | srv=<none> | subsrv=<none> | comp=Orion | op=logTracing.cpp[130]:logInfoRequestWithPayload | msg=Request received: POST /v2/entities, request payload (148 bytes): { "id": "Room2", "type": "Room", "temperature": { "value": 23, "type": "Number" }, "pressure": { "value": 720, "type": "Number" }}, response code: 201
time=2021-02-26T07:03:58.479Z | lvl=ERROR | corr=cc1d5934-7800-11eb-a28d-bea9c419835d | trans=1614321993-966-00000000003 | from=127.0.0.1 | srv=<none> | subsrv=<none> | comp=Orion | op=AlarmManager.cpp[235]:dbErrorReset | msg=Releasing alarm DatabaseError
time=2021-02-26T07:03:58.479Z | lvl=ERROR | corr=cc1d5934-7800-11eb-a28d-bea9c419835d | trans=1614321993-966-00000000003 | from=127.0.0.1 | srv=<none> | subsrv=<none> | comp=Orion | op=safeMongo.cpp[360]:getField | msg=Runtime Error (field '_id' is missing in BSONObj <{ ok: 0.0, code: 303, errmsg: "Legacy opcodes are not supported", operationTime: Timestamp 1614323038|1 }> from caller ContextElementResponse:109)
terminate called after throwing an instance of 'mongo::AssertionException'
what(): assertion src/mongo/bson/bsonelement.cpp:392
Pod exited and restarted during API call:
curl localhost:1026/v2/entities/Room2 -s -S --header 'Accept: application/json' | python -mjson.tool
command terminated with exit code 137
The following message shown in log traces is pretty significant
"Legacy opcodes are not supported"
Although the MongoDB driver used by Orion 2.5.2 and before works with official MongoDB version up to 4.4, probably it is not the case with MongoDB "clones" like AWS DocumentDB.
We are in the process to change the legacy driver used by Orion to a new one. Once this change lands in Orion master branch, I'd suggest to test it (using :latest dockerhub tag). In the meanwhile, as a workaround, I'd suggest to use a official MongoDB database.
EDIT: the process to change the MongoDB driver has finished and Orion is using the new driver since version 3.0.0. I think it would be a good idea to test with this new version and see how it goes. I can help with the test if you provide me with the access information (see here).
I am having the same problem as here and try to solve it, but I do not know how to properly format the datastore so cygnus will not throw the persistence error.
My orion suscription is this one:
(curl localhost:1026/v1/subscribeContext -s -S --header 'Content-Type: application/json' \
--header 'Accept: application/json' -d #- | python -mjson.tool) <<EOF
{
"entities": [
{
"type": "Event",
"isPattern": "false",
"id": "es-leon-0"
},
{
"type": "Event",
"isPattern": "false",
"id": "es-leon-1"
}
],
"attributes": [
"IdEvent", "IdUser", "Title"
],
"reference": "http://localhost:5050/notify",
"duration": "P1M",
"notifyConditions": [
{
"type": "ONCHANGE",
"condValues": [ ]
}
],
"throttling": "PT5S"
}
EOF
My cygnus config:
ygnusagent.sources = http-source
cygnusagent.sinks = ckan-sink
cygnusagent.channels = ckan-channel
cygnusagent.sources.http-source.channels = ckan-channel
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnusagent.sources.http-source.port = 5050
cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
cygnusagent.sources.http-source.handler.notification_target = /notify
cygnusagent.sources.http-source.handler.default_service = Papel
cygnusagent.sources.http-source.handler.default_service_path = Test
cygnusagent.sources.http-source.handler.events_ttl = 5
cygnusagent.sources.http-source.interceptors = ts gi
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
cygnusagent.sources.http-source.interceptors.gi.gropuing_rules_conf_file = /Applications/apache-flume-1.4.0-bin/conf/grouping_rules.conf
cygnusagent.channels.ckan-channel.type = memory
cygnusagent.channels.ckan-channel.capacity = 1000
cygnusagent.channels.ckan-channel.transactionCapacity = 100
# ============================================
# OrionCKANSink configuration
# channel name from where to read notification events
cygnusagent.sinks.ckan-sink.channel = ckan-channel
# sink class, must not be changed
cygnusagent.sinks.ckan-sink.type = com.telefonica.iot.cygnus.sinks.OrionCKANSink
# true if the grouping feature is enabled for this sink, false otherwise
cygnusagent.sinks.ckan-sink.enable_grouping = false
# true if lower case is wanted to forced in all the element names, false otherwise
cygnusagent.sinks.hdfs-sink.enable_lowercase = false
# the CKAN API key to use
cygnusagent.sinks.ckan-sink.api_key = xxxxx
# the FQDN/IP address for the CKAN API endpoint
cygnusagent.sinks.ckan-sink.ckan_host = ckan-demo.ckan.io
# the port for the CKAN API endpoint
cygnusagent.sinks.ckan-sink.ckan_port = 80
# Orion URL used to compose the resource URL with the convenience operation URL to query it
cygnusagent.sinks.ckan-sink.orion_url = http://localhost:1026
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.ckan-sink.attr_persistence = column
# enable SSL for secure Http transportation; 'true' or 'false'
cygnusagent.sinks.ckan-sink.ssl = false
# number of notifications to be included within a processing batch
cygnusagent.sinks.ckan-sink.batch_size = 100
# timeout for batch accumulation
cygnusagent.sinks.ckan-sink.batch_timeout = 60
# number of retries upon persistence error
cygnusagent.sinks.ckan-sink.batch_ttl = 10
Cygnus is receiving right it but then shows the following error:
time=2016-04-21T07:44:57.504CDT | lvl=INFO | trans=1461242686-614-0000000001 | srv=Papel | subsrv=Test | function=getEvents | comp=Cygnus | msg=com.telefonica.iot.cygnus.handlers.OrionRestHandler[231] : Starting transaction (1461242686-614-0000000001)
time=2016-04-21T07:44:57.528CDT | lvl=INFO | trans=1461242686-614-0000000001 | srv=Papel | subsrv=Test | function=getEvents | comp=Cygnus | msg=com.telefonica.iot.cygnus.handlers.OrionRestHandler[258] : Received data ({ "subscriptionId" : "571897360e94f9fa53829885", "originator" : "localhost", "contextResponses" : [ { "contextElement" : { "type" : "Event", "isPattern" : "false", "id" : "es-leon-0", "attributes" : [ { "name" : "IdEvent", "type" : "text", "value" : "1084" }, { "name" : "IdUser", "type" : "text", "value" : "18" }, { "name" : "Title", "type" : "text", "value" : "Papes" } ] }, "statusCode" : { "code" : "200", "reasonPhrase" : "OK" } } ]})
time=2016-04-21T07:44:57.528CDT | lvl=INFO | trans=1461242686-614-0000000001 | srv=Papel | subsrv=Test | function=getEvents | comp=Cygnus | msg=com.telefonica.iot.cygnus.handlers.OrionRestHandler[280] : Event put in the channel, id=2024732986
time=2016-04-21T07:45:50.771CDT | lvl=INFO | trans=1461242686-614-0000000001 | srv=Papel | subsrv=Test | function=persistAggregation | comp=Cygnus | msg=com.telefonica.iot.cygnus.sinks.OrionCKANSink[417] : [ckan-sink] Persisting data at OrionCKANSink (orgName=papel, pkgName=papel_test, resName=es-leon-0_event, data={"recvTime": "2016-04-21T12:44:57.497Z","fiwareServicePath": "Test","entityId": "es-leon-0","entityType": "Event","Title": "Papes"},{"recvTime": "2016-04-21T12:44:57.528Z","fiwareServicePath": "Test","entityId": "es-leon-0","entityType": "Event","IdEvent": "1084","IdUser": "18","Title": "Papes"})
time=2016-04-21T07:45:51.875CDT | lvl=ERROR | trans=1461242686-614-0000000001 | srv=Papel | subsrv=Test | function=processNewBatches | comp=Cygnus | msg=com.telefonica.iot.cygnus.sinks.OrionSink[426] : Runtime error (Cannot persist the data (orgName=papel, pkgName=papel_test, resName=es-leon-0_event))
As said in here, I created the corresponding datastore: http://ckan-demo.ckan.io/dataset/papel-test/resource/8d7cb489-878e-465e-8c8c-60ea537411e0
But don't know how to format it or if the csv is the correct format.
Thanks
*Note: I tried in row mode and all works, but it's not what I want.
**Note: I also found an error in the previewer software changing the title of my column "Title" to the title of the page "CKAN Demo".
EDITED:
I have done what is said in the documentation:
Column: A single row is upserted for all the notified context attributes. This kind of row will contain two fields per each entity's attribute (one for the value, called <attrName>, and other for the metadata, called <attrName>_md), plus four additional fields:
recvTime: UTC timestamp in human-redable format (ISO 8601).
fiwareServicePath: The notified one or the default one.
entityId: Notified entity identifier.
entityType: Notified entity type.
But still have the same error:
time=2016-04-25T05:17:48.790CDT | lvl=ERROR | trans=1461579403-571-0000000000 | srv=Papel | subsrv=Test | function=processNewBatches | comp=Cygnus | msg=com.telefonica.iot.cygnus.sinks.OrionSink[426] : Runtime error (Cannot persist the data (orgName=papel, pkgName=papel_test, resName=es-leon-0_event))
First of all, you'll need a CKAN organization and package/dataset before creating a resource and an associated datastore in order to persist the data.
Creating an organization, let's say in demo.ckan.org; the organization name is frb, because our entity will be in that FIWARE service:
$ curl -X POST "http://demo.ckan.org/api/3/action/organization_create" -d '{"name":"frb"}' -H "Authorization: xxxxxxxx"
Creating a package/dataset within the above organization; the package name is frb_test, because our entity will be in the FIWARE service frb and in the FIWARE service path test:
$ curl -X POST "http://demo.ckan.org/api/3/action/package_create" -d '{"name":"frb_test","owner_org":"frb"}' -H "Authorization: xxxxxxxx"
Creating a resource within the above package/dataset (the package ID is given in the response to the above package creation request); the name of the resource is room1_room because the entity ID will be room1 and its type room:
$ curl -X POST "http://demo.ckan.org/api/3/action/resource_create" -d '{"name":"room1_room","url":"none","format":"","package_id":"d35fca28-732f-4096-8376-944563f175ba"}' -H "Authorization: xxxxxxxx"
Finally, and answering to your question, creating a datastore associated to the above resource and suitable for receiving Cgynus data in column mode (the resource ID is given in the response to the above resource creation request):
$ curl -X POST "http://demo.ckan.org/api/3/action/datastore_create" -d '{"fields":[{"id":"recvTime","type":"text"}, {"id":"fiwareServicePath","type":"text"}, {"id":"entityId","type":"text"}, {"id":"entityType","type":"text"}, {"id":"temperature","type":"float"}, {"id":"temperature_md","type":"json"}],"resource_id":"48c120df-5bcd-48c7-81fa-8ecf4e4ef9d7","force":"true"}' -H "Authorization: xxxxxxxx"
Now, Cygnus is able to persist data for an entity with ID room1 of type room in the frb service, test service path:
time=2016-04-26T15:54:45.753CEST | lvl=INFO | corr=b465ffb8-710f-4cd3-9573-dc3799f774f9 | trans=b465ffb8-710f-4cd3-9573-dc3799f774f9 | svc=frb | subsvc=/test | function=getEvents | comp=cygnusagent | msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[240] : Starting internal transaction (b465ffb8-710f-4cd3-9573-dc3799f774f9)
time=2016-04-26T15:54:45.754CEST | lvl=INFO | corr=b465ffb8-710f-4cd3-9573-dc3799f774f9 | trans=b465ffb8-710f-4cd3-9573-dc3799f774f9 | svc=frb | subsvc=/test | function=getEvents | comp=cygnusagent | msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[256] : Received data ({ "subscriptionId" : "51c0ac9ed714fb3b37d7d5a8", "originator" : "localhost", "contextResponses" : [ { "contextElement" : { "attributes" : [ { "name" : "temperature", "type" : "centigrade", "value" : "26.5" } ], "type" : "room", "isPattern" : "false", "id" : "room1" }, "statusCode" : { "code" : "200", "reasonPhrase" : "OK" } } ]})
time=2016-04-26T15:55:07.843CEST | lvl=INFO | corr=b465ffb8-710f-4cd3-9573-dc3799f774f9 | trans=b465ffb8-710f-4cd3-9573-dc3799f774f9 | svc=frb | subsvc=/test | function=processNewBatches | comp=cygnusagent | msg=com.telefonica.iot.cygnus.sinks.NGSISink[342] : Batch accumulation time reached, the batch will be processed as it is
time=2016-04-26T15:55:07.844CEST | lvl=INFO | corr=b465ffb8-710f-4cd3-9573-dc3799f774f9 | trans=b465ffb8-710f-4cd3-9573-dc3799f774f9 | svc=frb | subsvc=/test | function=processNewBatches | comp=cygnusagent | msg=com.telefonica.iot.cygnus.sinks.NGSISink[396] : Batch completed, persisting it
time=2016-04-26T15:55:07.846CEST | lvl=INFO | corr=b465ffb8-710f-4cd3-9573-dc3799f774f9 | trans=b465ffb8-710f-4cd3-9573-dc3799f774f9 | svc=frb | subsvc=/test | function=persistAggregation | comp=cygnusagent | msg=com.telefonica.iot.cygnus.sinks.NGSICKANSink[419] : [ckan-sink] Persisting data at OrionCKANSink (orgName=frb, pkgName=frb_test, resName=room1_room, data={"recvTime": "2016-04-26T13:54:45.756Z","fiwareServicePath": "/test","entityId": "room1","entityType": "room","temperature": "26.5"})
time=2016-04-26T15:55:08.948CEST | lvl=INFO | corr=b465ffb8-710f-4cd3-9573-dc3799f774f9 | trans=b465ffb8-710f-4cd3-9573-dc3799f774f9 | svc=frb | subsvc=/test | function=processNewBatches | comp=cygnusagent | msg=com.telefonica.iot.cygnus.sinks.NGSISink[400] : Finishing internal transaction (b465ffb8-710f-4cd3-9573-dc3799f774f9)
The insertion can be checked through the CKAN API as well:
$ curl -X POST "http://demo.ckan.org/api/3/action/datastore_search" -d '{"resource_id":"48c120df-5bcd-48c7-81fa-8ecf4e4ef9d7"}' -H "Authorization: xxxxxxxx"
I approvisionned my device like this in the IoT agent:
curl -X POST http://192.168.1.110:80/iot/devices \
-i \
-H "Content-Type: application/json" \
-H "Fiware-Service: egmsmartcity" \
-H "Fiware-ServicePath: /egmsmartcitypath" \
-d ' { "devices": [ { "device_id": "test",
"entity_name": "sensors:test",
"entity_type": "sensors",
"protocol": "PDI-IoTA-MQTT-UltraLight",
"timezone": "Europe/Paris",
"endpoint": "notUsed",
"commands": [{ "name": "ping", "type": "command", "value": ""}] } ] }'
It is successufl registered in the context Broker:
GET on: http://192.168.1.77:1026/v1/contextEntities/sensors:test
with headers:
Accept: application/json
Fiware-Service: egmsmartcity
Fiware-ServicePath: /egmsmartcitypath
I got the following response:
{
contextElement: {
type: "sensors"
isPattern: "false"
id: "sensors:test"
attributes: [1]
0: {
name: "TimeInstant"
type: "ISO8601"
value: "2015-11-30T16:53:45.556269Z"
}-
-
}-
statusCode: {
code: "200"
reasonPhrase: "OK"
}-
}
and then when I try to update the command like this:
curl -X POST http://192.168.1.77:1026/v1/updateContext \
-i \
-H "Content-Type: application/json" \
-H "Fiware-Service: egmsmartcity" \
-H "Fiware-ServicePath: /egmsmartcitypath" \
-d ' {"updateAction":"UPDATE","contextElements":[{"id":"sensors:test","type":"sensors","isPattern":"false","attributes":[{"name":"ping","type":"command","value":"22" } ]} ]}'
I got this response:
HTTP/1.1 200 OK
Content-Length: 157
Content-Type: application/xml
Date: Mon, 30 Nov 2015 17:05:11 GMT
<updateContextResponse>
<errorCode>
<code>404</code>
<reasonPhrase>No context element found</reasonPhrase>
</errorCode>
</updateContextResponse>
why is it returning "No context element found" ? What have I missed ?
Thanks in advance for your help!
[Updates]
I tried a simple get on "/iot/devices"
Here is the log in the Iot Agent lg file:
In case the operation was successuful, from localhost
HTTP Version: 1.1
Resource requested: /iot/devices
Resource delivered: /iot/devices
Query string
Content length: 0
Headers: Fiware-ServicePath: /egmsmartcitypath
Fiware-Service: egmsmartcity
User-Agent: curl/7.29.0
Host: localhost
Content-Type: application/json
Accept: */*
time=2015-12-02T10:41:41,516.800CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=operator() | file=[140373934458624:rest_handle.cc:396] | msg=Processing request /iot-754c76a3-b95a-4aac-976a-08ab37119c05-34
time=2015-12-02T10:41:41,516.813CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=execute_filters | file=[140373934458624:rest_handle.cc:410] | msg=Processing request /iot-754c76a3-b95a-4aac-976a-08ab37119c05-34
time=2015-12-02T10:41:41,516.820CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=execute_filters | file=[140373934458624:rest_handle.cc:411] | msg=execute_filters status: 200
time=2015-12-02T10:41:41,516.953CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=handle_request | file=[140374294497024:media_filter.cc:56] | msg=MediaFilter handle_request /iot-754c76a3-b95a-4aac-976a-08ab37119c05-34/iot/devices
time=2015-12-02T10:41:41,516.982CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=handle_request | file=[140374294497024:media_filter.cc:95] | msg=Accept: */*
time=2015-12-02T10:41:41,517.035CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=handle_end_filters | file=[140373934458624:rest_handle.cc:446] | msg=Processing request /iot-754c76a3-b95a-4aac-976a-08ab37119c05-34
time=2015-12-02T10:41:41,517.046CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=handle_end_filters | file=[140373934458624:rest_handle.cc:447] | msg=End filters status 200
time=2015-12-02T10:41:41,517.056CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=handle_request | file=[140373934458624:rest_handle.cc:492] | msg=Processing request /iot-754c76a3-b95a-4aac-976a-08ab37119c05-34
time=2015-12-02T10:41:41,517.062CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=handle_request | file=[140373934458624:rest_handle.cc:493] | msg=Proccessing in handle /iot
time=2015-12-02T10:41:41,517.180CET | lvl= INFO | comp=iota:IoTAgent-MQTT-UL20 | op=devices | file=[140373934458624:admin_service.cc:586] | msg=iota::AdminService::devices method:GET trace_message:/iot-754c76a3-b95a-4aac-976a-08ab37119c05-34
time=2015-12-02T10:41:41,517.215CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=get_all_devices_json | file=[140373934458624:admin_service.cc:1748] | msg=get_all_devices_json service=egmsmartcity service_path=/egmsmartcitypath detailed=
time=2015-12-02T10:41:41,517.234CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=conn | file=[140373934458624:mongo_connection.cc:261] | msg=MongoConnection::conn returns 0x20ba8f0
time=2015-12-02T10:41:41,517.262CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=count | file=[140373934458624:collection.cc:833] | msg=Collection:count bbdd=iot.DEVICE options= query={ service: "egmsmartcity", service_path: "/egmsmartcitypath" }
time=2015-12-02T10:41:41,517.524CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=find | file=[140373934458624:collection.cc:704] | msg=Collection:find bbdd=iot.DEVICE query={ service: "egmsmartcity", service_path: "/egmsmartcitypath" } limit=20 offset=0 options=-2147483648
time=2015-12-02T10:41:41,517.546CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=find | file=[140373934458624:collection.cc:718] | msg=before getConnection
time=2015-12-02T10:41:41,517.553CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=find | file=[140373934458624:collection.cc:723] | msg=before query
time=2015-12-02T10:41:41,517.737CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=create_response | file=[140373934458624:admin_service.cc:1309] | msg=create_response: 200 { "count": 2,"devices": [{ "device_id" : "sensorTmp" },{ "device_id" : "test" }]}
time=2015-12-02T10:41:41,517.766CET | lvl= INFO | comp=iota:IoTAgent-MQTT-UL20 | op=devices | file=[140373934458624:admin_service.cc:718] | msg=iota::AdminService::devices method:GET trace_message:/iot-754c76a3-b95a-4aac-976a-08ab37119c05-34 code: 200 response:{ "count": 2,"devices": [{ "device_id" : "sensorTmp" },{ "device_id" : "test" }]}
time=2015-12-02T10:41:41,517.902CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=finish | file=[140373934458624:rest_handle.cc:567] | msg=finish connection 2
In case the operation was not successuful, from Rest client
HTTP Version: 1.1
Resource requested: /iot/devices/
Resource delivered: /iot/devices/
Query string
Content length: 0
Headers: Fiware-ServicePath: /egmsmartcitypath
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
Fiware-Service: egmsmartcity
Connection: keep-alive
Host: 192.168.1.110
Accept-Language: fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4
Accept-Encoding: gzip, deflate, sdch
Accept: */*
Content-Type: application/json
time=2015-12-02T10:39:55,246.079CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=operator() | file=[140373942851328:rest_handle.cc:396] | msg=Processing request /iot-c7a1ab03-73e6-4f9b-82de-d8f569927296-33
time=2015-12-02T10:39:55,246.079CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=execute_filters | file=[140373942851328:rest_handle.cc:410] | msg=Processing request /iot-c7a1ab03-73e6-4f9b-82de-d8f569927296-33
time=2015-12-02T10:39:55,246.079CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=execute_filters | file=[140373942851328:rest_handle.cc:411] | msg=execute_filters status: 200
time=2015-12-02T10:39:55,246.156CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=handle_request | file=[140374294497024:media_filter.cc:56] | msg=MediaFilter handle_request /iot-c7a1ab03-73e6-4f9b-82de-d8f569927296-33/iot/devices/
time=2015-12-02T10:39:55,246.191CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=handle_request | file=[140374294497024:media_filter.cc:95] | msg=Accept: */*
time=2015-12-02T10:39:55,246.338CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=handle_end_filters | file=[140373942851328:rest_handle.cc:446] | msg=Processing request /iot-c7a1ab03-73e6-4f9b-82de-d8f569927296-33
time=2015-12-02T10:39:55,246.350CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=handle_end_filters | file=[140373942851328:rest_handle.cc:447] | msg=End filters status 200
time=2015-12-02T10:39:55,246.360CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=handle_request | file=[140373942851328:rest_handle.cc:492] | msg=Processing request /iot-c7a1ab03-73e6-4f9b-82de-d8f569927296-33
time=2015-12-02T10:39:55,246.366CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=handle_request | file=[140373942851328:rest_handle.cc:493] | msg=Proccessing in handle /iot
time=2015-12-02T10:39:55,246.586CET | lvl=DEBUG | comp=iota:IoTAgent-MQTT-UL20 | op=finish | file=[140373942851328:rest_handle.cc:567] | msg=finish connection 2
Seems like the iot agent finish the opration before teminating the request handling
It seems the IoT Agent has not been able to answer the updateContext request. When you declare an attribute as a command, the IoT Agent registers itself as the Context Provider of that attribute for the Device's entity in the Context Broker. If an UPDATE request comes to the CB for that attribute, that request will be forwarded to the IoTA. If the IoTA can't answer or is not found, the request will fail.
The probable cause of this problem is the misconfiguration of the public IP. If the Context Broker can't find the provider of the attribute you want to update, it will return a 404 error. Did you configure the public IP in your IoT Agent?
This is a cygnus agent log:
16 Sep 2015 12:30:19,820 INFO [521330370#qtp-1739580287-1] (com.telefonica.iot.cygnus.handlers.OrionRestHandler.getEvents:236) - Received data (<notifyContextRequest><subscriptionId>55f932e6c06c4173451bbe1c</subscriptionId><originator>localhost</originator>...<contextAttribute><name>utctime</name><type>string</type><contextValue>2015-9-16 9:37:52</contextValue></contextAttribute></contextAttributeList></contextElement><statusCode><code>200</code><reasonPhrase>OK</reasonPhrase></statusCode></contextElementResponse></contextResponseList></notifyContextRequest>)
16 Sep 2015 12:30:19,820 INFO [521330370#qtp-1739580287-1] (com.telefonica.iot.cygnus.handlers.OrionRestHandler.getEvents:258) - Event put in the channel (id=1145647744, ttl=0)
16 Sep 2015 12:30:19,820 WARN [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:184) -
16 Sep 2015 12:30:19,820 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:193) - Finishing transaction (1442395508-572-0000013907)
We conserve the same configuration than this question:
Fiware Cygnus Error.
Although the Cygnus agent receives data correctly from the Context Broker suscription, Cosmos doesn't receive any data.
Thanks in advance, again :)
Independentely of the reason that leaded you to comment the grouing rules part (nevertheless, I think it was because my own wrong advice at https://jira.fiware.org/browse/HELC-986 :)), that part cannot be commented and must be added to the configuration:
# Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts gi
# TimestampInterceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
# GroupinInterceptor, do not change
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
# Grouping rules for the GroupingInterceptor, put the right absolute path to the file if necessary
# See the doc/design/interceptors document for more details
cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
Once added that part, most probably another problem will arise: the performance of Cygnus will be very poor (that was the reason I wrongly adviced the user at https://jira.fiware.org/browse/HELC-986 to comment the grouping feature, in an attempt to increase the performance by removing processing steps). The reason is the latest version of Cygnus (0.8.2) is not ready to deal with the HiveServer2 running in the Cosmos side (this server was recently upgraded from old HiveServer1 to HiveServer2) and each persistence operation delays for a lot. For instance:
time=2015-09-21T12:42:30.405CEST | lvl=INFO | trans=1442832138-143-0000000000 | function=getEvents | comp=Cygnus | msg=com.telefonica.iot.cygnus.handlers.OrionRestHandler[150] : Starting transaction (1442832138-143-0000000000)
time=2015-09-21T12:42:30.407CEST | lvl=INFO | trans=1442832138-143-0000000000 | function=getEvents | comp=Cygnus | msg=com.telefonica.iot.cygnus.handlers.OrionRestHandler[236] : Received data ({ "subscriptionId" : "51c0ac9ed714fb3b37d7d5a8", "originator" : "localhost", "contextResponses" : [ { "contextElement" : { "attributes" : [ { "name" : "temperature", "type" : "centigrade", "value" : "26.5" } ], "type" : "Room", "isPattern" : "false", "id" : "Room1" }, "statusCode" : { "code" : "200", "reasonPhrase" : "OK" } } ]})
time=2015-09-21T12:42:30.409CEST | lvl=INFO | trans=1442832138-143-0000000000 | function=getEvents | comp=Cygnus | msg=com.telefonica.iot.cygnus.handlers.OrionRestHandler[258] : Event put in the channel (id=1966649489, ttl=10)
time=2015-09-21T12:42:30.462CEST | lvl=INFO | trans=1442832138-143-0000000000 | function=process | comp=Cygnus | msg=com.telefonica.iot.cygnus.sinks.OrionSink[128] : Event got from the channel (id=1966649489, headers={fiware-servicepath=rooms, destination=other_rooms, content-type=application/json, fiware-service=deleteme2, ttl=10, transactionId=1442832138-143-0000000000, timestamp=1442832150410}, bodyLength=460)
time=2015-09-21T12:42:30.847CEST | lvl=INFO | trans=1442832138-143-0000000000 | function=persist | comp=Cygnus | msg=com.telefonica.iot.cygnus.sinks.OrionHDFSSink[330] : [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (deleteme2/rooms/other_rooms/other_rooms.txt), Data ({"recvTimeTs":"1442832150","recvTime":"2015-09-21T10:42:30.410Z","entityId":"Room1","entityType":"Room","attrName":"temperature","attrType":"centigrade","attrValue":"26.5","attrMd":[]})
time=2015-09-21T12:42:31.529CEST | lvl=INFO | trans=1442832138-143-0000000000 | function=provisionHiveTable | comp=Cygnus | msg=com.telefonica.iot.cygnus.backends.hdfs.HDFSBackendImpl[185] : Creating Hive external table=frb_deleteme2_rooms_other_rooms_row
(a big timeout)
A workaround is to configure as hive_host an unreachable IP address such as fake.cosmos.lab.fiware.org:
time=2015-09-21T12:44:58.278CEST | lvl=INFO | trans=1442832280-746-0000000001 | function=getEvents | comp=Cygnus | msg=com.telefonica.iot.cygnus.handlers.OrionRestHandler[150] : Starting transaction (1442832280-746-0000000001)
time=2015-09-21T12:44:58.280CEST | lvl=INFO | trans=1442832280-746-0000000001 | function=getEvents | comp=Cygnus | msg=com.telefonica.iot.cygnus.handlers.OrionRestHandler[236] : Received data ({ "subscriptionId" : "51c0ac9ed714fb3b37d7d5a8", "originator" : "localhost", "contextResponses" : [ { "contextElement" : { "attributes" : [ { "name" : "temperature", "type" : "centigrade", "value" : "26.5" } ], "type" : "Room", "isPattern" : "false", "id" : "Room1" }, "statusCode" : { "code" : "200", "reasonPhrase" : "OK" } } ]})
time=2015-09-21T12:44:58.280CEST | lvl=INFO | trans=1442832280-746-0000000001 | function=getEvents | comp=Cygnus | msg=com.telefonica.iot.cygnus.handlers.OrionRestHandler[258] : Event put in the channel (id=1640732993, ttl=10)
time=2015-09-21T12:44:58.283CEST | lvl=INFO | trans=1442832280-746-0000000001 | function=process | comp=Cygnus | msg=com.telefonica.iot.cygnus.sinks.OrionSink[128] : Event got from the channel (id=1640732993, headers={fiware-servicepath=rooms, destination=other_rooms, content-type=application/json, fiware-service=deleteme3, ttl=10, transactionId=1442832280-746-0000000001, timestamp=1442832298280}, bodyLength=460)
time=2015-09-21T12:44:58.527CEST | lvl=INFO | trans=1442832280-746-0000000001 | function=persist | comp=Cygnus | msg=com.telefonica.iot.cygnus.sinks.OrionHDFSSink[330] : [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (deleteme3/rooms/other_rooms/other_rooms.txt), Data ({"recvTimeTs":"1442832298","recvTime":"2015-09-21T10:44:58.280Z","entityId":"Room1","entityType":"Room","attrName":"temperature","attrType":"centigrade","attrValue":"26.5","attrMd":[]})
time=2015-09-21T12:44:59.148CEST | lvl=INFO | trans=1442832280-746-0000000001 | function=provisionHiveTable | comp=Cygnus | msg=com.telefonica.iot.cygnus.backends.hdfs.HDFSBackendImpl[185] : Creating Hive external table=frb_deleteme3_rooms_other_rooms_row
time=2015-09-21T12:44:59.304CEST | lvl=ERROR | trans=1442832280-746-0000000001 | function=doCreateTable | comp=Cygnus | msg=com.telefonica.iot.cygnus.backends.hive.HiveBackend[77] : Runtime error (The Hive table cannot be created. Hive query='create external table frb_deleteme3_rooms_other_rooms_row (recvTimeTs bigint, recvTime string, entityId string, entityType string, attrName string, attrType string, attrValue string, attrMd array<string>) row format serde 'org.openx.data.jsonserde.JsonSerDe' location '/user/frb/deleteme3/rooms/other_rooms''. Details=Could not establish connection to fake.cosmos.lab.fiware.org:10000/default?user=frb&password=llBl3dQsMhX2sEPtPuf3izUGS92RZo: java.net.UnknownHostException: fake.cosmos.lab.fiware.org)
time=2015-09-21T12:44:59.305CEST | lvl=WARN | trans=1442832280-746-0000000001 | function=provisionHiveTable | comp=Cygnus | msg=com.telefonica.iot.cygnus.backends.hdfs.HDFSBackendImpl[210] : The HiveQL external table could not be created, but Cygnus can continue working... Check your Hive/Shark installation
time=2015-09-21T12:44:59.305CEST | lvl=INFO | trans=1442832280-746-0000000001 | function=process | comp=Cygnus | msg=com.telefonica.iot.cygnus.sinks.OrionSink[193] : Finishing transaction (1442832280-746-0000000001)
This will allow Cygnus to continue, despite the Hive tables are not automatically created, a minor problem (anyway, they would have never been created because of the current incompatibility with HiveServer2). Of course, this will be fixed in Cygnus 0.9.0 (it will be released at the end of September 2015).