How to process sensor time series data in batch (e.g. every 1 minute) to time series database (e.g. CrateDB) via FIWARE ORION CONTEXT BROKER? - fiware

Question

I want to process sensor data into a time series database via FIWARE ORION CONTEXT BROKER. However, instead of writing each data individually, I want to write it collectively as a 1-minute time series. Is something like this possible?
Could you help me? What path should I follow?

Answer 1

You need to split this problem into separate parts. Firstly define the payload of your batch operation. If the device can collect a series of readings and then deliver them as NGSI you can use the /v2/op/update/ for NGSI-v2 or /ngsi-ld/v1/entityOperations/upsert for NGSI-LD
For NGSI-v2 it would be something like
curl -L -X POST 'http://localhost:1026/v2/op/update/' \
-H 'Content-Type: application/json' \
--data-raw '{
"actionType":"append",
"entities":[
{
"id": "urn:ngsi-ld:TemperatureSensor:002",
"type": "TemperatureSensor",
"temperature": {
"type": "Property",
"value": 21,
"metadata":{
"unitCode": "CEL",
"TimeInstant": {
"type": "DateTime",
"value": "XXXX-XXX-XXXXX"
}
}
}
},
{
"id": "urn:ngsi-ld:TemperatureSensor:003",
"type": "TemperatureSensor",
"temperature": {
"type": "Property",
"value": 27,
"metadata":{
"unitCode": "CEL",
"TimeInstant": {
"type": "DateTime",
"value": "XXXX-XXX-XXXXX"
}
}
}
}
]
}'
For NGSI-LD it would be something like this:
curl -L -X POST 'http://localhost:1026/ngsi-ld/v1/entityOperations/upsert' \
-H 'Content-Type: application/json' \
-H 'Link: <http://context/ngsi-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
--data-raw '[
{
"id": "urn:ngsi-ld:TemperatureSensor:002",
"type": "TemperatureSensor",
"temperature": {
"type": "Property",
"value": 21,
"unitCode": "CEL",
"observedAt": "XXXXX-XXXX-XXXXXX"
}
},
{
"id": "urn:ngsi-ld:TemperatureSensor:003",
"type": "TemperatureSensor",
"temperature": {
"type": "Property",
"value": 27,
"unitCode": "CEL",
"observedAt": "XXXXX-XXXX-XXXXXX"
}
}
]'
The important thing here is to include the observedAt or TimeInstant in the payload so that it can be passed when creating a subscription into QuantumLeap (amend the fiware-service to the correct Tenant:
curl -iX POST \
'http://localhost:1026/v2/subscriptions/' \
-H 'Content-Type: application/json' \
-H 'fiware-service: openiot' \
-H 'fiware-servicepath: /' \
-d '{
"description": "Notify QuantumLeap of temperature changes",
"subject": {
"entities": [
{
"idPattern": "TemperatureSensor.*"
}
],
"condition": {
"attrs": [
"count"
]
}
},
"notification": {
"http": {
"url": "http://quantumleap:8668/v2/notify"
},
"attrs": [
"temperature"
],
"metadata": ["dateCreated", "TimeInstant"]
},
"throttling": 1
}'
Now it could be the case that you have a device writing to a file and periodically uploading a batch of readings. In this case you can create a Custom IoT Agent which deciphers the payload and transforms each reading into a batch upsert request - an example can be found in this FIWARE Video on YouTube.
The steps are always the same:
Ingest the payload holding the readings (in format X)
Transform format X to an in-memory representation.
Tweak the representation into the entities part of an NGSI request
Send the upsert to the context broker.
The subscription to QuantumLeap will persist the data to a database where you can make time-series queries. The FIWARE Tutorial will cover the steps.
Now if you are using NGSI-LD you can still use QuantumLeap, but you could also look at using the Temporal interface directly. More details in the video here