Device is created when sending measures to non existing device - IoTAgent-UL - fiware

We've discovered a bug in the IoT Agent Ultralight.
If we try to send a measure to a non existing device, we'll get a 404 - DEVICE_NOT_FOUND error but at the same time a device without any attribute will be created in IoTA's and Orion CB's database.
When I say a device without any attribute I refer to the following:
{
"device_id": "test",
"service": "MyService",
"service_path": "/MyServicePath",
"entity_name": "MyEntity:test",
"entity_type": "MyEntity",
"attributes": [],
"lazy": [],
"commands": [],
"static_attributes": []
}
This is a very important bug, because it's really simple to create as many devices as someone wants and that could eat our database space.
Does someone know how to solve it?

Which version of IotAgent-ul and external components, I mean, (Fiware OCB and MongoDB) are you using? It's highly recommend to use the latest one.
It's a good way to run OCB and MongoDB using docker images
Could you provided us your commands/code that you used to send a meassure?
From my self, I tested it and it works great:
Curl commands used to send a meassure:
#!/usr/local/bin
mosquitto_pub -t /TEF/sensor03/attrs -m 't|45|c|extreme'
Response from IotAgent-ul:
{"op":"IOTAUL.Executable","time":"2018-09-21T09:59:17.906Z","lvl":"INFO","msg":"Ultralight 2.0 IoT Agent started"}
time=2018-09-21T09:59:32.679Z | lvl=DEBUG | corr=dbbcd94d-50fc-4062-a05b-bccfa76c52c8 | trans=dbbcd94d-50fc-4062-a05b-bccfa76c52c8 | op=IoTAgentNGSI.MongoDBDeviceRegister | srv=n/a | subsrv=n/a | msg=Looking for device with filter [{"id":"sensor03"}]. | comp=IoTAgent
Mongoose: mpromise (mongoose's default promise library) is deprecated, plug in your own promise library instead: http://mongoosejs.com/docs/promises.html
time=2018-09-21T09:59:32.717Z | lvl=ERROR | corr=dbbcd94d-50fc-4062-a05b-bccfa76c52c8 | trans=dbbcd94d-50fc-4062-a05b-bccfa76c52c8 | op=IOTAUL.IoTUtils | srv=n/a | subsrv=n/a | msg=MEASURES-001: Couldn't find device data for APIKey [TEF] and DeviceId[sensor03] | comp=IoTAgent
time=2018-09-21T09:59:32.717Z | lvl=ERROR | corr=dbbcd94d-50fc-4062-a05b-bccfa76c52c8 | trans=dbbcd94d-50fc-4062-a05b-bccfa76c52c8 | op=IOTAUL.Common.Binding | srv=n/a | subsrv=n/a | msg=MEASURES-005: Error before processing device measures [/TEF/sensor03/attrs] | comp=IoTAgent
time=2018-09-21T10:09:51.484Z | lvl=DEBUG | corr=dbbcd94d-50fc-4062-a05b-bccfa76c52c8 | trans=dbbcd94d-50fc-4062-a05b-bccfa76c52c8 | op=IoTAgentNGSI.MongoDBDeviceRegister | srv=n/a | subsrv=n/a | msg=Looking for device with filter [{"id":"sensor03"}]. | comp=IoTAgent
time=2018-09-21T10:09:51.504Z | lvl=ERROR | corr=dbbcd94d-50fc-4062-a05b-bccfa76c52c8 | trans=dbbcd94d-50fc-4062-a05b-bccfa76c52c8 | op=IOTAUL.IoTUtils | srv=n/a | subsrv=n/a | msg=MEASURES-001: Couldn't find device data for APIKey [TEF] and DeviceId[sensor03] | comp=IoTAgent
time=2018-09-21T10:09:51.509Z | lvl=ERROR | corr=dbbcd94d-50fc-4062-a05b-bccfa76c52c8 | trans=dbbcd94d-50fc-4062-a05b-bccfa76c52c8 | op=IOTAUL.Common.Binding | srv=n/a | subsrv=n/a | msg=MEASURES-005: Error before processing device measures [/TEF/sensor03/attrs] | comp=IoTAgent
Show data from MongoDB
root#727724bdd3d9:/# mongo --shell
MongoDB shell version: 3.2.21
connecting to: test
type "help" for help
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
http://docs.mongodb.org/
Questions? Try the support group
http://groups.google.com/group/mongodb-user
> show dbs
iotagentul 0.000GB
local 0.000GB
orion 0.000GB
It seems to me that you have something wrong about your config set up. Please provide us all the points before comment and for a nice follow up, could you open it in the Github site ? IotAgent-UL Take up there
Thanks,
Fernando Méndez - Research Junior Software Engineer

Related

How to enable NGSIv2 in iotagent-ul helmchart?

I received {"name":"BAD_REQUEST","message":"Request error connecting to the Context Broker: 501"} error when when I send a POST to iotagent-ul deployed via iotagent-ul helm chart.
time=2022-10-08T09:03:18.182Z | lvl=DEBUG | corr=d7ed414d-a132-45ea-b4a3-38ee14a828dc | trans=d7ed414d-a132-45ea-b4a3-38ee14a828dc | op=IoTAgentNGSI.Request | from=n/a | srv=openiot | subsrv=/ | msg=Response {
"error": "NotImplemented",
"description": "Only NGSIv1-based forwarding supported at the present moment. Set explictely legacyForwarding to true"
}
Following the iotagent-ul documentation to enable NGSIv2, it must be set ngsiVersion in contextBroker:
{
host: '192.168.56.101',
port: '1026',
ngsiVersion: 'v2'
}
Since I am using the official fiware helm-chart, I have updated the configmap.yaml to include the ngsiVersion
contextBroker: {
/**
* Host where the Context Broker is located.
*/
host: '{{ .Values.iota.contextBroker.host }}',
/**
* Port where the Context Broker is listening.
*/
port: '{{ .Values.iota.contextBroker.port }}',
/**
* ngsiVersion supported by the Context Broker.
*/
ngsiVersion: '{{ .Values.iota.contextBroker.ngsiVersion }}'
}
Now, I get this error:
time=2022-10-08T18:17:24.614Z | lvl=ERROR | corr=841d2b61-463e-42ff-acd7-d0f4b867b7b3 | trans=841d2b61-463e-42ff-acd7-d0f4b867b7b3 | op=IoTAgentNGSI.DeviceService | srv=openiot | subsrv=/ | msg=Registration error connecting to the Context Broker: {"code":"400","reasonPhrase":"Bad Request","details":"missing isDomain value for registration attribute"} | comp=IoTAgent
According to comments on the question, you are using Orion-LD instead of official Orion. The Orion-LD support to NGSIv2 is limited, so I'd suggest to change your context broker to official Orion newest version (3.7.0 at the moment of writing this).

"leaking due to improper teardown" in Jest

I am trying to write tests for an express API that uses Axios and is connected to a mySQL database. I am getting the following error when I run my tests in Jest:
A worker process has failed to exit gracefully and has been force exited. This is
likely caused by tests leaking due to improper teardown. Try running with --
detectOpenHandles to find leaks. Active timers can also cause this, ensure that
.unref() was called on them.
After running detectOpenHandles, I get the following:
Jest has detected the following 3 open handles potentially keeping Jest from exiting:
● TCPWRAP
15 |
16 |
> 17 | const connection = mysql.createConnection(process.env.MYSQL_CONNECTION)
| ^
18 |
19 | /**
20 | * #swagger
at new Connection (node_modules/mysql2/lib/connection.js:45:27)
at Object.createConnection (node_modules/mysql2/index.js:10:10)
at Object.<anonymous> (src/users/router.js:17:26)
at Object.<anonymous> (src/index.js:9:25)
at Object.<anonymous> (src/server.js:1:13)
at Object.<anonymous> (__tests__/app.test.js:3:13)
● TCPSERVERWRAP
3 |
4 |
> 5 | app.listen(serverPort, () =>
| ^
6 | console.log(`API Server listening on port ${serverPort}`)
7 | );
8 |
at Function.listen (node_modules/express/lib/application.js:618:24)
at Object.<anonymous> (src/server.js:5:5)
at Object.<anonymous> (__tests__/app.test.js:3:13)
● TLSWRAP
16 | const getMgmtApiJwt = async () => {
17 | try {
> 18 | const resp = await axios(newRequest);
| ^
19 | return resp.data
20 | } catch (e) {
21 | console.log("did not work");
at RedirectableRequest.Object.<anonymous>.RedirectableRequest._performRequest (node_modules/follow-redirects/index.js:279:24)
at new RedirectableRequest (node_modules/follow-redirects/index.js:61:8)
at Object.request (node_modules/follow-redirects/index.js:487:14)
at dispatchHttpRequest (node_modules/axios/lib/adapters/http.js:202:25)
at httpAdapter (node_modules/axios/lib/adapters/http.js:46:10)
at dispatchRequest (node_modules/axios/lib/core/dispatchRequest.js:53:10)
at Axios.request (node_modules/axios/lib/core/Axios.js:108:15)
at axios (node_modules/axios/lib/helpers/bind.js:9:15)
at getMgmtApiJwt (src/users/controller.js:18:24)
at Object.<anonymous> (__tests__/app.test.js:182:24)
What can I try next?
Implement a global tear down setup. That should fix this issue.
Take a look at Fiehra's answer here: jest and mongoose - jest has detected opened handles

MySQL: Cannot set MySQL server for InnoDB Cluster with dba.configureLocalInstance()

I am trying to setup a Production Deployment of InnoDB Cluster (https://dev.mysql.com/doc/refman/8.0/en/mysql-innodb-cluster-production-deployment.html) by following the steps outlined in the documentation.
I am using MySQL 5.7.25
My MySQL server starts successfully and I run (from the MySQL Shell)
dba.checkInstanceConfiguration('root#localhost:3306')
which prints
Validating local MySQL instance listening at port 3306 for use in an InnoDB cluster...
This instance reports its own address as ip-10-0-3-184.ap-south-1.compute.internal
Clients and other cluster members will communicate with it through this address by default. If this is not correct, the report_host MySQL system variable should be changed.
Checking whether existing tables comply with Group Replication requirements...
No incompatible tables detected
Checking instance configuration...
Some configuration options need to be fixed:
+----------------------------------+---------------+----------------+--------------------------------------------------+
| Variable | Current Value | Required Value | Note |
+----------------------------------+---------------+----------------+--------------------------------------------------+
| enforce_gtid_consistency | OFF | ON | Update the config file and restart the server |
| gtid_mode | OFF | ON | Update the config file and restart the server |
| log_bin | <not set> | <no value> | Update the config file |
| log_bin | OFF | ON | Update read-only variable and restart the server |
| log_slave_updates | OFF | ON | Update the config file and restart the server |
| master_info_repository | FILE | TABLE | Update the config file and restart the server |
| relay_log_info_repository | FILE | TABLE | Update the config file and restart the server |
| server_id | 0 | <unique ID> | Update the config file and restart the server |
| transaction_write_set_extraction | OFF | XXHASH64 | Update the config file and restart the server |
+----------------------------------+---------------+----------------+--------------------------------------------------+
Some variables need to be changed, but cannot be done dynamically on the server: an option file is required.
Please use the dba.configureInstance() command to repair these issues.
{
"config_errors": [
{
"action": "config_update+restart",
"current": "OFF",
"option": "enforce_gtid_consistency",
"required": "ON"
},
{
"action": "config_update+restart",
"current": "OFF",
"option": "gtid_mode",
"required": "ON"
},
{
"action": "config_update",
"current": "<not set>",
"option": "log_bin",
"required": "<no value>"
},
{
"action": "restart",
"current": "OFF",
"option": "log_bin",
"required": "ON"
},
{
"action": "config_update+restart",
"current": "OFF",
"option": "log_slave_updates",
"required": "ON"
},
{
"action": "config_update+restart",
"current": "FILE",
"option": "master_info_repository",
"required": "TABLE"
},
{
"action": "config_update+restart",
"current": "FILE",
"option": "relay_log_info_repository",
"required": "TABLE"
},
{
"action": "config_update+restart",
"current": "0",
"option": "server_id",
"required": "<unique ID>"
},
{
"action": "config_update+restart",
"current": "OFF",
"option": "transaction_write_set_extraction",
"required": "XXHASH64"
}
],
"status": "error"
(So far so good)
After this I try running dba.configureLocalInstance() but I get the following error message:
Dba.configureLocalInstance: Dba.configureLocalInstance: An open session is required to perform this operation. (RuntimeError)
I am not sure what this means but to try to get around it I run dba.configureLocalInstance('root#localhost:3306') which seems to work:
Configuring local MySQL instance listening at port 3306 for use in an InnoDB cluster...
This instance reports its own address as ip-10-0-3-184.ap-south-1.compute.internal
Clients and other cluster members will communicate with it through this address by default. If this is not correct, the report_host MySQL system variable should be changed.
WARNING: User 'root' can only connect from localhost.
If you need to manage this instance while connected from other hosts, new account(s) with the proper source address specification must be created.
1) Create remotely usable account for 'root' with same grants and password
2) Create a new admin account for InnoDB cluster with minimal required grants
3) Ignore and continue
4) Cancel
Please select an option [1]: 3
Some configuration options need to be fixed:
+----------------------------------+---------------+----------------+--------------------------------------------------+
| Variable | Current Value | Required Value | Note |
+----------------------------------+---------------+----------------+--------------------------------------------------+
| enforce_gtid_consistency | OFF | ON | Update the config file and restart the server |
| gtid_mode | OFF | ON | Update the config file and restart the server |
| log_bin | <not set> | <no value> | Update the config file |
| log_bin | OFF | ON | Update read-only variable and restart the server |
| log_slave_updates | OFF | ON | Update the config file and restart the server |
| master_info_repository | FILE | TABLE | Update the config file and restart the server |
| relay_log_info_repository | FILE | TABLE | Update the config file and restart the server |
| server_id | 0 | <unique ID> | Update the config file and restart the server |
| transaction_write_set_extraction | OFF | XXHASH64 | Update the config file and restart the server |
+----------------------------------+---------------+----------------+--------------------------------------------------+
Some variables need to be changed, but cannot be done dynamically on the server: an option file is required.
Detecting the configuration file...
Found configuration file at standard location: /etc/my.cnf
Do you want to modify this file? [y/N]: y
Do you want to perform the required configuration changes? [y/n]: y
Configuring instance...
The instance 'localhost:3306' was configured for InnoDB cluster usage.
MySQL server needs to be restarted for configuration changes to take effect.
But then after I restart MySQL server, I get the exact same output for dba.checkInstanceConfiguration('root#localhost:3306')...
I have also noticed that /etc/my.cnf has not been modified from it's default from the initial installation. In case it was a permissions issues I changed the owner of /etc/my.cnf to be owned my the user "mysql" but the result is the same...
Anybody that can shed some light onto it? Any help is appreciated.
I have been able to get past it by manually modifying my /etc/my.cnf file to include the necessary values to the variables mentioned in the output of dba.checkInstanceConfiguration('root#localhost:3306')

Error occurring as : w0 Fail to read json file with error:No JSON object could be decoded while launching couchbase image using docker-compose

I am using couchbase image with base version 4.6.3 in docker-compose . But I am getting the below error while launching docker-compose up : -
couchbase_1 | SUCCESS: init/edit couchbase.docker
couchbase_1 | SUCCESS: set hostname for couchbase.docker
couchbase_1 | SUCCESS: bucket-create
couchbase_1 | ....2017-11-13 09:57:06,301: w0 Fail to read json file with error:No JSON object could be decoded
couchbase_1 | .
couchbase_1 | bucket: ., msgs transferred...
couchbase_1 | : total | last | per sec
couchbase_1 | byte : 251515 | 251515 | 3987233.9
couchbase_1 | done
couchbase_1 | /entrypoint.sh couchbase-server
Making the Json file to compact single line resolved the error.

Rush not working on my Orion Context Broker

I have deployed an Orion Context Broker v0.20.0 instance using AWS. I'm trying to show an entity from it on the MapViewer widget of the Fiware Lab Wirecloud Mashup.
It seems like I'm having some kind of trouble with the NGSI source operator. I have configured it with:
NGSI server URL: http://MyORionInstanceIP:1026
NGSI proxy URL: https://ngsiproxy.lab.fi-ware.org (I have also tried ngsiproxy.lab.fiware.org)
I run my Conntext broker instance like this: contextBroker -port 1026 -logDir /var/log/contextBroker -pidpath /var/run/contextBroker/contextBroker.pid -dbhost localhost -db orion -t 0-255 -rush localhost:5001, and it works, but when I accept the operator configuration to create the subscription, this is what I read on my Context Broker:
DEBUG#12:07:40 senderThread.cpp[47]: sending to: host='ngsiproxy.lab.fi-ware.org', port=443, verb=POST, tenant='', service-path: '', xauthToken: '', path='/callbacks/14:27:47-1:15:08:34-1', content-type: application/xml
DEBUG#12:07:40 sem.cpp[124]: transactionIdSet taking the 'trans' semaphore for 'changing the transaction id'
DEBUG#12:07:40 sem.cpp[126]: transactionIdSet has the 'trans' semaphore
DEBUG#12:07:40 sem.cpp[181]: transactionIdSet gives the 'trans' semaphore for 'changing the transaction id'
INFO#12:07:40 clientSocketHttp.cpp[154]: Starting transaction to ngsiproxy.lab.fi-ware.org:443/callbacks/14:27:47-1:15:08:34-1
DEBUG#12:07:40 clientSocketHttp.cpp[240]: HTTP-HEADERS: 'X-relayer-host: ngsiproxy.lab.fi-ware.org:443'
DEBUG#12:07:40 clientSocketHttp.cpp[247]: HTTP-HEADERS: 'X-relayer-protocol: https'
DEBUG#12:07:40 clientSocketHttp.cpp[260]: HTTP-HEADERS: 'User-Agent: orion/0.20.0 libcurl/7.19.7'
DEBUG#12:07:40 clientSocketHttp.cpp[268]: HTTP-HEADERS: 'Host: localhost:5001'
DEBUG#12:07:40 clientSocketHttp.cpp[305]: HTTP-HEADERS: 'Content-length: 1141'
DEBUG#12:07:40 clientSocketHttp.cpp[353]: Sending message 18 to HTTP server: sending message of 1370 bytes to HTTP server
WARNING#12:07:40 clientSocketHttp.cpp[358]: Notification failure for localhost:5001 (curl_easy_perform failed: Couldn't connect to server)
INFO#12:07:40 clientSocketHttp.cpp[375]: Transaction ended
I know there is a similar question here, but it hasn't solved my problem so far, neither the Orion documentation.
I really appreciate any help you can provide.
EDIT:
It looks like that rush wasn't event installed, so I did it.
But everytime I try to run the listener it gives me this error:
time=2015-05-07T13:56:17.331Z | lvl=ERROR | op=RESPUSH BUCKET TASKS | msg=Error getting bucket elements | corr=N/A | trans=N/A | hostname=***** | component=retryBuckets | error=[Error: ERR unknown command 'evalsha']
Now, when I accept the operator, this are the traces:
-ORION:
DEBUG#15:04:28 senderThread.cpp[47]: sending to: host='ngsiproxy.lab.fi-ware.org', port=443, verb=POST, tenant='', service-path: '', xauthToken: '', path='/callbacks/13:35:20-1:18:05:22-1', content-type: application/xml
DEBUG#15:04:28 sem.cpp[124]: transactionIdSet taking the 'trans' semaphore for 'changing the transaction id'
DEBUG#15:04:28 sem.cpp[126]: transactionIdSet has the 'trans' semaphore
DEBUG#15:04:28 sem.cpp[181]: transactionIdSet gives the 'trans' semaphore for 'changing the transaction id'
INFO#15:04:28 clientSocketHttp.cpp[154]: Starting transaction to ngsiproxy.lab.fi-ware.org:443/callbacks/13:35:20-1:18:05:22-1
DEBUG#15:04:28 clientSocketHttp.cpp[240]: HTTP-HEADERS: 'X-relayer-host: ngsiproxy.lab.fi-ware.org:443'
DEBUG#15:04:28 clientSocketHttp.cpp[247]: HTTP-HEADERS: 'X-relayer-protocol: https'
DEBUG#15:04:28 clientSocketHttp.cpp[260]: HTTP-HEADERS: 'User-Agent: orion/0.20.0 libcurl/7.19.7'
DEBUG#15:04:28 clientSocketHttp.cpp[268]: HTTP-HEADERS: 'Host: localhost:5001'
DEBUG#15:04:28 clientSocketHttp.cpp[305]: HTTP-HEADERS: 'Content-length: 1141'
DEBUG#15:04:28 clientSocketHttp.cpp[353]: Sending message 1 to HTTP server: sending message of 1370 bytes to HTTP server
INFO#15:04:28 clientSocketHttp.cpp[364]: Notification Successfully Sent to localhost:5001/callbacks/13:35:20-1:18:05:22-1
INFO#15:04:28 clientSocketHttp.cpp[375]: Transaction ended
-Listener:
time=2015-05-07T15:08:12.803Z | lvl=INFO | op=RELAY REQUEST | msg=Relay Request received | corr=N/A | trans=N/A | hostname=Orion-Njoy | component=listener | userID='127.0.0.1' | reqInfo={ url: '/callbacks/13:35:20-1:18:09:06-1', method: 'POST', remoteAddress: '127.0.0.1', headers: { 'x-relayer-host': 'ngsiproxy.lab.fi-ware.org:443', 'x-relayer-protocol': 'https', 'x-relayer-proxy': undefined, 'x-relayer-retry': undefined, 'x-relayer-httpcallback': undefined, 'x-relayer-persistence': undefined, 'x-relayer-traceid': undefined, 'x-relayer-encoding': undefined, 'content-type': 'application/xml' }, responseTime: 1, statusCode: 500, bodyLength: 1141, id: { exceptionId: 'SVR1000', exceptionText: 'Generic Server Error: Error: ERR unknown command \'evalsha\'' } }
time=2015-05-07T15:08:12.803Z | lvl=INFO | op=PERSISTENCE | msg=Persistence Completed | corr=N/A | trans=e1467620-f4ca-11e4-a50f-ebe0dffc0e2e | hostname=Orion-Njoy | component=evPersistence | userID='127.0.0.1' | state='error'
-Consumer:
time=2015-05-07T13:56:17.331Z | lvl=ERROR | op=RESPUSH BUCKET TASKS | msg=Error getting bucket elements | corr=N/A | trans=N/A | hostname=Orion-Njoy | component=retryBuckets | error=[Error: ERR unknown command 'evalsha']
Any ideas, please?
EDIT 2:
I've manage to get it working thanks to the responses. The problem was that I was using redis 2.4, and it's required the 2.6 or supperior version. Now I can see my entities in the Map Viewer.
In case someone runs with the same problem, don't forget to install Rush and follow this instructions to install Redis 2.6:
Thank you all for your assistance.
Maybe the redis version is too old. EVAL/EVALSHA commands were introduced in redis 2.6.
I would check the redis version (>2.6.0)
It seems like the broker is unable to connect to rush, running in the same host and accepting connections on port 5001.
Are you sure rush is running? (and that it is on port 5001?).
The broker doesn't implement notifications (yet) in https itself, but needs rush for this purpose. But of course, rush must be running for this to work ...