Are esper time intervals supported in fiware perseo? - fiware

I'm trying to model a rule for perseo CEP using esper time intervals (timer:interval), like the one documented on http://esper.espertech.com/release-6.1.0/esper-reference/html/event_patterns.html#pattern-atoms.
In particular this example which must be fire every 20 seconds is never executed:
{
"name": "timer_interval_update",
"text": "select *,\"timer_interval_update\" as ruleName from pattern [every timer:interval(20 sec)]",
"action": {
"type": "update",
"parameters": {
"id":"${id}",
"type":"A",
"attributes": [
{
"name": "status",
"value": "open",
"type": "text"
}
]
}
}
}
In the perseo-core log I'm receiving:
time=2018-07-31T19:17:11.821Z | lvl=INFO | from= | corr=n/a | trans=n/a | srv=n/a | subsrv=n/a | op=update | comp=perseo-core | msg=rule fired: WrapperEventBean [event=MapEventBean eventType=com.espertech.esper.event.map.MapEventType#7c584419] [properties={ruleName=timer_interval_update}]
time=2018-07-31T19:17:11.827Z | lvl=ERROR | from= | corr=n/a | trans=n/a | srv=n/a | subsrv=n/a | op=DoHTTPPost | comp=perseo-core | msg=exception IOException: java.net.SocketException: Unexpected end of file from server
time=2018-07-31T19:17:11.828Z | lvl=ERROR | from= | corr=n/a | trans=n/a | srv=n/a | subsrv=n/a | op=update | comp=perseo-core | msg=action post failed
I'm using a dockerized environment using this docker-compose:
https://github.com/aguirrea/fiware-perseo-test.git

Related

Etcd: how to check that each node can see each other

I have 3 etcd nodes on VMs (not k8s).
There was such problem that nodes are alive but can't see each other, error "connection timeout" during health check. But every single node has "alive" status and zabbix with "etcd by http" template doesn't generate any alerts.
Is there any way to check nodes visibility and to monitor it using zabbix?

Depending upon the version you run, here's an example to do this with 3.5.2
Command
ETCDCTL_API=3 ./bin/etcdctl endpoint status --cluster -w table --endpoints="member1.etcd:2384,member2.etcd:2384,member3.etcd:2384"
Output:
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://member1.etcd:2384 | 17ef476d9d7fec5f | 3.5.2 | 1.5 MB | false | false | 7 | 20033 | 20033 | |
| http://member2.etcd:2384 | 31e0ca30ec3c9d94 | 3.5.2 | 1.5 MB | false | false | 7 | 20033 | 20033 | |
| http://member3.etcd:2384 | 721948abbb0522bd | 3.5.2 | 1.5 MB | false | false | 7 | 20033 | 20033 | |
+--------------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

SQOOP export HDFS to MYSQL db

I'm trying to export a HDFS to MYSQL database. I found various different solution but none of them worked, I even tried to remove the WINDOWS-1251 chars from the file.
As a small summary - I'm using virtualbox with Hortonworks image for this operations.
My HIVE in the default database:
CREATE EXTERNAL TABLE `airqualitydata`(
`sensor_id` VARCHAR(100),
`sensor_type` VARCHAR(100),
`location` VARCHAR(100),
`lat` VARCHAR(100),
`lon` VARCHAR(100),
`timestamp` timestamp,
`p1` VARCHAR(100),
`durp1` VARCHAR(100),
`ratiop1` VARCHAR(100),
`p2` VARCHAR(100),
`durp2` VARCHAR(100),
`ratiop2` VARCHAR(100))
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\073'
LOCATION 'hdfs://sandbox-hdp.hortonworks.com:8020/hadoop/airqualitydata'
TBLPROPERTIES ("skip.header.line.count"="1");
The file contained in /hadoop/airqualitydata HDFS (removed the win1251 chars just to be sure).
Note that this data can be visualized by querying SELECT * FROM airqualitydata in the hive.
sensor_id;sensor_type;location;lat;lon;timestamp;P1;durP1;ratioP1;P2;durP2;ratioP2
9710;SDS011;4894;43.226;27.934;2021-09-09T00:00:12;70;;;20;;
9710;SDS011;4894;43.226;27.934;2021-09-09T00:02:41;83;;;0.93;;
9710;SDS011;4894;43.226;27.934;2021-09-09T00:05:14;0.80;;;0.73;;
9710;SDS011;4894;43.226;27.934;2021-09-09T00:07:42;0.50;;;0.50;;
9710;SDS011;4894;43.226;27.934;2021-09-09T00:10:10;57;;;0.80;;
9710;SDS011;4894;43.226;27.934;2021-09-09T00:12:39;0.40;;;0.40;;
9710;SDS011;4894;43.226;27.934;2021-09-09T00:15:07;0.70;;;0.70;;
9710;SDS011;4894;43.226;27.934;2021-09-09T00:17:35;2;;;0.47;;
9710;SDS011;4894;43.226;27.934;2021-09-09T00:20:04;90;;;0.63;;
9710;SDS011;4894;43.226;27.934;2021-09-09T00:22:34;0.57;;;0.57;;
9710;SDS011;4894;43.226;27.934;2021-09-09T00:25:01;0.73;;;0.60;;
MYSQL DB & TABLE:
CREATE DATABASE airquality CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE TABLE `airqualitydata`(
`sensor_id` VARCHAR(100),
`sensor_type` VARCHAR(100),
`location` VARCHAR(100),
`lat` VARCHAR(100),
`lon` VARCHAR(100),
`timestamp` timestamp,
`p1` VARCHAR(100),
`durp1` VARCHAR(100),
`ratiop1` VARCHAR(100),
`p2` VARCHAR(100),
`durp2` VARCHAR(100),
`ratiop2` VARCHAR(100)
);
SQOOP CLI call:
sqoop export --connect "jdbc:mysql://localhost:3306/airquality?useUnicode=true&characterEncoding=WINDOWS-1251" --username root --password hortonworks1 --export-dir hdfs://sandbox-hdp.hortonworks.com:8020/hadoop/airqualitydata --table airqualitydata --input-fields-terminated-by "\073" --input-lines-terminated-by "\n" -m 1
I removed the ?useUnicode=true&characterEncoding=WINDOWS-1251 with no success.
I also cannot access the log from the URL given in the terminal, so I got only this as failure:
Warning: /usr/hdp/2.6.5.0-292/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
21/09/12 04:04:40 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.6.5.0-292
21/09/12 04:04:40 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
21/09/12 04:04:40 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
21/09/12 04:04:40 INFO tool.CodeGenTool: Beginning code generation
Sun Sep 12 04:04:40 UTC 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
21/09/12 04:04:40 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `airqualitydata` AS t LIMIT 1
21/09/12 04:04:40 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `airqualitydata` AS t LIMIT 1
21/09/12 04:04:40 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.6.5.0-292/hadoop-mapreduce
Note: /tmp/sqoop-raj_ops/compile/41fba9933b913b974b70403656a13287/airqualitydata.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
21/09/12 04:04:42 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-raj_ops/compile/41fba9933b913b974b70403656a13287/airqualitydata.jar
21/09/12 04:04:42 INFO mapreduce.ExportJobBase: Beginning export of airqualitydata
21/09/12 04:04:43 INFO client.RMProxy: Connecting to ResourceManager at sandbox-hdp.hortonworks.com/172.18.0.2:8032
21/09/12 04:04:43 INFO client.AHSProxy: Connecting to Application History server at sandbox-hdp.hortonworks.com/172.18.0.2:10200
21/09/12 04:04:50 INFO input.FileInputFormat: Total input paths to process : 1
21/09/12 04:04:50 INFO input.FileInputFormat: Total input paths to process : 1
21/09/12 04:04:50 INFO mapreduce.JobSubmitter: number of splits:1
21/09/12 04:04:51 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1631399426919_0028
21/09/12 04:04:51 INFO impl.YarnClientImpl: Submitted application application_1631399426919_0028
21/09/12 04:04:51 INFO mapreduce.Job: The url to track the job: http://sandbox-hdp.hortonworks.com:8088/proxy/application_1631399426919_0028/
21/09/12 04:04:51 INFO mapreduce.Job: Running job: job_1631399426919_0028
21/09/12 04:04:59 INFO mapreduce.Job: Job job_1631399426919_0028 running in uber mode : false
21/09/12 04:04:59 INFO mapreduce.Job: map 0% reduce 0%
21/09/12 04:05:03 INFO mapreduce.Job: map 100% reduce 0%
21/09/12 04:05:04 INFO mapreduce.Job: Job job_1631399426919_0028 failed with state FAILED due to: Task failed task_1631399426919_0028_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
21/09/12 04:05:04 INFO mapreduce.Job: Counters: 8
Job Counters
Failed map tasks=1
Launched map tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=2840
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=2840
Total vcore-milliseconds taken by all map tasks=2840
Total megabyte-milliseconds taken by all map tasks=710000
21/09/12 04:05:04 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
21/09/12 04:05:04 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 21.2627 seconds (0 bytes/sec)
21/09/12 04:05:04 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
21/09/12 04:05:04 INFO mapreduce.ExportJobBase: Exported 0 records.
21/09/12 04:05:04 ERROR mapreduce.ExportJobBase: Export job failed!
21/09/12 04:05:04 ERROR tool.ExportTool: Error during export: Export job failed!
Any directions will be helpful, Thanks!
EDIT #1:
As per the comments above, using:
sqoop export --connect jdbc:mysql://localhost:3306/airquality --table airqualitydata --username root --password hortonworks1 --hcatalog-database default --hcatalog-table airqualitydata --verbose
or basically (for people reproducing)
sqoop export --connect jdbc:mysql://<host:port>/<mysql db> --table <mysql table> --username <mysql_user> --password <mysqlpass> --hcatalog-database <hive_db> --hcatalog-table <hive_table> --verbose
I got it to put the data in the MYSQL. However it is putting the header row as well. Also when ran twice (I believe it should overwrite the data) it results in the data been in the table twice.
+-----------+-------------+----------+--------+--------+---------------------+------+-------+---------+------+-------+---------+
| sensor_id | sensor_type | location | lat | lon | timestamp | p1 | durp1 | ratiop1 | p2 | durp2 | ratiop2 |
+-----------+-------------+----------+--------+--------+---------------------+------+-------+---------+------+-------+---------+
| sensor_id | sensor_type | location | lat | lon | 2021-09-12 05:55:49 | P1 | durP1 | ratioP1 | P2 | durP2 | ratioP2 |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:55:49 | 70 | | | 20 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:55:49 | 83 | | | 0.93 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:55:49 | 0.80 | | | 0.73 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:55:49 | 0.50 | | | 0.50 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:55:49 | 57 | | | 0.80 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:55:49 | 0.40 | | | 0.40 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:55:49 | 0.70 | | | 0.70 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:55:49 | 2 | | | 0.47 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:55:49 | 90 | | | 0.63 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:55:49 | 0.57 | | | 0.57 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:55:49 | 0.73 | | | 0.60 | | |
| sensor_id | sensor_type | location | lat | lon | 2021-09-12 05:58:02 | P1 | durP1 | ratioP1 | P2 | durP2 | ratioP2 |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:58:02 | 70 | | | 20 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:58:02 | 83 | | | 0.93 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:58:02 | 0.80 | | | 0.73 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:58:02 | 0.50 | | | 0.50 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:58:02 | 57 | | | 0.80 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:58:02 | 0.40 | | | 0.40 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:58:02 | 0.70 | | | 0.70 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:58:02 | 2 | | | 0.47 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:58:02 | 90 | | | 0.63 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:58:02 | 0.57 | | | 0.57 | | |
| 9710 | SDS011 | 4894 | 43.226 | 27.934 | 2021-09-12 05:58:02 | 0.73 | | | 0.60 | | |
+-----------+-------------+----------+--------+--------+---------------------+------+-------+---------+------+-------+---------+
The data in HIVE is okay (no header row in there). What might cause this?
Also I have an exception but it completed overall, is this important?
21/09/12 05:57:41 INFO mapreduce.Job: Running job: job_1631399426919_0035
21/09/12 05:57:55 INFO mapreduce.Job: Job job_1631399426919_0035 running in uber mode : false
21/09/12 05:57:55 INFO mapreduce.Job: map 0% reduce 0%
21/09/12 05:58:03 INFO mapreduce.Job: map 100% reduce 0%
21/09/12 05:58:05 INFO mapreduce.Job: Job job_1631399426919_0035 completed successfully
21/09/12 05:58:06 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=345759
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=2597
HDFS: Number of bytes written=0
HDFS: Number of read operations=2
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
Job Counters
Launched map tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=4966
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=4966
Total vcore-milliseconds taken by all map tasks=4966
Total megabyte-milliseconds taken by all map tasks=1241500
Map-Reduce Framework
Map input records=12
Map output records=12
Input split bytes=1800
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=211
CPU time spent (ms)=3490
Physical memory (bytes) snapshot=217477120
Virtual memory (bytes) snapshot=1972985856
Total committed heap usage (bytes)=51380224
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=0
21/09/12 05:58:06 INFO mapreduce.ExportJobBase: Transferred 2.5361 KB in 62.3328 seconds (41.6635 bytes/sec)
21/09/12 05:58:06 INFO mapreduce.ExportJobBase: Exported 12 records.
21/09/12 05:58:06 INFO mapreduce.ExportJobBase: Publishing HCatalog export job data to Listeners
21/09/12 05:58:06 WARN mapreduce.PublishJobData: Unable to publish export data to publisher org.apache.atlas.sqoop.hook.SqoopHook
java.lang.ClassNotFoundException: org.apache.atlas.sqoop.hook.SqoopHook
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.sqoop.mapreduce.PublishJobData.publishJobData(PublishJobData.java:46)
at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:457)
at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:81)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:225)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.main(Sqoop.java:243)
21/09/12 05:58:06 DEBUG util.ClassLoaderStack: Restoring classloader: sun.misc.Launcher$AppClassLoader#4232c52b

Solution to your first problem -
--hcatalog-database mydb --hcatalog-table airquality and remove --export dir parameter.
Sqoop export cannot replace data. Pls issue a sqoop eval statement before loading main table to truncate it.
sqoop eval --connect conn_parameters --username xx --password yy --query "truncate table mytab;"
You can also use update statement to update the table too. https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html
Now, for your header issue, i think the original table may have header row. I am not sure about the data in original table. Check if the source table is properly defined in hive.

Command sent to ContextBroker doesn't reach IOT Agent

I've some problems with the commands in the IoTAgent JSON using MQTT. I follow these instructions:
POST ioat-agent:4041/iot/devices
{
"devices": [
{
"device_id": "id_lamp1",
"entity_name": "lamp1",
"entity_type": "lamp",
"attributes": [
{
"object_id": "l",
"name": "luminosity",
"type": "float"
}
],
"commands":[
{
"object_id": "u",
"name": "ping",
"type": "command"
}
],
"protocol": "IoTA-JSON",
"transport": "MQTT"
}
]
}
I try to send a command to the OCB. Something like that:
POST ocb:1026/v2/entities/lamp1/attrs?type=lamp
{
"ping":{
"type": "command",
"value": "Ping Request"
}
}
This appears to work fine, because I received in the log:
fiware-iot-agent | time=2019-05-16T09:42:00.436Z | lvl=DEBUG | corr=da768116-77be-11e9-8e17-0242ac140004 | trans=ed164f14-ffd8-404a-8da4-53b901d103b5 | op=IoTAgentNGSI.GenericMiddlewares | srv=myhome | subsrv=/environment | msg=Body:
fiware-iot-agent | {
fiware-iot-agent | "entities": [
fiware-iot-agent | {
fiware-iot-agent | "type": "lamp",
fiware-iot-agent | "isPattern": "false",
fiware-iot-agent | "id": "lamp1"
fiware-iot-agent | }
fiware-iot-agent | ],
fiware-iot-agent | "attributes": [
fiware-iot-agent | "ping"
fiware-iot-agent | ]
fiware-iot-agent | }
fiware-iot-agent | | comp=IoTAgent
fiware-iot-agent | time=2019-05-16T09:42:00.457Z | lvl=DEBUG | corr=da768116-77be-11e9-8e17-0242ac140004 | trans=ed164f14-ffd8-404a-8da4-53b901d103b5 | op=IoTAgentNGSI.ContextServer | srv=myhome | subsrv=/environment | msg=Handling query from [iot-agent:4041] | comp=IoTAgent
fiware-iot-agent | time=2019-05-16T09:42:00.462Z | lvl=DEBUG | corr=da768116-77be-11e9-8e17-0242ac140004 | trans=ed164f14-ffd8-404a-8da4-53b901d103b5 | op=IoTAgentNGSI.MongoDBDeviceRegister | srv=myhome | subsrv=/environment | msg=Looking for device with name [lamp1]. | comp=IoTAgent
fiware-iot-agent | time=2019-05-16T09:42:00.481Z | lvl=DEBUG | corr=3273f192-1cdc-42dc-80b1-9e0cab5a8a2b | trans=3273f192-1cdc-42dc-80b1-9e0cab5a8a2b | op=IoTAgentNGSI.MongoDBGroupRegister | srv=n/a | subsrv=n/a | msg=Looking for group params ["service","subservice","type"] with queryObj {"service":"myhome","subservice":"/environment","type":"lamp"} | comp=IoTAgent
fiware-iot-agent | time=2019-05-16T09:42:00.489Z | lvl=DEBUG | corr=3273f192-1cdc-42dc-80b1-9e0cab5a8a2b | trans=3273f192-1cdc-42dc-80b1-9e0cab5a8a2b | op=IoTAgentNGSI.MongoDBGroupRegister | srv=n/a | subsrv=n/a | msg=Device group for fields [["service","subservice","type"]] not found: [{"service":"myhome","subservice":"/environment","type":"lamp"}] | comp=IoTAgent
fiware-iot-agent | time=2019-05-16T09:42:00.491Z | lvl=ERROR | corr=3273f192-1cdc-42dc-80b1-9e0cab5a8a2b | trans=3273f192-1cdc-42dc-80b1-9e0cab5a8a2b | op=IoTAgentNGSI.Alarms | srv=n/a | subsrv=n/a | msg=Raising [MONGO-ALARM]: {"name":"DEVICE_GROUP_NOT_FOUND","message":"Couldn\t find device group","code":404} | comp=IoTAgent
fiware-iot-agent | time=2019-05-16T09:42:00.495Z | lvl=DEBUG | corr=3273f192-1cdc-42dc-80b1-9e0cab5a8a2b | trans=3273f192-1cdc-42dc-80b1-9e0cab5a8a2b | op=IoTAgentNGSI.ContextServer | srv=n/a | subsrv=n/a | msg=Handling received set of attributes: ["ping"] | comp=IoTAgent
fiware-iot-agent | time=2019-05-16T09:42:00.496Z | lvl=DEBUG | corr=3273f192-1cdc-42dc-80b1-9e0cab5a8a2b | trans=3273f192-1cdc-42dc-80b1-9e0cab5a8a2b | op=IoTAgentNGSI.MongoDBDeviceRegister | srv=n/a | subsrv=n/a | msg=Looking for device with name [lamp1]. | comp=IoTAgent
fiware-iot-agent | time=2019-05-16T09:42:00.506Z | lvl=ERROR | corr=3273f192-1cdc-42dc-80b1-9e0cab5a8a2b | trans=3273f192-1cdc-42dc-80b1-9e0cab5a8a2b | op=IoTAgentNGSI.Alarms | srv=n/a | subsrv=n/a | msg=Releasing [MONGO-ALARM] | comp=IoTAgent
fiware-iot-agent | time=2019-05-16T09:42:00.509Z | lvl=DEBUG | corr=3273f192-1cdc-42dc-80b1-9e0cab5a8a2b | trans=3273f192-1cdc-42dc-80b1-9e0cab5a8a2b | op=IoTAgentNGSI.ContextServer | srv=n/a | subsrv=n/a | msg=Query from [iot-agent:4041] handled successfully. | comp=IoTAgent
fiware-iot-agent | time=2019-05-16T09:42:00.511Z | lvl=DEBUG | corr=3273f192-1cdc-42dc-80b1-9e0cab5a8a2b | trans=3273f192-1cdc-42dc-80b1-9e0cab5a8a2b | op=IoTAgentNGSI.ContextServer | srv=n/a | subsrv=n/a | msg=Generated query response: {"contextResponses":[{"contextElement":{"type":"lamp","isPattern":false,"id":"lamp1","attributes":[{"name":"ping","type":"command","value":""}]},"statusCode":{"code":200,"reasonPhrase":"OK"}}]} | comp=IoTAgent
But I don't receive any message in the topic /1234/id_lamp1/cmd, and I don't see PENDING in the ring_status attribute of the entity of te OCB. Although when I simulate lamp1 device and publish a message in the /1234/id_lamp1/cmdexe the ring_status and ring_info change to OK and 22 respectively.

The problem is due to the usage of POST verb (in POST /v2/entities/lamp1/attrs?type=lamp) to send the command. POST has create semantics, so it will add the ping attribute to the entity instead of forwarding it to IOTA. Thus, the command never reaches the IOTA, so the IOTA never published in the MQTT broker. This is explained with more detail in the "Important Note" of this section of the documentation.
However, if you use PATCH instead POST (i.e.PATCH /v2/entities/lamp1/attrs?type=lamp) the forwarding takes place so IOTA receives the command and the IOTA will publish {"ping":"Ping Request"} in the /1234/id_lamp1/cmd topic in the MQTT broker.
If you already did POST /v2/entities/lamp1/attrs?type=lamp then first you have to fix the situation deleting the attribute with DELETE /v2/entities/lamp1/attrs/ping on the Orion Context Broker API. Otherwise the local ping attribute you created with POST will take preference over the ping registered attribute at IOTA when you do PATCH and forwarding will not take place.

Orion Iot/Idas has reached the maximum mongo pool

We are using Idas/Orion/Mongo(docker build) and Cygnus together in order to send data to Ckan and Cosmos.
We simulated 100 sensors that send data each 3 minutes, This approach stopped working after 2 days, I've checked both IDAS and Orion logs and see these Mongodb erros in the logs, there is not any notification coming both of the components no more.
Idas log:
failed time=2016-05-25T11:30:13,852.191UTC | lvl=ERROR | comp=iota:Manager | op=checkIndexes | file=[140414053451808:admin_service.cc:148] | msg=Check configuration, error in checkIndexes DBException can't connect couldn't connect to server 172.17.0.2:27017 (172.17.0.2), connection attempt failed
time=2016-05-25T11:30:13,853.966UTC | lvl=ERROR | comp=iota:Manager | op=conn | file=[140414053451808:mongo_connection.cc:254] | msg=It has reached the maximum mongo pool
time=2016-05-25T11:30:13,853.993UTC | lvl=ERROR | comp=iota:Manager | op=conn | file=[140414053451808:mongo_connection.cc:258] | msg=create a new con
Orion log:
time=2016-05-25T11:30:04.948UTC | lvl=INFO | trans=N/A | srv=N/A | subsrv=N/A | from=N/A | function=main | comp=Orion | msg=contextBroker.cpp[1719]: Orion Context Broker is running
time=2016-05-25T11:30:04.964UTC | lvl=ERROR | trans=N/A | srv=N/A | subsrv=N/A | from=N/A | function=mongoConnect | comp=Orion | msg=mongoConnectionPool.cpp[140]: Database Startup Error (cannot connect to mongo - doing 100 retries with a 1000 microsecond interval)
time=2016-05-25T11:30:05.969UTC | lvl=INFO | trans=N/A | srv=N/A | subsrv=N/A | from=N/A | function=mongoConnect | comp=Orion | msg=mongoConnectionPool.cpp[205]: Successful connection to database
time=2016-05-25T11:30:05.970UTC | lvl=INFO | trans=N/A | srv=N/A | subsrv=N/A | from=N/A | function=setWriteConcern | comp=Orion | msg=connectionOperations.cpp[681]: Database Operation Successful (setWriteConcern: 1)
time=2016-05-25T11:30:05.970UTC | lvl=INFO | trans=N/A | srv=N/A | subsrv=N/A | from=N/A | function=getWriteConcern | comp=Orion | msg=connectionOperations.cpp[724]: Database Operation Successful (getWriteConcern)
`
Do you think that this is related with the number of data that is being sent to Idas? and mongodb stopped due to maximum connections exceeding?
thanks

Orion shows the cannot connect to mongo - doing 100 retries with a 1000 microsecond interval error when the DB cannot be accessed, e.g. when the mongod server is down. I'm not an expert on IDAS, but I'd say that couldn't connect to server 172.17.0.2:27017 (172.17.0.2), connection attempt failed error is poiting to the same cause.
Thus, the solution to the problem is to ensure that MongoDB is up and running and accesible from Orion and IDAS.

Context Broker only responds when asked twice

So the problem is that when i try to establish a connection with the Context Broker whether i'm trying to update the entity or read the values. It only responds ok when i ask a second time.
Context Broker Version: 0.24.0 (i updated from 0.20.0 because i thought that was the problem)
Example:
python2.7 GetEntity.py entity_1
Output:
* Asking to http://188.***.**.***:1026/ngsi10/queryContext
* Headers: {'Fiware-Service': 'fiwarefinal', 'content-type': 'application/json', 'accept': 'application/json', 'X-Auth-Token': 'NULL'}
* Sending PAYLOAD:
{
"entities": [
{
"type": "",
"id": "entity_1",
"isPattern": "false"
}
],
"attributes": []
}
...
Traceback (most recent call last):
File "GetEntity.py", line 73, in <module>
r = requests.post(URL, data=PAYLOAD, headers=HEADERS)
File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 109, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))
When i do it a second time:
* Asking to http://188.***.**.***:1026/ngsi10/queryContext
* Headers: {'Fiware-Service': 'fiwarefinal', 'content-type': 'application/json', 'accept': 'application/json', 'X-Auth-Token': 'NULL'}
* Sending PAYLOAD:
{
"entities": [
{
"type": "",
"id": "entity_1",
"isPattern": "false"
}
],
"attributes": []
}
...
* Status Code: 200
* Response:
{
"contextResponses" : [
{
"contextElement" : {
"type" : "",
"isPattern" : "false",
"id" : "entity_1",
"attributes" : [
{
"name" : "humidity",
"type" : "int",
"value" : "4"
},
{
"name" : "latitude",
"type" : "float",
"value" : "3"
},
{
"name" : "longitude",
"type" : "float",
"value" : "5"
},
{
"name" : "temperature",
"type" : "int",
"value" : "8"
}
]
},
"statusCode" : {
"code" : "200",
"reasonPhrase" : "OK"
}
}
]
}
Example Nrº 2:
curl http://188.***.**.***:1026/version
Output:
curl: (56) Failure when receiving data from the peer
And when i do it a second time:
<orion>
<version>0.24.0</version>
<uptime>0 d, 0 h, 41 m, 27 s</uptime>
<git_hash><deleted for safety reasons></git_hash>
<compile_time>Mon Sep 14 17:52:44 CEST 2015</compile_time>
<compiled_by>fermin</compiled_by>
<compiled_in>centollo</compiled_in>
</orion>
Sometimes i even have to restart because when i check the status of the Context Broker it gives me an error. I'm not sure of the error but i think it's something like "the pid file doesn't exist"
EDIT:
EDIT 2:
So i tried using the script from Fiware-Figway:
python2.7 GetEntity.py entity_1
and as expected because of the problem, it worked half the times.
Full log trace for the Context Broker:
time=2015-10-01T11:31:09.269EDT | lvl=INFO | trans=N/A | function=main | comp=Orion | msg=contextBroker.cpp[1509]: Orion Context Broker is running
time=2015-10-01T11:31:09.280EDT | lvl=INFO | trans=N/A | function=mongoConnect | comp=Orion | msg=mongoConnectionPool.cpp[196]: Successful connection to database
time=2015-10-01T11:31:09.280EDT | lvl=INFO | trans=N/A | function=mongoConnect | comp=Orion | msg=mongoConnectionPool.cpp[196]: Successful connection to database
time=2015-10-01T11:31:09.281EDT | lvl=INFO | trans=N/A | function=mongoConnect | comp=Orion | msg=mongoConnectionPool.cpp[196]: Successful connection to database
time=2015-10-01T11:31:09.282EDT | lvl=INFO | trans=N/A | function=mongoConnect | comp=Orion | msg=mongoConnectionPool.cpp[196]: Successful connection to database
time=2015-10-01T11:31:09.282EDT | lvl=INFO | trans=N/A | function=mongoConnect | comp=Orion | msg=mongoConnectionPool.cpp[196]: Successful connection to database
time=2015-10-01T11:31:09.283EDT | lvl=INFO | trans=N/A | function=mongoConnect | comp=Orion | msg=mongoConnectionPool.cpp[196]: Successful connection to database
time=2015-10-01T11:31:09.283EDT | lvl=INFO | trans=N/A | function=mongoConnect | comp=Orion | msg=mongoConnectionPool.cpp[196]: Successful connection to database
time=2015-10-01T11:31:09.284EDT | lvl=INFO | trans=N/A | function=mongoConnect | comp=Orion | msg=mongoConnectionPool.cpp[196]: Successful connection to database
time=2015-10-01T11:31:09.284EDT | lvl=INFO | trans=N/A | function=mongoConnect | comp=Orion | msg=mongoConnectionPool.cpp[196]: Successful connection to database
time=2015-10-01T11:31:09.285EDT | lvl=INFO | trans=N/A | function=mongoConnect | comp=Orion | msg=mongoConnectionPool.cpp[196]: Successful connection to database
time=2015-10-01T11:31:09.285EDT | lvl=INFO | trans=N/A | function=mongoInit | comp=Orion | msg=contextBroker.cpp[1289]: Connected to mongo at localhost:orion
time=2015-10-01T11:31:09.286EDT | lvl=INFO | trans=N/A | function=getOrionDatabases | comp=Orion | msg=MongoGlobal.cpp[283]: Database Operation Successful (listDatabases command)
time=2015-10-01T11:31:09.286EDT | lvl=INFO | trans=N/A | function=subscriptionsTreat | comp=Orion | msg=MongoGlobal.cpp[2911]: Database Operation Successful ({})
time=2015-10-01T11:31:09.287EDT | lvl=INFO | trans=N/A | function=subscriptionsTreat | comp=Orion | msg=MongoGlobal.cpp[2911]: Database Operation Successful ({})
time=2015-10-01T11:31:09.288EDT | lvl=INFO | trans=N/A | function=treatOnTimeIntervalSubscriptions | comp=Orion | msg=MongoGlobal.cpp[553]: Database Operation Successful ({ conditions.type: "ONTIMEINTERVAL" })
time=2015-10-01T11:31:09.288EDT | lvl=INFO | trans=N/A | function=main | comp=Orion | msg=contextBroker.cpp[1597]: Startup completed
time=2015-10-01T11:31:18.112EDT | lvl=INFO | trans=1443713469-276-00000000001 | function=connectionTreat | comp=Orion | msg=rest.cpp[910]: Starting transaction from 127.0.0.1:45867/version
time=2015-10-01T11:31:18.114EDT | lvl=INFO | trans=1443713469-276-00000000001 | function=requestCompleted | comp=Orion | msg=rest.cpp[442]: Transaction ended
time=2015-10-01T11:31:19.289EDT | lvl=INFO | trans=N/A | function=getOrionDatabases | comp=Orion | msg=MongoGlobal.cpp[283]: Database Operation Successful (listDatabases command)
time=2015-10-01T11:31:19.290EDT | lvl=INFO | trans=N/A | function=subscriptionsTreat | comp=Orion | msg=MongoGlobal.cpp[2911]: Database Operation Successful ({})
time=2015-10-01T11:31:19.290EDT | lvl=INFO | trans=N/A | function=subscriptionsTreat | comp=Orion | msg=MongoGlobal.cpp[2911]: Database Operation Successful ({})
time=2015-10-01T11:31:29.292EDT | lvl=INFO | trans=N/A | function=getOrionDatabases | comp=Orion | msg=MongoGlobal.cpp[283]: Database Operation Successful (listDatabases command)
time=2015-10-01T11:31:29.293EDT | lvl=INFO | trans=N/A | function=subscriptionsTreat | comp=Orion | msg=MongoGlobal.cpp[2911]: Database Operation Successful ({})
time=2015-10-01T11:31:29.293EDT | lvl=INFO | trans=N/A | function=subscriptionsTreat | comp=Orion | msg=MongoGlobal.cpp[2911]: Database Operation Successful ({})
time=2015-10-01T11:31:39.294EDT | lvl=INFO | trans=N/A | function=getOrionDatabases | comp=Orion | msg=MongoGlobal.cpp[283]: Database Operation Successful (listDatabases command)
time=2015-10-01T11:31:39.295EDT | lvl=INFO | trans=N/A | function=subscriptionsTreat | comp=Orion | msg=MongoGlobal.cpp[2911]: Database Operation Successful ({})
time=2015-10-01T11:31:39.295EDT | lvl=INFO | trans=N/A | function=subscriptionsTreat | comp=Orion | msg=MongoGlobal.cpp[2911]: Database Operation Successful ({})
time=2015-10-01T11:31:39.427EDT | lvl=INFO | trans=1443713469-276-00000000002 | function=connectionTreat | comp=Orion | msg=rest.cpp[910]: Starting transaction from 188.***.**.***:58440/ngsi10/queryContext
time=2015-10-01T11:31:39.442EDT | lvl=INFO | trans=1443713469-276-00000000002 | function=entitiesQuery | comp=Orion | msg=MongoGlobal.cpp[1616]: Database Operation Successful ({ query: { $or: [ { _id.id: "entity_1" } ], _id.servicePath: { $in: [ null, /^$/, /^/.*/ ] } }, orderby: { creDate: 1 } })
time=2015-10-01T11:31:39.444EDT | lvl=INFO | trans=1443713469-276-00000000002 | function=registrationsQuery | comp=Orion | msg=MongoGlobal.cpp[2206]: Database Operation Successful ({ query: { $or: [ { contextRegistration.entities: { $in: [] } }, { contextRegistration.entities.id: { $in: [ "entity_1" ] } } ], expiration: { $gt: 1443713499 }, servicePath: { $in: [ null, /^$/, /^/.*/ ] } }, orderby: { _id: 1 } })
time=2015-10-01T11:31:39.453EDT | lvl=INFO | trans=1443713469-276-00000000002 | function=requestCompleted | comp=Orion | msg=rest.cpp[442]: Transaction ended

So as refered on this post's answer, there is a RAM memory issue regarding this problem. In fact my VM was running low on memory after a lot of testing and that itself made the Context Broker have this problem. I freed the memory and now the issue is gone.