ORION: How to reduce the response time for /v2/entities - fiware

I've deployed the iot stack with docker. These are the used containers:
fiware/orion:2.1.0
Command line options used: -corsOrigin __ALL -corsMaxAge 600 -dbhost mongo-orion -logLevel DEBUG -logForHumans -t 180-199
fiware/iotagent-json:1.9.0
GIVEN: The following device template:
{
"devices": [
{
"device_id": "sen-n",
"entity_name": "sen-n",
"entity_type": "sensor",
"transport": "HTTP",
"protocol": "IoTA-JSON",
"endpoint": "http://nodered:1880/notification",
"commands": [
{
"name": "ping",
"type": "command"
}
],
"attributes": [
{
"object_id": "status",
"name": "status",
"type": "Text"
}
],
"static_attributes": [
{
"name": "name",
"type": "Text",
"value": "Sensor n"
}
]
}
]
}
WHEN: Register a large amount of devices on the IoT agent running the following shell script:
# Bash execution example: sudo chmod +x run.sh --> ./run.sh 1100
# Sh execution example: sh run.sh 1100
#!/bin/bash
for n in $(seq "$1")
do
echo "Inserting $n device"
curl --request POST \
--url 'http://localhost:4041/iot/devices' \
--header 'Content-Type: application/json' \
--header 'fiware-service: test' \
--header 'fiware-servicepath: /test' \
--data '{"devices":[{"device_id":"sen-'$n'","entity_name":"sen-'$n'","entity_type":"sensor","transport":"HTTP","protocol":"IoTA-JSON","endpoint":"http://nodered:1880/notification","commands":[{"name":"ping","type":"command"}],"attributes":[{"object_id":"status","name":"status","type":"Text"}],"static_attributes":[{"name":"name","type":"Text","value":"Sensor '$n'"}]}]}'
done
AND: Trying to get these entities from Orion.
curl --request GET \
--url 'http://localhost:1026/v2/entities/?limit=1000' \
--header 'fiware-service: test' \
--header 'fiware-servicepath: /test'
THEN: The response time is too long. It is between 13447ms and 15516ms.
I have checked the Orion command line argument -cprForwardLimit equal to 0 but this causes the execution of commands not to work. It through the following 404 error.
orion | INFO#07:32:34 logMsg.h[1832]: Starting transaction from 172.18.0.1:38752/v1/updateContext
orion | DEBUG#07:32:34 rest.cpp[1414]: Got 417 of payload of 417 bytes
orion | INFO#07:32:34 rest.cpp[885]: Service Path 0: '/test'
orion | INFO#07:32:34 connectionOperations.cpp[94]: Database Operation Successful (query: { _id.id: "sen1", _id.type: "sensor", _id.servicePath: { $in: [ /^/test$/ ] } })
orion | INFO#07:32:34 connectionOperations.cpp[177]: Database Operation Successful (query: { query: { $or: [ { contextRegistration.entities: { $in: [ { id: "sen1", type: "sensor" }, { type: "sensor", id: "sen1" } ] } }, { contextRegistration.entities.id: { $in: [] } } ], expiration: { $gt: 1552548754 }, contextRegistration.attrs.name: { $in: [ "ping" ] }, servicePath: { $in: [ /^/test$/ ] } }, orderby: { _id: 1 } })
orion | INFO#07:32:34 connectionOperations.cpp[177]: Database Operation Successful (query: { query: { $or: [ { contextRegistration.entities: { $in: [ { id: "sen1", type: "sensor" }, { type: "sensor", id: "sen1" } ] } }, { contextRegistration.entities.id: { $in: [] } } ], expiration: { $gt: 1552548754 }, servicePath: { $in: [ /^/test$/ ] } }, orderby: { _id: 1 } })
orion | DEBUG#07:32:34 restReply.cpp[75]: Response 6: responding with 237 bytes, Status Code 200
orion | DEBUG#07:32:34 restReply.cpp[76]: Response payload: '{"contextResponses":[{"contextElement":{"type":"sensor","isPattern":"false","id":"sen1","attributes":[{"name":"ping","type":"command","value":""}]},"statusCode":{"code":"404","reasonPhrase":"No context element found","details":"sen1"}}]}'
orion | INFO#07:32:34 logMsg.h[1916]: Transaction ended
So, is there a way to reduce this response time without breaking the command execution?

Related

Looping the list of variables of json into stage in Jenkins pipeline

I have been able to get the list of the value for a particular key which is array in the provided json to be in the shell script :
The "echo ${list.jvm.pega['jenkins-name']}" output these values :
["ent-giem-sasw02","ent-giem-sasw03","ent-giem-sasw04"]
How would I be able to loop each of the list in the in the array and pass it to the node under stage. Expected solution is to loop each of the list as
stage('delete_pegarules_schema_for_each_node') {
///loop 3 times based on the ${list.jvm.pega['jenkins-name'] output list
node("${list.jvm.pega['jenkins-name']") {
sh """
echo -e "-------------------------------System Information of current node running ----------------------------"
echo -e "Hostname:\t\t"`hostname`
"""
}
}
The script is :
#!/usr/bin/env groovy
node{
properties([
parameters([
choice(
name: 'environment',
choices: ['','upgrade', 'BV' ],
description: 'environment to choose'
),
])
])
deleteDir()
dir('dir-switch') {
stage('Checkout') {
// git branch: 'test-upgrade-json', url: 'https://gitlab.xxxxxxx/pipeline.git'
// stash includes: '**', name: 'upgrade-pega'
checkout([$class: 'GitSCM', branches: [[name: '*/test-upgrade-json']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'jenkins-user-github', url: 'https://gitlab.xxxxx/pipeline.git']]])
}
stage('Get Environment') {
sh """
ls -lart ./*
ls ${env.WORKSPACE}
cp -R ./upgrade-pega/environment/env.json /${env.WORKSPACE}/dir-switch
ls /${env.WORKSPACE}/dir-switch
"""
}
def obj = readJSON file: './env.json'
def list = obj[params.environment];
println list
list.each { println (it) }
stage('JVM check content') {
sh "echo ${list.jvm.pega['jenkins-name']}"
}
stage('delete_pegarules_schema_for_each_node') {
///loop
node("Expecting the each of the loop in the list of ${list.jvm.pega['jenkins-name']}") {
sh """
echo -e "-------------------------------System Information of current node running ----------------------------"
echo -e "Hostname:\t\t"`hostname`
"""
}
}
}
}
The json file :
{
"upgrade": {
"level": 1,
"domain": "develop.autosample.co.uk",
"resources": {
"db-schemas": {
"rule": {
"schema": "pegarules",
"database": "sas_develop",
"jvm": ["primary", "secondary"]
}
}
},
"jvm": {
"load-balancer": null,
"pega": [{
"jenkins-name": "ent-giem-sasw02",
"host": "ent-giem-sasw02",
"ip": "x.x.x.x"
},
{
"jenkins-name": "ent-giem-sasw03",
"host": "ent-giem-sasw03",
"ip": "x.x.x.x"
},
{
"jenkins-name": "ent-giem-sasw04",
"host": "ent-giem-sasw04",
"ip": "x.x.x.x"
}
],
"db": {
"primary": {
"jenkins-name": "ent-giem-sasrd26",
"host": "ent-giem-sasrd26",
"ip": "x.x.x.x",
"port": 5432
},
"secondary": {
"jenkins-name": "ent-giem-sasrd98",
"host": "ent-giem-pgrd98",
"ip": "x.x.x.x",
"port": 5432
}
}
}
}
}
I added this :
list.jvm.pega['jenkins-name'].each { elemupdate ->
echo "Item: ${elemupdate}"
stage('delete_pegarules_schema') {
node("$elemupdate") {
......
}
}

Error when invoke a PATCH command to Orion Context Broker

We're trying to invoke commands from Orion to IoT Device using PATCH and POST requests, but we're always having the following error:
Error [MANDATORY_PARAMS_NOT_FOUND] handing request:
Some of the mandatory params weren't found in the request: ["API Key","Device Id","Payload"]
Steps:
Create a "Device Entity" in Orion Context Broker
Provisioning a new Device
Create a Service in IoT JSON Agent
{
"services": [
{
"apikey": "my_api_key",
"protocol": [
"IoTA-JSON"
],
"cbroker": "orion:1026",
"entity_type": "Device",
"resource": "/iot/json"
}
]
}
Send a POST request to http://iot-agent-host:4041/v2/op/update to execute a command, via gateway, into the IoT Device (Works OK):
{
"actionType": "update",
"entities": [
{
"id": "shelly-1pm-test",
"type": "Device",
"cmd": {
"type": "command",
"value": {
"turn": "off",
"idDevice": "device001"
}
}
}
]
}
Send a PATCH request to http://context-broker-host:1026/v2/entities/entity-name/attrs to execute a command, via Orion CB, into the IoT Device, getting 204 in the response:
{
"cmd": {
"type": "command",
"value": {
"turn": "off",
"idDevice": "98cdac2e6101"
}
}
}
The command is not executed. Viewing the IoT JSON Agent logfile we can see:
[MANDATORY_PARAMS_NOT_FOUND] handing request: Some of the mandatory params weren't found in the request: ["API Key","Device Id","Payload"]
Any ideas?
Thanks in advance
MANDATORY_PARAMS_NOT_FOUND is an error message specific to the IoT Agent for JSON. It is usually invoked by an incoming measure which has failed to supply the necessary anti-spoofing information. It should not occur when a command is sent. You have not supplied the actuator provisioning call, but I assume it is something like this:
curl -L -X POST 'http://localhost:4041/iot/devices' \
-H 'fiware-service: openiot' \
-H 'fiware-servicepath: /' \
-H 'Content-Type: application/json' \
--data-raw '{
"devices": [
{
"device_id": "bell001",
"entity_name": "urn:ngsi-ld:Bell:001",
"entity_type": "Bell",
"protocol": "PDI-IoTA-JSON",
"transport": "HTTP",
"endpoint": "http://my-device/iot/bell001",
"commands": [
{
"name": "ring",
"type": "command"
}
]
}
]
}'
Where http://my-device/iot/bell001 is the listening endpoint of the actuator. At a guess your device is responding to this by sending a response to http://iot-agent/iot/json - probably a GET - specifically a response without i or k or d parameters.
i: is the device ID.
k: the API Key for the device's service.
d: the data payload
If you set the IOTA_LOG_LEVEL=DEBUG, the following transactions should occur:
Incoming Request:
op=IoTAgentNGSI.ContextServer-v2 | from=n/a | srv=openiot | subsrv=/ | msg=Handling v2 update from [iot-agent:4041] | comp=IoTAgent
op=IoTAgentNGSI.ContextServer-v2 | from=n/a | srv=openiot | subsrv=/ | msg={
"entities": [
{
"id": "Bell:001",
"type": "Bell",
"ring": {
"type": "command",
"value": "",
"metadata": {}
}
}
],
"actionType": "update"
} | comp=IoTAgent
Actuation Request:
op=IoTAgentNGSI.Request | from=n/a | srv=n/a | subsrv=n/a | msg=Options: {
"url": "http://iot-sensors:3001/iot/bell001",
"method": "POST",
"body": "{ring:''}",
"headers": {
"fiware-service": "openiot",
"fiware-servicepath": "/",
"content-type": "application/json"
},
"responseType": "text"
} | comp=IoTAgent
Set status to pending
op=IoTAgentNGSI.Entities-v2 | from=n/a | srv=n/a | subsrv=n/a | msg=Using the following NGSI v2 request:
{
"url": "http://orion:1026/v2/entities/Bell:001/attrs?type=Bell",
"method": "PATCH",
"headers": {
"fiware-service": "openiot",
"fiware-servicepath": "/"
},
"json": {
"ring_status": {
"type": "commandStatus",
"value": "PENDING",
"metadata": {
"TimeInstant": {
"type": "DateTime",
"value": "2022-07-12T06:47:59.795Z"
}
}
}
}
If the device responds with 200, set status to OK
op=IoTAgentNGSI.Entities-v2 | from=n/a | srv=openiot | subsrv=/ | msg=Updating device value in the Context Broker at [http://orion:1026/v2/op/update] | comp=IoTAgent
op=IoTAgentNGSI.Entities-v2 | from=n/a | srv=openiot | subsrv=/ | msg=Using the following NGSI v2 request:
2022-07-12T06:47:59.944482296Z
{
"url": "http://orion:1026/v2/op/update",
"method": "POST",
"headers": {
"fiware-service": "openiot",
"fiware-servicepath": "/"
},
"json": {
"actionType": "update",
"entities": [
{
"ring_status": {
"type": "commandStatus",
"value": "OK",
"metadata": {
"TimeInstant": {
"type": "DateTime",
"value": "2022-07-12T06:47:59.942Z"
}
}
},
"ring_info": {
"type": "commandResult",
"value": " ring OK",
"metadata": {
"TimeInstant": {
"type": "DateTime",
"value": "2022-07-12T06:47:59.942Z"
}
}
}
]
}
}
I'm guessing that your device is responding incorrectly to the actuation request (number 2 in the list)

curl -X POST with JSON how can I filtered Integer with less then or greater then (API is from Zabbix)"

i try to make a "maintenance.get" on Zabbix and what I need is the current active maintenance. which just working is:
curl --noproxy '*' -H "Content-Type: application/json" -X POST http://local.zabbix.lan/api_jsonrpc.php -d'
{
"jsonrpc": "2.0",
"method": "maintenance.get",
"params": {
"output": "extend",
"search": {
"name": ["Auto_"]
},
"filter": {
"active_till": "1657562960"
}
}
},
"auth": "07fxxx574444444444a65034c6xxxxx0",
"id": 1
}
' | sed 's/","/\n/g'
{"jsonrpc":"2.0
result":[{"maintenanceid":"199
name":"Auto_Maintenance_2022-07-11_16:56:53\u00b4
maintenance_type":"0
description":"
active_since":"1657551360
active_till":"1657572960
tags_evaltype":"0
timeperiods":[{"timeperiodid":"199
timeperiod_type":"3
every":"1
month":"0
dayofweek":"64
day":"0
start_time":"64800
period":"3600
start_date":"0"}]},{"maintenanceid":"200
name":"Auto_Maintenance_2022-07-11_17:51:06
maintenance_type":"0
description":"
active_since":"1657554660
active_till":"1657558260
tags_evaltype":"0
timeperiods":[{"timeperiodid":"200
timeperiod_type":"3
every":"1
month":"0
dayofweek":"64
day":"0
start_time":"64800
period":"3600
start_date":"0"}]}],"id":1}
but I get the whole list. I would need set a greater than operator to get only the number greater from today (so date in the future). what I tried with my knowledge is that but unfortunately it does not work:
{
"jsonrpc": "2.0",
"method": "maintenance.get",
"params": {
"output": "extend",
"search": {
"name": ["Auto_"]
},
"filter": {
"active_till": {
">=": [1657562960]
}
}
},
"auth": "07f186572af264203ba65034c6cc83a0",
"id": 1
}

jq command to add onto a map

Is there a command to use jq to add onto this type of map?
append an array of maps using keys (ie, vm1, vm2, vm3)
Note: I have an existing vm_map {} in a json file and i want to add to the vm_map
this is my new_json.json file
{
"gcs_config": [
{
"bucket_name": "somebucket",
"bucket_readers": [],
"bucket_writers": []
}
],
"label_application": "someapp",
"label_environment": "dev",
"lits_vm_zone": "somezone",
"project_id": "someproject",
"region": "someregion",
"storage_bucket_required": true,
"vm_map" : {}
}
expected: using jq to add onto vm_maps map. I will have an empty vm_map and each time it runs, i will add a x amount of new entries.
{
"gcs_config": [
{
"bucket_name": "somebucket",
"bucket_readers": [],
"bucket_writers": []
}
],
"label_application": "someapp",
"label_environment": "dev",
"lits_vm_zone": "zone-a",
"project_id": "someproject",
"region": "someregion",
"storage_bucket_required": true,
"vm_map": {
"vm1": {
"host": "vm1",
"network": "10.1.1.1",
"name": "vm1"
},
"vm2": {
"host": "123",
"network": "10.1.12",
"name": "vm2"
}
}
}
The file you describe is not valid JSON. I'm assuming you mean
{
"vm_map": {
"vm1": {
"host": "vm1",
"network": "xxxxx",
"name": "xxxxxxx"
},
"vm2": {
"host": "vm2",
"network": "xxxxx",
"name": "xxxxxxx"
}
}
}
You can use this:
jq \
--arg VMHOST "$VMHOST" \
--arg NETWORK_IP "$NETWORK_IP" \
--arg VM_NAME "$VM_NAME" \
'
.vm_map[ $VMHOST ] = {
host: $VMHOST,
network: $NETWORK_IP,
name: $VM_NAME
}
'

How to search a json with jq for values?

I have a json of this structure:
{
"nodes": {
"60e327ee58a0": {
"nodeinfo": {
"network": {
"mesh": {
"bat0": {
"interfaces": {
"wireless": [
"<mac-address-removed>"
],
"tunnel": [
"<mac-address-removed>"
]
}
}
},
"mac": "<mac removed>",
"addresses": [
"<ipv6 removed>",
"<ipv6 removed>"
]
},
"hardware": {
"model": "TP-Link TL-WR841N/ND v10",
"nproc": 1
},
"software": {
"batman-adv": {
"compat": 15,
"version": "2015.1"
},
"autoupdater": {
"branch": "stable",
"enabled": true
},
"firmware": {
"release": "v2016.1+1.0.1",
"base": "gluon-v2016.1"
},
"status-page": {
"api": 1
},
"fastd": {
"enabled": true,
"version": "v17"
}
},
"hostname": "Antoniusweg12",
"system": {
"site_code": "ffmsd03"
},
"node_id": "60e327ee58a0"
},
"lastseen": "2016-04-14T12:39:04",
"flags": {
"gateway": false,
"online": true
},
"firstseen": "2016-03-16T15:14:04",
"statistics": {
"clients": 1,
"gateway": "de:ad:be:ef:43:02",
"rootfs_usage": 0.6041666666666667,
"loadavg": 0.09,
"uptime": 1822037.41,
"memory_usage": 0.8124737210932025,
"traffic": {
"rx": {
"packets": 50393821,
"bytes": 5061895206
},
"forward": {
"packets": 173,
"bytes": 17417
},
"mgmt_rx": {
"packets": 47453745,
"bytes": 6623785282
},
"tx": {
"packets": 1205695,
"bytes": 173509528,
"dropped": 5683
},
"mgmt_tx": {
"packets": 37906725,
"bytes": 11475209742
}
}
}
},
"30b5c2b042f4": {
<next block...>
And I want to query it with jq for the hostname, the mac or the IPv6.
cat nodes.json |jq -c '.nodes[] | select(.nodes[]| contains("Antoniusweg12"))'
Most examples do not fit this kind of json structure as the objects have an index
Thanks for help in advance.
If you're going to filter, you need to drill down to the property that you want to check for and see if it matches your criteria. You can't expect to just give a name and you'll magically be presented with the results you want.
Searching by hostname, it is found on the .nodeinfo.hostname property of each node:
$ jq -c --arg hostname "Antoniusweg12" \
'.nodes[] | select(.nodeinfo.hostname == $hostname)' nodes.json
Similarly for the mac address, it's found on the .nodeinfo.network.mac property:
$ jq -c --arg mac "aa:bb:cc:dd:ee:ff" \
'.nodes[] | select(.nodeinfo.network.mac == $mac)' nodes.json
For the ip addresses, there's an array of them but it's not that much different in the query. They're found on the .nodeinfo.network.addresses property:
$ jq -c --arg ip "aaaa:bbbb:cccc:dddd::1" \
'.nodes[] | select(.nodeinfo.network.addresses[] == $ip)' nodes.json
Here's another take on the question. Suppose you want to find all occurrences of the key "hostname" for which the value is "Antoniusweg12",
no matter where the key/value combination occurs.
The following will reveal the path to the key/value combination of interest:
paths as $p
| select ( $p[-1] == "hostname" and getpath($p) == "Antoniusweg12" )
| $p
The result for the given input JSON:
[
"nodes",
"60e327ee58a0",
"nodeinfo",
"hostname"
]
If you wanted the path to the containing object, then replace the final $p with $p[0:-1]; and if you want the containing object itself: getpath($p[0:-1])
Here is a solution which searches for nodes where the specified $needle is present in any of the addresses, mac or hostname fields.
"<ipv6 removed>" as $needle # set to whatever you like
| foreach (.nodes|keys[]) as $k (
.
; .
; ( .nodes[$k].nodeinfo.network.addresses?
+ [ .nodes[$k].nodeinfo.network.mac?
, .nodes[$k].nodeinfo.hostname?
]
) as $haystack
| if $haystack | index($needle)
then {($k): .nodes[$k]}
else empty
end
)
EDIT: I now realize a filter of the form foreach E as $X (.; .; R) can almost always be rewritten as E as $X | R so the above is really just
"<ipv6 removed>" as $needle
| (.nodes|keys[]) as $k
| ( .nodes[$k].nodeinfo.network.addresses?
+ [ .nodes[$k].nodeinfo.network.mac?
, .nodes[$k].nodeinfo.hostname?
]
) as $haystack
| if $haystack | index($needle)
then {($k): .nodes[$k]}
else empty
end