No substitution in httpCustom payload - fiware

I would like to make a httpCustom payload in OCB but no replace info properly. I think i have tested all the ways i know but no results, someone who could help me. This is my code:
"notification": {
"httpCustom": {
"url": "http://xxxx.xxxx.xxxx:8080/api/v1/telemetry",
"payload": "[{ %22temperature%22: %22${id}%22, %22humidity%22: %22${humidity}%22, %22battery%22: %22${battery}%22 }]"
},
"attrs": [
"temperature","humidity","battery"
]
},
I have no error when i do subscription, but when i test in my end point no replace for the macros ${...}, the payload take object information compose but with no values.
I have test to write/send ${id} as then value of a field in payload and no substitution at all. Test it with URL encode and %22 and no success, I think could be disable substitution ? but i have check it and it has FALSE value.
This is a http response:
{
"method": "POST",
"path": "/",
"query": {},
"headers": {
"x-forwarded-for": "3.124.211.58",
"x-forwarded-proto": "https",
"x-forwarded-port": "443",
"host": "83efe9565d48d8bc8cf298d7786b8042.m.pipedream.net",
"x-amzn-trace-id": "Root=1-5f1bddce-7f2b4277458e77b98c0920d1",
"content-length": "54",
"user-agent": "orion/2.1.0 libcurl/7.29.0",
"fiware-service": "example",
"fiware-servicepath": "/example",
"accept": "application/json",
"content-type": "application/json",
"fiware-correlator": "a7fcfe32-ce47-11ea-9723-0242ac14000a",
"ngsiv2-attrsformat": "custom"
},
"bodyRaw": "[{ \"temperature\": \"\", \"humidity\": \"\", \"battery\": \"\" }]",
"body": [
{
"temperature": "",
"humidity": "",
"battery": ""
}
]
}
Orion VersiĆ³n: 2.1.0 and tested in 2.4.0
Any help ? Thanks in advance !!

I have done the following test, with Orion 2.4.0. Orion database is empty before starting the test.
First, create this subscription:
curl -v localhost:1026/v2/subscriptions -s -S -H 'Content-Type: application/json' -d #- <<EOF
{
"subject": {
"entities": [
{
"id": "Device1",
"type": "Device"
}
]
},
"notification": {
"httpCustom": {
"url": "http://localhost:1027/api/v1/telemetry",
"payload": "[{ %22temperature%22: %22\${id}%22, %22humidity%22: %22\${humidity}%22, %22battery%22: %22\${battery}%22 }]"},
"attrs": [
"temperature","humidity","battery"
]
}
}
EOF
Next, start a listining process at port 1027:
nc -l -p 1027
Next, create entity as follows (which triggers a notification):
curl localhost:1026/v2/entities -s -S -H 'Content-Type: application/json' -d #- <<EOF
{
"id": "Device1",
"type": "Device",
"temperature": {
"value": 23,
"type": "Number"
},
"humidity": {
"value": 99,
"type": "Number"
},
"battery": {
"value": 15,
"type": "Number"
}
}
EOF
What I get in nc is:
POST /api/v1/telemetry HTTP/1.1
Host: localhost:1027
User-Agent: orion/2.4.0 libcurl/7.52.1
Fiware-Servicepath: /
Accept: application/json
Content-Length: 65
Content-Type: text/plain; charset=utf-8
Fiware-Correlator: 4ae608b0-d248-11ea-81de-000c29df7908
Ngsiv2-AttrsFormat: custom
[{ "temperature": "Device1", "humidity": "99", "battery": "15" }]
which is the expected result, with replacements.
Next, restart the listening process and update the entity this way (triggering a new notification):
curl localhost:1026/v2/entities/Device1/attrs?options=forcedUpdate -s -S -H 'Content-Type: application/json' -d #- <<EOF
{
"temperature": {
"value": 32,
"type": "Number"
},
"humidity": {
"value": 79,
"type": "Number"
},
"battery": {
"value": 25,
"type": "Number"
}
}
EOF
and I get now in nc:
POST /api/v1/telemetry HTTP/1.1
Host: localhost:1027
User-Agent: orion/2.4.0 libcurl/7.52.1
Fiware-Servicepath: /
Accept: application/json
Content-Length: 65
Content-Type: text/plain; charset=utf-8
Fiware-Correlator: 66278dd8-d248-11ea-822d-000c29df7908
Ngsiv2-AttrsFormat: custom
[{ "temperature": "Device1", "humidity": "79", "battery": "25" }]
Conclusion: according my tests, Orion is working as expected.
I'd suggest to have a close look to the steps above and try to identify any possible diference comparing with your case. Note the \$ in the subscription creation payload: it is needed to avoid bash vars replacement in curl. Maybe you are facing a similar problem? You can check how your subscription is with GET /v2/subscriptions or checking directly in the database (csubs collection).

Related

IFC conversionMethods "modern" and "v3" don't seem to be working

Good Day,
I'm trying to convert an IFC file using the "Revit engine".
I've tried conversionMethods "modern" and "v3".
https://forge.autodesk.com/en/docs/model-derivative/v2/reference/http/job-POST/
SVF2 Output > Formats > advanced
But the "Navisworks engine" is always used.
Here is the Request Header:
Content-Type: application/json
x-ads-derivative-format: fallback
x-ads-force: true
Authorization: Bearer ...
User-Agent: PostmanRuntime/7.29.0
Accept: */*
Cache-Control: no-cache
Postman-Token: 60ace5f6-a41c-4314-b3de-e2057f4f8795
Host: developer.api.autodesk.com
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 521
Cookie: PF=zlUuSLBAdPBYRrnJUXxwq1
Here is the Request Body:
{
"input": {
"urn": "dXJuOmFkc2sub2JqZWN0czpvcy5vYmplY3Q6bmVhbmV4LXN0YWdpbmcxLTE2MTk2MTgyOTM4MjMvMTY1NTIxOTY2OTY5MS1BQzIwLUZaSy1IYXVzLmlmYw"
},
"output": {
"destination": {
"region": "us"
},
"formats": [
{
"type": "svf2",
"views": [
"3d"
],
"advanced": {
"conversionMethod": "v3"
}
}
]
}
}
The Response:
{
"result": "success",
"urn": "dXJuOmFkc2sub2JqZWN0czpvcy5vYmplY3Q6bmVhbmV4LXN0YWdpbmcxLTE2MTk2MTgyOTM4MjMvMTY1NTIxOTY2OTY5MS1BQzIwLUZaSy1IYXVzLmlmYw",
"acceptedJobs": {
"output": {
"destination": {
"region": "us"
},
"formats": [
{
"type": "svf2",
"views": [
"3d"
],
"advanced": {
"conversionMethod": "v3"
}
}
]
}
}
}
And here is a snippet from the Manifest:
"name": "........ifc",
"progress": "complete",
"outputType": "svf2",
"properties": {
"Document Information": {
"Navisworks File Creator": "LcNwcLoaderPlugin:lcldifc",
"IFC Application Name": "ARCHICAD-64",
"IFC Application Version": "20.0.0",
"IFC Organization": "GRAPHISOFT",
"IFC Schema": "IFC4",
"IFC Loader": "3"
}
},
"status": "success"
Any ideas?
TIA
Wolfgang
It looks like you missed passing x-ads-force: true to the request header. Could you call the API like this way?
curl --location --request POST 'https://developer.api.autodesk.com/modelderivative/v2/designdata/job' \
--header 'Authorization: Bearer ' \
--header 'Content-Type: application/json' \
--header 'x-ads-force: true' \
--data-raw '{
"input": {
"urn": ""
},
"output": {
"formats": [
{
"type": "svf2",
"views": [
"3d"
],
"advanced": {
"conversionMethod": "v3"
}
}
]
}
}'
Everything works as designed, but the documentation is not clear on this issue.
The documentation says that the Revit IFC Loader is used for modern and V3:
Also, this blog post mentions that Revit will be used instead of Navisworks:
https://forge.autodesk.com/blog/model-derivative-ifc-pipeline-call-change
So you would expect that the external IDs in the properties to be Revit element IDs or uniqueIDs when using modern or V3.
But all IFC conversion methods are running on the Cloud version of Navisworks on Forge env.
Both modern and v3 IFC conversion methods or loaders are based on the Revit technics integrated with Navisworks, so they are called Revit-based IFC loaders.
You can see a similar thing on Navisworks Desktop.
You can check which conversion method was used in the Manifest (> "IFC Loader": "3").

Why a curl POST request is returning a different result than this Common Library POST request to the same address with the same body?

I am using Steel Bank Common Lisp (SBCL), Emacs, Slime, and a library called Dexador.
In the REPL, I can do:
CL-USER> (dex:post "https://httpbin.org/post"
:content "{\"user\": \"user1\", \"pass\":\"abcd\"}")
"{
\"args\": {},
\"data\": \"{\\\"user\\\": \\\"user1\\\", \\\"pass\\\": \\\"abcd\\\"}\",
\"files\": {},
\"form\": {},
\"headers\": {
\"Accept\": \"*/*\",
\"Content-Length\": \"33\",
\"Content-Type\": \"text/plain\",
\"Host\": \"httpbin.org\",
\"User-Agent\": \"Dexador/0.9.15 (SBCL 2.1.9.nixos); Linux; 5.10.94\",
\"X-Amzn-Trace-Id\": \"Root=1-62956dc4-1b95d37752a67b8420180f71\"
},
\"json\": {
\"pass\": \"abcd\",
\"user\": \"user1\"
},
\"origin\": \"189.2.84.243\",
\"url\": \"https://httpbin.org/post\"
}
"
Removing the escape character, notice the JSON key of the JSON output:
"json": {
"pass": "abcd",
"user": "user1"
}
Now, if I try doing the same on curl following this tutorial, this is what I get:
$ curl -d "user=user1&pass=abcd" -X POST https://httpbin.org/post
{
"args": {},
"data": "",
"files": {},
"form": {
"pass": "abcd",
"user": "user1"
},
"headers": {
"Accept": "*/*",
"Content-Length": "20",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "httpbin.org",
"User-Agent": "curl/7.79.1",
"X-Amzn-Trace-Id": "Root=1-62956e50-4e880d101ac23b781fe0f9d5"
},
"json": null,
"origin": "189.2.84.243",
"url": "https://httpbin.org/post"
}
Notice the "json": null,.
I was expecting both results to be the same since they are pointing to the same address and have the same body being sent.
Why are they different? Did I miss something?
Set up netcat:
nc -l -p 3500
Run your curl command against that:
curl -d "user=user1&pass=abcd" -X POST http://localhost:3500/post
Netcat prints:
POST /post HTTP/1.1
Host: localhost:3500
User-Agent: curl/7.68.0
Accept: */*
Content-Length: 20
Content-Type: application/x-www-form-urlencoded
user=user1&pass=abcd
So first, Content-Type is application/x-www-form-urlencoded, because that's what -d means to curl.
Second, the data is sent as user=user1&pass=abcd, not as JSON. That's because curl's default data mode is application/x-www-form-urlencoded, it doesn't just send it as JSON because you hoped it would.
In fact, it doesn't implement JSON at all.
If you want to send JSON, you have to do that explicitly:
jq --null-input '
.user |= "user1" |
.pass |= "abcd"' \
| curl \
-H 'Content-Type: text/plain' \
-d #- \
-X POST \
https://httpbin.org/post
Response:
{
"args": {},
"data": "{ \"user\": \"user1\", \"pass\": \"abcd\"}",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Content-Length": "36",
"Content-Type": "text/plain",
"Host": "httpbin.org",
"User-Agent": "curl/7.68.0",
"X-Amzn-Trace-Id": "Root=1-6295a78d-0d828c3c6d3891174fd22d01"
},
"json": {
"pass": "abcd",
"user": "user1"
},
"origin": "71.122.242.250",
"url": "https://httpbin.org/post"
}
If one open the main github repository, he would be able to read the very first example in the Usage part:
(dex:post "https://example.com/login"
:content '(("name" . "fukamachi") ("password" . "1ispa1ien")))
The form content is defined as a a-list or association list. Association lists are a very important thing in Common Lisp and if the function need a a-list in parameter, you should not provide a string:
(dex:post "https://httpbin.org/post" :content "{\"user\": \"user1\", \"pass\": \"abcd\"}")
^ this is a string not a a-list
The correct function call would be:
(dex:post "https://httpbin.org/post"
:content '(("user" . "user1") ("pass" . "abcd")))
And this returns:
(dex:post "https://httpbin.org/post"
:content '(("user" . "user1") ("pass" . "abcd")))
#<(SIMPLE-ARRAY CHARACTER (464)) {
"args": {},
"data": "",
"files": {},
"form": { ;;
"pass": "abcd", ;; The data is here
"user": "user1" ;;
}, ;;
"headers": {
"Content-Length": "20",
"Content-Type": "application/x-www-form-urlen... {10076E5CCF}>
200

Eclipse Ditto - Create a MQTT Connection - fails with invalid json 400 response

Establishing a connection to an MQTT 3.1.1 endpoint following the description here and
the Operating Devops Commands end up in an invalid json 400 response. Even the example MQTT-Bidirectional gets refused with a 400. So this is why i am posting this question here to get hints what i am currently doing wrong and what i can do to get it right to help others running in the same issue. Here is my curl request:
gd#gd:~/ditto/mosquitto$ curl -X POST 'http://{ditto-url}/devops/piggyback/connectivity?timeout=10000' -u devops:foobar -H 'Content-Type: application/json' -d createMQTT_connection.json
{"status":400,"error":"json.invalid","message":"Failed to parse JSON string 'createMQTT_connection.json'!","description":"Check if the JSON was valid (e.g. on https://jsonlint.com) and if it was in required format."}
The hint to check my json file gives back that my json is valid.
Here is how my json file currently looks like:
{
"targetActorSelection": "/system/sharding/connection",
"headers": {
"aggregate": false
},
"piggybackCommand": {
"type": "connectivity.commands:createConnection",
"connection": {
"id": "mqtt-example-connection-123",
"name": "mmqtt-example-connection-123",
"connectionType": "mqtt",
"connectionStatus": "open",
"failoverEnabled": true,
"uri": "tcp://{mqtt-broker-url}:1883",
"sources": [
{
"addresses": [
"{ditto-url}/#"
],
"authorizationContext": ["nginx:ditto"],
"qos": 0,
"filters": []
}
],
"targets": [
{
"address": "{ditto-url}/{{ thing:id }}",
"topics": [
"_/_/things/twin/events"
],
"authorizationContext": ["nginx:ditto"],
"qos": 0
}
]
}
}
}
Someone got an idea why this json is not valid?
Thanks for the support and hints to solve this issue!
[EDIT]
First of all a "-f" makes more sense for the curl request:
curl -X POST -u devops:foobar 'http://{ditto-url}:8080/devops/piggyback/connectivity?timeout=10000' -f createMQTT_connection_1.json
curl: (22) The requested URL returned error: 400 Bad Request
curl: (6) Could not resolve host: createMQTT_connection_1.json
Second here the update json (with the result shown above)
{
"targetActorSelection": "/system/sharding/connection",
"headers": {
"aggregate": false
},
"piggybackCommand": {
"type": "connectivity.commands:createConnection",
"connection": {
"id": "mqtt-example-connection-123",
"connectionType": "mqtt",
"connectionStatus": "open",
"failoverEnabled": true,
"uri": "tcp://{MQTT-Broker-url}:1883",
"sources": [{
"addresses": ["ditto-tutorial/#"],
"authorizationContext": ["nginx:ditto"],
"qos": 0,
"filters": []
}],
"targets": [{
"address": "ditto-tutorial/{{ thing:id }}",
"topics": [
"_/_/things/twin/events",
"_/_/things/live/messages"
],
"authorizationContext": ["nginx:ditto"],
"qos": 0
}]
}
}
}
I think the problem you face is curl related.
Please have a look here on how to send json data from a file: https://stackoverflow.com/a/18614411/5058051
Seems the # is missing in your case when specifying the file location.

Orion CB doesn't update lazy attributes on IoT Agent

I'm trying to use Orion CB as Contex Provider for an IoT Agent in which I have registred a device with lazy attributes only.
On the IoT Agent I need to handle updateContext requests so I did a handler for these requests like this:
iotAgentLib.setDataUpdateHandler(updateContextHandler);
And in the updateContextHandler function I have only one instruction:
console.log(attributes);
In order to see if all the values I want to update have been received correctly.
Now if I do an update on one of the attributes of the entity represented by the device:
curl -i -X POST \
-H "fiware-service:service1" \
-H "fiware-servicepath:/subservice1" \
-H "X-Auth-Token:wNRwDwqYlLoLD8U9sFkTAEE6PfYMbQ" \
-H "Content-Type:application/json" \
-d \
'{
"contextElements": [
{
"id": "ncc_estimate",
"attributes": [
{
"name": "arrival",
"type": "string",
"value": "some_value"
}
]
}
],
"updateAction": "UPDATE"
} ' \
'http://{orion_address}/v1/updateContext'
What I see on the IoT Agent output console is:
time=2018-01-09T08:14:59.539Z | lvl=DEBUG | corr=2f4fdb0c-f515-11e7-86b2-0242ac110003 | trans=6ac5c35d-d7bf-419c-8f64-bc843b991d47 | op=IoTAgentNGSI.GenericMiddlewares | srv=service1 | subsrv=/subservice1 | msg=Body:
{
"contextElements": [
{
"type": "nccestimate",
"isPattern": "false",
"id": "ncc_estimate",
"attributes": [
{
"name": "arrival",
"type": "string",
"value": ""
}
]
}
],
"updateAction": "UPDATE"
}
Where as you can see the value field is empty, as I can also see from the console.log() output in the UpdateHandler function that is:
[ { name: 'arrival', type: 'string', value: '' } ]
It seems that Orion is deleting the value before sending it to the IoT Agent. What could be the problem? Am I wrong doing something?
edit:
Here is the response for the call to: /v1/registry/contextEntities/ncc_estimate
{"contextRegistrationResponses":[
{"contextRegistration":
{"entities":[
{
"type":"nccestimate",
"isPattern":"false",
"id":"ncc_estimate"
}
],
"attributes":[
{
"name":"transport_type",
"type":"string",
"isDomain":"false"
},
{
"name":"arrival",
"type":"string",
"isDomain":"false"
}
],
"providingApplication":"http://192.168.199.151:4044"}
}
]}
edit2:
This is what Orion is sending to the iot agent when performing the updateContext operation described before:
POST //updateContext HTTP/1.1
User-Agent: orion/1.10.0-next libcurl/7.19.7
Host: 192.168.199.151:4044
fiware-service: service1
Fiware-ServicePath: /subservice1
X-Auth-Token: M62UkJc7yKX5aQwaHrsODfIrV4Ou85
Accept: application/json
Content-length: 169
Content-type: application/json; charset=utf-8
Fiware-Correlator: 42561e9a-f615-11e7-8610-0242ac110003
{"contextElements":[{"type":"nccestimate","isPattern":"false","id":"ncc_estimate","attributes":[{"name":"arrival","type":"string","value":""}]}],"updateAction":"UPDATE"}
As you can see the "value" field for the attribute is empty.
I'm using Orion version 1.10.0 and iot agent node lib version 2.5.1.
I have done the following test using CB same version as yours (i.e. 1.10.0)
First, create a registration as the one that IOTA would create:
(curl -s -S localhost:1026/v1/registry/registerContext -H "fiware-service:service1" -H "fiware-servicepath:/subservice1" -H 'Content-Type: application/json' -d #- | python -mjson.tool) <<EOF
{
"contextRegistrations": [
{
"entities": [
{
"type": "nccestimate",
"isPattern": "false",
"id": "ncc_estimate"
}
],
"attributes": [
{
"name": "transport_type",
"type": "string",
"isDomain": "false"
},
{
"name": "arrival",
"type": "string",
"isDomain": "false"
}
],
"providingApplication": "http://localhost:4044"
}
],
"duration": "P1M"
}
EOF
Next, check that it is exactly the same registration shown in the question post (except by the providingApplication, that points to localhost):
curl localhost:1026/v1/registry/contextEntities/ncc_estimate -s -S -H "fiware-service:service1" -H "fiware-servicepath:/subservice1" -H 'Accept: application/json' | python -mjson.tool
which response is
{
"contextRegistrationResponses": [
{
"contextRegistration": {
"attributes": [
{
"isDomain": "false",
"name": "transport_type",
"type": "string"
},
{
"isDomain": "false",
"name": "arrival",
"type": "string"
}
],
"entities": [
{
"id": "ncc_estimate",
"isPattern": "false",
"type": "nccestimate"
}
],
"providingApplication": "http://localhost:4044"
}
}
]
}
Next, run nc process at localhost on providingApplication port.
nc -l -p 4044
Once the setup is done, let's test first with an update based on the one in the question.
curl -s -S -X POST http://localhost:1026/v1/updateContext -H "fiware-service:service1" -H "fiware-servicepath:/subservice1" -H "Content-Type:application/json" -d #- <<EOF
{
"contextElements": [
{
"id": "ncc_estimate",
"attributes": [
{
"name": "arrival",
"type": "string",
"value": "some_value"
}
]
}
],
"updateAction": "UPDATE"
}
EOF
In this case, Orion doesn't recognized the registration and returns a Not Found response:
{
"contextResponses": [{
"contextElement": {
"type": "",
"isPattern": "false",
"id": "ncc_estimate",
"attributes": [{
"name": "arrival",
"type": "string",
"value": ""
}]
},
"statusCode": {
"code": "404",
"reasonPhrase": "No context element found",
"details": "ncc_estimate"
}
}]
}
In other words, Orion is not forwarding the response. I don't know why in your case is forwarded and leaves a trace in IOTA log file.
Next test uses the same request but adding a type field for the entity.
curl -s -S -X POST http://localhost:1026/v1/updateContext -H "fiware-service:service1" -H "fiware-servicepath:/subservice1" -H "Content-Type:application/json" -d #- <<EOF
{
"contextElements": [
{
"id": "ncc_estimate",
"type": "nccestimate",
"attributes": [
{
"name": "arrival",
"type": "string",
"value": "some_value"
}
]
}
],
"updateAction": "UPDATE"
}
EOF
In this case, the request is forwarded and I get this in the nc terminal.
POST //updateContext HTTP/1.1
User-Agent: orion/1.10.0 libcurl/7.38.0
Host: localhost:4044
fiware-service: service1
Fiware-ServicePath: /subservice1
Accept: application/json
Content-length: 179
Content-type: application/json; charset=utf-8
Fiware-Correlator: 42e75f8a-fa0d-11e7-93f1-000c29173617
{"contextElements":[{"type":"nccestimate","isPattern":"false","id":"ncc_estimate","attributes":[{"name":"arrival","type":"string","value":"some_value"}]}],"updateAction":"UPDATE"}
Note the some_value in the response. Orion seems to be progressing correctly the request in this case.
EDIT: according to user's feedback, entity type solved the problem. We are highlighting it in the documentation regarding Context Providers:
You should include entity type in the query/update in order for the ContextBroker to be able to forward to Context Providers

how to load data in Big Query using POST request?

I want to load data in Big Query using POST request.Getting 404 error. Please suggest where i am going wrong?
headers:
Authorization: Bearer xxxxx
Content-Type : multipart/related; boundary=foo_bar_baz
scope: https://www.googleapis.com/auth/bigquery
https://www.googleapis.com/upload/bigquery/v2/projects/demo_project/jobs?uploadType=multipart HTTP/1.1
--foo_bar_baz
Content-Type: application/json; charset=UTF-8
{
"configuration": {
"load": {
"sourceFormat": "NEWLINE_DELIMITED_JSON",
"schema": {
"fields": [
{"name": "username", "type": "STRING"},
{"name": "password", "type": "INTEGER"}
]
},
"destinationTable": {
"projectId": "universal-torch-116906",
"datasetId": "demo_project",
"tableId": "test_table"
}
}
}
}
--foo_bar_baz
Content-Type: application/octet-stream
{"username":"Kutubuddin","password":123}
--foo_bar_baz--
The issue is resolved. Removed HTTP/1.1 from URL.