Related
just want to know if and how I can parse a HTTP response with a dynamic name in a JSON?
I used the Azure Management API to receive the managed identities (system- and user assigned managed identities) to receive all managed identities.
With a foreach I am iterating the results.
If a resource has a system assigned managed identity and user assigned managed identity, the response looks like this:
{
"principalId": "<principalId1>",
"tenantId": "<tenantId>",
"type": "SystemAssigned, UserAssigned",
"userAssignedIdentities": {
"/subscriptions/<subscriptionId>/resourcegroups/<resourceGroupName>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<userAssignedIdentitiesName>": {
"principalId": "<principalId2>",
"clientId": "<clientId>"
}
}
}
Now, I would like to get the <principalId2>.
Unfortunately, the Name of the object is dynamic related to the scope of the resource /subscriptions/<subscriptionId>/resourcegroups/<resourceGroupName>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<userAssignedIdentitiesName>.
How can I parse the JSON to receive the needed <principalId2>?
For all other responses I can easily use the Data operations Parse JSON with the payload I inserted from the HTTP response.
Is there a way to use a wildcard? Otherwise, could I somehow just select the first object of userAssignedIdentities to receive the needed value?
Ok, this should work for you. This is the flow I tested with ...
Initialise JSON
Your JSON as a string, how you do that in your solution may differ slightly.
Initialize XPath Result
Defined as an Array and the expression is ...
xpath(xml(json(concat('{ root: ', replace(variables('JSON'), 'PrincipalId', 'principalId'), '}'))), '(//principalId)[2]')
Initialize Result
A bit more work again but defined as a String and the expression is ...
array(xpath(xml(base64ToString(variables('XPath Result')[0]?['$content'])), '//text()'))[0]
The end result should be your value ...
I have orchestrated a data pipe line using AWS Step function.
In last state I want to send a custom notification. I'm using an Intrinsic function States.Format to format my message and subject. It works fine for Context object element. Here, I have tested that in Message parameter.
But it doesn't work with input JSON. This is my input JSON
{
"job-param":{
"pipe-line-name":"My pipe line name", "other-keys":"other values"
}
}
"Success State": {
"Type": "Task",
"Resource": "arn:aws:states:::sns:publish",
"Parameters": {
"Message.$": "States.Format('Execution Id:{}, completed successfully!', $$.Execution.Id)",
"Subject.$": "States.Format('[INFO] {} completed successfully!', $.job-param.pipe-line-name)",
"TopicArn": "arn:aws:sns:us-east-1:************:sns-topic"
},
"End": true
}
While saving this state machine, it gives me following error message:
The value for the field 'Subject.$' must be a valid JSON Path
I checked Input and Result path. They have this value. I can directly use this value as parameter. This is working fine. But I can't format with other string.
"Subject.$": "$.job-param.pipe-line-name"
Alternate approach would be to call lambda to customize and trigger SNS. But I want to avoid that.
Can I request some suggestions to fix this error?
Thanks in advance!
If you want to use any name with - in your JSON then you can write your JSON Path like this:
"Subject.$": "States.Format('[INFO] {} completed successfully!', $['job-param']['pipe-line-name'])",
But it would be easier if you change your input JSON and replace - with _:
"Subject.$": "States.Format('[INFO] {} completed successfully!', $.job_param.pipe_line_name)",
I want to import data of type JSON in MongoDB compass,
the import function gives this error
" unexpected end of JSON input "
there is a some of my JSON file
[
{
"id":4,
"user":"test#example.com",
"date1":"2019-03-01",
"date2":"2019-04-01",
"statut":"Good",
"guest_number":4
}
]
the solution is to write all JSON in one line, but if we have a big doc !!
I just found a solution that I can import data with this command in terminal :
mongoimport --jsonArray --db YourDatabase --collection YourCollection --file Yourfile.json
I had this issue 6 month ago, the solution is write all JSON in one line.
[{"id":4,"user":"test#example.com","date1":"2019-03-01","date2":"2019-04-01","statut":"Good","guest_number":4}]
MongoDB Compass will told you:
Import success!
But definitely the document will not appear in your collection, so better use Robo3T if you gonna insert json. Then you can use again Compass like I do.
It is weird, yes, but I didnt found other solution yet.
[UPDATE]
I achieve import data with Compass, but I achieve exporting first a document from Compass to see how it write the json.
{"_id":{"$oid":"5e4cf105c9ba1a21143d04a2"},"tPreguntas":["Pregunta 1","Pregunta 2","Pregunta 3","Pregunta 4","Pregunta 5"],"tCategorias":[],"tPublico":true,"tFechaCreacion":{"$date":{"$numberLong":"1582100741716"}},"tCodigo":"test1","tTitulo":"Test 1","tDescripcion":"Test de muestreo número uno para comprobar.","tCreadoPor":"eoeo#eoeo.com"}
It look to different to the json online I have post in my first post. (look that objectId "$oid" for example). So if you follow that pattern Compass will import you fine.
This parsing error can be solved using minification. So, minify json like this. Although, it is quite a hectic process to do this for each object.
And this kind of minification like this worked for me.
{
"_id" : ObjectId("5b9ecf9a64f634289ca895bb"),
"name" : "Mark"
}
{
"_id" : ObjectId("5b9edd9064f634289ca895e4"),
"name" : "David"
}
To :
{"_id":"ObjectId(\"5b9ecf9a64f634289ca895bb\")","name":"Mark"}
{"_id":"ObjectId(\"5b9edd9064f634289ca895e4\")","name":"David"}
Just copy the contents of your json file then in Mongodb Compass select your database then click on Add Data which will drop down then click on insert document a dialog pops up then paste it in there and click insert.
This parsing error can be solved using minification. So, minify json like this. Although, it is quite a hectic process to do this for each object.
{
"_id" : "123456",
"name" : "stackoverflow"
}
change to :
{"_id":"123456","name":"stackoverflow"}
This answer here Solution solved the issue for me. It seems to be a formatting issue.
It's an issue with the end-of-line characters (EOL).
In a Windows environment line terminations are normally CR NL (\r\n), while MongoDB Compass seems to only support CR (\r).
You can open the file in Notepad++, enable the "Show all characters" toggle in the toolbar and inspect your current end-of-line character.
To fix the issue, select Edit > EOL Conversion > Macintosh (CR).
The structure of your JSON is incorrect, you might want to read info regarding JSON standards
A value can be a string in double quotes, or a number, or true or false or null, or an object or an array. These structures can be nested.
try using double quotes instead of single ones:
JSON validators could help you aswell
[
{
"id" : 4,
"user" : "test#example.com",
"date1" : "2019-03-01",
"date2" : "2019-04-01",
"statut" : "Good",
"guest_number" : 4
}
]
I had a similar issue but it turned out to be additional line feeds at the end of the file. Removing these fixed the issue. I suggest opening your file in an editor that shows line feeds e.g. Notepad++
Add --jsonFormat=canonical to your mongoexport script:
mongoexport --db=quotes --collection=quotes --jsonFormat=canonical --out=data/quotes.json
JSON can only directly represent a subset of the types supported by BSON. To preserve type information, MongoDB adds the following extensions to the JSON format.
Source
You can also use the command line of mongodb like this :
db.user.insert(
[
{
"id" : 4,
"user" : "test#example.com",
"date1" : "2019-03-01",
"date2" : "2019-04-01",
"statut" : "Good",
"guest_number" : 4
},
{
"id" : 5,
"user" : "test2#example.com",
"date1" : "2019-03-01",
"date2" : "2019-04-01",
"statut" : "Good",
"guest_number" : 4
}
]
Run this command in cmd and the cmd path should be in the same folder where the JSON file occurs.
mongoimport --jsonArray --db YourDatabase --collection YourCollection --file Yourfile.json
I have been trying for a few days to get a parameter sent from the API Gateway in AWS to a Lambda function and I am having no success.
I decided to start from the beginning so I followed their walkthrough (http://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started.html#getting-started-new-lambda)
I have checked this walkthrough twice and I have followed the steps to the letter.
Problem
When I test the API from Postman or in Swift I am getting the error:
{"message": "Could not parse request body into json: Unexpected character (\'-\' (code 45)) in numeric value: expected digit (0-9) to follow minus sign, for valid numeric value\n at [Source: [B#c036d15; line: 1, column: 3]"}
In postman, When I view the result as JSON I just get
Bad String
Lambda Function
The function is the basic example from the Walkthrough:
console.log('Loading event');
exports.handler = function(event, context) {
var name = (event.name === undefined ? 'No-Name' : event.name);
console.log('"Hello":"' + name + '"');
context.done(null, {"Hello":name}); // SUCCESS with message
};
When Tested from the Lambda Console and with the Test data I get the result:
{
"Hello": "TestUser123"
}
When Tested from the API Gateway Test, The result is also:
{
"Hello": "TestUser123"
}
Can anyone see why both test consoles are allowing this work but when tested with POSTMAN or used within a Swift Script it does not work ?
Edit 1
In postman, I have set the content-type to application/json
The script returns the default:
{
"Hello": "user"
}
However, When I add in the parameters name and TestUser123 in POSTMAN, this is when it returns the error.
Update 1
Ok, so I changed the mapping template to one that I found on another answer:
{ "name": "$input.params('name')" }
Now the result is:
{
"Hello": ""
}
Any Ideas why it is not getting the name?
I just got stuck with this today.
your mapping template is:
{ "name": "$input.params('name')" }
AWS uses AWS Velocity templates, which; even though looks like JSON, is different.
if you use
{ "name": $input.params('name') } // notice no quotes
for the mapping template right at the integration request step, then it should work as expected.
Read the error message very carefully, it actually tells you the problem. For example, I got
Could not parse request body into json: Unexpected character (\'\"\' (code 34)): was expecting comma to separate Object entries
So the problem is that I'm missing a comma. I check my Lambda's Integration Request - Body Mapping Template:
{
"age" : $input.json('$.persondata.age'),
"income" : $input.json('$.persondata.income')
"height" : $input.json('$.persondata.height')
}
Can you spot the problem? I am missing a comma after the income line.
Here is another example.
Could not parse request body into json: Unexpected character (\'}\' (code 125)): expected a value
When I look at the Integration Request - Body Mapping Template:
#set($inputRoot = $input.path('$'))
{
"age" : $inputRoot.age,
"height" : $inputRoot.height,
"income" : $inputRootincome
}
Can you spot the problem? I am missing a dot in $inputRootincome.
Error Message :
Could not parse request body into json: Could not parse payload into json: Unrecognized token \' \': was expecting (\'true\', \'false\' or \'null\')
Cause of the error : When string values inside the json are not assigned using double quotations in the aws mapping template the error occurs.
Solution : (Please Note : This example is for application/json type request template)
Actually the solution for the problem is, if you are using a value of type string in json then its value should be assigned inside a ("" - double quotation marks) in the mapping template.
The below shown example has the following attributes :
customerId - string (Please note : this value comes from a query parameter)
customerName - string
customerAge - integer
isPermanentEmployee - boolean
customerAddress - string (Please note this is an optional parameter)
And the mapping template should be defined like the example shown below
Refer the example below :
#set($inputRoot = $input.path('$'))
{
"CustomerId": "$input.params('customerId')",
"CustomerName": "$inputRoot.customerName",
"CustomerAge": $inputRoot.customerAge,
"IsPermanentEmployee": $inputRoot.isPermanentEmployee
"CustomerAddress ": #if($inputRoot.customerAddress == "") "" #elseif($inputRoot.customerAddress != "") "$inputRoot.customerAddress" #end
}
If you note the above mapping template, I would have given string values inside double quotation marks("") which will solve the error
Also this example contains how to handle optional parameters in aws mapping templates using #if#else statements.
It is likely that you had copy-pasted multiple lines in your "Integration Request" in the API gateway.
When copying a line and pasting it below, you might have copied the hidden character '\n' at the end of that line. This is probably causing issues at the lambda function.
Example: Copying the line containing age and pasting it twice and modifying them to have height and income respectively.
#set($inputRoot = $input.path('$'))
{
"age" : $inputRoot.age,
"height": $inputRoot.height,
"income": $inputRoot.income
}
Instead of copy-pasting, just type the line out for height and income.
I am working with Node-RED in Bluemix for IoT.
How can I parse out the individual pieces of information (like the cmdmsg and the tempr) in a function node so I can use it in other nodes in the flow?
I'm getting an error when I try (see below)
I am receiving the JSON complete message object (from an IoT in Node) that I see in my "debug" node when I set it to look at the complete message object. see the object below.
It appears to me that the JSON is formatted correctly.
I tried putting the following in the function node, but I'm getting an error that says "TypeError: Cannot read property 'tempr' of undefined"
Here is what the function parameter is:
return {payload:msg.payload.d.tempr};
and here is the message object
{
"topic": "iot-2/type/Arduino-tempsensor/id/FFFFFFFFFFFF/evt/status/fmt/json",
"payload": "{\n\"d\": {\n\"myName\": \"Arduino CF\",\n\"cmdmsg\": \"Weekly\",\n\"tempr\": -3,\n}\n}",
"deviceId": "FFFFFFFFFFFF",
"deviceType": "Arduino-tempsensor",
"eventType": "status",
"format": "json",
"_msgid": "ffffffff.55555"
}
note: I obfuscated the device ID (mac address) and msgid
Any ideas on how to parse the data out and why I'm getting an error?
Sorry, but your JSON Payload is completely messed up, it should look like this: {"d": {"myName": "Arduino CF","cmdmsg": "Weekly","tempr": -3}}
You shouldn't see any \or \nin the payload, they look like escape characters from the client side. I also believe that the last comma after the tempr value shouldn't be there for valid JSON.
I am not an Arduino expert but I have experimented with a Raspberry Pi and the Mosquitto client, this is how I can successfully send an event to IoTF:
mosquitto_pub -h <org>.messaging.internetofthings.ibmcloud.com -p 1883 -u "use-token-auth" -P "<token>" -i d:<org>:raspi:raspi2 -t iot-2/evt/message/fmt/json -m {"d":{"text":"Hello World"}}
If the paylod is correct JSON your statement return {payload:msg.payload.d.tempr}; will work.
Have you seen this: http://www.ibm.com/developerworks/cloud/library/cl-bluemix-arduino-iot2/
The JSON string you are showing should be converted to a Javascript object before you try to access its fields. To do that is as simple as wiring the arduino output to a "JSON" node, which does the conversion for you (or throws an error if the string is not valid JSON).
Wire the output of the JSON node to a debug node, if you want to see the structure of the msg object. You can also wire it to a "change" node, if you simply want to replace the msg.payload with the temperature value, for instance. You don't need any custom javascript code in a function node to do simple changes like that.
Here is a sample flow that you can import... the arduino output string is simulated by pasting your payload into a "template" node:
[{"id":"1a79abfe.b8abb4","type":"inject","z":"58c8eb7a.5496c4","name":"send output","topic":"iot-2/type/Arduino-tempsensor/id/FFFFFFFFFFFF/evt/status/fmt/json","payload":"true","payloadType":"bool","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":170,"y":2740,"wires":[["9fc678fb.ae18e8"]]},{"id":"69e91778.e0c6e8","type":"json","z":"58c8eb7a.5496c4","name":"","property":"payload","action":"","pretty":false,"x":390,"y":2800,"wires":[["d066800f.60a9b","cf991eb1.f2a1a"]]},{"id":"9d8d7da2.2a7da","type":"debug","z":"58c8eb7a.5496c4","name":"msg string","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":610,"y":2740,"wires":[]},{"id":"9fc678fb.ae18e8","type":"template","z":"58c8eb7a.5496c4","name":"arduino string","field":"payload","fieldType":"msg","format":"json","syntax":"plain","template":"{\n \"d\": {\n \"myName\": \"Arduino CF\",\n \"cmdmsg\": \"Weekly\",\n \"tempr\": -3\n }\n}","output":"str","x":360,"y":2740,"wires":[["69e91778.e0c6e8","9d8d7da2.2a7da"]]},{"id":"d066800f.60a9b","type":"change","z":"58c8eb7a.5496c4","name":"extract tempr","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.d.tempr","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":450,"y":2860,"wires":[["af730d72.2995a"]]},{"id":"af730d72.2995a","type":"debug","z":"58c8eb7a.5496c4","name":"tempr","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":630,"y":2860,"wires":[]},{"id":"cf991eb1.f2a1a","type":"debug","z":"58c8eb7a.5496c4","name":"msg object","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":610,"y":2800,"wires":[]}]
As was noted, your original JSON string was not valid, because of the trailing comma -- but it is ok to have double-quotes and newlines as long as they are escaped with a "\". When the string is parsed, they are stripped out anyway.