AWS IoT Rule SQL republish get thing name as JSON object key - json

I'm publishing Shadow updates from multiple iot devices.
I want to filter those updates and extract only some data that will update another thing shadow (let's call it main shadow) with a list of Json objects. Each json object key is the thing id.
For that, I have a rule that is subscribed to $aws/things/+/shadow/update/documents to capture all the devices shadow updates. So for example for the following updates
to $aws/things/device101/shadow/update/documents
{
state: {
reported: {
"temperature": "28"
}
}
}
to $aws/things/device202/shadow/update/documents
{
state: {
reported: {
"temperature": "31"
}
}
the main shadow will result in
{
state: {
reported: {
"device101": {
"temperature": "28"
}
"device202": {
"temperature": "31"
}
}
}
In my SQL I still can't find the way to use the thing id as a key in the json data that will be published to the main shadow.
I'm using topic(3) as the function to get the thing name and Literals to build the json but it keeps telling me that it expects a key. I tried to cast the topic(3) function but same result
SELECT { topic(3): {'temperature': current.state.reported.temperature }} as state.reported FROM '$aws/things/+/shadow/update/documents'
SqlParseException
Expected a key, but got IDENT(topic). Object literals should follow the format {"keyName": any-expression, "anotherKey": any-expression} topic(3):
Any idea how this can be achieved ?
thanks

As of today, it seems that there is no way to achieve this with just SQL in IoT rules.
A lambda must be invoked to transform the payload
more details in this post

Related

Best Schema for a Data List in JSON for RestFul API to use in Angular

I've been wondering for some days what kind of scheme would be more appropriate to use a data list in json in a web application.
I'm developing a REST Web Application, and im using Angular for front end, i should order, filter and print these data list also in xml ...
For you what scheme is better and why?
1) {
"datas": [
{ "first":"","second":""},
{ "first":"","second":""},
{ "first":"","second":""}
]
}
2) {
"datas": [{
"data": { "first":"","second":""},
"data": { "first":"","second":""},
"data": { "first":"","second":""}
}]
}
3) [
{ "first":"","second":""},
{ "first":"","second":""},
{ "first":"","second":""}
]
Thanks so much.
The first and third notations are quite similar because the third notation is included in your first.
So the question is "Should I return my datas as an array or should I return an object with a property that contain the array ?
It will depend on either you want to have more information alongside your datas or not.
For exemple, if your API might return an error, you will want to manage it from the front end.
In case of error, the JSON will looks like this :
{
"datas": null,
"error": "An error occured because of some reasons..."
}
At the opposite, if everything goes well and your API actually return the results, it will looks like this :
{
"datas": [
{ "first":"","second":""},
{ "first":"","second":""},
{ "first":"","second":""}
],
"error": null
}
Then your front end can use the error property to manage errors sent from the API.
var result = getDatas(); // Load datas from the API
if(result.error){
// Handle the error, display a message to the user, ...
} else {
doSomething(result.datas); // Use your datas
}
If you don't need to have extra properties like error then you can stick with the third schema.
The second notation is invalid. The datas array will contain only one object which will have one property named data. In this case data is a property that is defined multiple times so the object in the array will contain only the last occurence:
var result = {
"datas": [{
"data": { "first":"a","second":"b"},
"data": { "first":"c","second":"d"},
"data": { "first":"e","second":"f"}
}]
}
console.log("Content of result.datas[0].data : ")
console.log(result.datas[0].data)
Obviously the first option would be easy to use. Once you will access datas it'll give you an array. Any operation (filter, sort, print) on that array will be easy in comparison to anything else. Everywhere you just need to pass datas not datas.data.

Issue with formatting json array in new file

After creating a new file for json, I am needing to create two arrays, when creating the first one, an error popped up where it doesn't want to accept type1:[ as the array's start.
Tried using a validator, inside, I switched to a comma after it confirmed the error, that didn't give the desired effect.
{
"type1":[
"product1":{
},
"product2":{
},
"product3":{
}
]
}
It threw up an error message when I expected to be able to expand on products and then have the basic system to practice with.
{
"type1":[
{
"product1":{
}
},
{
"product2":{
}
},
{
"product3":{
}
}
]
}
Please use this format. your format is invalid. as #kmr said
"JSON arrays cant contain key value pairs"

JSON Deserialization on Talend

Trying to figuring out how to deserialize this kind of json in talend components :
{
"ryan#toofr.com": {
"confidence":119,"email":"ryan#toofr.com","default":20
},
"rbuckley#toofr.com": {
"confidence":20,"email":"rbuckley#toofr.com","default":15
},
"ryan.buckley#toofr.com": {
"confidence":18,"email":"ryan.buckley#toofr.com","default":16
},
"ryanbuckley#toofr.com": {
"confidence":17,"email":"ryanbuckley#toofr.com","default":17
},
"ryan_buckley#toofr.com": {
"confidence":16,"email":"ryan_buckley#toofr.com","default":18
},
"ryan-buckley#toofr.com": {
"confidence":15,"email":"ryan-buckley#toofr.com","default":19
},
"ryanb#toofr.com": {
"confidence":14,"email":"ryanb#toofr.com","default":14
},
"buckley#toofr.com": {
"confidence":13,"email":"buckley#toofr.com","default":13
}
}
This JSON comes from the Toofr API where documentation can be found here .
Here the actual sitation :
For each line retreived in the database, I call the API and I got this (the first name, the last name and the company change everytime.
Does anyone know how to modify the tExtractJSONField (or use smthing else) to show the results in tLogRow (for each line in the database) ?
Thank you in advance !
EDIT 1:
Here's my tExtractJSONfields :
When using tExtractJSONFields with XPath, you need
1) a valid XPath loop point
2) valid XPath mapping to your structure relative to the loop path
Also, when using XPath with Talend, every value needs a key. The key cannot change if you want to loop over it. Meaning this is invalid:
{
"ryan#toofr.com": {
"confidence":119,"email":"ryan#toofr.com","default":20
},
"rbuckley#toofr.com": {
"confidence":20,"email":"rbuckley#toofr.com","default":15
},
but this structure would be valid:
{
"contact": {
"confidence":119,"email":"ryan#toofr.com","default":20
},
"contact": {
"confidence":20,"email":"rbuckley#toofr.com","default":15
},
So with the correct data the loop point might be /contact.
Then the mapping for Confidence would be confidence (the name from the JSON), the mapping for Email would be email and vice versa for default.
EDIT
JSONPath has a few disadvantages, one of them being you cannot go higher up in the hierarchy. You can try finding out the correct query with jsonpath.com
The loop expression could be $.*. I am not sure if that will satisfy your need, though - it has been a while since I've been using JSONPath in Talend because of the downsides.
I have been ingesting some complex json structures and did this via minimal json libraries, and tjava components within talend.

Cloudant Query on dynamic index object

I would like query using the selector for the below JSON object. Based on "Type" my Msgs array displays details.i.e if Type is PAM ,Msgs array will have 'PAMDetails' object. if Type is NAS then NASDetails. Only one type will arrive in the json object. My requirement is to get documents for a given 'LotNumber' and 'messageDate' prior 30 days documents. I tried with default cloudant query but struggling to write the selector for this dynamically creating PAMDetails/NASDetails. tried with Msgs.[].MessageDate but failed to get docs.
{
"_id": "65c5e4c919871f7365f4d814f6e4783g",
"Type": "PAM",
"LotNumber": "11680555",
"Msgs": [
{
"PAMDetails": {
"MessageDate": "2017-01-19"
}
},{
"PAMDetails": {
"MessageDate": "2017-01-21"
}
}
]
}

Couchbase View not returning array value

I am trying to create a view to group on a particular attribute inside an array. However, the below map function is not returning any result.
JSON Document Structure :
{
"jsontype": "survey_response",
"jsoninstance": "xyz",
"jsonlanguage": "en_us",
"jsonuser": "test#test.com",
"jsoncontact": "test#mayoclinic.com",
"pages": [
{
q-placeholder": "q1-p1",
q:localized": "q1-localized-p1",
q-answer-placeholder": "jawaabu121",
q-answer-localized": "localized jawaabu1"
},
{
q-placeholder": "q2-p2",
q:localized": "q2-localized-p2",
q-answer-placeholder": "jawaabu221",
q-answer-localized": "localized jawaabu2"
},
{
"q-placeholder": "q3-p3",
"q:localized": "q3-localized-p3",
"q-answer-placeholder": "jawaabu313",
"q-answer-localized": "localized jawaabu3"
}
]
}
Map Function :
function(doc, meta){
emit(doc.jsoninstance,[doc.pages[0].q-placeholder, doc.pages[0].q-localized,doc.pages[0].q-answer-placeholder,q-answer-localized]);
}
It looks like you made a typo at the end of your emit statement:
doc.pages[0].q-answer-placeholder,q-answer-localized.
Instead q-answer-localized should be changed to doc.pages[0].q-answer-localized.
Further to this it seems that you have defined a field as q-localized in your emit statement, but actually according to the sample document that you posted this should actually be q:localized, I assume that this was a mistake in the snippet of the document and not the emit statement, but if not then will also need amending.
I would imagine errors like this would be flagged up in the view engine's map-reduce error log, in future you should check this log so that you will be able to debug errors like this yourself.
The location of the mapreduce_errors log can be found in the Couchbase documentation