I'm following this tutorial to push data from my API Gateway to a Kinesis stream :
http://docs.aws.amazon.com/apigateway/latest/developerguide/integrating-api-with-aws-services-kinesis.html#api-gateway-get-and-add-records-to-stream
I have my Body Mapping Template setup as.....
{
"StreamName": "my-stream-name",
"Data": "$util.base64Encode($input.path('$.Data'))",
"PartitionKey": "$input.path('$.PartitionKey')"
}
...and have put the following in the Request Body of an API test...
{
"Data": {
"Foo": "A",
"Bar": "B"
},
"PartitionKey": "some key"
}
I've then created a Lambda Function which has a trigger set up against the same Kinesis Stream. However, I'm struggling to decode/deserialise the records coming in from Kinesis.
exports.handler = (event, context, callback) => {
event.Records.forEach(function(record) {
let payload = JSON.parse(Buffer(record.kinesis.data, 'base64').toString('ascii'))
});
};
It seems that the data is being serialised to Kinesis in a non-JSON format. The value for record.kinesis.data in the forEach loop is
e0Zvbz1BLCBCYXI9Qn0=
...which when push through Buffer(record.kinesis.data, 'base64').toString('ascii')
returns as
{Foo=A, Bar=B}
not
{"Foo":"A", "Bar":"B"}
Main aim is obviously to get payload to in a state where I can say console.log(payload.Foo)
Any hints as to what I should be doing/looking for would be appreciated.
For anyone else out there
I had my Body Mapping Template setup as.....
{
"StreamName": "my-stream-name",
"Data": "$util.base64Encode($input.path('$.Data'))",
"PartitionKey": "$input.path('$.PartitionKey')"
}
Now changed to handle the json....
{
"StreamName": "my-stream-name",
"Data": "$util.base64Encode($input.json('$.Data'))",
"PartitionKey": "$input.path('$.PartitionKey')"
}
where...
($input.json('$.Data'))
is the change :)
Related
My goal is to loop over a JSON response, grab two values, and build out an API call to load information into a POST to create a policy I am building.
To start out on this, I am trying to get two values from a JSON response to assign as variables to build the POST call, which will be the second step to this. As each different "id" and "name" key is assigned, I would like to build out a JSON payload and send the POST calls one at a time. The keys "id" and "name" occurs multiple times in the response payload and I am having issues capturing the two keys.
JSON response
data = {
"data":[
{
"id":"02caf2be-3245-4d3d",
"name":"ORA-FIN-ACTG",
"description":"Oracle",
"links":{
"web":"https://com/",
"api":"https://com/"
}
},
{
"id":"03af2f46-fad6-41a1",
"name":"NBCMAINFRAME",
"description":"Network",
"links":{
"web":"https://com/",
"api":"https://com/"
}
},
{
"id":"0649628b-0e3b-48df",
"name":"CAMS",
"description":"Customer",
"links":{
"web":"https://com/",
"api":"https://com/"
}
},
{
"id":"069d4bcf-3e50-4105",
"name":"SHAREPOINTSITES",
"description":"Sharepoint",
"links":{
"web":"https://com/",
"api":"https://com/"
}
}
],
"took":0.013,
"requestId":"1f364470"
}
I have tried various "for loops" to grab the data. Here is one of the loops below:
data = json.loads(data)
data[0]['data'][0]['name']
for item in range(len(data)):
print(data[item]['data'][0]['name'])
I have also tried reading it as a dictionary:
for data_dict in data:
for key, value in data_dict.items():
team.append(key)
id.append(value)
print('name = ', team)
print('id = ', id)
I am also getting KeyError's and TypeError: the JSON object must be str, bytes or bytearray, not 'dict'.
Any help is appreciated.
FYI, this is the payload I am wanting to populate with the "name" and "id" values:
data= {
"type":"alert",
"description":"policy",
"enabled":"true",
"filter":{
"type":"match-any-condition",
"conditions":[
{
"field":"extra-properties",
"key":"alertOwner",
"operation":"equals",
"expectedValue":name
}
]
},
"ignoreOriginalResponders": "true",
"ignoreOriginalTags": "true",
"continue": "true",
"name": str(name) + " Policy",
"message":"{{message}}",
"responders":[{"type":"team","id":id}],
"alias":"{{alias}}",
"tags":["{{tags}}"],
"alertDescription":"{{description}}"
}
The JSON response which you have given is already a dictionary so no need to use json.loads for that. The multiple item list is actually nested under the data key. So you could just simply iterate through the array of items like this:
for item in data["data"]:
print("{} : {}".format(item["id"],item["name"]))
This is the output:
02caf2be-3245-4d3d : ORA-FIN-ACTG
03af2f46-fad6-41a1 : NBCMAINFRAME
0649628b-0e3b-48df : CAMS
069d4bcf-3e50-4105 : SHAREPOINTSITE
I have done this before, but I am not sure what error I am making here in order to populate the select drop down with options retrieved from JSON response. The API call is successfull and I can see the data pulled but not able to display it in options of select drop down.
varibale in concern is idsFetched
let idsFetched = this.state.storeRanges.Stores
console.log("ranges I/O ", idsFetched);
let options3 = idsFetched.map(store => {
return {value: store.value.description, label: store.value.description};
})
---- JSON RESPONSE ---- Stored in variable = storeRanges
{
"Stores": [
{
"mode": "LOADED",
"value": {
"description": "Ontario-Example-123",
}
},
{
"mode": "LOADED",
"value": {
"description": "Ontario-Example-456",
}
},
{
"mode": "LOADED",
"value": {
"description": "Ontario-Example-789",
}
},
]
}
First make sure you have done JSON.parse(apiResponse) otherwise you'll just be storing a string, making storeRanges.Stores === undefined probably causing your issue.
Secondly, make sure the State has been set first, by wrapping the function in a check. If you've done it correctly the re-render after setting state should fire it off no worries
I was able to fix the issue by declaring an empty variable in state and then updating the state in ComponentDidUpdate hook and accessing that in onChange method.
componentDidUpdate(prevProp, prevState){
this.setState({idsFetched: this.state.storeRanges.Stores})
}
onChange(){
options = ranges.map(range => {
return {value: range.value.description, label: range.value.description};
})
}
This is how I could access the array of stores fetched from JSON response!
Thank you everyone for looking into my query!
I've been wondering for some days what kind of scheme would be more appropriate to use a data list in json in a web application.
I'm developing a REST Web Application, and im using Angular for front end, i should order, filter and print these data list also in xml ...
For you what scheme is better and why?
1) {
"datas": [
{ "first":"","second":""},
{ "first":"","second":""},
{ "first":"","second":""}
]
}
2) {
"datas": [{
"data": { "first":"","second":""},
"data": { "first":"","second":""},
"data": { "first":"","second":""}
}]
}
3) [
{ "first":"","second":""},
{ "first":"","second":""},
{ "first":"","second":""}
]
Thanks so much.
The first and third notations are quite similar because the third notation is included in your first.
So the question is "Should I return my datas as an array or should I return an object with a property that contain the array ?
It will depend on either you want to have more information alongside your datas or not.
For exemple, if your API might return an error, you will want to manage it from the front end.
In case of error, the JSON will looks like this :
{
"datas": null,
"error": "An error occured because of some reasons..."
}
At the opposite, if everything goes well and your API actually return the results, it will looks like this :
{
"datas": [
{ "first":"","second":""},
{ "first":"","second":""},
{ "first":"","second":""}
],
"error": null
}
Then your front end can use the error property to manage errors sent from the API.
var result = getDatas(); // Load datas from the API
if(result.error){
// Handle the error, display a message to the user, ...
} else {
doSomething(result.datas); // Use your datas
}
If you don't need to have extra properties like error then you can stick with the third schema.
The second notation is invalid. The datas array will contain only one object which will have one property named data. In this case data is a property that is defined multiple times so the object in the array will contain only the last occurence:
var result = {
"datas": [{
"data": { "first":"a","second":"b"},
"data": { "first":"c","second":"d"},
"data": { "first":"e","second":"f"}
}]
}
console.log("Content of result.datas[0].data : ")
console.log(result.datas[0].data)
Obviously the first option would be easy to use. Once you will access datas it'll give you an array. Any operation (filter, sort, print) on that array will be easy in comparison to anything else. Everywhere you just need to pass datas not datas.data.
My question is how can I pull the values for events.payload.media.name?
I am posting to a raw zapier webhook from another app. If I check it using requestb.in it comes through as "Content-Type: application/json". The output is also validating as JSON.
{
"hook":{
"uuid":"1asdfasd5-asdf-4f52-bd31-c7a544897808"
},
"events":[
{
"uuid":"0asdfasdfasdf0",
"type":"viewing_session.turnstile.converted",
"payload":{
"visitor":{
"id":"28b606b_7853753-3868-4f07-9543-70da084452cc-7442322af-407bdc31d8fc-2739"
},
"viewing_session":{
"id":"154284_b40c5358-1faf-40e9-a44e-60aa641a11cd-fd3c69d8d-302471c603f4-8245"
},
"name":null,
"media":{
"url":"https://things.wistia.com/medias/asdfasdf",
"thumbnail":{
"url":"http://embed.wistia.com/deliveries/asdfasdfasdfasdfasdfasdfasd.jpg?image_crop_resized=200x120"
},
"name":"this is what I want!",
"id":"asdfasdfasdf",
"duration":52.872
},
"last_name":null,
"foreign_data":{
},
"first_name":null,
"email":"email#email.com"
},
"metadata":{
"account_id":"asdfasdfasdf"
},
"generated_at":"2017-05-02T07:31:08Z"
}
]
}
However, when I check the typeof data in the output it is telling me that it is a string (see my code to check below). This prevents me from pulling the info out of it using:
return {stuff: typeof inputData.thing.events.payload.media.name};
I'm a huge noob, am I missing something fundamental here?
screenshot to check typeof data
events is an array, so you would access it like this:
inputData.thing.events[0].payload.media.name
is there a way to have the whole payload without creating a new App in Zapier? inputData didn't work
Folks,
Designing my first API in Node.JS using restify.js. My background is not webapis, pardon my amateur questions. In any case, I would like to have the res.send(data); responses to comply with the http://jsonapi.org/format/ so that my mobile application can start utilizing the api calls. At the moment if you were to call my api, it would return data in the following format:
{"Count":1,"Items":[{"dbsource":{"S":"foo"},"id":{"S":"5002820"},"name":{"S":"fnameblah,lnameblah"},"expiration":{"S":"06/13/2015"},"type":{"S":"bar"}}]}
Actually what you see above is just a return of a DynamoDB Query call.
So the question is... do you use a special library that you can pass data to, which would format and return the data in JSON format. Which in turn you can return it via res.send(data) to the clients, or is it up to us to make 'data' JSON compliant, then return it? At the end of the day we all want the results to look like:
{
"posts": [{
"id": "1",
"title": "Rails is Omakase",
"links": {
"author": "9",
"comments": [ "5", "12", "17", "20" ]
}
}]
}
Thanks!
In server side, stringify JSON object,
//...
res.statusCode = 200;
res.setHeader('Content-Type', 'application/json');
res.send(JSON.stringify(data)); //data is JSON object
res.end();
In client side, parse JSON string accordingly.
EDIT: Corrected response content type.
JSON data from server should be a JSON string
You have to parse it back the JSON format in client.
JSON.parse(string); // return JSON object