How to dynamically read external json files in node.js? - json

I am creating a website that reads externally hosted json files and then uses node.js to populate the sites content.
Just to demonstrate what I'm after, this is a really simplified version of what I'm trying to do in node.js
var ids = [111, 222, 333];
ids.forEach(function(id){
var json = getJSONsomehow('http://www.website.com/'+id+'.json');
buildPageContent(json);
});
Is what I want to do possible?
(Marked as a duplicate of "How do I return the response from an asynchronous call?" see my comment below for my rebuttal)

You are trying to get it synchronously. What you should aim for instead, is not a function used like this:
var json = getJSONsomehow('http://www.website.com/'+id+'.json');
but more like this:
getJSONsomehow('http://www.website.com/'+id+'.json', function (err, json) {
if (err) {
// error
} else {
// your json can be used here
}
});
or like this:
getJSONsomehow('http://www.website.com/'+id+'.json')
.then(function (json) {
// you can use your json here
})
.catch(function (err) {
// error
});
You can use the request module to get your data with something like this:
var request = require('request');
var url = 'http://www.website.com/'+id+'.json';
request.get({url: url, json: true}, (err, res, data) => {
if (err) {
// handle error
} else if (res.statusCode === 200) {
// you can use data here - already parsed as json
} else {
// response other than 200 OK
}
});
For a working example see this answer.
For more info see: https://www.npmjs.com/package/request

I think problem is in async request. Function will return result before request finished.
AJAX_req.open( "GET", url, true );
Third parameter specified async request.
You should add handler and do all you want after request finished.
For example:
function AJAX_JSON_Req( url ) {
var AJAX_req = new XMLHttpRequest.XMLHttpRequest();
AJAX_req.open( "GET", url, true );
AJAX_req.setRequestHeader("Content-type", "application/json");
AJAX_req.onreadystatechange = function() {
if (AJAX_req.readyState == 4 && AJAX_req.status == 200) {
console.log(AJAX_req.responseText);
}
};
}

Related

How to make the function wait until response comes back

I am trying to add this response data to a zip object and later calling the createZip() method to download file as a zip file. Problem is my file getting downloaded first and the response is coming back. If I try to run the function again then right file is getting downloaded because in the previous API call I already got the response data.
Can anyone help me with this. I am new to angular and don't know how to use async/await properly.
zipFile = new JSZip();
exportIcsReportInJSONFormat() {
this.icsService.getICSReport()
.subscribe(response => {
console.log("JSONFile",response)
this.jsonFile = response;
this.zipFile.file("ics-report_.json", response, {binary:true});
});
To create zip file and download.
createZip() {
this.zipFile.generateAsync({type:"blob"})
.then(function(content) {
saveAs(content, "example.zip");
});
}
You can use the async/await pattern with Promises, something like this:
zipFile = new JSZip();
async mapZip() {
try {
var response = await this.exportIcsReportInJSONFormat();
console.log("JSONFile", response)
this.jsonFile = response;
this.zipFile.file("ics-report_.json", response, { binary: true });
var content = await this.zipFile.generateAsync({ type: "blob" });
saveAs(content, "example.zip");
}
catch {
...
}
}
exportIcsReportInJSONFormat() {
this.icsService.getICSReport().toPromise();
}

JSON.parse from a URL

I have several URL similar to https://zkillboard.com/api/stats/solarSystemID/31000007/
I am trying to extract the JSON from the url into an object.
I have been able to get as far as this which returns a Promise, PromiseState: fulfilled and PromiseResults contains an object with the data I am looking for.
async function readJSON(url:string) {
var request = new XMLHttpRequest();
request.open ('get', url, false)
request.send(null)
if (request.status == 200) {
return JSON.parse(request.responseText)
}
}
const systemJSON = readJSON('https://zkillboard.com/api/stats/solarSystemID/31000007/')
console.log(systemJSON)
How can I ensure that my console.log only returns the PromiseResult?
This seems to have fixed it for me, removed the async from the function as well as .responseText in the JSON.parse()
function readJSON(url:string) {
var request = new XMLHttpRequest();
request.open ('get', url, false)
request.send(null)
if (request.status == 200) {
return JSON.parse(request.response)
}
}
const systemJSON = readJSON('https://zkillboard.com/api/stats/solarSystemID/31000007/')
const printJSON = () =>{
console.log(systemJSON)
}
printJSON();
First off, when handling json from an external source I would suggest wrapping it in a try/catch function, to avoid unsuspecting errors.
Secondly I think the issue is that readJSON returns a promise, so you might need to await it.
try {
const json = await readJSON('https://zkillboard.com/api/stats/solarSystemID/31000007/')
const systemJSON = JSON.parse(json);
} catch (error) {
// Woops something happend - see error variable
}

NodeJS request not giving any response in AWS Lambda

I am using NodeJS request module to pass a JSON request to a URL and generate a JSON response from it. I tried this code and it generates a valid response. I am pasting the link for a StackOverflow question I asked for the same.
NodeJS Request returning an empty array inside a JSON response
However, when I utilize the same logic in AWS Lambda, there is no response at all from the module. Since there is no response at all, I cannot understand what the problem is.
This is the handling function for the AWS Lambda with Alexa as a trigger.
'use strict';
var request = require('request');
var accountNumberRequest = {};
var balanceResponse = {};
const url = "https://ibluatapig.indusind.com/app/uat/balinq/AccountEnquiry?client_id=6867b781-9b21-45c5-9c55-948f7cd1a33f&client_secret=hP3yB3hM2oH4pH4hM1kV3uY8vR3qV7jY8cF6bG2sF5jX8lT1vN";
var bal = {};
exports.handler = function (event,context) {
try{
console.log("Try Started");
var req = event.request;
console.log("Request Generated");
if(req.type === "LaunchRequest") {
console.log("Launch Request! Calling handleLaunchRequest");
handleLaunchRequest(context);
} else if(req.type === "IntentRequest") {
console.log("IntentRequest");
let options = {};
console.log(0);
if(req.intent.name === "BalanceIntent") {
console.log("Balance Intent");
//Got the account number from Alexa request
let accNo = req.intent.slots.AccountNumber.value;
console.log(accNo);
accountNumberRequest = {
"AERequest":{
"serviceType":"BE",
"deviceId":"Test",
"accountId":accNo
}
};
console.log(accountNumberRequest);
console.log("Calling NodeJS.Request");
request({
url: url,
method: "POST",
json: true,
header: {
"content-type": "application/json",
},
body: accountNumberRequest
},
function(error, response,body){
if(!error && response.statusCode === 200){
console.log(body.AEResponse.AcctBal[1].BalAmt);
} else {
//options.speechText = `The account <say-as interepret-as = "digits">${accNo}</say-as> does not exist`;
console.log("error: "+error);
console.log("response.statusCode"+response.statusCode);
console.log("response.statusText"+response.statusText);
}
}
);
console.log("Balance Response should be assigned by now");
console.log(bal);
/* if(accountNumbers.hasOwnProperty(accNo)) {
var balance = accountNumbers[accNo];
accountExists = true;
}
if(accountExists == true){
options.speechText = `The balance of account number <say-as interpret-as = "digits">${accNo}</say-as> is <say-as interpret-as = "cardinal">${balance}</say-as>`;
} else {
options.speechText = `The account <say-as interepret-as = "digits">${accNo}</say-as> does not exist`;
}*/
context.succeed(buildResponse(options));
}
} else if(req.type === "SessionEndedRequest") {
//Code here
} else {
throw("Unknown Intent Type");
}
} catch(e){
context.fail("Exception "+e);
}
};
function getBalance(){
//Code to parse the JSON response and extract values from the response.
}
function handleLaunchRequest(context){
//Code for handling launch requests }
function buildResponse(options){
//Code for generating response
}
This is the problem...
// You're sending an asynchronous HTTP request here.
request();
// But you sent the response here without waiting for the above request to finish.
context.succeed();
Basically, you're executing context.succeed() before request() finishes. So you're basically ending your Lambda invocation without the response from that HTTP request.
To fix your code, put the context.succeed() inside the callback that you pass to the request() call.
P.S.
You should be using callback instead of the deprecated context.succeed()/context.fail() API.

AWS Lambda http request: Unable to stringify body as json: Converting circular structure to JSON

I would like to return the result of an HTTP request in my AWS Lambda function:
var http = require('http');
exports.someFunction = function(event, context) {
var url = "http://router.project-osrm.org/trip?loc=47.95,12.95&loc=47.94,12.94";
http.get(url, function(res) {
context.succeed(res);
}).on('error', function(e) {
context.fail("Got error: " + e.message);
});
}
It should return exactly what I get when I open the url directly in my browser (try it to see the expected json).
AWS Lambda return the following error message when I call context.succeed(res):
{
"errorMessage": "Unable to stringify body as json: Converting circular structure to JSON",
"errorType": "TypeError"
}
I assume that I need to use some property of res instead of res itself, but I couldn't figure out which one contains the actual data I want.
If you are using the raw http module you need to listen for data and end events.
exports.someFunction = function(event, context) {
var url = "http://router.project-osrm.org/trip?loc=47.95,12.95&loc=47.94,12.94";
http.get(url, function(res) {
// Continuously update stream with data
var body = '';
res.on('data', function(d) {
body += d;
});
res.on('end', function() {
context.succeed(body);
});
res.on('error', function(e) {
context.fail("Got error: " + e.message);
});
});
}
Using another module such as request https://www.npmjs.com/package/request would make it so you don't have to manage those events and your code could go back to almost what you had before.

Node Express 4 get header in middleware missing

I have a middleware function using Node's Express4 to log each request & response for debugging. I use the res.json call in the request handler to send back JSON to the client for all but static files. So I do not want to log the response for static files, but only the JSON responses. I have the following code:
function logRequests(req, res, next) {
// do logging (will show user name before authentication)
logger.reqLog('IN '+req.method+' '+req.url, req);
var oldEnd = res.end,
oldWrite = res.write,
chunks = [];
res.write = function(chunk) {
chunks.push(chunk);
oldWrite.apply(res, arguments);
};
res.end = function(chunk, encoding) {
if(chunk) {
chunks.push(chunk);
}
oldEnd.apply(res, arguments);
// the content-type prints "undefined" in some cases
// even though the browser shows it returned as "application/json"
console.log('type='+res.get('content-type'));
if(res.get('content-type') === 'application/json') {
var body = Buffer.concat(chunks).toString('utf8');
logger.info(body, req);
}
logger.reqLog('OUT '+req.method+' '+req.path, req);
};
next(); // make sure we go to the next routes and don't stop here
}
So why do some requests show the correct content type in the middleware meaning they also print the response fine and others do not? All of them look good in the REST client when inspecting the returned headers.
EDIT: Some more info discovered tonight while trying to figure this out - if I append any character as a dummy request parameter, it logs the response type correctly:
http://localhost:8081/node/ionmed/api/logout?0 WORKS
where
http://localhost:8081/node/ionmed/api/logout DOES NOT
Also, I can always get a response type logging in the middleware function if I replace the .json() call with .end() so this:
res.json({ item: 'logout', success: true });
becomes:
res.set('content-type', 'application/json');
res.end({ item: 'logout', success: true });