JSON.parse from a URL - json

I have several URL similar to https://zkillboard.com/api/stats/solarSystemID/31000007/
I am trying to extract the JSON from the url into an object.
I have been able to get as far as this which returns a Promise, PromiseState: fulfilled and PromiseResults contains an object with the data I am looking for.
async function readJSON(url:string) {
var request = new XMLHttpRequest();
request.open ('get', url, false)
request.send(null)
if (request.status == 200) {
return JSON.parse(request.responseText)
}
}
const systemJSON = readJSON('https://zkillboard.com/api/stats/solarSystemID/31000007/')
console.log(systemJSON)
How can I ensure that my console.log only returns the PromiseResult?

This seems to have fixed it for me, removed the async from the function as well as .responseText in the JSON.parse()
function readJSON(url:string) {
var request = new XMLHttpRequest();
request.open ('get', url, false)
request.send(null)
if (request.status == 200) {
return JSON.parse(request.response)
}
}
const systemJSON = readJSON('https://zkillboard.com/api/stats/solarSystemID/31000007/')
const printJSON = () =>{
console.log(systemJSON)
}
printJSON();

First off, when handling json from an external source I would suggest wrapping it in a try/catch function, to avoid unsuspecting errors.
Secondly I think the issue is that readJSON returns a promise, so you might need to await it.
try {
const json = await readJSON('https://zkillboard.com/api/stats/solarSystemID/31000007/')
const systemJSON = JSON.parse(json);
} catch (error) {
// Woops something happend - see error variable
}

Related

How can I invoke and use Google Cloud Functions in a Flutter app?

I have created a url scraper function, working and tested on Google Cloud, but I am really drawing a blank on how to invoke it. I have tried two methods, one using the cloud_functions package, and the other using a standard HTTPS get. I've tried looking online, but none of the solutions/guides involve functions with an input from the Flutter app, and an output back to the app.
Here's the structure of the function (which is working alright). I've named this function Parse in Google Cloud Platform.
<PYTHON PACKAGE IMPORTS>
def Parser(url):
<URL PARSE FUNCTIONS>
return source, datetime, imageurl, keyword
def invoke_parse(request):
request_json = request.get_json(silent=True)
file = Parser(request_json['url'])
return jsonify({
"source": file[0],
"datetime": file[1],
"imageurl": file[2],
"keyword": file[3],
})
The first method I tried was using an HTTP CALL to get the function. But that isn't working, even though there are no errors - I suspect it's just returning nothing.
parser(String url) async{ // Here I honestly don't know where to use the url input within the function
var uri = Uri.parse(<Function URL String>);
HttpClient client;
try {
var request = await client.getUrl(uri);
var response = await request.close();
if (response.statusCode == HttpStatus.ok) {
var json = await response.transform(utf8.decoder).join();
Map data = jsonDecode(json) as Map;
source = data['source']; // These are the variables used in the main Flutter app
postedAt = data['datetime'];
_imageUrl = data['image'];
keyword = data['keyword'];
} else {
print('Error running parse:\nHttp status ${response.statusCode}');
}
} catch (exception) {
print('Failed invoking the parse function.');
}
}
That didn't work, so I thought I might alternatively use the cloud_functions package as follows (in lieu of the previous):
parser(String url) async {
var functionUrl = <FUNCTION URL>;
HttpsCallable callable = CloudFunctions.instance.getHttpsCallable(functionName: 'Parse')
..timeout = const Duration(seconds: 30);
try {
final HttpsCallableResult result = await callable.call(
<String, dynamic>{
'url': url,
}
);
setState(() {
source = result.data['source']; //THESE ARE VARIABLES USED IN THE FLUTTER APP
postedAt = result.data['datetime'];
_imageUrl = result.data['image'];
keyword = result.data['keyword'];
});
}
on CloudFunctionsException catch (e) {
print('caught firebase functions exception');
print(e.code);
print(e.message);
print(e.details);
} catch (e) {
print('caught generic exception');
print(e);
}
}
In the latter case, the code ran without errors but doesn't work. My flutter log states the following error:
I/flutter ( 2821): caught generic exception
I/flutter ( 2821): PlatformException(functionsError, Cloud function failed with exception., {code: NOT_FOUND, details: null, message: NOT_FOUND})
which I'm assuming is also an error at not being able to read the function.
Any help on how I should go about processing my function would be appreciated. Apologies if something is a really obvious solution, but I am not familiar as much with HTTP requests and cloud platforms.
Thanks and cheers.
Node Js Backend Function
const functions = require("firebase-functions");
const admin = require("firebase-admin");
admin.initializeApp();
exports.test = functions.https.onCall(async (data, context) => {
functions.logger.info("Hello logs: ", {structuredData: true});
functions.logger.info( data.token, {structuredData: true});
}
Flutter frontend
1- pubspec.yaml
cloud_functions: ^1.1.2
2 - Code
HttpsCallable callable = FirebaseFunctions.instance.httpsCallable('test');
final HttpsCallableResult results = await callable.call<Map>( {
'token': token,
});

Dart Future HttpCLientRequest Returns Null

The class Requests set up an HttpClientRequest. The method getTeamsJsonForRequest is supposed to return the JSON response. However, the variable 'return' is not being assigned properly I assum. The print 'CONTS' in the .then response successfully prints the correct response, but printing 'myres' sections says result is null. Not sure why result is not being assigned in the response.transform section.
class Requests {
static Future getTeamsJsonForRequest(String reqPath) async{
var result;
HttpClient myhttp = new HttpClient();
String path = '/api/v3' + reqPath;
myhttp.get('www.thebluealliance.com', 80, path)
.then((HttpClientRequest request) {
request.headers.set("accept", "application/json");
request.headers.set("X-TBA-Auth-Key", "XXXXX");
return request.close();
})
.then((HttpClientResponse response) {
response.transform(utf8.decoder).transform(json.decoder).listen((conts) {
print('CONTS: ' + conts.toString());
result = json.decode(conts).toString();
});
});
print('myres: ' + result.toString());
return result;
}
}
Short answer: avoid using Future.then inside an async method.
Your print is executing before the response comes back. https://www.dartlang.org/tutorials/language/futures#async-await
Without an await any work that is done asynchronously will happen after subsequent statements in this function are executed.
Here is how I'd write this:
Future<String> teamsJsonForRequest(String requestPath) async {
var client = new HttpClient();
var path = '/api/v3$requestPath';
var request = (await client.get('www.thebluealliance.com', 80, path))
..headers.set("accept", "application/json")
..headers.set("X-TBA-Auth-Key", "XXXXX");
var response = await request.close();
var result =
await response.transform(utf8.decoder).transform(json.decoder).single;
print('myres: $result');
return result;
}

NodeJS request not giving any response in AWS Lambda

I am using NodeJS request module to pass a JSON request to a URL and generate a JSON response from it. I tried this code and it generates a valid response. I am pasting the link for a StackOverflow question I asked for the same.
NodeJS Request returning an empty array inside a JSON response
However, when I utilize the same logic in AWS Lambda, there is no response at all from the module. Since there is no response at all, I cannot understand what the problem is.
This is the handling function for the AWS Lambda with Alexa as a trigger.
'use strict';
var request = require('request');
var accountNumberRequest = {};
var balanceResponse = {};
const url = "https://ibluatapig.indusind.com/app/uat/balinq/AccountEnquiry?client_id=6867b781-9b21-45c5-9c55-948f7cd1a33f&client_secret=hP3yB3hM2oH4pH4hM1kV3uY8vR3qV7jY8cF6bG2sF5jX8lT1vN";
var bal = {};
exports.handler = function (event,context) {
try{
console.log("Try Started");
var req = event.request;
console.log("Request Generated");
if(req.type === "LaunchRequest") {
console.log("Launch Request! Calling handleLaunchRequest");
handleLaunchRequest(context);
} else if(req.type === "IntentRequest") {
console.log("IntentRequest");
let options = {};
console.log(0);
if(req.intent.name === "BalanceIntent") {
console.log("Balance Intent");
//Got the account number from Alexa request
let accNo = req.intent.slots.AccountNumber.value;
console.log(accNo);
accountNumberRequest = {
"AERequest":{
"serviceType":"BE",
"deviceId":"Test",
"accountId":accNo
}
};
console.log(accountNumberRequest);
console.log("Calling NodeJS.Request");
request({
url: url,
method: "POST",
json: true,
header: {
"content-type": "application/json",
},
body: accountNumberRequest
},
function(error, response,body){
if(!error && response.statusCode === 200){
console.log(body.AEResponse.AcctBal[1].BalAmt);
} else {
//options.speechText = `The account <say-as interepret-as = "digits">${accNo}</say-as> does not exist`;
console.log("error: "+error);
console.log("response.statusCode"+response.statusCode);
console.log("response.statusText"+response.statusText);
}
}
);
console.log("Balance Response should be assigned by now");
console.log(bal);
/* if(accountNumbers.hasOwnProperty(accNo)) {
var balance = accountNumbers[accNo];
accountExists = true;
}
if(accountExists == true){
options.speechText = `The balance of account number <say-as interpret-as = "digits">${accNo}</say-as> is <say-as interpret-as = "cardinal">${balance}</say-as>`;
} else {
options.speechText = `The account <say-as interepret-as = "digits">${accNo}</say-as> does not exist`;
}*/
context.succeed(buildResponse(options));
}
} else if(req.type === "SessionEndedRequest") {
//Code here
} else {
throw("Unknown Intent Type");
}
} catch(e){
context.fail("Exception "+e);
}
};
function getBalance(){
//Code to parse the JSON response and extract values from the response.
}
function handleLaunchRequest(context){
//Code for handling launch requests }
function buildResponse(options){
//Code for generating response
}
This is the problem...
// You're sending an asynchronous HTTP request here.
request();
// But you sent the response here without waiting for the above request to finish.
context.succeed();
Basically, you're executing context.succeed() before request() finishes. So you're basically ending your Lambda invocation without the response from that HTTP request.
To fix your code, put the context.succeed() inside the callback that you pass to the request() call.
P.S.
You should be using callback instead of the deprecated context.succeed()/context.fail() API.

How to dynamically read external json files in node.js?

I am creating a website that reads externally hosted json files and then uses node.js to populate the sites content.
Just to demonstrate what I'm after, this is a really simplified version of what I'm trying to do in node.js
var ids = [111, 222, 333];
ids.forEach(function(id){
var json = getJSONsomehow('http://www.website.com/'+id+'.json');
buildPageContent(json);
});
Is what I want to do possible?
(Marked as a duplicate of "How do I return the response from an asynchronous call?" see my comment below for my rebuttal)
You are trying to get it synchronously. What you should aim for instead, is not a function used like this:
var json = getJSONsomehow('http://www.website.com/'+id+'.json');
but more like this:
getJSONsomehow('http://www.website.com/'+id+'.json', function (err, json) {
if (err) {
// error
} else {
// your json can be used here
}
});
or like this:
getJSONsomehow('http://www.website.com/'+id+'.json')
.then(function (json) {
// you can use your json here
})
.catch(function (err) {
// error
});
You can use the request module to get your data with something like this:
var request = require('request');
var url = 'http://www.website.com/'+id+'.json';
request.get({url: url, json: true}, (err, res, data) => {
if (err) {
// handle error
} else if (res.statusCode === 200) {
// you can use data here - already parsed as json
} else {
// response other than 200 OK
}
});
For a working example see this answer.
For more info see: https://www.npmjs.com/package/request
I think problem is in async request. Function will return result before request finished.
AJAX_req.open( "GET", url, true );
Third parameter specified async request.
You should add handler and do all you want after request finished.
For example:
function AJAX_JSON_Req( url ) {
var AJAX_req = new XMLHttpRequest.XMLHttpRequest();
AJAX_req.open( "GET", url, true );
AJAX_req.setRequestHeader("Content-type", "application/json");
AJAX_req.onreadystatechange = function() {
if (AJAX_req.readyState == 4 && AJAX_req.status == 200) {
console.log(AJAX_req.responseText);
}
};
}

How to read HttpRequest data sent from client, on server

How do i read HttpRequest data sent by POST method from client, on the server, in Dart?
I send a message from the client like this:
HttpRequest request = new HttpRequest();
var url = "http://127.0.0.1:8081";
request.open("POST", url, async: false);
String data = 'hello from client';
request.send(data);
On server i am catching the request like this:
HttpServer.bind('127.0.0.1', 8081).then((server) {
server.listen((HttpRequest request) {
//DATA SHOULD BE READ HERE
});
});
But i cant figure out how to actually read the data... There is not data property in HttpRequest nor anything else...
EDIT This is how i get the answer now:
HttpServer.bind('127.0.0.1', 8081).then((server) {
server.listen((HttpRequest request) {
//DATA SHOULD BE READ HERE
print("got it");
print(request.method);
if(request.method == "POST") {
print("got it 2");
List<int> dataBody = new List<int>();
request.listen(dataBody.addAll, onDone: () {
var postData = new String.fromCharCodes(dataBody);
print(postData);
});
}
});
});
But for some reason the request.method is not "POST" but "OPTIONS", and if i change to if(request.method == "OPTIONS") , then print(postData) will still return nothing...
You can use the StringDecoder to tranform from "List of Int" to "String" from the HttpRequest. Since no matter if you send json, plain text, or png, Dart always send data in form of
"List of Int" to the server.Another means is to use the Streams (http://www.dartlang.org/articles/feet-wet-streams/) tested on Heroku Steam v0.6.2 Dart Editor 0.4.3_r20602 Dat SDK 0.4.3.5_r26062
For example,
the client:
import 'dart:html';
import 'dart:json' as Json;
import 'dart:async';
import 'dart:uri';
final String data = 'Hello World!';
void _sendPNG(String pngData) {
HttpRequest request = new HttpRequest(); // create a new XHR
// add an event handler that is called when the request finishes
request.onReadyStateChange.listen((_)
{
if (request.readyState == HttpRequest.DONE &&
(request.status == 200 || request.status == 0)) {
// data saved OK.
print(request.responseText); // output the response from the server
}
}
);
// POST the data to the server Async
print('Sending Photos to the server...');
var url = "/png";
request.open("POST", url);
request.setRequestHeader("Content-Type", "text/plain");
request.send(data);
}
the server:
import 'dart:io';
import 'dart:async';
import 'dart:json' as Json;
import "package:stream/stream.dart";
import 'package:xml/xml.dart' as xml;
import 'package:unittest/unittest.dart';
import 'package:rikulo_commons/mirrors.dart';
void receivePNG(HttpConnect connect){
var request = connect.request;
var response = connect.response;
if(request.uri.path == '/png' && request.method == 'POST')
{
String png='';
response.write('The server received png request!');
//read incoming List<int> data from request and use StringDecoder to transform incoming data to string
var stream = request.transform(new StringDecoder());
stream.listen((value){
print(value);
//Hello World!
}
else
{
response.write('error');
response.statusCode = HttpStatus.NOT_FOUND;
connect.close();
}
}
configure.dart
var _mapping = {
"/": home,
"/png": receivePNG,
};
Right now, the handling of POST data is a little difficult. But essentially the HttpRequest itself has to be 'listened' to. HttpRequest is a stream itself. In particular it's a Stream<List<int>>. So basically your data may be passed to your HttpRequest as multiple List<int>'s. So we need to reconstruct the data then convert it into a string (assuming you're expecting a string, not binary data, etc). Here's more or less what I do:
HttpServer.bind('127.0.0.1', 8081).then((server) {
server.listen((HttpRequest request) {
if(request.method == "POST") {
List<int> dataBody = new List<int>();
request.listen(dataBody.addAll, onDone: () {
var postData = new String.fromCharCodes(dataBody);
// Do something with the data now.
});
}
request.response.close();
});
Note that the request.listen(dataBody.AddAll, ...) basically calls List.addAll() each time data is to the server (in cases of larger data or multi-part forms it may not come all at once). This ensures we buffer it all until the stream indicates it is 'done' In which case we can now do something with the data we received, like convert it to a string.
I have found this useful example with client/side code
GitHub json send to server Example
// XXX: Dart Editor thinks this is OK, but I haven't run it.
import 'dart:html';
String encodeMap(Map data) {
return data.keys.map((k) {
return '${Uri.encodeComponent(k)}=${Uri.encodeComponent(data[k])}';
}).join('&');
}
loadEnd(HttpRequest request) {
if (request.status != 200) {
print('Uh oh, there was an error of ${request.status}');
return;
} else {
print('Data has been posted');
}
}
main() {
var dataUrl = '/registrations/create';
var data = {'dart': 'fun', 'editor': 'productive'};
var encodedData = encodeMap(data);
var httpRequest = new HttpRequest();
httpRequest.open('POST', dataUrl);
httpRequest.setRequestHeader('Content-type',
'application/x-www-form-urlencoded');
httpRequest.onLoadEnd.listen((e) => loadEnd(httpRequest));
httpRequest.send(encodedData);
}