How to make a nodejs server asynchronous - json

Question

I have a nodeJS server, that takes JSON from three websites and sends it to be displayed on my website(in JSON). The JSON on the websites that I'm taking from is constantly updated, every 10 seconds. How can I make my NodeJS server constantly update so it has the most up to date data?
I'm assuming this isn't possible without refreshing the page, but it would be optimal if the page wasn't refreshed.
If this is impossible to do with NodeJS and there is a different method of accomplishing this, I would be extremely appreciative if you told me.
Code:
router.get("/", function(req, res){
var request = require('request-promise');
var data1;
var data2;
var data3;
request("website1.json").then(function(body){
data1 = JSON.parse(body);
return request("website2.json");
})
.then(function(body) {
data2 = JSON.parse(body);
return request("website3.json");
})
.then(function(body){
data3 = JSON.parse(body);
res.render("app.ejs", {data1: data1, data2: data2, data3: data3});
})
});

Answer 1

Here's some general guidelines:
Examine the APIs you have available from your external service. Find out if there is anything it offers that lets you make a webSocket connection or some other continuous TCP connection so you can get realtime (or close to realtime) notifications when things change. If so, have your server use that.
If there is no realtime notification API from the external server, then you are just going to have to poll it every xx seconds. To decide how often you should poll it, you need to consider: a) How often you really need new data in your web pages (for example, maybe data that is current within 5 minutes is OK), b) What the terms of service and rate limiting are for the 3rd party service (e.g. how often will they let you poll it) and c) how much your server can afford to poll it (from a server load point of view).
Once you figure out how often you're going to poll the external service, then you build yourself a recurring polling mechanism. The simplest way would be using setInterval() that is set for your polling interval time. I have a raspberry pi node.js server that uses a setInterval() to repeatedly check several temperature sensors. That mechanism works fine as long as you pick an appropriate interval time for your situation.
Then for communication of new information back to a connected web page, the best way to get near "real time" updates form the server is for the web page to make a webSocket or socket.io connection to your server. This is a continuously connected socket over which messages can be sent either way. So, using this mechanism, the client makes a socket.io connection to your server. The server receives that connection and the connection stays open for the lifetime of that web page. Then, anytime your server has new data that needs to be sent to that web page, it can just send a message over that socket.io connection. The web page will receive that message and can then update the contents of the web page accordingly based on the data in the message. No page refresh is needed.
Here's an outline of the server code:
// start up socket.io listener using your existing web server
var io = require('socket.io')(app);
// recurring interval to poll several external web sites.
setInterval(function () {
var results = {};
request("website1.json").then(function (body) {
results.data1 = JSON.parse(body);
return request("website2.json");
}).then(function (body) {
results.data2 = JSON.parse(body);
return request("website3.json");
}).then(function (body) {
results.data3 = JSON.parse(body);
// decide if anything has actually changed on external service data
// and whether anything needs to be sent to connected clients
io.emit("newData", results);
}).catch(function(err) {
// decide what to do if external service causes an error
});
}, 10000);
The client code would then be generally like this:
<script src="/socket.io/socket.io.js"></script>
<script>
var socket = io();
socket.on("newData", function(data) {
// process new data here and update the web page
});
</script>