What i'm trying to do is to save all the static of my website in a json file that i want to read in angular, i was thinking about two ways of do it:
Call a json file directly from AngularJS
Send the json file from node to AngularJS
I don't know how to do either of them, i've tried the second way like this(no luck):
Nodejs code:
app.get( '/content', function ( require, response ) {
response.setHeader('Content-Type', 'application/json');
response.json( readJSONFile( './client/content.json', function ( err, json ) {
if ( err ) {
throw err;
}
console.log( json );
} )
)
} );
function readJSONFile( filename, callback ) {
require( "fs" ).readFile( filename, function ( err, data ) {
if ( err ) {
callback( err );
return;
}
try {
callback( null, JSON.parse( data ) );
} catch ( exception ) {
callback( exception );
}
} );
}
When i access to localhost:3000/content and i check the network on the browser the file is sent, but i can't see the data in the preview tab, not sure of what i'm doing wrong...
Also how could i make a service to get this data from the server in AngularJS?
So your question can be broken down into 2 (quite) separate questions. Let's go one by one:
1. How to serve a json file from a Node.js app
The reason your code didn't work was that you did it in a "wrong order". Your readJSONFile function is an asynchronous one, where the content of the json file is only available after a while, when the callback block of Node's readFile function is invoked. Here, you are trying to use the immediate return value of readJSONFile as a parameter to response.json, which means undefined. And that is why you couldn't see any data in the preview tab.
The solutions:
a. You can use readFileSync instead for simplicity:
app.get( '/content', function ( request, response ) {
response.json( require('fs').readFileSync('./client/content.json', 'utf8') );
}
b. Or to live by Node.js' asynchrony signature (also a better practice), you could do something like this:
app.get( '/content', function ( request, response ) {
readJSONFile( './client/content.json', function ( err, json ) {
if ( err ) {
throw err;
}
return response.json( json );
} );
} );
I'd suggest you google readFile and readFileSync to have a more solid understanding on these native Node.js APIs. Either way, both approaches should give you some data in the console preview tab. Now what's left is for the next question:
2. How to make requests to an HTTP end-point from AngularJS
To be honest this is quite a "big question", not because it's difficult or too complicated to explain, but because there are many different ways to do it. The quick answer would be to tell you to use either Angular's native $http service, or ngResource, or the versatile Restangular; and while I can go on and on about those here in this post, I think there has been much better answers on StackOverflow about this topic, for example this one.
Again, I'd personally recommend that you research a bit more on the 3 keywords above. If things still look so murky afterwards, you may always come back here and fire a more detailed question, for example: "How can I make an HTTP request to the server from AngularJS using <NAME>", with <NAME> being $http, ngResource or Restangular, which I (or any other active user) would then be more than happy to answer!
Related
I am writing a private plugin for nodebb (open forum software). In the nodebb's webserver.js file there is a line that seems to be hogging all incoming json data.
app.use(bodyParser.json(jsonOpts));
I am trying to convert all incoming json data for one of my end-points into raw data. However the challenge is I cannot remove or modify the line above.
The following code works ONLY if I temporarily remove the line above.
var rawBodySaver = function (req, res, buf, encoding) {
if (buf && buf.length) {
req.rawBody = buf.toString(encoding || 'utf8');
}
}
app.use(bodyParser.json({ verify: rawBodySaver }));
However as soon as I put the app.use(bodyParser.json(jsonOpts)); middleware back into the webserver.js file it stops working. So it seems like body-parser only processes the first parser that matches the incoming data type and then skips all the rest?
How can I get around that? I could not find any information in their official documentation.
Any help is greatly appreciated.
** Update **
The problem I am trying to solve is to correctly handle an incoming stripe webhook event. In the official stripe documentation they suggested I do the following:
// Match the raw body to content type application/json
app.post('/webhook', bodyParser.raw({type: 'application/json'}),
(request, response) => {
const sig = request.headers['stripe-signature'];
let event;
try {
event = stripe.webhooks.constructEvent(request.body, sig,
endpointSecret);
} catch (err) {
return response.status(400).send(Webhook Error:
${err.message});
}
Both methods, the original at the top of this post and the official stripe recommended way, construct the stripe event correctly but only if I remove the middleware in webserver. So my understanding now is that you cannot have multiple middleware to handle the same incoming data. I don't have much wiggle room when it comes to the first middleware except for being able to modify the argument (jsonOpts) that is being passed to it and comes from a .json file. I tried adding a verify field but I couldn't figure out how to add a function as its value. I hope this makes sense and sorry for not stating what problem I am trying to solve initially.
The only solution I can find without modifying the NodeBB code is to insert your middleware in a convenient hook (that will be later than you want) and then hack into the layer list in the app router to move that middleware earlier in the app layer list to get it in front of the things you want to be in front of.
This is a hack so if Express changes their internal implementation at some future time, then this could break. But, if they ever changed this part of the implementation, it would likely only be in a major revision (as in Express 4 ==> Express 5) and you could just adapt the code to fit the new scheme or perhaps NodeBB will have given you an appropriate hook by then.
The basic concept is as follows:
Get the router you need to modify. It appears it's the app router you want for NodeBB.
Insert your middleware/route as you normally would to allow Express to do all the normal setup for your middleware/route and insert it in the internal Layer list in the app router.
Then, reach into the list, take it off the end of the list (where it was just added) and insert it earlier in the list.
Figure out where to put it earlier in the list. You probably don't want it at the very start of the list because that would put it after some helpful system middleware that makes things like query parameter parsing work. So, the code looks for the first middleware that has a name we don't recognize from the built-in names we know and insert it right after that.
Here's the code for a function to insert your middleware.
function getAppRouter(app) {
// History:
// Express 4.x throws when accessing app.router and the router is on app._router
// But, the router is lazy initialized with app.lazyrouter()
// Express 5.x again supports app.router
// And, it handles the lazy construction of the router for you
let router;
try {
router = app.router; // Works for Express 5.x, Express 4.x will throw when accessing
} catch(e) {}
if (!router) {
// Express 4.x
if (typeof app.lazyrouter === "function") {
// make sure router has been created
app.lazyrouter();
}
router = app._router;
}
if (!router) {
throw new Error("Couldn't find app router");
}
return router;
}
// insert a method on the app router near the front of the list
function insertAppMethod(app, method, path, fn) {
let router = getAppRouter(app);
let stack = router.stack;
// allow function to be called with no path
// as insertAppMethod(app, metod, fn);
if (typeof path === "function") {
fn = path;
path = null;
}
// add the handler to the end of the list
if (path) {
app[method](path, fn);
} else {
app[method](fn);
}
// now remove it from the stack
let layerObj = stack.pop();
// now insert it near the front of the stack,
// but after a couple pre-built middleware's installed by Express itself
let skips = new Set(["query", "expressInit"]);
for (let i = 0; i < stack.length; i++) {
if (!skips.has(stack[i].name)) {
// insert it here before this item
stack.splice(i, 0, layerObj);
break;
}
}
}
You would then use this to insert your method like this from any NodeBB hook that provides you the app object sometime during startup. It will create your /webhook route handler and then insert it earlier in the layer list (before the other body-parser middleware).
let rawMiddleware = bodyParser.raw({type: 'application/json'});
insertAppMethod(app, 'post', '/webhook', (request, response, next) => {
rawMiddleware(request, response, (err) => {
if (err) {
next(err);
return;
}
const sig = request.headers['stripe-signature'];
let event;
try {
event = stripe.webhooks.constructEvent(request.body, sig, endpointSecret);
// you need to either call next() or send a response here
} catch (err) {
return response.status(400).send(`Webhook Error: ${err.message}`);
}
});
});
The bodyParser.json() middleware does the following:
Check the response type of an incoming request to see if it is application/json.
If it is that type, then read the body from the incoming stream to get all the data from the stream.
When it has all the data from the stream, parse it as JSON and put the result into req.body so follow-on request handlers can access the already-read and already-parsed data there.
Because it reads the data from the stream, there is no longer any more data in the stream. Unless it saves the raw data somewhere (I haven't looked to see if it does), then the original RAW data is gone - it's been read from the stream already. This is why you can't have multiple different middleware all trying to process the same request body. Whichever one goes first reads the data from the incoming stream and then the original data is no longer there in the stream.
To help you find a solution, we need to know what end-problem you're really trying to solve? You will not be able to have two middlewares both looking for the same content-type and both reading the request body. You could replace bodyParser.json() that does both what it does now and does something else for your purpose in the same middleware, but not in separate middleware.
I see that JSON.stringify and JSON.parse are both sycnhronous.
I would like to know if there a simple npm library that does this in an asynchonous way .
Thank you
You can make anything "asynchronous" by using Promises:
function asyncStringify(str) {
return new Promise((resolve, reject) => {
resolve(JSON.stringify(str));
});
}
Then you can use it like any other promise:
asyncStringfy(str).then(ajaxSubmit);
Note that because the code is not asynchronous, the promise will be resolved right away (there's no blocking operation on stringifying a JSON, it doesn't require any system call).
You can also use the async/await API if your platform supports it:
async function asyncStringify(str) {
return JSON.stringify(str);
}
Then you can use it the same way:
asyncStringfy(str).then(ajaxSubmit);
// or use the "await" API
const strJson = await asyncStringify(str);
ajaxSubmit(strJson);
Edited: One way of adding true asynchrnous parsing/stringifying (maybe because we're parsing something too complex) is to pass the job to another process (or service) and wait on the response.
You can do this in many ways (like creating a new service that shares a REST API), I will demonstrate here a way of doing this with message passing between processes:
First create a file that will take care of doing the parsing/stringifying. Call it async-json.js for the sake of the example:
// async-json.js
function stringify(value) {
return JSON.stringify(value);
}
function parse(value) {
return JSON.parse(value);
}
process.on('message', function(message) {
let result;
if (message.method === 'stringify') {
result = stringify(message.value)
} else if (message.method === 'parse') {
result = parse(message.value);
}
process.send({ callerId: message.callerId, returnValue: result });
});
All this process does is wait a message asking to stringify or parse a JSON and then respond with the right value.
Now, on your code, you can fork this script and send messages back and forward. Whenever a request is sent, you create a new promise, whenever a response comes back to that request, you can resolve the promise:
const fork = require('child_process').fork;
const asyncJson = fork(__dirname + '/async-json.js');
const callers = {};
asyncJson.on('message', function(response) {
callers[response.callerId].resolve(response.returnValue);
});
function callAsyncJson(method, value) {
const callerId = parseInt(Math.random() * 1000000);
const callPromise = new Promise((resolve, reject) => {
callers[callerId] = { resolve: resolve, reject: reject };
asyncJson.send({ callerId: callerId, method: method, value: value });
});
return callPromise;
}
function JsonStringify(value) {
return callAsyncJson('stringify', value);
}
function JsonParse(value) {
return callAsyncJson('parse', value);
}
JsonStringify({ a: 1 }).then(console.log.bind(console));
JsonParse('{ "a": "1" }').then(console.log.bind(console));
Note: this is just one example, but knowing this you can figure out other improvements or other ways to do it. Hope this is helpful.
Check this out, another npm package-
async-json is a library that provides an asynchronous version of the standard JSON.stringify.
Install-
npm install async-json
Example-
var asyncJSON = require('async-json');
asyncJSON.stringify({ some: "data" }, function (err, jsonValue) {
if (err) {
throw err;
}
jsonValue === '{"some":"data"}';
});
Note-Didn't test it, you need to manually check it's dependency and
required packages.
By asynchronous I assume you actually mean non-blocking asynchronous - i.e., if you have a large (megabytes large) JSON string, and you stringify, you don't want your web server to hard freeze and block newly incoming web requests for 500+ milliseconds while it processes the object.
Option 1
The generic answer is to iterate through your object piece by piece, and to then call setImmedate whenever a threshold is reached. This then allows other functions in the event queue to run for a bit.
For JSON (de)serialization, the yieldable-json library does this very well. It does however drastically sacrifice JSON processing time (which is somewhat intentional).
Usage example from the yieldable-json readme:
const yj = require('yieldable-json')
yj.stringifyAsync({key:"value"}, (err, data) => {
if (!err)
console.log(data)
})
Option 2
If processing speed is extremely important (such as with real-time data), you may want to consider spawning multiple Node threads instead. I've used used the PM2 Process Manager with great success, although initial setup was quite daunting. Once it works however, the final result is magic, and does not require modifying your source code, just your package.json file. It acts as a proxy, load balancer, and monitoring tool for Node applications. It's somewhat analogous to Docker swarm, but bare metal, and does not require a special client on the server.
I have a loopback 2.x app, in which I have a model Conversation and a model Message, with a relationship "Conversation has many messages". I want to customize the response for POST conversations/:id/messages with a json response different than the default, say {status: 'success'}. I tried to use remote hook for the method __create__messages, but it did not work:
Conversation.afterRemote('__create__messages', function(ctx, next) {
ctx.result.data = {
success: 'yes'
};
next();
});
This still returns the default response. How can I return a custom json for a remote method? I have seen examples only for all models, or for all methods: multiple models, multiple methods
Maybe you can try a version of following code below. Also, I think you are meaning to to manipulate data before the method finishes, not after. If you wait, the response will already be created, preventing your intended goal. Let me know if this works (replace with methods that will work for your use case).
Conversation.observe('before save', function(context, next) {
var instance = context.instance || context.data;
if (!instance) return next();
// Your code here
next();
});
What would be a good way to work with Chrome's incoming 1MB limit for native messaging extensions? The data that we would be sending to the extension is json-serialized gpx, if that matters.
When the original message is >1MB, it seems like this question really has two parts:
how to partition the data on the sending end (i.e. the client)
this part should be pretty trivial. Even if we need to split into separate self-contained complete gpx strings, that is pretty straightforward.
how to join the <1MB messages back in to the original >1MB
is there a standard known solution for this question? We can call background.js (ie. the function passed to chrome.runtime.onMessageExternal.addListener) once for each <1MB incoming message, but, how would we combine the strings from those repeated calls in to one response for the extension?
UPDATE 8-18-16:
what we've been doing is just appending each message 'chunk' on a buffer variable in background.js, and not send it back to Chrome until disconnection:
var gpxText="";
port.onMessage.addListener(function(msg) {
// msg must be a JSON-serialized simple string;
// append each incoming msg to the collective gpxText string
// but do not send it to Chrome until disconnection
// console.log("received " + msg);
gpxText+=msg;
});
port.onDisconnect.addListener(function(msg) {
if (gpxText!="") {
sendResponse(JSON.parse(gpxText));
gpxText="";
} else {
sendResponse({status: 'error', message: 'something really bad happened'});
}
// build the response object here with msg, status, error tokens, and always send it
console.log("disconnected");
});
We will have to make that appending a bit smarter to handle and send both status and message keys/values, but that should be easy.
I have this same issue and have been scouring the web for the past couple days to figure out something to do. In my application, I am currently shipping a JSON string over to the background script in chunks, having to create a subprotocol to handle this special case. e.g. my initial question might look like:
{action:"getImage",guid:"123"}
and the response for <1MB might look like:
{action:"getImage",guid:"123",status:"success",return:"ABBA..."}
where ABBA... represents a base64 encoding of the bytes. when >1MB, however, the response will look like:
{action:"getStream",guid:"123",status:"success",return:"{action:\"getImage\",guid:\"123\",return:\"ABBA...",more:true}
and upon receipt of the payload with method==='stream', the background page will immediately issue a new request like:
{action:"getStream",guid:"123"}
and the next response might look like:
{action:"getStream",guid:"123",status:"success",return:"...DEAF==",more:false}
so your onMessage handler would look something like:
var streams;
function onMessage( e ) {
var guid = e.guid;
if ( e.action === 'getStream' ) {
if ( !streams[ guid ] ) streams[ guid ] = '';
streams[ guid ] += e[ 'return' ];
if ( e.more ) {
postMessage( { action: 'getStream', guid: guid } );
// no more processing to do here, bail
return;
}
e = JSON.parse( streams[ guid ] );
streams[ guid ] = null;
}
// do something with e as if it was never chunked
...
}
it works, but I am somewhat convinced that it is slower than it should be (though this could be due to the slow feeling of the STDIO signaling and, in my particular app, additional signaling that has to happen for each new chunk).
Ideally I'd like to stream the file in a more efficient protocol supported natively by Chrome. I looked into WebRTC, but it would mean that I'd need to implement the API into my native messaging host (as best I can tell), which is not an option I'm willing to take on. I played with 'passing' the contents by file as such:
if ( e.action = 'getFile' ) {
xhr = new XMLHttpRequest();
xhr.onreadystatechange = function( e ) {
if ( e.target.readyState === 4 ) {
onMessage( e.target.responseText );
}
};
xhr.open( 'GET', chrome.extension.getURL( '/' + e.file ), true );
xhr.send();
return;
}
where I have my native message host write a .json file out to extension's install directory and it seems to work, but there is no way for me to reliably derive the path (without fudging things and hoping for the best), because as best I can the location of the extensions install path is determined by your Chrome user profile and there's no API I could find to give me that path. Additionally, there's a 'version' folder created under your extension id which includes an _0 that I don't know how to calculate (is the _0 constant for some future use? does it tick up when an extension is published anew to the web store, but the version is not adjusted?).
At this point I'm out of ideas and I'm hoping someone will stumble across this question with some guidance.
I am using the following package for Meteor https://atmospherejs.com/ajbarry/yahoo-finance
I cant seem to get a specified field to work, here is a link that contains a list of all the available fields, however 'j2' and some others I tested don't work, in the sense there is no response in the result object, or no json key pair values.
Heres is my client side code.
Template.stock.rendered = function (){
if ( _.isEmpty(Session.get('ENW.V')) ) {
Meteor.call('getQuote', 'ENW.V', function(err, result) {
Session.set('ENW.V', result['ENW.V']);
console.log(result)
});
}
}
Template.stock.helpers({
stock: function() {
return Session.get('ENW.V');
}
})
Server side Method
Meteor.methods({
getQuote: function( stockname ) {
return YahooFinance.snapshot({symbols: [stockname] , fields:['n','a','b','j2'] });
}
});
Thanks for any Help in Advance. Happy to add any additional info if needed.
Did a test run after commenting out that line and it seems to work fine. Create an issue with the package owner to see if you can have it fixed for the long run.
The package you are using is deliberately excluding those fields. For what reason, I cannot say. For a full list of fields that it is avoiding, look here:
https://github.com/pilwon/node-yahoo-finance/blob/master/lib/index.js#L122