JSON streaming with Oboe.js, MongoDB and Express.js - json

Question

I'm experimenting with JSON streaming through HTTP with Oboe.js, MongoDB and Express.js.
The point is to do a query in MongoDB (Node.js's mongodb native drive), pipe it (a JavaScript array) to Express.js and parse it in the browser with Oboe.js.
The benchmarks I did compared streaming vs. blocking in both the MongoDB query server-side and the JSON parsing in the client-side.
Here is the source code for the two benchmarks. The first number is the number of milli-seconds for 1000 queries of 100 items (pagination) in a 10 million documents collection and the second number between parenthesis, represents the number of milli-seconds before the very first item in the MongoDB result array is parsed.
The streaming benchmark server-side:
// Oboe.js - 20238 (16.887)
// Native - 16703 (16.69)
collection
.find()
.skip(+req.query.offset)
.limit(+req.query.limit)
.stream()
.pipe(JSONStream.stringify())
.pipe(res);
The blocking benchmark server-side:
// Oboe.js - 17418 (14.267)
// Native - 13706 (13.698)
collection
.find()
.skip(+req.query.offset)
.limit(+req.query.limit)
.toArray(function (e, docs) {
res.json(docs);
});
These results really surprise me because I would have thought that:
Streaming would be quicker than blocking every single time.
Oboe.js would be quicker to parse the entire JSON array compared to the native JSON.parse method.
Oboe.js would be quicker to parse the first element in the array compared to the native JSON.parse method.
Does anyone have an explanation ?
What am I doing wrong ?
Here is the source-code for the two client-side benchmarks too.
The streaming benchmark client-side:
var limit = 100;
var max = 1000;
var oboeFirstTimes = [];
var oboeStart = Date.now();
function paginate (i, offset, limit) {
if (i === max) {
console.log('> OBOE.js time:', (Date.now() - oboeStart));
console.log('> OBOE.js avg. first time:', (
oboeFirstTimes.reduce(function (total, time) {
return total + time;
}, 0) / max
));
return true;
}
var parseStart = Date.now();
var first = true;
oboe('/api/spdy-stream?offset=' + offset + '&limit=' + limit)
.node('![*]', function () {
if (first) {
first = false;
oboeFirstTimes.push(Date.now() - parseStart);
}
})
.done(function () {
paginate(i + 1, offset + limit, limit);
});
}
paginate(0, 0, limit);
The blocking benchmark client-side:
var limit = 100;
var max = 1000;
var nativeFirstTimes = [];
var nativeStart = Date.now();
function paginate (i, offset, limit) {
if (i === max) {
console.log('> NATIVE time:', (Date.now() - nativeStart));
console.log('> NATIVE avg. first time:', (
nativeFirstTimes.reduce(function (total, time) {
return total + time;
}, 0) / max
));
return true;
}
var parseStart = Date.now();
var first = true;
var req = new XMLHttpRequest();
req.open('GET', '/api/spdy-stream?offset=' + offset + '&limit=' + limit, true);
req.onload = function () {
var json = JSON.parse(req.responseText);
json.forEach(function () {
if (first) {
first = false;
nativeFirstTimes.push(Date.now() - parseStart);
}
});
paginate(i + 1, offset + limit, limit);
};
req.send();
}
paginate(0, 0, limit);
Thanks in advance !

Answer 1

I found those comments in Oboe doc at the end of the "Why Oboe?" section:
Because it is a pure Javascript parser, Oboe.js requires more CPU time than JSON.parse. Oboe.js works marginally more slowly for small messages that load very quickly but for most real-world cases using i/o effectively beats optimising CPU time.
SAX parsers require less memory than Oboe’s pattern-based parsing model because they do not build up a parse tree. See Oboe.js vs SAX vs DOM.
If in doubt, benchmark, but don’t forget to use the real internet, including mobile, and think about perceptual performance.