edit it seems I'm trying to create some concat gulp pipe that first transforms the contents of the various files.
I have been using through2 to write a custom pipe. Scope: Using gulp-watch, run a task that loads all of them, transform them in memory and output a few files.
Through2 comes with a "highWaterMark" option that defaults to 16 files being read at a time. My pipe doesn't need to be memory optimized (it reads dozens of <5kb jsons, runs some transforms and outputs 2 or 3 jsons). But I'd like to understand the preferred approach.
I'd like to find a good resource / tutorial explaining how such situations are handled, any lead's welcome.
Thanks,
Ok, found my problem.
When using through2 to create a custom pipe, in order to "consume" the data (and not hit the highWaterMark limit), one simply has to add an .on('data', () => ...) handler, like in the following example:
function processAll18nFiles(done) {
const dictionary = {};
let count = 0;
console.log('[i18n] Rebuilding...');
gulp
.src(searchPatternFolder)
.pipe(
through.obj({ highWaterMark: 1, objectMode: true }, (file, enc, next) => {
const { data, path } = JSON.parse(file.contents.toString('utf8'));
next(null, { data, path });
})
)
// this line fixes my issue, the highWaterMark doesn't cause a limitation now
.on('data', ({ data, path }) => ++count && composeDictionary(dictionary, data, path.split('.')))
.on('end', () =>
Promise.all(Object.keys(dictionary).map(langKey => writeDictionary(langKey, dictionary[langKey])))
.then(() => {
console.log(`[i18n] Done, ${count} files processed, language count: ${Object.keys(dictionary).length}`);
done();
})
.catch(err => console.log('ERROR ', err))
);
rem mind the "done" parameter forcing the developper to make a call when it's done()
Related
I'm relatively new to puppeteer and I'm trying to understand the patterns that can be used to build more complex apis with it. I am building a cli where I am running a WebGL app in puppeteer which i call various functions in, and with my current implementation i have to copy and paste a lot of setup code.
Usually, in every cli command i have to setup pupeteer, setup the app and get access to its api object, and then run an arbitrary command on that api, and get the data back in node.
It looks something like this.
const {page, browser} = await createBrowser() // Here i setup the browser and add some script tags.
let data;
page.exposeFunction('extractData', (data) => {
data = data;
})
await page.evaluate(async (input) => {
// Setup work
const requestEvent = new CustomEvent('requestAppApi', {
api: undefined;
})
window.dispatchEvent(requestEvent);
const api = requestEvent.detail.api;
// Then i call some arbitrary function, that will
always return some data that gets extracted by the exposed function.
const data = api.arbitraryFunction(input);
window.extractData(data)
}, input)
What i would like is to wrap all of the setup code in a function, so that i could call it and just specify what to do with the api object once i have it.
My initial idea was to have a function that will take a callback that has this api object as a parameter.
const { page, browser } = wait createBrowser();
page.exposeFunction(async (input) =>
setupApiObject(((api) =>
api.callSomeFunction(input)
), input)
However, this does not work. I understand that puppeteer requires any communication between the node context and the browser to be serialised as json, and obviously a function cant be. Whats tripping me up is that I'm not actually wanting to call these methods in the node context, just have a way to reuse them. The actual data transfer is already handled by page.exposeFunction.
How would a more experienced puppeteer dev accomplish this?
I'll answer my own question here, since i managed to figure out a way to do it. Basically, you can use page.evaluate to create a function on the window object that can later be reused.
So i did something like
await page.evaluate(() => {
window.useApiObject = function(callback: (api) => void){
// Perform setup code
callback()
}
})
Meaning that later on i could use that method in the browser context and avoid redoing the setup code.
page.evaluate(() => {
window.useApiObject((api) => {
api.someMethod()
})
})
onMounted(() => {
productService.value
.getProducts()
.then((data) => (products.value = data));
console.log((products))
});
When I print products with console.log, here what I have.
capture of the console
I see that the data I want are in RawValue but I don't know how to access them.
I tried Object.values(products) or just console.log(products._rawValue) or console.log(products.rawValue) it print undefined.
Do you know what function call ?
Thanks
There are 2 issues
#1 - you're using console.log(products) which shows you the reactive object, what you need instead is console.log(products.value) which will only show the value, which should match the content of data.produtcs
#2 - you might find that 👆 now shows an empty result. The reason that's happening is that you're calling the console log after the async function, but before it finishes, so you're calling it before it has a chance to update. To fix that, you can log as part of the async function
onMounted(() => {
productService.value
.getProducts()
.then((data) => {
products.value = data;
console.log(products.value);
})
});
If you're using the products inside a template, you don't need to worry about what's before or after the async function since it will re-render the component on change.
Also, you probably don't need to define productService as a ref, the class is likely not something that needs to be reactive, so you can just do simple assignment and then skip the .value to call getProducts
with axios what I do is take out the data with response.data you could try
onMounted(() => {
productService.value.getProducts().then((response) => (
products = response.data
));
console.log(products.length);
});
I am using node.js to export a collection that I have in firestore.
Until now the connection and the selection of the documents of the collection work perfectly.
I am trying to save the structure in a json file but the result is not what I expected.
This is the output in the physical file:
enter image description here
On the right of the photo, it will be seen as presented by the console.log and on the right it is displayed as recorded by json.push
I can't get the physical file to have the structure shown in the console.log.
I appreciate all your help.
as you will see the structure here is failing: "USBCALI|AFILIADO|uqcMSoxdwCTQoLzS2J6brNTZ3Dy2",
":",
{.....
should be: USBCALI|AFILIADO|uqcMSoxdwCTQoLzS2J6brNTZ3Dy2 : {.....
this is the code
const jsonfile = require('jsonfile')
function start (db, file, collection) {
const ref = db.collection(collection)
const json = []
console.log(`Getting ${collection}...`)
ref
.get()
.then((snapshot) => {
snapshot.forEach((doc) => {
console.log(doc.id, ':', doc.data())
json.push(doc.id, ':', doc.data())
})
console.log(`Writing ${file}...`)
jsonfile.writeFile(file, json, {spaces: 2}, err => {
if (err) {
console.error(err)
} else {
console.log(`Collection ${collection} successfully written to ${file}.`)
}
})
})
.catch((err) => {
console.log('Error getting documents', err)
})
}
module.exports = start
Let me explain whats going on here
console.log(doc.id, ':', doc.data())
Is logging out the information how you would think you would like it to be
json.push(doc.id, ':', doc.data()) is pushing 3 elements into the array every time its called.
What you really want to do is manipulate the data properly into an object using a map function. I'm making some assumptions based on all the info I have here, but I'm guessing you want to do something like
var docs snapshot.map( doc => ({[doc.id] : doc.data()})
json.push(docs)
instead of
snapshot.forEach((doc) => {
console.log(doc.id, ':', doc.data())
json.push(doc.id, ':', doc.data())
})
You could also do (this is a bit more verbose than the map method but will essentially do the same thing. The map function is like select in sql or Linq, its handy to transform data from one form into another 😀
snapshot.forEach((doc) => {
var currentItem = {[doc.id] : doc.data()}
json.push(currentItem)
})
Since you have used some es6 notation, I'm assuming that Computed Property Names are supports, if not let me know and Ill come up with another soloution
I'm writing a method that uses async/await and promises to write some JSON to a file and then render a pug template. But for some reason the code that writes the JSON conflicts with the res.render() method resulting in the browser not being able to connect to the server.
The weird thing is that I don't get any errors in the console, and the JSON file is generated as expected — the page just won't render.
I'm using the fs-extra module to write to disk.
const fse = require('fs-extra');
exports.testJSON = async (req, res) => {
await fse.writeJson('./data/foo.json', {Key: '123'})
.then(function(){
console.log('JSON updated.')
})
.catch(function(err){
console.error(err);
});
res.render('frontpage', {
title: 'JSON Updated...',
});
}
I'm starting to think that there is something fundamental I'm not getting that conflicts with promises, writing to disk and/or express' res.render method. It's worth noting that res.send() works fine.
I've also tried a different NPM module to write the file (write-json-file). It gave me the exact same issue.
UPDATE:
So I'm an idiot. The problem has nothing to do with Express og the JSON file. It has to do with the fact that I'm running nodemon to automatically restart the server when files are changed. So as soon as the JSON file was saved the server would restart, stopping the process of rendering the page. Apologies to the awesome people trying to help me anyway. You still helped me get to the problem, so I really appreciate it!
Here's the actual problem:
The OP is running nodemon to restart the server whenever it see filechanges, and this is what stops the code from running, because as soon as the json file is generated the server restarts.
Efforts to troubleshoot:
It's going to take some trouble shooting to figure this out and since I need to show you code, I will put it in an answer even though I don't yet know what is causing the problem. I'd suggest you fully instrument things with this code:
const fse = require('fs-extra');
exports.testJSON = async (req, res) => {
try {
console.log(`1:cwd - ${process.cwd()}`);
await fse.writeJson('./data/foo.json', {Key: '123'})
.then(function(){
console.log('JSON updated.')
}).catch(function(err){
console.error(err);
});
console.log(`2:cwd - ${process.cwd()}`);
console.log("about to call res.render()");
res.render('frontpage', {title: 'JSON Updated...',}, (err, html) => {
if (err) {
console.log(`res.render() error: ${err}`);
res.status(500).send("render error");
} else {
console.log("res.render() success 1");
console.log(`render length: ${html.length}`);
console.log(`render string (first part): ${html.slice(0, 20}`);
res.send(html);
console.log("res.render() success 2");
}
});
console.log("after calling res.render()");
} catch(e) {
console.log(`exception caught: ${e}`);
res.status(500).send("unknown exception");
}
}
I see that JSON.stringify and JSON.parse are both sycnhronous.
I would like to know if there a simple npm library that does this in an asynchonous way .
Thank you
You can make anything "asynchronous" by using Promises:
function asyncStringify(str) {
return new Promise((resolve, reject) => {
resolve(JSON.stringify(str));
});
}
Then you can use it like any other promise:
asyncStringfy(str).then(ajaxSubmit);
Note that because the code is not asynchronous, the promise will be resolved right away (there's no blocking operation on stringifying a JSON, it doesn't require any system call).
You can also use the async/await API if your platform supports it:
async function asyncStringify(str) {
return JSON.stringify(str);
}
Then you can use it the same way:
asyncStringfy(str).then(ajaxSubmit);
// or use the "await" API
const strJson = await asyncStringify(str);
ajaxSubmit(strJson);
Edited: One way of adding true asynchrnous parsing/stringifying (maybe because we're parsing something too complex) is to pass the job to another process (or service) and wait on the response.
You can do this in many ways (like creating a new service that shares a REST API), I will demonstrate here a way of doing this with message passing between processes:
First create a file that will take care of doing the parsing/stringifying. Call it async-json.js for the sake of the example:
// async-json.js
function stringify(value) {
return JSON.stringify(value);
}
function parse(value) {
return JSON.parse(value);
}
process.on('message', function(message) {
let result;
if (message.method === 'stringify') {
result = stringify(message.value)
} else if (message.method === 'parse') {
result = parse(message.value);
}
process.send({ callerId: message.callerId, returnValue: result });
});
All this process does is wait a message asking to stringify or parse a JSON and then respond with the right value.
Now, on your code, you can fork this script and send messages back and forward. Whenever a request is sent, you create a new promise, whenever a response comes back to that request, you can resolve the promise:
const fork = require('child_process').fork;
const asyncJson = fork(__dirname + '/async-json.js');
const callers = {};
asyncJson.on('message', function(response) {
callers[response.callerId].resolve(response.returnValue);
});
function callAsyncJson(method, value) {
const callerId = parseInt(Math.random() * 1000000);
const callPromise = new Promise((resolve, reject) => {
callers[callerId] = { resolve: resolve, reject: reject };
asyncJson.send({ callerId: callerId, method: method, value: value });
});
return callPromise;
}
function JsonStringify(value) {
return callAsyncJson('stringify', value);
}
function JsonParse(value) {
return callAsyncJson('parse', value);
}
JsonStringify({ a: 1 }).then(console.log.bind(console));
JsonParse('{ "a": "1" }').then(console.log.bind(console));
Note: this is just one example, but knowing this you can figure out other improvements or other ways to do it. Hope this is helpful.
Check this out, another npm package-
async-json is a library that provides an asynchronous version of the standard JSON.stringify.
Install-
npm install async-json
Example-
var asyncJSON = require('async-json');
asyncJSON.stringify({ some: "data" }, function (err, jsonValue) {
if (err) {
throw err;
}
jsonValue === '{"some":"data"}';
});
Note-Didn't test it, you need to manually check it's dependency and
required packages.
By asynchronous I assume you actually mean non-blocking asynchronous - i.e., if you have a large (megabytes large) JSON string, and you stringify, you don't want your web server to hard freeze and block newly incoming web requests for 500+ milliseconds while it processes the object.
Option 1
The generic answer is to iterate through your object piece by piece, and to then call setImmedate whenever a threshold is reached. This then allows other functions in the event queue to run for a bit.
For JSON (de)serialization, the yieldable-json library does this very well. It does however drastically sacrifice JSON processing time (which is somewhat intentional).
Usage example from the yieldable-json readme:
const yj = require('yieldable-json')
yj.stringifyAsync({key:"value"}, (err, data) => {
if (!err)
console.log(data)
})
Option 2
If processing speed is extremely important (such as with real-time data), you may want to consider spawning multiple Node threads instead. I've used used the PM2 Process Manager with great success, although initial setup was quite daunting. Once it works however, the final result is magic, and does not require modifying your source code, just your package.json file. It acts as a proxy, load balancer, and monitoring tool for Node applications. It's somewhat analogous to Docker swarm, but bare metal, and does not require a special client on the server.