Converting comma delimited JSON to a newline delimited node - json

Question

I have a JSON file which I am reading with node, modifying and saving it as a json file.
I'm looking to save the new json as newline delimited vs being in an array.
I came across https://github.com/CrowdProcess/newline-json but don't fully understand streams. If i have the following stream setup, how can I pipe it though the parser and stringifier?
fileStream = fs.createReadStream('source.json')
writeStream = fs.createWriteStream('output.txt');
var Parser = require('newline-json').Parser;
var Stringifier = require('newline-json').Stringifier;
var parser = new Parser();
var stringifier = new Stringifier();
But running the following only outputs a blank file.
fileStream.pipe(parser).pipe(stringifier).pipe(writeStream)
what am I missing about streams?

Answer 1

One way to convert a JSON array to a stream of newline-delimited JSON entities is to use jq with the -c option, e.g.
$ jq -c ".[]"
Input:
[[1,2], 3, {"4":5}]
Output:
[1,2]
3
{"4":5}
See https://stedolan.github.io/jq

Answer 2

For anybody who is looking to a solution on how to convert json array of objects to nd-json. Here is the solution:
Input:
const arrObj = [{
id: 1,
name: 'joe'
}, {
id: 2,
name: 'ben'
}, {
id: 3,
name: 'jake'
}, {
id: 4,
name: 'marsh'
}];
// stringify the object and join with \n
const ndJson = arrObj.map(JSON.stringify).join('\n');
console.log(ndJson);
Output:
{"id":1,"name":"joe"}
{"id":2,"name":"ben"}
{"id":3,"name":"jake"}
{"id":4,"name":"marsh"}
Example Usecase: When importing bulk request from a json file to elasticsearch.
Happy Coding :)

Answer 3

In node.js you can use the node-jq package to do what #peak has shown above.
var stream = require('stream');
var fs = require('fs');
const jq = require('node-jq');
var fileName = 'YOUR_FILE_NAME'; //e.g abc.json
var bucketName = 'YOUR_BUCKET NAME'; // e.g gs://def
var dataStream = new stream.PassThrough();
async function formatJson() {
jq.run('.[]', fileName, {output: 'compact'})
.then((output) => {
dataStream.push(output)
dataStream.push(null)
console.log(dataStream)
})
.catch((err) => {
console.log(err)
})
}
formatJson()
I am not an experienced node person, so apologies if the code is clumsy but it works.