Following the example on Piping results with Streams2, I'm trying to stream results from MySQL to stdout in node.js.
Code looks like this:
connection.query('SELECT * FROM table')
.stream()
.pipe(process.stdout);
I get this error: TypeError: invalid data
Explanation
From this github issue for the project:
.stream() returns stream in "objectMode". You can't pipe it to stdout or network
socket because "data" events have rows as payload, not Buffer chunks
Fix
You can fix this using the csv-stringify module.
var stringify = require('csv-stringify');
var stringifier = stringify();
connection.query('SELECT * FROM table')
.stream()
.pipe(stringifier).pipe(process.stdout);
notice the extra .pipe(stringifier) before the .pipe(process.stdout)
There is another solution now with the introduction of pipelinein Node v10 (view documentation).
The pipeline method does several things:
Allows you to pipe through as many streams as you like.
Provides a callback once completed.
Importantly, it provides automatic clean up. Which is a benefit over the standard pipe method.
const fs = require('fs')
const mysql = require('mysql')
const {pipeline} = require('stream')
const stringify = require('csv-stringify')
const stringifier = stringify()
const output = fs.createWriteStream('query.csv')
const connection = mysql.createConnection(...)
const input = connection.query('SELECT * FROM table').stream()
pipeline(input, stringifier, process.stdout, err => {
if (err) {
console.log(err)
} else {
console.log('Output complete')
}
}
Related
I am currently tasked with finding the amount of times a specific email has contacted us. The contacts are stored in JSON files and the key should be "email".
The thing is there are potentially infinite JSON files so I would like to merge them in to a single object and iterate to count the email frequency.
So to be clear I need to read in the JSON content. Produce it as a log
consume the message
transform that message into a tally of logs per email used.
My thought process may be wrong but I am thinking I need to merge all JSON files into a single object that I can then iterate over and manipulate if needed. However I believe I am having issues with the synchronicity of it.
I am using fs to read in (I think in this case 100 JSON files) running a forEach and attempting to push each into an array but the array comes back empty. I am sure I am missing something simple but upon reading the documentation for fs I think I just may be missing it.
const fs = require('fs');
let consumed = [];
const fConsume = () => {
fs.readdir(testFolder, (err, files) => {
files.forEach(file => {
let rawData = fs.readFileSync(`${testFolder}/${file}`);
let readable = JSON.parse(rawData);
consumed.push(readable);
});
})
}
fConsume();
console.log(consumed);
For reference this is what each JSON object looks like, and there are several per imported file.
{
id: 'a7294140-a453-4f3c-91b6-210819e2c43e',
email: 'ethan.hernandez#microsoft.com',
message: 'successfully handled skipped operation.'
},
fs.readdir() is async, so your function returns before it executes the callback. If you want to use synchronous code here, you need to use fs.readdirSync() instead:
const fs = require('fs');
let consumed = [];
const fConsume = () => {
const files = fs.readdirSync(testFolder)
files.forEach(file => {
let rawData = fs.readFileSync(`${testFolder}/${file}`);
let readable = JSON.parse(rawData);
consumed.push(readable);
});
}
fConsume();
console.log(consumed);
In the Appwrite console, I'm adding a test environment variable to pass into function...
In my function code (NodeJs) index.js, I'm logging out the value of above variable...
I save the code and use the Appwrite CLI (createTag) to push/publish the code.
Then in Appwrite console, I activate the new function then I execute it and I see this in log...
Clearly I'm missing something but I'm searching the Appwrite docs and I don't see it.
What am I doing incorrectly?
Thank you for helping :-)
Ok looks like as of this post, this is a bug in the Appwrite web UI code. You can do this 2 ways right now. You can set the environment vars in code or you can use the Appwrite CLI. I ended up putting the CLI commend in my NodeJs package.json scripts for quick easy access.
Here are both ways that worked for me...
appwrite functions create --functionId=regions_get_all --name=regions_get_all --execute=[] --runtime=node-16.0 --vars={ 'LT_API_ENDPOINT': 'https://appwrite.league-tracker.com/v1', 'LT_PROJECT_ID': '61eb...7e4ff', 'LT_FUNCTIONS_SECRET': '3b4b478e5a5576c1...ef84ba44e5fc2261cb8a8b3bfee' }
const sdk = require('node-appwrite');
const endpoint = 'https://appwrite.league-tracker.com/v1';
const projectId = '61eb3...7e4ff';
const funcionsSecret = '3b4b478e5a557ab8a...c121ff21977a';
const functionId = process.argv[2];
const name = process.argv[2];
const execute = [];
const runtime = 'node-16.0';
const env_vars = {
"LT_API_ENDPOINT": endpoint,
"LT_PROJECT_ID": projectId,
"LT_FUNCTIONS_SECRET": funcionsSecret
};
// Init SDK
const client = new sdk.Client();
const functions = new sdk.Functions(client);
client
.setEndpoint(endpoint) // Your API Endpoint
.setProject(projectId) // Your project ID
.setKey('33facd6c0d792e...359362efbc35d06bfaa'); // Your secret API key
functions.get(functionId)
.then(
func => {
// Does this function already exist?
if ((typeof (func) == 'object' && func['$id'] == functionId)) {
throw `Function '${functionId}' already exists. Cannot 'create'.\n\n`;
}
// Create the function
functions.create(functionId, name, execute, runtime, env_vars)
.then(
response => console.log(response),
error => console.error(`>>> ERROR! ${error}`)
);
}).catch(
error => console.error(`>>> ERROR! ${error}`)
);
As of Appwrite 0.13.0, an Appwrite Function must expose a function that accepts a request and response. To return data, you would use the response object and either call response.json() or response.send(). The request object has an env object with all function variables. Here is an example NodeJS Function:
module.exports = async (req, res) => {
const payload =
req.payload ||
'No payload provided. Add custom data when executing function.';
const secretKey =
req.env.SECRET_KEY ||
'SECRET_KEY environment variable not found. You can set it in Function settings.';
const randomNumber = Math.random();
const trigger = req.env.APPWRITE_FUNCTION_TRIGGER;
res.json({
message: 'Hello from Appwrite!',
payload,
secretKey,
randomNumber,
trigger,
});
};
In the example above, you can see req.env.SECRET_KEY being referenced. For more information, refer to the Appwrite Functions docs.
I am writing a backend API in node.js and need the functionality for users to be able to upload files with data and then calling stored procedures for inserting data into MySQL. I'm thinking of using fast-csv as parser, however I am struggling with how to set up the call to stored procedure in csv stream. the idea is something like this:
var fs = require("fs");
var csv = require("fast-csv");
var stream1 = fs.createReadStream("files/testCsvFile.csv");
csv
.fromStream(stream2, { headers: true })
.on("data", function(data) {
//CALL TO SP with params from "data"//
numlines++;
})
.on("end", function() {
console.log("done");
});
In other parts of application I have set up routes as follows:
auth.post("/verified", async (req, res) => {
var user = req.session.passwordless;
if (user) {
const rawCredentials = await admin.raw(getUserRoleCredentials(user));
const { user_end, role } = await normalizeCredentials(rawCredentials);
const user_data = { user_end, role };
res.send(user_data);
} else {
res.sendStatus(401);
}
});
..that is - routes are written in async/await way with queries (all are Stored Procedures called) being defined as Promises.. I would like to follow this pattern in upload/parse csv/call SP for every line function
This is doing the job for me - - can you please describe how to achive that with your framework - - I believe it should be done somehowe, I just need to configure it correctli
//use fast-csv to stream data from a file
csv
.fromPath(form.FileName, { headers: true })
.on("data", async data => {
const query = await queryBuilder({
schema,
routine,
parameters,
request
}); //here we prepare query for calling the SP with parameters from data
winston.info(query + JSON.stringify(data));
const rawResponse = await session.raw(query); //here the query gets executed
fileRows.push(data); // push each row - for testing only
})
.on("end", function() {
console.log(fileRows);
fs.unlinkSync(form.FileName); // remove temp file
//process "fileRows" and respond
res.end(JSON.stringify(fileRows)) // - for testing
});
As mentioned in the comment, I made my scramjet to handle such a use case with ease... Please correct me if I understood it wrong, but I understand you want to call the two await lines for every CSV row in the test.
If so, your code would look like this (updated to match your comment/answer):
var fs = require("fs");
var csv = require("fast-csv");
var stream1 = fs.createReadStream("files/testCsvFile.csv");
var {DataStream} = require("scramjet");
DataStream
// the following line will convert any stream to scramjet.DataStream
.from(csv.fromStream(stream2, { headers: true }))
// the next lines controls how many simultaneous operations are made
// I assumed 16, but if you're fine with 40 or you want 1 - go for it.
.setOptions({maxParallel: 16})
// the next line will call your async function and wait until it's completed
// and control the back-pressure of the stream
.do(async (data) => {
const query = await queryBuilder({
schema,
routine,
parameters,
request
}); //here we prepare query for calling the SP with parameters from data
winston.info(query + JSON.stringify(data));
const rawResponse = await session.raw(query); //here the query gets executed
return data; // push each row - for testing only)
})
// next line will run the stream until end and return a promise
.toArray()
.then(fileRows => {
console.log(fileRows);
fs.unlinkSync(form.FileName); // remove temp file
//process "fileRows" and respond
res.end(JSON.stringify(fileRows)); // - for testing
})
.catch(e => {
res.writeHead(500); // some error handling
res.end(e.message);
})
;
// you may want to put an await statement before this, or call then to check
// for errors, which I assume is your use case.
;
To answer your comment question - if you were to use an async function in the on("data") event - you would need to create an array of promises and await Promise.all of that array on stream end - but that would need to be done synchronously - so async function in an event handler won't do it.
In scramjet this happens under the hood, so you can use the function.
I'm attempting to parse a fairly large JSON file (~500Mb) in NodeJS. My implementation is based on the Async approach given in this answer:
var fileStream = require('fs');
var jsonObj;
fileStream.readFile('./data/exporttest2.json', fileCallback);
function fileCallback (err, data) {
return err ? (console.log(err), !1):(jsonObj = JSON.parse(data));
//Process JSON data here
}
That's all well and good, but I'm getting hit with the following error message:
buffer.js:495
throw new Error('"toString()" failed');
^
Error: "toString()" failed
at Buffer.toString (buffer.js:495:11)
at Object.parse (native)
at fileCallback (C:\Users\1700675\Research\Experiments\NodeJS\rf_EU.js:49:18)
at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:445:3)
I understand from this answer that this is caused by the maximum buffer length in the V8 engine set at 256Mb.
My question then is this, is there a way I can asynchronously read my JSON file in chunks that do not exceed the buffer length of 256Mb, without manually disseminating my JSON data into several files?
is there a way I can asynchronously read my JSON file in chunks that do not exceed the buffer length of 256Mb, without manually disseminating my JSON data into several files?
This is acommon problem and there are several modules than can help you with that:
https://www.npmjs.com/package/JSONStream
https://www.npmjs.com/package/stream-json
https://www.npmjs.com/package/json-stream
https://www.npmjs.com/package/json-parse-stream
https://www.npmjs.com/package/json-streams
https://www.npmjs.com/package/jsonparse
Example with JSONStream:
const JSONStream = require('JSONStream');
const fs = require('fs');
fs.createReadStrem('./data/exporttest2.json')
.pipe(JSONStream.parse('...'))...
See the docs for details of all of the arguments.
Try using streams:
let fs = require("fs");
let s = fs.createReadStream('./a.json');
let data = [];
s.on('data', function (chunk) {
data.push(chunk);
}).on('end', function () {
let json = Buffer.concat(data).toString();
console.log(JSON.parse(json));
});
I am trying to write a JSON object to a JSON file. The code executes without errors, but instead of the content of the object been written, all that gets written into the JSON file is:
[object Object]
This is the code that actually does the writing:
fs.writeFileSync('../data/phraseFreqs.json', output)
'output' is a JSON object, and the file already exists. Please let me know if more information is required.
You need to stringify the object.
fs.writeFileSync('../data/phraseFreqs.json', JSON.stringify(output));
I don't think you should use the synchronous approach, asynchronously writing data to a file is better also stringify the output if it's an object.
Note: If output is a string, then specify the encoding and remember the flag options as well.:
const fs = require('fs');
const content = JSON.stringify(output);
fs.writeFile('/tmp/phraseFreqs.json', content, 'utf8', function (err) {
if (err) {
return console.log(err);
}
console.log("The file was saved!");
});
Added Synchronous method of writing data to a file, but please consider your use case. Asynchronous vs synchronous execution, what does it really mean?
const fs = require('fs');
const content = JSON.stringify(output);
fs.writeFileSync('/tmp/phraseFreqs.json', content);
Make the json human readable by passing a third argument to stringify:
fs.writeFileSync('../data/phraseFreqs.json', JSON.stringify(output, null, 4));
When sending data to a web server, the data has to be a string (here). You can convert a JavaScript object into a string with JSON.stringify().
Here is a working example:
var fs = require('fs');
var originalNote = {
title: 'Meeting',
description: 'Meeting John Doe at 10:30 am'
};
var originalNoteString = JSON.stringify(originalNote);
fs.writeFileSync('notes.json', originalNoteString);
var noteString = fs.readFileSync('notes.json');
var note = JSON.parse(noteString);
console.log(`TITLE: ${note.title} DESCRIPTION: ${note.description}`);
Hope it could help.
Here's a variation, using the version of fs that uses promises:
const fs = require('fs');
await fs.promises.writeFile('../data/phraseFreqs.json', JSON.stringify(output)); // UTF-8 is default