MySQL nodejs crash upon selecting data from big table - mysql

Question

I'm attempting to convert data from one database to the other but I'm getting some issues trying to get data from a big table, save it to an object and insert into another database. This is my code:
let sql;
let resultsToFetch = true;
while (resultsToFetch) {
sql = `SELECT X FROM Y LIMIT ${index}, 1000`;
DB1.query(sql, (err, result) => {
if (err) {
resultsToFetch = false;
throw err;
} else if (result.length == 0) {
resultsToFetch = false;
} else {
result.forEach(res => {
const obj = {
id: res.id,
name: res.name
};
sql = "INSERT INTO X SET ?";
DB2.query(sql, obj, (err, result) => {
if (err) throw err;
});
});
}
});
index += 1000;
}
I'm trying to use LIMIT so I'm not selecting all 6 million entries right away but I still get a Javascript heap out of memory error. I think I misunderstood something related to Node.js, but I'm not quite sure what it is. This is the error:
<--- Last few GCs --->
[11256:000002A5D2CBB600] 22031 ms: Mark-sweep 1418.5 (1482.0) -> 1418.5 (1451.5) MB, 918.3 / 0.0 ms last resort GC in old space requested
[11256:000002A5D2CBB600] 22947 ms: Mark-sweep 1418.5 (1451.5) -> 1418.5 (1451.5) MB, 915.2 / 0.0 ms last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 000000B356525529 <JSObject>
1: /* anonymous */ [\index.js:~1] [pc=00000042DA416732](this=000000C326B04AD1 <Object map = 0000027D35B023B9>,exports=000000C326B04AD1 <Object map = 0000027D35B023B9>,require=000000C326B04A89 <JSFunction require (sfi = 00000229888651E9)>,module=000000C326B04A39 <Module map = 0000027D35B44F69>,__filename=000002298886B769 <String[52]\
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::DecodeWrite
2: node_module_register
3: v8::internal::FatalProcessOutOfMemory
4: v8::internal::FatalProcessOutOfMemory
5: v8::internal::Factory::NewUninitializedFixedArray
6: v8::internal::WasmDebugInfo::SetupForTesting
7: v8::internal::interpreter::BytecodeArrayRandomIterator::UpdateOffsetFromIndex
8: 00000042DA2843C1
Edit: #Grégory NEUT
let query = DB1.query("SELECT * FROM X");
let index = 0;
query
.on("error", function(err) {
// Handle error, an 'end' event will be emitted after this as well
})
.on("fields", function(fields) {
// the field packets for the rows to follow
})
.on("result", function(row) {
// Pausing the connnection is useful if your processing involves I/O
DB1.pause();
const obj = {
id: row.id,
};
console.log(obj);
const sql = `INSERT INTO X SET ?`;
DB2.query(sql, obj, (err, result) => {
if (err) {
throw err;
}
DB1.resume();
});
console.log(index);
index++;
})
.on("end", function() {
// all rows have been received
});

Answer 1

I don't know how mysql driver is done in node.js but maybe it load everything and then limit the data. Or maybe 1000 entry are too much.
Anyway the soluce is to use streams
var query = connection.query('SELECT * FROM posts');
query
.on('error', function(err) {
// Handle error, an 'end' event will be emitted after this as well
})
.on('fields', function(fields) {
// the field packets for the rows to follow
})
.on('result', function(row) {
// Pausing the connnection is useful if your processing involves I/O
connection.pause();
processRow(row, function() {
connection.resume();
});
})
.on('end', function() {
// all rows have been received
});
So it will load in memory only the processed data at a time. Using it you will be sure that whatever the amount of data you have, you won't hit allocation failure.