Getting Error When I Start multiple select query inside loop TimeoutError: ResourceRequest timed out - mysql

Question

I'm using nodeJs Express Framework.
I'm using mysql database with sequelizejs library and using querying for retrieve data.
I am getting timeout error when I fired select query for almost 50,00,000 records.
I have done the server timeout but not worked.
I have done the pooling method in sequlizeJs But not worked.
function fetchNamesData(req, name) {
return new Promise((resolve, reject) => {
const names = req.app.locals.models.names_data;
names.findAll({
where: {
name: name
},
order: [['date', 'DESC']],
limit: 50
})
.then(function (dbRes) {
console.log(dbRes.length);
resolve(dbRes);
})
.catch(function (dbErr) {
console.log(dbErr);
return reject(dbErr);
});
});
}
allNames.forEach(element => {
//console.log(element.dataValues.name);
fetchNamesData(req, element.dataValues.name).then((dbRes) => {
//here I will have all the records
}).catch((dbErr) => { console.log(dbErr) });
var allNames = {having almost 7000 names}
now I iterate this obj and each names having 50 record in database
I want to get that all record like 50*7000 = 3,50,000.

Answer 1

What happens in your case is :
Looping through 7000 names and at same time hitting 7000 queries in mySql , and mysql will create queue for executing 7000 queries at same time cause load on machine. Either you can update your configuration to handle such load OR
Solution to this : Try to put some timeout b/w each queries , this way you will be able to fetch more records ,
allNames.forEach(element => {
setTimeout(() => { // <----------- HERE -------------
fetchNamesData(req, element.dataValues.name).then((dbRes) => {
//here I will have all the records
}).catch((dbErr) => {
console.log(dbErr)
});
},500); // <----------- HERE -------------
});

Answer 2

I have found the solution like
- Remove the unwanted console.log() and
- Also your hardware configuration depend upon it for timeout error.
- When query is firing do not start or run any other work it will lead to time out error[when there are multiple crud operation is going on].
- Also give index to table field when particular field is going to use in where clause.