TypeORM bulk create with relations - mysql

Question

Is there a way to insert large amount of datas without blowing the JS heap memory ? I have a model which is Email as follow :
#Entity("email")
export class Email extends BaseEntity {
#PrimaryGeneratedColumn()
public id: number;
#ManyToOne((type) => Category, (cat) => cat.category, {nullable: false, cascade: ['insert']})
public category: Category;
#Column({type: "text", name: "email"})
public email: string;
}
and Category :
#Entity("category")
export class Category extends BaseEntity {
#PrimaryGeneratedColumn()
public id: number;
#Column({type: "text", name: "category"})
public category: string;
#OneToMany((type) => Email, (email) => email.category, {nullable: true})
public emails: Email[];
}
First problem I had is when I try to save {email: 'blabal#blalbah.com', category: 'default'} it says that Category must be an ID, but the thing is I want to add emails and create the category if it doesnt exist or asign the ID to the email if it exists. I did the following code :
public async bulkCreate(emails: Email[]): Promise<any> {
try {
const emailRepo = await getRepository(Email);
const categoryRepo = await getRepository(Category);
await Promise.all(emails.map(async (mail) => {
const cat = await categoryRepo.findOne({where: {category: mail.category}});
if (cat) {
// #ts-ignore
mail.category = cat.id;
} else {
const newCat = await categoryRepo.save(Object.assign(new Category(), mail));
// #ts-ignore
mail.category = newCat.id;
}
await emailRepo.save(mail);
}));
} catch (e) {
console.log(e);
throw new Error(e);
}
}
Worked for a few emails, but when I try to add even only 1,000 memory goes up to Like 4Gig and just crash.
What should I do? I'd like to add more than 1,000 emails at once.

Answer 1

I know it's little bit late, but solution for this use Bluebird Promise.map so you can define concurrency. instead executing in one run.