How can I create a proper sql statement from this django model? - mysql

Question

I'm trying to insert data retrieved by scraping into the DB created by the following models.
However, I realized that using django's bulk_create or the external library based on it, bulk_create_or_update, is likely to make the logic too complex. (I felt that orm should be used for simple CRUD, etc.)
So I'm thinking of using Row SQL to save the data, for both maintainability and speed.
I'm not familiar with sql at all, so I'd like to get some advice from you guys.
What SQL code is preferable to this?
Each page to be scraped has multiple pieces of information, and there are multiple pages in total. I'd like to scrape all the pages first, add them to a dictionary, and then save them in a batch using sql, but I don't know the best way to do this.
from django.db import models
from django.forms import CharField
# Create your models here.
# province
class Prefecture(models.Model):
name=models.CharField("都道府県名",max_length=10)
# city
class City(models.Model):
prefecture = models.ForeignKey(Prefecture, on_delete=models.CASCADE, related_name='city')
name=models.CharField("市区町村名",max_length=10)
# seller
class Client(models.Model):
prefecture = models.ForeignKey(Prefecture, on_delete=models.CASCADE, related_name='client',null=True,blank=True)
city = models.ForeignKey(City, on_delete=models.CASCADE, related_name='client',null=True,blank=True)
department = models.CharField("部局",max_length=100)
# detail
class Project(models.Model):
name = models.CharField("案件名",max_length=100)
serial_no = models.CharField("案件番号",max_length=100,null=True,blank=True)
client = models.ForeignKey(Client, on_delete=models.CASCADE, related_name='project')
# etc...
# file
class AttachedFile(models.Model):
project = models.ForeignKey(Project, on_delete=models.CASCADE, related_name='attach_file')
name = models.CharField(max_length=100)
path = models.CharField(max_length=255)
# bid company
class Bidder(models.Model):
name = models.CharField("入札業者名",max_length=100)
prefecture = models.ForeignKey(Prefecture, on_delete=models.CASCADE, related_name='bidder',null=True,blank=True)
city = models.ForeignKey(City, on_delete=models.CASCADE, related_name='bidder',null=True,blank=True)
# etc...
# result
class BidResult(models.Model):
project = models.ForeignKey(Project, on_delete=models.CASCADE, related_name='bid_result')
bidder = models.ForeignKey(Bidder, on_delete=models.CASCADE, related_name='bid_result')

Answer 1

I don't think you will have a drastic performance boost by using raw SQL instead of orm. Also, Orm can be used for complex operations, and operations such as bulk update and bulk create are not complex and as fast as normal raw SQL. Things may go slow with ORM when you try to fetch records into memory and then do the operations but in your case, it is updated and created which can be done easily using Django ORM. As far as using a function doing update_or_create, using an external library won't affect your performance but using raw SQL for marginal speed gains may impact your code maintainability as you already told, You are not much familiar with raw sql.