Insert large text file in MySQL using Python - mysql

Question

I have a big text file which contains comma separeted data. But inside the file, there are multiple data chunks separeted by tags and .
I need to insert these chunks in different tables in mySQL.
I'm reading the file line by line.
When is found, I 'create if not exists' a new table and start inserting line by line.
But this is taking a lot of time as the file has around 1M rows.
Is there a better way to speedup the inserting process?
Below is the code I'm using
with open(file_name_with_path) as csv_file:
csv_reader = csv.reader(csv_file,delimiter=',')
line_count = 0
start_count = 0
new_file = False
file_end = False
ignore_line = False
record_list = []
file_name = ''
temp_data = []
for row in csv_reader:
ignore_line = False
if row[0].find('<START>') ==0:
file_name = row[1]
line_count = 0
new_file = True
ignore_line = True
if row[0].find('<END>') == 0:
file_end = True
record_list.append(file_name + '_' + str(line_count))
line_count = 0
ignore_line = True
print('End of a file.')
if new_file == True:
print('Start of a new file.')
new_file = False
if line_count == 0 and ignore_line is False and new_file is False:
line_count += 1
create_table(file_name,row)
temp_data.append(row)
elif ignore_line == False:
line_count += 1
add_data(file_name, row)
temp_data.append(row)
print(f'Processed the file - {",".join(record_list)}' )
Sample text file.
<START> File1
col1,col2,col3
1,A,B
2,C,D
<END>
<START> File2
col1,col2,col3,col4,col5
1,A,B,1,2
2,C,D,3,4
<END>

Answer 1

You didn't show us the code you're running.
You suggested that, optimistically, you're getting
app-level throughput of around 40 row / sec (50k / 1200),
which we all agree is "low".
You should be able to easily achieve
one or two orders of magnitude better performance.
It's not clear if you're using a local or remote instance.
Possibly you're doing 40 commits per second to local disk.
Or possibly you're managing 40 WAN round-trip messages per second.
What you want to be using is .executemany() with a batch size
of about ten thousand rows inserted per COMMIT.
That would be about a hundred COMMITs for your million rows.
Numerous examples are available on the web, e.g.
https://pynative.com/python-mysql-insert-data-into-database-table/