MyDumper / MyLoader unable to import large tables into azure MySql database - mysql

Question

I’m in the progress of migration a large mysql database to a “Azure-database voor MySQL Flexible Server”.
The database has a few tables that are larger than 1GB, the largest one being 200GB. All tables are InnoDB tables.
Because of the size of the tables, a normal mysql dump didn’t work, so as suggested here, I resorted to MyDumper/MyLoader: https://learn.microsoft.com/en-us/azure/mysql/concepts-migrate-mydumper-myloader
I dumped one of the large tables (a 31GB table) with the following command:
mydumper --database mySchema \
--tables-list my_large_table
--host database
--user root
--ask-password
--compress-protocol
--chunk-filesize 500
--verbose 3
--compress
--statement-size 104857600
I then copied the files over to a VM in the same region/zone as the Azure database and started the import with the following command:
myloader --directory mydumpdir \
--host dbname.mysql.database.azure.com \
--user my_admin \
--queries-per-transaction 100 \
--ask-password \
--verbose 3 \
--enable-binlog \
--threads 4 \
--overwrite-tables \
--compress-protocol
MyLoader seems to start loading and produced the following output:
** Message: 08:37:56.624: Server version reported as: 5.7.32-log
** Message: 08:37:56.674: Thread 1 restoring create database on `mySchema` from mySchema-schema-create.sql.gz
** Message: 08:37:56.711: Thread 2 restoring table `mySchema`.`my_large_table` from export-20220217-073020/mySchema.my_large_table-schema.sql.gz
** Message: 08:37:56.711: Dropping table or view (if exists) `mySchema`.`my_large_table`
** Message: 08:37:56.979: Creating table `mySchema`.`my_large_table` from export-20220217-073020/mySchema.my_large_table-schema.sql.gz
** Message: 08:37:57.348: Thread 2 restoring `mySchema`.`my_large_table` part 3 of 0 from mySchema.my_large_table.00003.sql.gz. Progress 1 of 26 .
** Message: 08:37:57.349: Thread 1 restoring `mySchema`.`my_large_table` part 0 of 0 from mySchema.my_large_table.00000.sql.gz. Progress 2 of 26 .
** Message: 08:37:57.349: Thread 4 restoring `mySchema`.`my_large_table` part 1 of 0 from mySchema.my_large_table.00001.sql.gz. Progress 3 of 26 .
** Message: 08:37:57.349: Thread 3 restoring `mySchema`.`my_large_table` part 2 of 0 from mySchema.my_large_table.00002.sql.gz. Progress 4 of 26 .
When I execute a "show full processlist" command on the Azure database, I see the 4 connected threads, but I see they are all sleeping, it seems like nothing is happening.
When I don't kill the command, it errors out after a long time:
** (myloader:31323): CRITICAL **: 17:07:27.642: Error occours between lines: 6 and 1888321 on file mySchema.my_large_table.00002.sql.gz: Lost connection to MySQL server during query
** (myloader:31323): CRITICAL **: 17:07:27.642: Error occours between lines: 6 and 1888161 on file mySchema.my_large_table.00001.sql.gz: MySQL server has gone away
** (myloader:31323): CRITICAL **: 17:07:27.642: Error occours between lines: 6 and 1888353 on file mySchema.my_large_table.00003.sql.gz: Lost connection to MySQL server during query
** (myloader:31323): CRITICAL **: 17:07:27.642: Error occours between lines: 6 and 1888284 on file mySchema.my_large_table.00000.sql.gz: MySQL server has gone away
After these errors, the table is still empty.
I tried a few different settings when dumping/loading, but to no avail:
start only 1 thread
make smaller chunks (100mb)
remove --compress-protocol
I also tried importing a smaller table (400MB in chunks of 100MB ), with exactly the same settings, and that did actually work.
I tried to import the tables into a mysql database on my local machine, and there I experienced exactly the same problem: the large table (31GB) import created 4 sleeping threads and didn't do anything, while the smaller table import (400MB in chunks of 100MB) did work.
So the problem doesn't seem to be related to the Azure database.
I now have no clue what the problem is, any ideas?

Answer 1

I had a similar problem, for me it ended up being the instance I was restoring into was too small, the server kept running out of memory. Try temporarily increasing the instance size to a much larger size, and once the data is imported shrink the size of the instance back down.