How to diagnose cause of slow JDBC write operations while running PySpark job on AWS Glue - mysql

Question

I've been stuck on this one for a couple of days so any help is greatly appreciated. I have a Spark ETL job running on AWS Glue and am having a hard time optimizing write performance.
Summary of the job:
A large dataset is being read from S3 (~5 million records)
A smaller dataset is being read from an RDS MySQL instance (hundreds of records)
The task at hand calls for a pairwise comparison of each record from both datasets. I went with a cross join (cartesian product) which itself seems to work fine. I know that this is an expensive operation but I haven't been able to figure out how to avoid expanding the dataset in that manor in order to do this pairwise comparison. It's just the nature of the problem. The good news is that all of the core transformation work is done on a row by row basis. There are no groupbys or aggregate functions required. There is 1 numerical filter that is executed after the cross join and subsequent computations. If I use .show() as an action to make all of these operations run, it takes around 40 minutes. Not bad.
The problem arises when I go to write this data to MySQL using the built-in Spark JDBC connector as such:
` joined_df.write.format("jdbc") \
.option("url", url_write) \
.option("batchsize", 100) \
.option("dbtable","{0}.{1}".format(database, table)) \
.option("user", user).option("password", password).mode("append").save()`
After 4 hours, not a single record is written.
When I run the job over a smaller sample of data, the writes go slowly but they do occur so database connectivity isn't the issue. I have also tried modifying the connection string to set useServerPrepStmts=false and rewriteBatchedStatements=true and using a small batch size of 100.
I am using 10 DPUs with the G1X worker type. This provides 36 cores in total. I used the 3X rule of thumb to pick 108 as the number of partitions which I apply to the larger dataset before converting the DynamicFrame into a Dataframe as such:
df1 = glue_context.create_dynamic_frame.from_catalog(
database = glue_database,
table_name = "test_tbl1",
push_down_predicate = "(upload_date >= CAST('{0}' AS TIMESTAMP))".format(lookback_date)
).repartition(108).toDF()`
I know that 10 DPUs is relatively light on compute but I have a feeling there is something else is going on here since no records are being written after hours. I have also tried using various number of partitions. I set up Spark UI to get some additional metrics. I am including the metrics for my latest attempt that timed out after 4 hours with zero records written. I would have posed a screenshot if my ranking allowed it (Sorry that this is so ugly). My question is, what should I be focused on here that may help get me to the answer of why the writes are hanging? Any wisdom would be greatly appreciated.
Executor ID Address Status RDD Blocks Storage Memory Disk Used Cores Active Tasks Failed Tasks Complete Tasks Total Tasks Task Time (GC Time) Input Shuffle Read Shuffle Write
driver 172.31.33.169:37095 Active 0 0.0 B / 5.8 GiB 0.0 B 0 0 0 0 0 0.0 ms (0.0 ms) 0.0 B 0.0 B 0.0 B
1 172.31.45.233:45821 Active 0 0.0 B / 5.8 GiB 0.0 B 4 3 0 4 7 1.5 min (4 s) 91.1 MiB 0.0 B 306.3 MiB
2 172.31.47.118:44467 Active 0 0.0 B / 5.8 GiB 0.0 B 4 4 0 5 9 1.2 min (3 s) 82.1 MiB 0.0 B 281.3 MiB
3 172.31.42.73:45151 Active 0 0.0 B / 5.8 GiB 0.0 B 4 3 0 4 7 1.4 min (4 s) 89.7 MiB 0.0 B 303.4 MiB
4 172.31.43.123:34465 Active 0 0.0 B / 5.8 GiB 0.0 B 4 4 0 4 8 1.3 min (5 s) 79.5 MiB 0.0 B 274.1 MiB
5 172.31.40.44:46117 Active 0 0.0 B / 5.8 GiB 0.0 B 4 3 0 4 7 1.5 min (5 s) 89.1 MiB 0.0 B 304.6 MiB
6 172.31.36.20:42763 Active 0 0.0 B / 5.8 GiB 0.0 B 4 4 0 4 8 1.1 min (3 s) 21.9 MiB 0.0 B 74.3 MiB
7 172.31.32.240:45533 Active 0 0.0 B / 5.8 GiB 0.0 B 4 4 0 7 11 1.3 min (4 s) 91.6 MiB 59 B 312.4 MiB
8 172.31.36.49:34471 Active 0 0.0 B / 5.8 GiB 0.0 B 4 3 0 7 10 1.5 min (5 s) 101.6 MiB 0.0 B 347.4 MiB
9 172.31.43.177:40385 Active 0 0.0 B / 5.8 GiB 0.0 B 4 3 0 5 8 1.4 min (5 s) 85.7 MiB 0.0 B 294.1 MiB