Scala to read Kafka topic and write to MySQL table - mysql

Question

I wrote a Scala program, created a Kafka topic, show data in console. Now, I'm trying to modify the existing code so it can sink to a MySQ table.
db name: books
table name: authors
Can you help me to modify the code below so data from Kafka topic_json can be sent to a MySQL table?
df.selectExpr("CAST(id AS STRING) AS key", "to_json(struct(*)) AS value")
.writeStream
.format("jdbc")
.outputMode("overwrite")
.option("url", "jdbc:mysql://127.0.0.1:3306/books")
.option("books", "authors")
.start()
.awaitTermination()
ReadStream code for reference:
val df = spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "json_topic")
.option("startingOffsets", "earliest") // From starting
.load()
Appreciate all your help.

Answer 1

How about try to use “for each batch”,
and the essential logic to write is about writing a normal DF to MySQL.
See below for the plumbing part.
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#using-foreach-and-foreachbatch
If you struggle to get the logic of writing a normal data frame to MySQL, I can later paste my working example code which writes the stream to postgresql using for each batch here.
Code to write to postgresql, rushed POC , on Java. Pay attention to method writeToPostgresql. hopefully you can get the idea and get the scala version working. let me know if you need help on scala version.
import org.apache.spark.api.java.function.VoidFunction2;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SaveMode;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.streaming.OutputMode;
import java.util.HashMap;
import java.util.Map;
//todo: write a logging class and extends from it.
public class PostgresqlService {
// Transform and write batchDF.
static <T> void postgresSink(Dataset<T> dataset, Long batchId, String tableName) {
dataset.persist();
dataset.write().format("jdbc").option("dbtable", tableName).options(postgresqlOptions()).mode(SaveMode.Append).save();
dataset.write().format("console").mode(SaveMode.Overwrite).save(); //note: on prod do not do this type of stuff.
dataset.unpersist();
}
//TODO(optional): pass in as an option for things like checkpoint.
//Method to write the dataset into postgresql
public static <T> void writeToPostgresql(Dataset<T> dataset, OutputMode outputMode, String tableName) {
try {
dataset
.writeStream()
.option("checkpointLocation", "/tmp/spark-checkpoint/"+tableName) //path/to/HDFS/dir
.outputMode(outputMode)
.foreachBatch(
new VoidFunction2<Dataset<T>, Long>() {
public void call(Dataset<T> dataset, Long batchId) {
postgresSink(dataset, batchId, tableName);
}
}
)
// .trigger(Trigger.Once())
.start()
.awaitTermination();
} catch (Exception e) {
System.out.println(e.toString());
System.exit(1);
}
}
/**
* Spark-PostgreSQL connection properties.
*
* #return Map of String -> String
*/
static Map<String, String> postgresqlOptions() {
//TODO (optional): current is POC level. if i have time: read from config
Map<String, String> map = new HashMap<String, String>() {
{
put("user", "sparkstreaming"); // Database username
put("password", "password"); // Password
put("driver", "org.postgresql.Driver");
put("url", "jdbc:postgresql://localhost:5432/sparkstreaming");
}
};
return map;
}
}
`
when i called above method, i used `OutputMode.Update()`, like
` writeToPostgresql(transformedAggregatedPayload, OutputMode.Update(), "my-table-name");`