Dynamic replacement of Spark dataframe columns per row - json

Question

I have here 2 JSON strings per record.
The first JSON string is variable, indicating column(s) to be replaced in second JSON string.
E.g. the input.
import spark.implicits._
// Leaving timestamp aspect out and multiple occurrence of changes in same run, for simplicity.
val df = Seq((1, """{"b": "new", "c": "new"}""", """{"k": 1, "b": "old", "c": "old", "d": "old"}""" ),
(2, """{"b": "new", "d": "new"}""", """{"k": 2, "b": "old", "c": "old", "d": "old"}""" )).toDF("id", "chg", "before")
Like so:
+---+------------------------+--------------------------------------------+
|id |chg |before |
+---+------------------------+--------------------------------------------+
|1 |{"b": "new", "c": "new"}|{"k": 1, "b": "old", "c": "old", "d": "old"}|
|2 |{"b": "new", "d": "new"}|{"k": 2, "b": "old", "c": "old", "d": "old"}|
+---+------------------------+--------------------------------------------+
What I want is - but I cannot get it completed - is this output, thus on a per row basis".
+---+--------------------------------------------+
|id |after |
+---+------------------------+-------------------+
|1 |{"k": 1, "b": "new", "c": "new", "d": "old"}|
|2 |{"k": 2, "b": "new", "c": "old", "d": "new"}|
+---+------------------------+-------------------+
We could have an array in the JSON, that's fine as well.

Answer 1

You can go the strongly typed route, which gives you a lot of control over the exact transformations you want to have occur.
I made the following assumptions:
your before column contains all of the fields in each row
the chg column never contains the k field.
import spark.implicits._
import org.apache.spark.sql.Encoders
// Creating case classes with the schema of your json objects. We're making
// these to make use of strongly typed Datasets. Notice that the MyChgClass has
// each field as an Option: this will enable us to choose between "chg" and
// "before"
case class MyChgClass(b: Option[String], c: Option[String], d: Option[String])
case class MyFullClass(k: Int, b: String, c: String, d: String)
case class MyEndClass(id: Int, after: MyFullClass)
// Creating schemas for the from_json function
val chgSchema = Encoders.product[MyChgClass].schema
val beforeSchema = Encoders.product[MyFullClass].schema
// Your dataframe from the example
val df = Seq(
(1, """{"b": "new", "c": "new"}""", """{"k": 1, "b": "old", "c": "old", "d": "old"}""" ),
(2, """{"b": "new", "d": "new"}""", """{"k": 2, "b": "old", "c": "old", "d": "old"}""" )
).toDF("id", "chg", "before")
// Parsing the json string into our case classes and finishing by creating a
// strongly typed dataset with the .as[] method
val parsedDf = df
.withColumn("parsedChg",from_json(col("chg"), chgSchema))
.withColumn("parsedBefore",from_json(col("before"), beforeSchema))
.drop("chg")
.drop("before")
.as[(Int, MyChgClass, MyFullClass)]
// Mapping over our dataset with a lot of control of exactly what we want. Since
// the "chg" fields are options, we can use the getOrElse method to choose
// between either the "chg" field or the "before" field
val output = parsedDf.map{
case (id, chg, before) => {
MyEndClass(id, MyFullClass(
before.k,
chg.b.getOrElse(before.b),
chg.c.getOrElse(before.c),
chg.d.getOrElse(before.d)
))
}
}
output.show(false)
+---+------------------+
|id |after |
+---+------------------+
|1 |[1, new, new, old]|
|2 |[2, new, old, new]|
+---+------------------+
So there you have it, the output dataset contains the values you want. Since you asked for it, we can turn it into a json string again (Opinion incoming: this is less useful within Spark IMO, so unless you absolutely need to output this json somewhere I would not do this) by doing the following:
output.select($"id", to_json($"after")).show(false)
+---+-------------------------------------+
|id |structstojson(after) |
+---+-------------------------------------+
|1 |{"k":1,"b":"new","c":"new","d":"old"}|
|2 |{"k":2,"b":"new","c":"old","d":"new"}|
+---+-------------------------------------
Hope this helps!