Spark Streaming MQTT - Apply schema on dataset - json

Question

I'm working on DataBricks (Spark 2.0.1-db1 (Scala 2.11)) and I am trying to use Spark Streaming functions. I am using the libraries :
- spark-sql-streaming-mqtt_2.11-2.1.0-SNAPSHOT.jar (see here : http://bahir.apache.org/docs/spark/current/spark-sql-streaming-mqtt/)
The following command gives me a dataset :
val lines = spark.readStream
.format("org.apache.bahir.sql.streaming.mqtt.MQTTStreamSourceProvider")
.option("clientId", "sparkTest")
.option("brokerUrl", "tcp://xxx.xxx.xxx.xxx:xxx")
.option("topic", "/Name/data")
.option("localStorage", "dbfs:/models/mqttPersist")
.option("cleanSession", "true")
.load().as[(String, Timestamp)]
with this printSchema :
root
|-- value : string (nullable : true)
|-- timestamp : timestamp (nullable : true)
And I would like to apply a schema on the "value" column of my dataset. you can see my json schema as folowing.
root
|-- id : string (nullable = true)
|-- DateTime : timestamp (nullable = true)
|-- label : double (nullable = true)
Is it possible to parse directly my json in the stream to obtain something like that :
root
|-- value : struct (nullable : true)
|-- id : string (nullable = true)
|-- DateTime : timestamp (nullable = true)
|-- label : double (nullable = true)
|-- timestamp : timestamp (nullable : true)
For now, I don't see any way to parse the json from a mqtt and any help would be very great.
Thanks in advance.

Answer 1

I had this same exact problem today! I used json4s and Jackson to parse the json.
How I am getting the streaming DataSet (pretty much the same as what you have):
val lines = spark.readStream
.format("org.apache.bahir.sql.streaming.mqtt.MQTTStreamSourceProvider")
.option("topic", topic)
.option("brokerUrl",brokerUrl)
.load().as[(String,Timestamp)]
I defined the schema using a case class:
case class DeviceData(devicename: String, time: Long, metric: String, value: Long, unit: String)
Parsing out the JSON column using org.json4s.jackson.JsonMethods.parse:
val ds = lines.map {
row =>
implicit val format = DefaultFormats
parse(row._1).extract[DeviceData]
}
Outputting the results:
val query = ds.writeStream
.format("console")
.option("truncate", false)
.start()
The results:
+----------+-------------+-----------+-----+----+
|devicename|time |metric |value|unit|
+----------+-------------+-----------+-----+----+
|dht11_4 |1486656575772|temperature|9 |C |
|dht11_4 |1486656575772|humidity |36 |% |
+----------+-------------+-----------+-----+----+
I am kind of disappointed I cannot come up with a solution that uses Sparks native json parsing. Instead we must rely on Jackson. You can use spark native json parsing if you are reading a file as a stream. As so:
val lines = spark.readStream
.....
.json("./path/to/file").as[(String,Timestamp)]
But for MQTT we cannot do this.