Extract Options from potentially null JSON values using for expression - json

Question

I have a JSON document where some values can be null. Using for expressions in json4s, how I can yield None, instead of nothing?
The following will fail to yield when the value for either of the fields FormattedID or PlanEstimate is null.
val j: json4s.JValue = ...
for {
JObject(list) <- j
JField("FormattedID", JString(id)) <- list
JField("PlanEstimate", JDouble(points)) <- list
} yield (id, points)
For example:
import org.json4s._
import org.json4s.jackson.JsonMethods._
scala> parse("""{
| "FormattedID" : "the id",
| "PlanEstimate" : null
| }""")
res1: org.json4s.JValue = JObject(List((FormattedID,JString(the id)),
(PlanEstimate,JNull)))
scala> for {
| JObject(thing) <- res1
| JField("FormattedID", JString(id)) <- thing
| } yield id
res2: List[String] = List(the id)
scala> for {
| JObject(thing) <- res1
| JField("PlanEstimate", JDouble(points)) <- thing
| } yield points
res3: List[Double] = List()
// Ideally res3 should be List[Option[Double]] = List(None)

Answer 1

scala> object OptExtractors {
|
| // Define a custom extractor for using in for-comprehension.
| // It returns Some[Option[Double]], instead of Option[Double].
| object JDoubleOpt {
| def unapply(e: Any) = e match {
| case d: JDouble => Some(JDouble.unapply(d))
| case _ => Some(None)
| }
| }
| }
defined object OptExtractors
scala>
scala> val j = parse("""{
| "FormattedID" : "the id",
| "PlanEstimate" : null
| }""")
j: org.json4s.JValue = JObject(List((FormattedID,JString(the id)), (PlanEstimate,JNull)))
scala>
scala> import OptExtractors._
import OptExtractors._
scala>
scala> for {
| JObject(list) <- j
| JField("FormattedID", JString(id)) <- list
| JField("PlanEstimate", JDoubleOpt(points)) <- list
| } yield (id, points)
res1: List[(String, Option[Double])] = List((the id,None))

Answer 2

According to the documentation,
Any value can be optional. Field and value is completely removed when
it doesn't have a value.
scala> val json = ("name" -> "joe") ~ ("age" -> (None: Option[Int]))
scala> compact(render(json))
res4: String = {"name":"joe"}
Explaining why your for comprehension doesn't yield anything.
Of course, a null value is mapped to None internally.

Answer 3

The last command should look like:
for {
JObject(thing) <- res1
} yield thing.collectFirst{case JField("PlanEstimate", JDouble(points)) => points}
Or like
for {
JObject(thing) <- res1
points = thing.collectFirst{case JField("PlanEstimate", JDouble(p)) => p}
} yield points

Answer 4

What about this
for {
JObject(thing) <- res1
x = thing.find(_._1 == "PlanEstimate").flatMap(_._2.toOption.map(_.values))
} yield x