Parse JSON with unknown attributes names into a Case Class - json

I have the following JSON file to be parsed into a case class:
{
"root": {
"nodes": [{
"id": "1",
"attributes": {
"name": "Node 1",
"size": "3"
}
},
{
"id": "2",
"attributes": {
"value": "4",
"name": "Node 2"
}
}
]
}
}
The problem is that the attributes could have any value inside it: name, size, value, anything ...
At this moment I have defined my case classes:
case class Attributes(
name: String,
size: String,
value: Sting
)
case class Nodes(
id: String,
attributes: Attributes
)
case class Root(
nodes: List[Nodes]
)
case class R00tJsonObject(
root: Root
)
Whats is the best way to deal with this scenario when I can receive any attribute ?
Currently I am using Json4s to handle son files.
Thanks!

Your attributes are arbitrarily many and differently named, but it seems you can store them in a Map[String, String] (at least, if those examples are anything to go by). In this case, using circe-parser (https://circe.github.io/circe/parsing.html), you could simply use code along these lines in order to convert your JSON directly into a simple case-class:
import io.circe._, io.circe.parser._
import io.circe.generic.semiauto._
case class Node(id: String, attributes: Map[String,String])
case class Root(nodes: List[Node])
implicit val nodeDecoder: Decoder[Node] = deriveDecoder[Node]
implicit val nodeEncoder: Encoder[Node] = deriveEncoder[Node]
implicit val rootDecoder: Decoder[Root] = deriveDecoder[Root]
implicit val rootEncoder: Encoder[Root] = deriveEncoder[Root]
def myParse(jsonString: String) = {
val res = parse(jsonString) match {
case Right(json) => {
val cursor = json.hcursor
cursor.get[Root]("root")
}
case _ => Left("Wrong JSON!")
}
println(res)
}
This snippet will print
Right(Root(List(Node(1,Map(name -> Node 1, size -> 3)), Node(2,Map(value -> 4, name -> Node 2)))))
on the console, for the JSON, you've given. (Assuming, the solution doesn't have to be in Json4s.)

Related

Read a JSON and parse the contents and display the result in Scala

I am a newbie in scala. i try to read a json and parse it using json4s library.
Already written the case class and code for reading and parsing the sample json file.
I need to iterate the json and print the details of each attribute's.
Case Class
case class VehicleDetails(
name: String,
manufacturer: String,
model: String,
year: String,
color: String,
seat: Int,
variants: Seq[String],
engine: Int,
dealer: Map[String, String],
franchise: Map[String, String])
The json data and the code i tried is given below.
import org.json4s._
import org.json4s.jackson.JsonMethods._
import org.json4s.DefaultFormats
object CarDetails extends App {
val json = parse("""{
"vehicle_details": [
{
"CAR": {
"name": "Brezza",
"manufacturer": "Maruti",
"model": "LDI",
"year": 2019,
"color": "Blue",
"seat": 5,
"engine": 1,
"cylinder": 4,
"variants": [
"LDI",
"LDI(O)",
"VDI",
"VDI(O)",
"ZDI",
"ZDI+"
],
"dealer": {
"kerala": "Popular"
},
"franchise": {
"ekm": "popular_ekm"
}
},
"SUV": {
"name": "Scross",
"manufacturer": "Maruti",
"model": "LDI",
"year": 2020,
"color": "Blue",
"variants": [
"LDI",
"VDI",
"ZDI"
],
"dealer": {
"kerala": "Popular"
},
"franchise": {
"ekm": "popular_ekm"
}
}
}
]
}""")
implicit val formats = DefaultFormats
val definition = json.extract[VehicleDetails.Definition]
val elements = (json \\ "vehicle_details").children
This pretty close, just a few small changes needed.
First, create a class that encapsulates all the JSON data:
case class AllDetails(vehicle_details: List[Map[String, VehicleDetails]])
Then just extract that class from the json
implicit val formats = DefaultFormats
val details = Extraction.extract[AllDetails](json)
With this particular JSON the seat and engine fields are not present in all the records so you need to modify VehicleDetails to make these Option values:
case class VehicleDetails(
name: String,
manufacturer: String,
model: String,
year: String,
color: String,
seat: Option[Int],
variants: Seq[String],
engine: Option[Int],
dealer: Map[String, String],
franchise: Map[String, String]
)
[ Other values that might be omitted in other records will also need to be Option values ]
You can unpick the result using standard Scala methods. For example
res.vehicle_details.headOption.foreach { cars =>
val typeNames = cars.keys.mkString(", ")
println(s"Car types: $typeNames")
cars.foreach { case (car, details) =>
println(s"Car type: $car")
println(s"\tName: ${details.name}")
val variants = details.variants.mkString("[", ", ", "]")
println(s"\tVariants: $variants")
}
}
To get back to the raw JSON, use Serialization:
import org.json4s.jackson.Serialization
val newJson = Serialization.write(res)
println(newJson)

Transforming JSON with Circe

Suppose I have some JSON data like this:
{
"data": {
"title": "example input",
"someBoolean": false,
"innerData": {
"innerString": "input inner string",
"innerBoolean": true,
"innerCollection": [1,2,3,4,5]
},
"collection": [6,7,8,9,0]
}
}
And I want to flatten it a bit and transform or remove some fields, to get the following result:
{
"data": {
"ttl": "example input",
"bool": false,
"collection": [6,7,8,9,0],
"innerCollection": [1,2,3,4,5]
}
}
How can I do this with Circe?
(Note that I'm asking this as a FAQ since similar questions often come up in the Circe Gitter channel. This specific example is from a question asked there yesterday.)
I've sometimes said that Circe is primarily a library for encoding and decoding JSON, not for transforming JSON values, and in general I'd recommend mapping to Scala types and then defining relationships between those (as Andriy Plokhotnyuk suggests here), but for many cases writing transformations with cursors works just fine, and in my view this kind of thing is one of them.
Here's how I'd implement this transformation:
import io.circe.{DecodingFailure, Json, JsonObject}
import io.circe.syntax._
def transform(in: Json): Either[DecodingFailure, Json] = {
val someBoolean = in.hcursor.downField("data").downField("someBoolean")
val innerData = someBoolean.delete.downField("innerData")
for {
boolean <- someBoolean.as[Json]
collection <- innerData.get[Json]("innerCollection")
obj <- innerData.delete.up.as[JsonObject]
} yield Json.fromJsonObject(
obj.add("boolean", boolean).add("collection", collection)
)
}
And then:
val Right(json) = io.circe.jawn.parse(
"""{
"data": {
"title": "example input",
"someBoolean": false,
"innerData": {
"innerString": "input inner string",
"innerBoolean": true,
"innerCollection": [1,2,3]
},
"collection": [6,7,8]
}
}"""
)
And:
scala> transform(json)
res1: Either[io.circe.DecodingFailure,io.circe.Json] =
Right({
"data" : {
"title" : "example input",
"collection" : [
6,
7,
8
]
},
"boolean" : false,
"collection" : [
1,
2,
3
]
})
If you look at it the right way, our transform method kind of resembles a decoder, and we can actually write it as one (although I'd definitely recommend not making it implicit):
import io.circe.{Decoder, Json, JsonObject}
import io.circe.syntax._
val transformData: Decoder[Json] = { c =>
val someBoolean = c.downField("data").downField("someBoolean")
val innerData = someBoolean.delete.downField("innerData")
(
innerData.delete.up.as[JsonObject],
someBoolean.as[Json],
innerData.get[Json]("innerCollection")
).mapN(_.add("boolean", _).add("collection", _)).map(Json.fromJsonObject)
}
This can be convenient in some situations where you want to perform the transformation as part of a pipeline that expects a decoder:
scala> io.circe.jawn.decode(myJsonString)(transformData)
res2: Either[io.circe.Error,io.circe.Json] =
Right({
"data" : {
"title" : "example input",
"collection" : [ ...
This is also potentially confusing, though, and I've thought about adding some kind of Transformation type to Circe that would encapsulate transformations like this without questionably repurposing the Decoder type class.
One nice thing about both the transform method and this decoder is that if the input data doesn't have the expected shape, the resulting error will include a history that points to the problem.

Parsing a file having content as Json format in Scala

I want to parse a file having content as json format.
From the file I want to extract few properties (name, DataType, Nullable) to create some column names dynamically.
I have gone through some examples but most of them are using case class but my problem is every time I will receive a file may have different content.
I tried to use the ujson library to parse the file but I am unable to understand how to use it properly.
object JsonTest {
def main(args: Array[String]): Unit = {
val source = scala.io.Source.fromFile("C:\\Users\\ktngme\\Desktop\\ass\\file.txt")
println(source)
val input = try source.mkString finally source.close()
println(input)
val data = ujson.read(input)
data("name") = data("name").str.reverse
val updated = data.render()
}
}
Content of the file example:
{
"Organization": {
"project": {
"name": "POC 4PL",
"description": "Implementation of orderbook"
},
"Entities": [
{
"name": "Shipments",
"Type": "Fact",
"Attributes": [
{
"name": "Shipment_Details",
"DataType": "StringType",
"Nullable": "true"
},
{
"name": "Shipment_ID",
"DataType": "StringType",
"Nullable": "true"
},
{
"name": "View_Cost",
"DataType": "StringType",
"Nullable": "true"
}
],
"ADLS_Location": "/mnt/mns/adls/raw/poc/orderbook/"
}
]
}
}
Expected output:
StructType(
Array(StructField("Shipment_Details",StringType,true),
StructField("Shipment_ID",DateType,true),
StructField("View_Cost",DateType,true)))
StructType needs to be added to the expected output programatically.
Try Using Playframework's Json utils - https://www.playframework.com/documentation/2.7.x/ScalaJson
Here's the solution to your issue-
\ Placed your json in text file
val fil_path = "C:\\TestData\\Config\\Conf.txt"
val conf_source = scala.io.Source.fromFile(fil_path)
lazy val json_str = try conf_source.mkString finally conf_source.close()
val conf_json: JsValue = Json.parse(json_str)
val all_entities: JsArray = (conf_json \ "Organization" \ "Entities").get.asInstanceOf[JsArray]
val shipments: JsValue = all_entities.value.filter(e => e.\("name").as[String] == "Shipments").head
val shipments_attributes: IndexedSeq[JsValue] = shipments.\("Attributes").get.asInstanceOf[JsArray].value
val shipments_schema: StructType = StructType(shipments_attributes.map(a => Tuple3(a.\("name").as[String], a.\("DataType").as[String], a.\("Nullable").as[String]))
.map(x => StructField(x._1, StrtoDatatype(x._2), x._3.toBoolean)))
shipments_schema.fields.foreach(println)
Output is -
StructField(Shipment_Details,StringType,true)
StructField(Shipment_ID,StringType,true)
StructField(View_Cost,StringType,true)
It depends if you want it to be completely dynamic or not, here are some options:
If you just want to read one field you can do:
import upickle.default._
val source = scala.io.Source.fromFile("C:\\Users\\ktngme\\Desktop\\ass\\file.txt")
val input = try source.mkString finally source.close()
val json = ujson.read(input)
println(json("Organization")("project")("name"))
the output will be: "POC 4PL"
If you just want just the Attributes to be with types, you can do:
import upickle.default.{macroRW, ReadWriter => RW}
import upickle.default._
val source = scala.io.Source.fromFile("C:\\Users\\ktngme\\Desktop\\ass\\file.txt")
val input = try source.mkString finally source.close()
val json = ujson.read(input)
val entitiesArray = json("Organization")("Entities")(0)("Attributes")
println(read[Seq[StructField]](entitiesArray))
case class StructField(name: String, DataType: String, Nullable: String)
object StructField{
implicit val rw: RW[StructField] = macroRW
}
the output will be: List(StructField(Shipment_Details,StringType,true), StructField(Shipment_ID,StringType,true), StructField(View_Cost,StringType,true))
another option, is to use a different library to do the class mapping. If you use Google Protobuf Struct and JsonFormat it can be 2-liner:
import com.google.protobuf.Struct
import com.google.protobuf.util.JsonFormat
val source = scala.io.Source.fromFile("C:\\Users\\ktngme\\Desktop\\ass\\file.txt")
val input = try source.mkString finally source.close()
JsonFormat.parser().merge(input, builder)
println(builder.build())
the output will be: fields { key: "Organization" value { struct_value { fields { key: "project" value { struct_value { fields { key: "name" value { string_value: "POC 4PL" } } fields { key: "description" value { string_value: "Implementation of orderbook" } } } } } fields { key: "Entities" value { list_value { values { struct_value { fields { key: "name" value { string_value: "Shipments" } }...

Scala play JSON, lookup and match defined field holding null value

I have the following Json block that I have returned as a JsObject
{
"first_block": [
{
"name": "demo",
"description": "first demo description"
}
],
"second_block": [
{
"name": "second_demo",
"description": "second demo description",
"nested_second": [
{
"name": "bob",
"value": null
},
{
"name": "john",
"value": null
}
]
}
]
}
From this, I want to return a list of all the possible values I could have in the second block, nested array for name and value. so with the example above
List([bob,null],[john,null]) or something along those lines.
The issue I am having is with the value section understanding null values. I've tried to match against it and return a string "null" but I can't get it to match on Null values.
What would be the best way for me to return back the name and values in the nested_second array.
I've tried using case classes and readAsNullable with no luck, and my latest attempt has gone along these lines:
val secondBlock = (jsObj \ "second_block").as[List[JsValue]]
secondBlock.foreach(nested_block => {
val nestedBlock = (nested_block \ "nested_second").as[List[JsValue]]
nestedBlock.foreach(value => {
val name = (value \ "name").as[String] //always a string
var convertedValue = ""
val replacement_value = value \ "value"
replacement_value match {
case JsDefined(null) => convertedValue = "null"
case _ => convertedValue = replacement_value.as[String]
}
println(name)
println(convertedValue)
})
}
)
It seems convertedValue returns as 'JsDefined(null)' regardless and I'm sure the way I'm doing it is horrifically bad.
Replace JsDefined(null) with JsDefined(JsNull).
You probably got confused, because println(JsDefined(JsNull)) prints as JsDefined(null). But that is not, how null value of a JSON field is represented. null is represented as case object JsNull. This is just a good API design, where possible cases are represented with a hierarchy of classes:
With play-json I use always case-classes!
I simplified your problem to the essence:
import play.api.libs.json._
val jsonStr = """[
{
"name": "bob",
"value": null
},
{
"name": "john",
"value": "aValue"
},
{
"name": "john",
"value": null
}
]"""
Define a case class
case class Element(name: String, value: Option[String])
Add a formatter in the companion object:
object Element {
implicit val jsonFormat: Format[Element] = Json.format[Element]
}
An use validate:
Json.parse(jsonStr).validate[Seq[Element]] match {
case JsSuccess(elems, _) => println(elems)
case other => println(s"Handle exception $other")
}
This returns: List(Element(bob,None), Element(john,Some(aValue)), Element(john,None))
Now you can do whatever you want with the values.

Argonaut: decoding a polymorphic array

The JSON object for which I'm trying to write a DecodeJson[T] contains an array of different "types" (meaning the JSON structure of its elements is varying). The only common feature is the type field which can be used to distinguish between the types. All other fields are different. Example:
{
...,
array: [
{ type: "a", a1: ..., a2: ...},
{ type: "b", b1: ...},
{ type: "c", c1: ..., c2: ..., c3: ...},
{ type: "a", a1: ..., a2: ...},
...
],
...
}
Using argonaut, is it possible to map the JSON array to a Scala Seq[Element] where Element is a supertype of suitable case classes of type ElementA, ElementB and so on?
I did the same thing with play-json and it was quite easy (basically a Reads[Element] that evaluates the type field and accordingly forwards to more specific Reads). However, I couldn't find a way to do this with argonaut.
edit: example
Scala types (I wish to use):
case class Container(id: Int, events: List[Event])
sealed trait Event
case class Birthday(name: String, age: Int) extends Event
case class Appointment(start: Long, participants: List[String]) extends Event
case class ... extends Event
JSON instance (not under my control):
{
"id":1674,
"events": {
"data": [
{
"type": "birthday",
"name": "Jones Clinton",
"age": 34
},
{
"type": "appointment",
"start": 1675156665555,
"participants": [
"John Doe",
"Jane Doe",
"Foo Bar"
]
}
]
}
}
You can create a small function to help you build a decoder that handles this format.
See below for an example.
import argonaut._, Argonaut._
def decodeByType[T](encoders: (String, DecodeJson[_ <: T])*) = {
val encMap = encoders.toMap
def decoder(h: CursorHistory, s: String) =
encMap.get(s).fold(DecodeResult.fail[DecodeJson[_ <: T]](s"Unknown type: $s", h))(d => DecodeResult.ok(d))
DecodeJson[T] { c: HCursor =>
val tf = c.downField("type")
for {
tv <- tf.as[String]
dec <- decoder(tf.history, tv)
data <- dec(c).map[T](identity)
} yield data
}
}
case class Container(id: Int, events: ContainerData)
case class ContainerData(data: List[Event])
sealed trait Event
case class Birthday(name: String, age: Int) extends Event
case class Appointment(start: Long, participants: List[String]) extends Event
implicit val eventDecoder: DecodeJson[Event] = decodeByType[Event](
"birthday" -> DecodeJson.derive[Birthday],
"appointment" -> DecodeJson.derive[Appointment]
)
implicit val containerDataDecoder: DecodeJson[ContainerData] = DecodeJson.derive[ContainerData]
implicit val containerDecoder: DecodeJson[Container] = DecodeJson.derive[Container]
val goodJsonStr =
"""
{
"id":1674,
"events": {
"data": [
{
"type": "birthday",
"name": "Jones Clinton",
"age": 34
},
{
"type": "appointment",
"start": 1675156665555,
"participants": [
"John Doe",
"Jane Doe",
"Foo Bar"
]
}
]
}
}
"""
def main(args: Array[String]) = {
println(goodJsonStr.decode[Container])
// \/-(Container(1674,ContainerData(List(Birthday(Jones Clinton,34), Appointment(1675156665555,List(John Doe, Jane Doe, Foo Bar))))))
}