How do I create a Json (Circe) looking like this:
{
"items": [{
"field1": "somevalue",
"field2": "somevalue2"
},
{
"field1": "abc",
"field2": "123abc"
}]
}
val result = Json.fromFields(List("items" -> ???))
You can do so using Circe's built in list typeclasses for encoding JSON. This code will work:
import io.circe.{Encoder, Json}
import io.circe.syntax._
case class Field(field1: String, field2: String)
object Field {
implicit val encodeFoo: Encoder[Field] = new Encoder[Field] {
final def apply(a: Field): Json = Json.obj(
("field1", Json.fromString(a.field1)),
("field2", Json.fromString(a.field2))
)
}
}
class Encoding(items: List[Field]) {
def getJson: Json = {
Json.obj(
(
"items",
items.asJson
)
)
}
}
If we instantiate an "Encoding" class and call getJson it will give you back the desired JSON. It works because with circe all you need to do to encode a list is provide an encoder for whatever is inside the list. Thus, if we provide an encoder for Field it will encode it inside a list when we call asJson on it.
If we run this:
val items = new Encoding(List(Field("jf", "fj"), Field("jfl", "fjl")))
println(items.getJson)
we get:
{
"items" : [
{
"field1" : "jf",
"field2" : "fj"
},
{
"field1" : "jfl",
"field2" : "fjl"
}
]
}
Related
I have a dynamic json object generated with a certain format and I would like to manipulate that object to map it to another format in scala.
The problem is that the names of the fields are dynamic so "field1" and "field2" can be anything.
Is there any way to do it dynamically in scala?
The original object:
{
"field1": {
"value" "some value 1",
"description": "some test description",
...
},
"field2": {
"value" "some value 2",
"description": "some test description",
...
}
}
And I'd like to convert it to something like:
{
"field1": "some value 1",
"field2": "some value 2"
}
You can collect all keys and then check if the downField("value") exists
import io.circe._
import io.circe.literal.JsonStringContext
object CirceFieldsToMap {
def main(args: Array[String]): Unit = {
val json: Json =
json"""{
"field1": {
"foo" : "bar1",
"value" : "foobar1"
},
"field2": {
"foo" : "bar2",
"value" : "foobar2"
},
"field3": {
"foo" : "bar2"
}
}"""
implicit val decodeFoo = new Decoder[Map[String, Option[String]]] {
final def apply(c: HCursor): Decoder.Result[Map[String, Option[String]]] = {
val result = c.keys.get //// You should handle the .get properly ( if None it breaks)
.toList
.foldLeft(List.empty[(String, Option[String])]) { case (acc, key) =>
acc :+ (key, c.downField(key).downField("value").as[String].toOption)
}
Right(result.toMap)
}
}
val myField01 = json.as[Map[String, Option[String]]]
println(myField01) //Right(Map(field1 -> Some(foobar1), field2 -> Some(foobar2), field3 -> None))
}
}
I have the following data classes to parse JSON. I can parse it easily with the decodeFromString method. However, the Info classes could contain the List<Int> type from time to time along with the Int type so that both are included in a single JSON. How can I handle this variation in serialization?
#Serializable
data class Node (#SerialName("nodeContent") val nodeContent: List<Info>)
#Serializable
data class Info (#SerialName("info") val info: Int)
p.s. The closest question to mine is this one: Kotlinx Serialization, avoid crashes on other datatype. I wonder if there are other ways?
EDIT:
An example is given below.
"nodeContent": [
{
"info": {
"name": "1",
},
},
{
"info": [
{
"name": "1"
},
{
"name": "2"
},
],
},
{
"info": {
"name": "2",
},
}
]
Here is an approach with a custom serializer similar to the link you provided. The idea is to return a list with just a single element.
// Can delete these two lines, they are only for Kotlin scripts
#file:DependsOn("org.jetbrains.kotlinx:kotlinx-serialization-json:1.2.0")
#file:CompilerOptions("-Xplugin=/snap/kotlin/current/lib/kotlinx-serialization-compiler-plugin.jar")
import kotlinx.serialization.*
import kotlinx.serialization.json.*
import kotlinx.serialization.encoding.Decoder
#Serializable
data class Node (val nodeContent: List<Info>)
#Serializable(with = InfoSerializer::class)
data class Info (val info: List<Name>)
#Serializable
data class Name (val name: Int)
#Serializer(forClass = Info::class)
object InfoSerializer : KSerializer<Info> {
override fun deserialize(decoder: Decoder): Info {
val json = ((decoder as JsonDecoder).decodeJsonElement() as JsonObject)
return Info(parseInfo(json))
}
private fun parseInfo(json: JsonObject): List<Name> {
val info = json["info"] ?: return emptyList()
return try {
listOf(Json.decodeFromString<Name>(info.toString()))
} catch (e: Exception) {
(info as JsonArray).map { Json.decodeFromString<Name>(it.toString()) }
}
}
}
Usage:
val ss2 = """
{
"nodeContent": [
{
"info":
{"name": 1}
},
{
"info": [
{"name": 1},
{"name": 2}
]
},
{
"info":
{"name": 2}
}
]
}
"""
val h = Json.decodeFromString<Node>(ss2)
println(h)
Result:
Node(nodeContent=[Info(info=[Name(name=1)]), Info(info=[Name(name=1), Name(name=2)]), Info(info=[Name(name=2)])])
For example, here payload is optional and it has 3 variants:
How can I parse the json with types like option[either[A,B,C]] but to use abstract data type using things sealed trait or sum type?
Below is a minimal example with some boiler plate:
https://scalafiddle.io/sf/K6RUWqk/1
// Start writing your ScalaFiddle code here
val json =
"""[
{
"id": 1,
"payload" : "data"
},
{
"id": 2.1,
"payload" : {
"field1" : "field1",
"field2" : 5,
"field3" : true
}
},
{
"id": 2.2,
"payload" : {
"field1" : "field1",
}
},
{
"id": 3,
payload" : 4
},
{
"id":4,
"
}
]"""
final case class Data(field1: String, field2: Option[Int])
type Payload = Either[String, Data]
final case class Record(id: Int, payload: Option[Payload])
import io.circe.Decoder
import io.circe.generic.semiauto.deriveDecoder
implicit final val dataDecoder: Decoder[Data] = deriveDecoder
implicit final val payloadDecoder: Decoder[Payload] = Decoder[String] either Decoder[Data]
implicit final val recordDecoder: Decoder[Record] = deriveDecoder
val result = io.circe.parser.decode[List[Record]](json)
println(result)
Your code is almost fine, you have just syntax issues in your json and Record.id should be Double instead of Int - because it is how this field present in your json ("id": 2.1). Please, find fixed version below:
val json =
s"""[
{
"id": 1,
"payload" : "data"
},
{
"id": 2.1,
"payload" : {
"field1" : "field1",
"field2" : 5,
"field3" : true
}
},
{
"id": 2.2,
"payload" : {
"field1" : "field1"
}
},
{
"id": 3,
"payload" : 4
},
{
"id": 4
}
]"""
type Payload = Either[String, Data]
final case class Data(field1: String, field2: Option[Int])
final case class Record(id: Double, payload: Option[Payload]) // id is a Double in your json in some cases
import io.circe.Decoder
import io.circe.generic.semiauto.deriveDecoder
implicit val dataDecoder: Decoder[Data] = deriveDecoder
implicit val payloadDecoder: Decoder[Payload] = Decoder[String] either Decoder[Data]
implicit val recordDecoder: Decoder[Record] = deriveDecoder
val result = io.circe.parser.decode[List[Record]](json)
println(result)
Which produced in my case:
Right(List(Record(1.0,Some(Left(data))), Record(2.1,Some(Right(Data(field1,Some(5))))), Record(2.2,Some(Right(Data(field1,None)))), Record(3.0,Some(Left(4))), Record(4.0,None)))
UPDATE:
The more general approach would be to use so-called Sum Types or in simple words - general sealed trait with several different implementations. Please, see for more details next Circe doc page : https://circe.github.io/circe/codecs/adt.html
In your case it can be achieved something like this:
import cats.syntax.functor._
import io.circe.Decoder
import io.circe.generic.semiauto.deriveDecoder
sealed trait Payload
object Payload {
implicit val decoder: Decoder[Payload] = {
List[Decoder[Payload]](
Decoder[StringPayload].widen,
Decoder[IntPayload].widen,
Decoder[ObjectPayload].widen
).reduce(_ or _)
}
}
case class StringPayload(value: String) extends Payload
object StringPayload {
implicit val decoder: Decoder[StringPayload] = Decoder[String].map(StringPayload.apply)
}
case class IntPayload(value: Int) extends Payload
object IntPayload {
implicit val decoder: Decoder[IntPayload] = Decoder[Int].map(IntPayload.apply)
}
case class ObjectPayload(field1: String, field2: Option[Int]) extends Payload
object ObjectPayload {
implicit val decoder: Decoder[ObjectPayload] = deriveDecoder
}
final case class Record(id: Double, payload: Option[Payload])
object Record {
implicit val decoder: Decoder[Record] = deriveDecoder
}
def main(args: Array[String]): Unit = {
val json =
s"""[
{
"id": 1,
"payload" : "data"
},
{
"id": 2.1,
"payload" : {
"field1" : "field1",
"field2" : 5,
"field3" : true
}
},
{
"id": 2.2,
"payload" : {
"field1" : "field1"
}
},
{
"id": 3,
"payload" : "4"
},
{
"id": 4
}
]"""
val result = io.circe.parser.decode[List[Record]](json)
println(result)
}
which produced in my case next output:
Right(List(Record(1.0,Some(StringPayload(data))), Record(2.1,Some(ObjectPayload(field1,Some(5)))), Record(2.2,Some(ObjectPayload(field1,None))), Record(3.0,Some(StringPayload(4))), Record(4.0,None)))
Hope this helps!
I want to parse a file having content as json format.
From the file I want to extract few properties (name, DataType, Nullable) to create some column names dynamically.
I have gone through some examples but most of them are using case class but my problem is every time I will receive a file may have different content.
I tried to use the ujson library to parse the file but I am unable to understand how to use it properly.
object JsonTest {
def main(args: Array[String]): Unit = {
val source = scala.io.Source.fromFile("C:\\Users\\ktngme\\Desktop\\ass\\file.txt")
println(source)
val input = try source.mkString finally source.close()
println(input)
val data = ujson.read(input)
data("name") = data("name").str.reverse
val updated = data.render()
}
}
Content of the file example:
{
"Organization": {
"project": {
"name": "POC 4PL",
"description": "Implementation of orderbook"
},
"Entities": [
{
"name": "Shipments",
"Type": "Fact",
"Attributes": [
{
"name": "Shipment_Details",
"DataType": "StringType",
"Nullable": "true"
},
{
"name": "Shipment_ID",
"DataType": "StringType",
"Nullable": "true"
},
{
"name": "View_Cost",
"DataType": "StringType",
"Nullable": "true"
}
],
"ADLS_Location": "/mnt/mns/adls/raw/poc/orderbook/"
}
]
}
}
Expected output:
StructType(
Array(StructField("Shipment_Details",StringType,true),
StructField("Shipment_ID",DateType,true),
StructField("View_Cost",DateType,true)))
StructType needs to be added to the expected output programatically.
Try Using Playframework's Json utils - https://www.playframework.com/documentation/2.7.x/ScalaJson
Here's the solution to your issue-
\ Placed your json in text file
val fil_path = "C:\\TestData\\Config\\Conf.txt"
val conf_source = scala.io.Source.fromFile(fil_path)
lazy val json_str = try conf_source.mkString finally conf_source.close()
val conf_json: JsValue = Json.parse(json_str)
val all_entities: JsArray = (conf_json \ "Organization" \ "Entities").get.asInstanceOf[JsArray]
val shipments: JsValue = all_entities.value.filter(e => e.\("name").as[String] == "Shipments").head
val shipments_attributes: IndexedSeq[JsValue] = shipments.\("Attributes").get.asInstanceOf[JsArray].value
val shipments_schema: StructType = StructType(shipments_attributes.map(a => Tuple3(a.\("name").as[String], a.\("DataType").as[String], a.\("Nullable").as[String]))
.map(x => StructField(x._1, StrtoDatatype(x._2), x._3.toBoolean)))
shipments_schema.fields.foreach(println)
Output is -
StructField(Shipment_Details,StringType,true)
StructField(Shipment_ID,StringType,true)
StructField(View_Cost,StringType,true)
It depends if you want it to be completely dynamic or not, here are some options:
If you just want to read one field you can do:
import upickle.default._
val source = scala.io.Source.fromFile("C:\\Users\\ktngme\\Desktop\\ass\\file.txt")
val input = try source.mkString finally source.close()
val json = ujson.read(input)
println(json("Organization")("project")("name"))
the output will be: "POC 4PL"
If you just want just the Attributes to be with types, you can do:
import upickle.default.{macroRW, ReadWriter => RW}
import upickle.default._
val source = scala.io.Source.fromFile("C:\\Users\\ktngme\\Desktop\\ass\\file.txt")
val input = try source.mkString finally source.close()
val json = ujson.read(input)
val entitiesArray = json("Organization")("Entities")(0)("Attributes")
println(read[Seq[StructField]](entitiesArray))
case class StructField(name: String, DataType: String, Nullable: String)
object StructField{
implicit val rw: RW[StructField] = macroRW
}
the output will be: List(StructField(Shipment_Details,StringType,true), StructField(Shipment_ID,StringType,true), StructField(View_Cost,StringType,true))
another option, is to use a different library to do the class mapping. If you use Google Protobuf Struct and JsonFormat it can be 2-liner:
import com.google.protobuf.Struct
import com.google.protobuf.util.JsonFormat
val source = scala.io.Source.fromFile("C:\\Users\\ktngme\\Desktop\\ass\\file.txt")
val input = try source.mkString finally source.close()
JsonFormat.parser().merge(input, builder)
println(builder.build())
the output will be: fields { key: "Organization" value { struct_value { fields { key: "project" value { struct_value { fields { key: "name" value { string_value: "POC 4PL" } } fields { key: "description" value { string_value: "Implementation of orderbook" } } } } } fields { key: "Entities" value { list_value { values { struct_value { fields { key: "name" value { string_value: "Shipments" } }...
I have a json file , which has some keys and values. I need to parse the Json and print the keys and their values. For example, the json file is like below. I want to print this as Keys and values
{
"Parcer":[
{
"key":"0203",
"value":{
"Encryption":
{
"enabled":"yes",
"encryption_type":"base64",
"key":"334848484",
"return":"name"
}
}
},
{
"key":"0405",
"value":{
"Encryption":
{
"enabled":"yes",
"encryption_type":"base64",
"key":"334848484",
"return":"none"
},
"Parcer":[
{
"key":"0102",
"value":"humidity"
},
{
"key":"0304",
"value":{
"Encryption":{
"enabled":"yes",
"encryption_type":"SHA1",
"key":"1211212",
"return":"none"
}
}
}
]
}
}],
}```
The easiest way is to create a case class, like:
case class MyObj(header:String, value: Seq[Map[String, String]])
Then you just need to add one line for marshalling, like:
import play.api.libs.json._
object MyObj {
implicit val jsonFormat: OFormat[MyObj] = Json.format[MyObj]
}
Now you get a nice case class that you can work with:
val json =
Json.parse(
"""{
"header" : "header value",
"value" : [
{
"a" : "a_val",
"b" : "b_val",
"c" : "c_val"
},
{
"a" : "a_val",
"b" : "b_val",
"c" : "c_val"
}
]
}""")
Here an example how to retrieve all "a".
json.validate[MyObj] match {
case JsSuccess(myObj, _) =>
val allAs =myObj.value.flatMap(m => m.get("a").toSeq)
println(allAs) // >> Vector(a_val, a_val)
case e:JsError => // handle error
}
This gives you:
json.validate[MyObj] returns JsSuccess(MyObj(header value,Vector(Map(a -> a_val, b -> b_val, c -> c_val), Map(a -> a_val, b -> b_val, c -> c_val))),)
The println returns: Vector(a_val, a_val)
This is described here in the Documentation: JSON automated mapping