How to write a Json Transformer that converts
{"id" : [1,2], "name":["a","b"]} to idname:[ {id:1, name:"a"}, {id:2 , name:"b"}]
I am following this guide for implementing transfomration https://www.playframework.com/documentation/2.4.x/ScalaJsonTransformers
Writing the general case transformer can be difficult. Instead you can try reasoning through the standard Scala objects of Map and List. Here's a way to do it using map twice:
import play.api.libs.json._
val a = Json.parse("""{"id" : [1,2], "name":["a","b"]}""").as[Map[String, List[JsValue]]]
val b = a.map {case (k, v) => v.map(k -> _)}
.transpose
.map(_.toMap)
Json.toJson(b)
// play.api.libs.json.JsValue = [{"id":1,"name":"a"},{"id":2,"name":"b"}]
Related
The JSON output that I am looking for is
{[[1, 1.5, "String1"], [-2, 2.3, "String2"]]}
So I want to have an Array of Arrays and the inner array is storing different types.
How should I store my variables so I can create such JSON in Scala?
I thought of List of Tuples. However, all the available JSON libraries try to convert a Tuple to a map instead of an Array. I am using json4s library.
Here is a custom serializer for those inner arrays using json4s:
import org.json4s._
class MyTupleSerializer extends CustomSerializer[(Int, Double, String)](format => ({
case obj: JArray =>
implicit val formats: Formats = format
(obj(0).extract[Int], obj(1).extract[Double], obj(2).extract[String])
}, {
case (i: Int, d: Double, s: String) =>
JArray(List(JInt(i), JDouble(d), JString(s)))
}))
The custom serialiser converts JArray into a tuple and back again. This will be used wherever the Scala object being read or written has a value of the appropriate tuple type.
To test this against the sample input I have modified it to make it valid JSON by adding a field name:
{"data": [[1, 1.5, "String1"], [-2, 2.3, "String2"]]}
I have defined a container class to match this:
case class MyTupleData(data: Vector[(Int, Double, String)])
The name of the class is not relevant but the field name data must match the JSON field name. This uses Vector rather than Array because Array is really a Java type rather than a Scala type. You can use List if preferred.
import org.json4s.jackson.Serialization.{read, write}
case class MyTupleData(data: Vector[(Int, Double, String)])
object JsonTest extends App {
val data = """{"data": [[1, 1.5, "String1"], [-2, 2.3, "String2"]]}"""
implicit val formats: Formats = DefaultFormats + new MyTupleSerializer
val td: MyTupleData = read[MyTupleData](data)
println(td) // MyTupleData(Vector((1,1.5,String1), (-2,2.3,String2)))
println(write(td)) // {"data":[[1,1.5,"String1"],[-2,2.3,"String2"]]}
}
If you prefer to use a custom class for the data rather than a tuple, the code looks like this:
case class MyClass(i: Int, d: Double, s: String)
class MyClassSerializer extends CustomSerializer[MyClass](format => ({
case obj: JArray =>
implicit val formats: Formats = format
MyClass(obj(0).extract[Int], obj(1).extract[Double], obj(2).extract[String])
}, {
case MyClass(i, d, s) =>
JArray(List(JInt(i), JDouble(d), JString(s)))
}))
Use a List of List rather than List of Tuples.
an easy way to convert list of tuples to list of list is:
val listOfList: List[List[Any]] = listOfTuples.map(_.productIterator.toList)
I would use jackson, which is a java library and can deal with arbitrary datatypes inside collections of type Any/AnyRef, rather than trying to come up with a custom serializer in one of scala json libraries.
To convert scala List to java List use
import collection.JavaConverters._
So, in summary the end list would be:
val javaListOfList: java.util.List[java.util.List[Any]] = listOfTuples.map(_.productIterator.toList.asJava).asJava
Using this solution, you could have arbitrary length tuples in your list and it would work.
import com.fasterxml.jackson.databind.ObjectMapper
import collection.JavaConverters._
object TuplesCollectionToJson extends App {
val tuplesList = List(
(10, false, 43.6, "Text1"),
(84, true, 92.1, "Text2", 'X')
)
val javaList = tuplesList.map(_.productIterator.toList.asJava).asJava
val mapper = new ObjectMapper()
val json = mapper.writeValueAsString(javaList)
println(json)
}
Would produce:
[[10,false,43.6,"Text1"],[84,true,92.1,"Text2","X"]]
PS: Use this solution only when you absolutely have to work with variable types. If your tuple datatype is fixed, its better to create a json4s specific serializer/deserializer.
There are many JSON parsers in Kotlin like Forge, Gson, JSON, Jackson... But they deserialize the JSON to a data class, meaning it's needed to define a data class with the properties corresponding to the JSON, and this for every JSON which has a different structure.
But what if you don't want to define a data class for every JSON you could have to parse?
I'd like to have a parser which wouldn't use data classes, for example it could be something like:
val jsonstring = '{"a": "b", "c": {"d: "e"}}'
parse(jsonstring).get("c").get("d") // -> "e"
Just something that doesn't require me to write a data class like
data class DataClass (
val a: String,
val b: AnotherDataClass
)
data class AnotherDataClass (
val d: String
)
which is very heavy and not useful for my use case.
Does such a library exist? Thanks!
With kotlinx.serialization you can parse JSON String into a JsonElement:
val json: Map<String, JsonElement> = Json.parseToJsonElement(jsonstring).jsonObject
You can use JsonPath
val json = """{"a": "b", "c": {"d": "e"}}"""
val context = JsonPath.parse(json)
val str = context.read<String>("c.d")
println(str)
Output:
Result: e
I want to read a generic CSV file with headers but unknown column number into a typed structure. My question is kind of the same as Strongly typed access to csv in scala? but with the fact I would have no schema to pass to the parser...
Until now, I was using Jackson CSV mapper to read each row as a Map[String,String], and it was working well.
import com.fasterxml.jackson.module.scala.DefaultScalaModule
def genericStringIterator(input: InputStream): Iterator[Map[String, String]] = {
val mapper = new CsvMapper()
mapper.registerModule(DefaultScalaModule)
val schema = CsvSchema.emptySchema.withHeader
val iterator = mapper
.readerFor(classOf[Map[String, String]])
.`with`(schema)
.readValues[Map[String, String]](input)
iterator.asScala
}
Now, we need the field to be typed, so 4.2 would be a Double but "4.2" would still be a String.
We are using play-json everywhere in our project, and so I know JsValue already has a great type inference for generic stuff like that.
As paly-json it is based on Jackson too, I thought it would be great to have something like
import play.api.libs.json.jackson.PlayJsonModule
def genericStringIterator(input: InputStream): Iterator[JsValue] = {
val mapper = new CsvMapper()
mapper.registerModule(PlayJsonModule)
val schema = CsvSchema.emptySchema.withHeader
val iterator = mapper
.readerFor(classOf[JsValue])
.`with`(schema)
.readValues[JsValue](input)
iterator.asScala
}
But when I try the former code, I get an exception :
val iterator = CSV.genericAnyIterator(input(
"""foo,bar,baz
|"toto",42,43
|"tata",,45
| titi,87,88
|"tutu",,
|""".stripMargin))
iterator
.foreach { a =>
println(a)
}
java.lang.RuntimeException: We should be reading map, something got wrong
at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:165)
at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:128)
at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:123)
at com.fasterxml.jackson.databind.MappingIterator.nextValue(MappingIterator.java:277)
at com.fasterxml.jackson.databind.MappingIterator.next(MappingIterator.java:192)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:40)
at scala.collection.Iterator.foreach(Iterator.scala:929)
at scala.collection.Iterator.foreach$(Iterator.scala:929)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
at my.company.csv.CSVSpec$$anon$4.<init>(CSVSpec.scala:240)
Is there something I'm doing wrong ?
I don't care particulary to have a play-json JsValue in the end, any Json structure with generic typed field would be ok. Is there another lib I could use for that ? For what I found, all other libs are based on a mapping given to the CSV Reader in advance, and what is important for me is to be able to infer the type from the CSV.
Ok, I was lazy to want to find something working out of the box :)
In fact it was easy to implement it myself.
I went looking to lib in other languages that does this infering (PapaParse in JS, Pandas in python), and discovered that they were doing a test-and-retry with priority to guess the types.
So I implemented it myself, and it works fine !
Here it is:
def genericAnyIterator(input: InputStream): Iterator[JsValue] = {
// the result of former code, mapping to Map[String,String]
val strings = genericStringIterator(input)
val decimalRegExp: Regex = "(\\d*){1}(\\.\\d*){0,1}".r
val jsValues: Iterator[Map[String, JsValue]] = strings.map { m =>
m.mapValues {
case "" => None
case "false" | "FALSE" => Some(JsFalse)
case "true" | "TRUE" => Some(JsTrue)
case value#decimalRegExp(g1, g2) if !value.isEmpty => Some(JsNumber(value.toDouble))
case "null" | "NULL" => Some(JsNull)
case value#_ => Some(JsString(value))
}
.filter(_._2.isDefined)
.mapValues(_.get)
}
jsValues.map(map => JsObject(map.toSeq))
}
which does in a test
it should "read any csv in JsObject" in new WithInputStream {
val iterator = CSV.genericAnyIterator(input(
"""foo,bar,baz
|"toto",NaN,false
|"tata",,TRUE
|titi,87.79,88
|"tutu",,null
|"tete",5.,.5
|""".stripMargin))
val result: Seq[JsValue] = iterator.toSeq
result should be(Stream(
Json.obj("foo" -> "toto", "bar" -> "NaN", "baz" -> false)
, Json.obj("foo" -> "tata", "baz" -> true)
, Json.obj("foo" -> "titi", "bar" -> 87.79, "baz" -> 88)
, Json.obj("foo" -> "tutu", "baz" -> JsNull)
, Json.obj("foo" -> "tete", "bar" -> 5, "baz" -> 0.5)
) )
}
I have many very large json-objects that I return from Play Framework with Scala.
In most cases the user doesn't need all the data in the objects, only a few fields. So I want to pass in the paths I need (as query parameters), and return a subset of the json object.
I have looked at using JSON Transformers for this task.
Filter code
def filterByPaths(paths: List[JsPath], inputObject: JsObject) : JsObject = {
paths
.map(_.json.pick)
.map(inputObject.transform)
.filter(_.isSuccess)
.map { case JsSuccess(value, path) => (value, path) }
.foldLeft(Json.obj()) { (obj, jsValueAndPath) =>
val(jsValue, path) = jsValueAndPath
val transformer = __.json.update(path.json.put(jsValue))
obj.transform(transformer).get
}
}
Usage:
val input = Json.obj(
"field1" -> Json.obj(
"field2" -> "right result"
),
"field4" -> Json.obj(
"field5" -> "not included"
),
)
val result = filterByPaths(List(JsPath \ "field1" \ "field2"), input)
// {"field1":{"field2":"right result"}}
Problem
This code works fine for JsObjects. But I can't make it work if there are JsArrays in the strucure. I had hoped that my JsPath could contain an index to look up the field, but that's not the case. (Don't know why I assumed that, maybe my head was too far in the JavaScript-world)
So this would fail to return the first entry in the Array:
val input: JsObject = Json.parse("""
{
"arr1" : [{
"field1" : "value1"
}]
}
""").as[JsObject]
val result = filterByPaths(List(JsPath \ "arr1" \ "0"), input)
// {}
Question
My question is: How can I return a subset of a json structure that contains arrays?
Alternative solution
I have the data as a case class first, and I serialize it to Json, and then run filterByPaths on it. Having a Reader that only creates the json I need in the first place might be a better solution, but creating a Reader on the fly, with configuration from queryparams seamed a more difficult task, then just stripping down the json afterwards.
The example of the returning array element:
val input: JsValue = Json.parse("""
{
"arr1" : [{
"field1" : "value1"
}]
}
""")
val firstElement = (input \ "arr1" \ 0).get
val firstElementAnotherWay = input("arr1")(0)
More about this in the Play Framework documentation: https://www.playframework.com/documentation/2.6.x/ScalaJson
Update
It looks like you got the old issue RuntimeException: expected KeyPathNode. JsPath.json.put, JsPath.json.update can't past an object to a nesting array.
https://github.com/playframework/playframework/issues/943
https://github.com/playframework/play-json/issues/82
What you can do:
Use the JSZipper: https://github.com/mandubian/play-json-zipper
Create a script to update arrays "manually"
If you can afford it, strip array in a resulting object
Example of stripping array (point 3):
def filterByPaths(paths: List[JsPath], inputObject: JsObject) : JsObject = {
paths
.map(_.json.pick)
.map(inputObject.transform)
.filter(_.isSuccess)
.map { case JsSuccess(value, path) => (value, path)}
.foldLeft(Json.obj()) { (obj, jsValueAndPath) =>
val (jsValue, path) = jsValueAndPath
val arrayStrippedPath = JsPath(path.path.filter(n => !(n.toJsonString matches """\[\d+\]""")))
val transformer = __.json.update(arrayStrippedPath.json.put(jsValue))
obj.transform(transformer).get
}
}
val result = filterByPaths(List(JsPath \ "arr1" \ "0"), input)
// {"arr1":{"field1":"value1"}}
The example
The best to handle JSON objects is by using case classes and create implicit Reads and Writes, by that you can handle errors every fields directly. Don't make it complicated.
Don't use .get() much recommended to use .getOrElse() because scala is a type-safe programming language.
Don't just use any Libraries except you know the process behind it, much better to create your own parsing method with simplified solution to save memory.
I hope it will help you..
I am trying to implement a generic pattern with which to generate marshallers and unmarshallers for an Akka HTTP REST service using Argonaut, handling both entity and collection level requests and responses. I have no issues in implementing the entity level as such:
case class Foo(foo: String)
object Foo {
implicit val FooJsonCodec = CodecJson.derive[Foo]
implicit val EntityEncodeJson = FooJson.Encoder
implicit val EntityDecodeJson = FooJson.Decoder
}
I am running into issues attempting to provide encoders and decoders for the following:
[
{ "foo": "1" },
{ "foo": "2" }
]
I have attempted adding the following to my companion:
object Foo {
implicit val FooCollectionJsonCodec = CodecJson.derive[HashSet[Foo]]
}
However, I am receiving the following error:
Error:(33, 90) value jencode0L is not a member of object argonaut.EncodeJson
I see this method truly does not exist but is there any other generic method to generate my expected result. I'm strongly avoiding using an additional case class to describe the collection since I am using reflection heavily in my use case.
At this point, I'd even be fine with a manually constructed Encoder and Decoder, however, I've found no documentation on how to construct it with the expected structure.
Argonaut have predefined encoders and decoders for Scala's immutable lists, sets, streams and vectors. If your type is not supported explicitly, as in the case of java.util.HashSet, you can easily add EncodeJson and DecodeJson for the type:
import argonaut._, Argonaut._
import scala.collection.JavaConverters._
implicit def hashSetEncode[A](
implicit element: EncodeJson[A]
): EncodeJson[java.util.HashSet[A]] =
EncodeJson(set => EncodeJson.SetEncodeJson[A].apply(set.asScala.toSet))
implicit def hashSetDecode[A](
implicit element: DecodeJson[A]
): DecodeJson[java.util.HashSet[A]] =
DecodeJson(cursor => DecodeJson.SetDecodeJson[A]
.apply(cursor)
.map(set => new java.util.HashSet(set.asJava)))
// Usage:
val set = new java.util.HashSet[Int]
set.add(1)
set.add(3)
val jsonSet = set.asJson // [1, 3]
jsonSet.jdecode[java.util.HashSet[Int]] // DecodeResult(Right([1, 3]))
case class A(set: java.util.HashSet[Int])
implicit val codec = CodecJson.derive[A]
val a = A(set)
val jsonA = a.asJson // { "set": [1, 3] }
jsonA.jdecode[A] // DecodeResult(Right(A([1, 3])))
Sample is checked on Scala 2.12.1 and Argonaut 6.2-RC2, but as far as I know it shouldn't depend on some latest changes.
Approach like this works with any linear or unordered homogenous data structure that you want to represent as JSON array. Also, this is preferable to creating a CodecJson: latter can be inferred automatically from JsonEncode and JsonDecode, but not vice versa. This way, your set will serialize and deserialize both when used independently or within other data type, as shown in example.
I don't use Argonaut but use spray-json and suspect solution can be similar.
Have you tried something like this ?
implicit def HashSetJsonCodec[T : CodecJson] = CodecJson.derive[Set[T]]
if it doesn't work I'd probably try creating more verbose implicit function like
implicit def SetJsonCodec[T: CodecJson](implicit codec: CodecJson[T]): CodecJson[Set[T]] = {
CodecJson(
{
case value => JArray(value.map(codec.encode).toList)
},
c => c.as[JsonArray].flatMap {
case arr: Json.JsonArray =>
val items = arr.map(codec.Decoder.decodeJson)
items.find(_.isError) match {
case Some(error) => DecodeResult.fail[Set[T]](error.toString(), c.history)
case None => DecodeResult.ok[Set[T]](items.flatMap(_.value).toSet[T])
}
}
)
}
PS. I didn't test this but hopefully it leads you to the right direction :)