I have a scala app in which I'm using json4s to do some json manipulation. I have a few fields that I would like to convert into a new object.
For example I have the following:
"start_datetime":"2016-12-11T01:00:05+0000",
"type":"absolute",
"start":"5",
"type":"offset"
That would like to make into:
"time":[
{
"type":"absolute",
"start_datetime":"2016-12-11T01:00:05+0000"
},
{
"type":"offset",
"start":"10"
}
]
Any way I can do this using json4s?
The below snippet uses native json4s DSL
A Json object is formed by tuples chained together by method ~ and the Json Array is created by creating a Sequence object in Scala. Other primitive types like String, Number, Boolean are mapped to the corresponding types in scala
import org.json4s.native.JsonMethods._
import org.json4s.JsonDSL._
val json = "time" -> Seq(
("type" -> "absolute") ~ ("start_datetime" -> "2016-12-11T01:00:05+0000"),
("type" -> "offset") ~ ("start" -> "10")
)
scala> compact(render(json))
res3: String = {"time":[{"type":"absolute","start_datetime":"2016-12-11T01:00:05+0000"},{"type":"offset","start":"10"}]}
Related
Scala - Couldn't remove double quotes for "{}" braces while building Json
import scala.util.Random
import math.Ordered.orderingToOrdered
import math.Ordering.Implicits.infixOrderingOps
import play.api.libs.json._
import play.api.libs.json.Writes
import play.api.libs.json.Json.JsValueWrapper
val data1 = (1 to 2)
.map {r => Json.toJson(Map(
"name" -> Json.toJson(s"Perftest${Random.alphanumeric.take(6).mkString}"),
"domainId"->Json.toJson("343RDFDGF4RGGFG"),
"value" ->Json.toJson("{}")))}
val data2 = Json.toJson(data1)
println(data2)
Result :
[{"name":"PerftestpXI1ID","domainId":"343RDFDGF4RGGFG","value":"{}"},{"name":"PerftestHoZSQR","domainId":"343RDFDGF4RGGFG","value":"{}"}]
Expected :
"value":{}
[{"name":"PerftestpXI1ID","domainId":"343RDFDGF4RGGFG","value":{}},{"name":"PerftestHoZSQR","domainId":"343RDFDGF4RGGFG","value":{}}]
Please suggest a solution
You are giving it a String so it is creating a string in JSON. What you actually want is an empty dictionary, which is a Map in Scala:
val data1 = (1 to 2)
.map {r => Json.toJson(Map(
"name" -> Json.toJson(s"Perftest${Random.alphanumeric.take(6).mkString}"),
"domainId"->Json.toJson("343RDFDGF4RGGFG"),
"value" ->Json.toJson(Map.empty[String, String])))}
More generally you should create a case class for the data and create a custom Writes implementation for that class so that you don't have to call Json.toJson on every value.
Here is how to do the conversion using only a single Json.toJson call:
import play.api.libs.json.Json
case class MyData(name: String, domainId: String, value: Map[String,String])
implicit val fmt = Json.format[MyData]
val data1 = (1 to 2)
.map { r => new MyData(
s"Perftest${Random.alphanumeric.take(6).mkString}",
"343RDFDGF4RGGFG",
Map.empty
)
}
val data2 = Json.toJson(data1)
println(data2)
The value field can be a standard type such as Boolean or Double. It could also be another case class to create nested JSON as long as there is a similar Json.format line for the new type.
More complex JSON can be generated by using a custom Writes (and Reads) implementation as described in the documentation.
I have a few json arrays
[{"key":"country","value":"aaa"},{"key":"region","value":"a"},{"key":"city","value":"a1"}]
[{"key":"city","value":"b"},{"key":"street","value":"1"}]
I need to extract city and street value into different columns.
Using get_json_object($"address", "$[2].value").as("city") to get element by it's number doesn't work because arrays can miss some fields.
Instead I decided to turn this array into a map of key -> value pairs, but have trouble doing it. So far I only managed to get an array of arrays.
val schema = ArrayType(StructType(Array(
StructField("key", StringType),
StructField("value", StringType)
)))
from_json($"address", schema)
Returns
[[country, aaa],[region, a],[city, a1]]
[[city, b],[street, 1]]
I'm not sure where to go from here.
val schema = ArrayType(MapType(StringType, StringType))
Fails with
cannot resolve 'jsontostructs(`address`)' due to data type mismatch: Input schema array<map<string,string>> must be a struct or an array of structs.;;
I'm using spark 2.2
Using UDF we can handle this easily. In the below code I have created a map using a UDF. I hope this will suffice the need
import org.apache.spark.sql.types._
import org.apache.spark.sql.functions._
val df1 = spark.read.format("text").load("file path")
val schema = ArrayType(StructType(Array(
StructField("key", StringType),
StructField("value", StringType)
)))
val arrayToMap = udf[Map[String, String], Seq[Row]] {
array => array.map { case Row(key: String, value: String) => (key, value) }.toMap
}
val dfJSON = df1.withColumn("jsonData",from_json(col("value"),schema))
.select("jsonData").withColumn("address", arrayToMap(col("jsonData")))
.withColumn("city", when(col("address.city").isNotNull, col("address.city")).otherwise(lit(""))).withColumn("street", when(col("address.street").isNotNull, col("address.street")).otherwise(lit("")))
dfJSON.printSchema()
dfJSON.show(false)
The JSON output that I am looking for is
{[[1, 1.5, "String1"], [-2, 2.3, "String2"]]}
So I want to have an Array of Arrays and the inner array is storing different types.
How should I store my variables so I can create such JSON in Scala?
I thought of List of Tuples. However, all the available JSON libraries try to convert a Tuple to a map instead of an Array. I am using json4s library.
Here is a custom serializer for those inner arrays using json4s:
import org.json4s._
class MyTupleSerializer extends CustomSerializer[(Int, Double, String)](format => ({
case obj: JArray =>
implicit val formats: Formats = format
(obj(0).extract[Int], obj(1).extract[Double], obj(2).extract[String])
}, {
case (i: Int, d: Double, s: String) =>
JArray(List(JInt(i), JDouble(d), JString(s)))
}))
The custom serialiser converts JArray into a tuple and back again. This will be used wherever the Scala object being read or written has a value of the appropriate tuple type.
To test this against the sample input I have modified it to make it valid JSON by adding a field name:
{"data": [[1, 1.5, "String1"], [-2, 2.3, "String2"]]}
I have defined a container class to match this:
case class MyTupleData(data: Vector[(Int, Double, String)])
The name of the class is not relevant but the field name data must match the JSON field name. This uses Vector rather than Array because Array is really a Java type rather than a Scala type. You can use List if preferred.
import org.json4s.jackson.Serialization.{read, write}
case class MyTupleData(data: Vector[(Int, Double, String)])
object JsonTest extends App {
val data = """{"data": [[1, 1.5, "String1"], [-2, 2.3, "String2"]]}"""
implicit val formats: Formats = DefaultFormats + new MyTupleSerializer
val td: MyTupleData = read[MyTupleData](data)
println(td) // MyTupleData(Vector((1,1.5,String1), (-2,2.3,String2)))
println(write(td)) // {"data":[[1,1.5,"String1"],[-2,2.3,"String2"]]}
}
If you prefer to use a custom class for the data rather than a tuple, the code looks like this:
case class MyClass(i: Int, d: Double, s: String)
class MyClassSerializer extends CustomSerializer[MyClass](format => ({
case obj: JArray =>
implicit val formats: Formats = format
MyClass(obj(0).extract[Int], obj(1).extract[Double], obj(2).extract[String])
}, {
case MyClass(i, d, s) =>
JArray(List(JInt(i), JDouble(d), JString(s)))
}))
Use a List of List rather than List of Tuples.
an easy way to convert list of tuples to list of list is:
val listOfList: List[List[Any]] = listOfTuples.map(_.productIterator.toList)
I would use jackson, which is a java library and can deal with arbitrary datatypes inside collections of type Any/AnyRef, rather than trying to come up with a custom serializer in one of scala json libraries.
To convert scala List to java List use
import collection.JavaConverters._
So, in summary the end list would be:
val javaListOfList: java.util.List[java.util.List[Any]] = listOfTuples.map(_.productIterator.toList.asJava).asJava
Using this solution, you could have arbitrary length tuples in your list and it would work.
import com.fasterxml.jackson.databind.ObjectMapper
import collection.JavaConverters._
object TuplesCollectionToJson extends App {
val tuplesList = List(
(10, false, 43.6, "Text1"),
(84, true, 92.1, "Text2", 'X')
)
val javaList = tuplesList.map(_.productIterator.toList.asJava).asJava
val mapper = new ObjectMapper()
val json = mapper.writeValueAsString(javaList)
println(json)
}
Would produce:
[[10,false,43.6,"Text1"],[84,true,92.1,"Text2","X"]]
PS: Use this solution only when you absolutely have to work with variable types. If your tuple datatype is fixed, its better to create a json4s specific serializer/deserializer.
I use json4s native,with a json string like this
val myjson = """
{
"normative":"C",
"prefixType":{
"cod":["smallint", "int", "varchar(5)"],
"des":["varchar", "string"],
"fec":["timestamp"],
"hms":["timestamp"],
"tim":["timestamp"],
"imp":["decimal","Float", "Double"]
},
"fixcolname":{
"aud_usuario":"varchar(8)",
"aud_fec":"timestamp",
"aud_tim":"timestamp"
},
"symSep":"_",
"maxLength":26
}"""
And a case class
case class colVerify(prefixType: Map[String, Array[String]], fixcolname: Map[String, String], symSep: String, maxLength: Int)
and I want to extract it from the json String
val t = parse(myjson)
implicit val formats = DefaultFormats
val myvfy = t.extract[colVerify]
then got an error like this
Exception in thread "main" org.json4s.package$MappingException: Parsed JSON values do not match with class constructor
args=Map(des -> [Ljava.lang.String;#d7b1517, fec -> [Ljava.lang.String;#16c0663d, tim -> [Ljava.lang.String;#23223dd8, hms -> [Ljava.lang.String;#4ec6a292, imp -> [Ljava.lang.String;#1b40d5f0, cod -> [Ljava.lang.String;#ea4a92b),Map(aud_usuario -> varchar(8), aud_fec -> timestamp, aud_tim -> timestamp),_,26
arg types=scala.collection.immutable.HashMap$HashTrieMap,scala.collection.immutable.Map$Map3,java.lang.String,java.lang.Integer
constructor=public colVerify(scala.collection.mutable.Map,scala.collection.mutable.Map,java.lang.String,int)
Seems like it has problem with the type of Map, but how can I convert it implicitly?
The problem is that the maps in your case class are mutable maps, is this intentional or did you accidently import collection.mutable.Map?
If you really want the mutable maps, you could implement a custom Serializer as described here: https://github.com/json4s/json4s#serializing-non-supported-types
My first idea to add another constructor with immutable maps in case class doesn't seem to work reliably.
I am attempting to produce JSON in a Scala app using json4s. Fairly straight forward, Here's some sample value I put together to test it in my Scalatra app:
import org.json4s._
import org.json4s.JsonDSL._
object JsonStub {
val getPeople =
("people" ->
("person_id" -> 5) ~
("test_count" -> 5))
}
In my controller, I simply have:
import org.json4s._
import org.json4s.JsonDSL._
import org.json4s.{DefaultFormats, Formats}
class FooController(mongoDb: MongoClient)(implicit val swagger: Swagger) extends ApiStack with NativeJsonSupport with SwaggerSupport {
get ("/people", operation(getPeople)) {
JsonStub.getPeople
}
}
The output I'm seeing in the browser however, is the following:
{
"_1": "people",
"_2": {
"person_id": 5,
"test_count": 5
}
}
Any clue where the _1 and _2 keys are coming from? I was expecting this output instead:
{
"people":{
"person_id": 5,
"test_count": 5
}
}
What you're seeing in the output is a reflectively serialized tuple, which has fields _1 and _2. This is because the return type that the compiler has inferred for JsonStub.getPeople is Tuple2[String, JObject].
The json4s DSL uses implicit conversions to turn values like the tuple into a JValue. But, if you don't tell the compiler you wanted a JValue, it won't apply the conversion.
Ideally, this would result in a compile error, because you tried to produce JSON from something that isn't the right type. Unfortunately, because your web framework assumes you want to fall back to reflection-based serialization, it means there is another way to turn the tuple into JSON, which isn't what you wanted.
If you explicitly tell the compiler that you want a JValue and not a Tuple2, the DSL's implicit conversion will be applied in the correct place.
val getPeople: JValue =
("people" ->
("person_id" -> 5) ~
("test_count" -> 5))