I need to create a Json with 2 elements. The First element is a List and the second element is simple key-value pair.
My output looks as follows:
"{
"tables":[
{"table": "sn: 2134"},
{"table": "sn: 5676"},
{"table": "sn: 4564"},
],
"paid": 219
}"
In the example , the first element is tables which is List of table. The second element is paid.
I tried it using play.api.libs.json lib , but stuck while adding second element.
My code looks as follows:
case class Input(table:String){
override def toString = s""""table" : "sn: $table""""
}
implicit val userFormat = Json.format[Input]
val inputsSeq = Seq(Input(table1),Input(table2),Input(table3))
val users = Json.obj("tables" -> inputsSeq)
println(users)
This code print Json as :
"{
"tables":[
{"table": "sn: 2134"},
{"table": "sn: 5676"},
{"table": "sn: 4564"},
]
}
I am not sure, how to add the second element in this json. any suggestion how to
resolve this.
Json.obj accepts multiple pairs of (String, JsValueWrapper) as its arguments:
object Json {
...
def obj(fields: (String, JsValueWrapper)*): JsObject = JsObject(fields.map(f => (f._1, f._2.asInstanceOf[JsValueWrapperImpl].field)))
...
}
So you can add both elements like this:
val users = Json.obj("tables" -> inputsSeq, "paid" -> 219)
Related
Suppose you have a bunch of data whose rows look like this:
{
'key': [
{'key1': 'value11', 'key2': 'value21'},
{'key1': 'value12', 'key2': 'value22'}
]
}
I would like to read this into a Spark Dataset. One way to do it is as follows:
case class ObjOfLists(k1: List[String], k2: List[String])
case class Data(k: ObjOfLists)
Then you can do:
sparkSession.read.json(pathToData).select(
struct($"key.key1" as "k1", $"key.key2" as "k2") as "k"
)
.as[Data]
This works fine, but it kind of butchers the data a little bit; after all in the data 'key' points to a list of objects rather than an object of lists. In other words, what I really want is:
case class Obj(k1: String, k2: String)
case class DataOfList(k: List[Obj])
My question: is there some other syntax I can put in select which allows the resulting Dataframe to be converted to a Dataset[DataOfList]?
I tried using the same select syntax as above, and got:
Exception in thread "main" org.apache.spark.sql.AnalysisException: need an array field but got struct<k1:array<string>,k2:array<string>>;
So I also tried:
sparkSession.read.json(pathToData).select(
array(struct($"key.key1" as "k1", $"key.key2" as "k2")) as "k"
)
.as[DataOfList]
This compiles and runs, but the data looks like this:
DataOfList(List(Obj(org.apache.spark.sql.catalyst.expressions.UnsafeArrayData#bb2a5516,org.apache.spark.sql.catalyst.expressions.UnsafeArrayData#bec5e4a7)))
Any other ideas?
Just recast data to reflect expected names:
case class Obj(k1: String, k2: String)
case class DataOfList(k: Seq[Obj])
val text = Seq("""{
"key": [
{"key1": "value11", "key2": "value21"},
{"key1": "value12", "key2": "value22"}
]
}""").toDS
val df = spark.read.json(text)
df
.select($"key".cast("array<struct<k1:string,k2:string>>").as("k"))
.as[DataOfList]
.first
DataOfList(List(Obj(value11,value21), Obj(value12,value22)))
With extraneous objects you define schema on read:
val textExtended = Seq("""{
"key": [
{"key0": "value01", "key1": "value11", "key2": "value21"},
{"key1": "value12", "key2": "value22", "key3": "value32"}
]
}""").toDS
val schemaSubset = StructType(Seq(StructField("key", ArrayType(StructType(Seq(
StructField("key1", StringType),
StructField("key2", StringType))))
)))
val df = spark.read.schema(schemaSubset).json(textExtended)
and proceed as before.
I've wrote a program which process JSON objects. Now I want to verify if I've missed something.
Is there an JSON-example of all allowed JSON structure combinations? Something like this:
{
"key1" : "value",
"key2" : 1,
"key3" : {"key1" : "value"},
"key4" : [
[
"string1",
"string2"
],
[
1,
2
],
...
],
"key5" : true,
"key6" : false,
"key7" : null,
...
}
As you can see at http://json.org/ on the right hand side the grammar of JSON isn't quite difficult, but I've got several exceptions because I've forgotten to handles some structure combinations which are possible. E.g. inside an array there can be "string, number, object, array, true, false, null" but my program couldn't handle arrays inside an array until I ran into an exception. So everything was fine until I got this valid JSON object with arrays inside an array.
I want to test my program with a JSON object (which I'm looking for). After this test I want to be feel certain that my program handle every possible valid JSON structure on earth without an exception.
I don't need nesting in depth 5 or so. I only need something in nested depth 2 or max 3. With all base types which nested all allowed base types, inside this base type.
Have you thought of escaped characters and objects within an object?
{
"key1" : {
"key1" : "value",
"key2" : [
"String1",
"String2"
],
},
"key2" : "\"This is a quote\"",
"key3" : "This contains an escaped slash: \\",
"key4" : "This contains accent charachters: \u00eb \u00ef",
}
Note: \u00eb and \u00ef are resp. charachters ë and ï
Choose a programming language that support json.
Try to load your json, on fail the exception's message is descriptive.
Example:
Python:
import json, sys;
json.loads(open(sys.argv[1]).read())
Generate:
import random, json, os, string
def json_null(depth = 0):
return None
def json_int(depth = 0):
return random.randint(-999, 999)
def json_float(depth = 0):
return random.uniform(-999, 999)
def json_string(depth = 0):
return ''.join(random.sample(string.printable, random.randrange(10, 40)))
def json_bool(depth = 0):
return random.randint(0, 1) == 1
def json_list(depth):
lst = []
if depth:
for i in range(random.randrange(8)):
lst.append(gen_json(random.randrange(depth)))
return lst
def json_object(depth):
obj = {}
if depth:
for i in range(random.randrange(8)):
obj[json_string()] = gen_json(random.randrange(depth))
return obj
def gen_json(depth = 8):
if depth:
return random.choice([json_list, json_object])(depth)
else:
return random.choice([json_null, json_int, json_float, json_string, json_bool])(depth)
print(json.dumps(gen_json(), indent = 2))
I can't understand, how can i add optional property with json transformer.
I want merge two json objects (list and calendars) without or with dynamic list of properties (for example without owner):
calendar1 = {id:1, name: "first", description:"my first calendar", owner: 1}
calendar2 = {id:2, name: "second", owner: 1}
list = [{id: 1, settings: []}, {id: 2, settings: []}]
and result must be
{calendars:
[
{id:1, name: "first", description:"my first calendar", settings: []},
{id:2, name: "second", settings: []}
]
}
I'll assume the following json trees
val calendar1 = Json.parse("""{"id":1, "name": "first", "description":"my first calendar", "owner": 1}""")
val calendar2 = Json.parse("""{"id":2, "name": "second", "owner": 1}""")
You need to add settings to each calendar, then remove the owner if it exists.
Putting a value in branch settings is explained in the documentation
val addSettings = __.json.update((__ \ "settings").json.put(Json.arr()))
Dropping the owner is also explained
val removeOwner = (__ \ "owner").json.prune
Now you can define the transformer to be applied to each of your calendar object
val transformer = addSettings andThen removeOwner
With that in place there are multiple options depending on how your data is actually modeled. If you have a Seq of calendars as in
val calendars = Seq(calendar1, calendar2)
you can do
val normalizedCalendars = calendars.map(_.transform(transformer))
This gives you a Seq[JsResult[JsObject]] which you want to transform into a JsResult[Seq[JsObject]].
I am pretty sure there is a way to do it using play's functional syntax (see play.api.libs.functional and play.api.libs.functional.syntax) but this part of play is not well documented and I haven't gotten around to studying Applicatives yet even though I have a feel for what they do.
Instead, I rely on the following code inspired by scala's Future#sequence
def sequence[A, M[X] <: TraversableOnce[X]](in: M[JsResult[A]])(implicit cbf: CanBuildFrom[M[JsResult[A]], A, M[A]]): JsResult[M[A]] = {
val empty: JsResult[mutable.Builder[A, M[A]]] = JsSuccess(cbf(in))
in.foldLeft(empty) {(jracc,jrel) => (jracc,jrel) match {
case (JsSuccess(builder,_), JsSuccess(a,p)) =>JsSuccess(builder+=a, p)
case (ra#JsError(builderErrors), re#JsError(errors)) =>JsError.merge(ra, re)
case (JsSuccess(_,_), re#JsError(errors)) => re
case (ra#JsError(builderErrors), JsSuccess(_,_)) => ra
}} map (_.result())
}
With that you can write :
val calendarArray = sequence(normalizedCalendars).map(v=>Json.obj("calendars"->JsArray(v)))
which will give you a JsResult[JsObject]. As long as your original calendars are indeed JsObjects you will get a JsSuccess. You can verify the output structure with :
calendarArray.foreach(println)
which returns :
{"calendars":[{"id":1,"name":"first","description":"my first calendar","settings":[]},{"id":2,"name":"second","settings":[]}]}
which is the same as what you asked modulo some whitespace
{
"calendars":[
{"id":1,"name":"first","description":"my first calendar","settings":[]},
{"id":2,"name":"second","settings":[]}
]
}
Start with:
scala> case class Calendar(id:Int,name:String,description:Option[String],owner:Int)
defined class Calendar
scala> case class CalendarRow(id:Int,name:String,description:Option[String],settings:Seq[String]=Seq.empty)
defined class CalendarRow
scala> def append(calendars:Calendar*) = calendars.map(c => CalendarRow(c.id,c.name,c.description))
append: (calendars: Calendar*)Seq[CalendarRow]
scala> val calendar1 = Calendar(1,"first",Option("my first calendar"),1)
calendar1: Calendar = Calendar(1,first,Some(my first calendar),1)
scala> val calendar2 = Calendar(2, "second",None,1)
calendar2: Calendar = Calendar(2,second,None,1)
scala> val list = append(calendar1,calendar2)
list: Seq[CalendarRow] = ArrayBuffer(CalendarRow(1,first,Some(my first calendar),List()), CalendarRow(2,second,None,List()))
Many thanks to #Jean and comments in "Unveiling Play 2.1 Json API - Part 3 : JSON Transformers"
It's hard understanding things like and, andThen, andKeep, keepAnd in JSON transformers for me (I can not find any detailed descriptions with examples), but i found some templates for my question:
Optional property in JSON:
With Reader
(__ \ "id").json.pick[JsString].flatMap{
case id if id.equals(JsString(accountId)) =>
(__ \ "primary").json.put(JsBoolean(true))
case _ =>
Reads.pure(Json.obj())
}
With Json.obj()
(__ \ "id").json.pick[JsString].map{
case id if id.equals(JsString(accountId)) =>
Json.obj("primary" -> true)
case _ =>
Json.obj()
}
Given a JSON array like this one:
{
"success": true,
"data": [
{
"id": 600,
"title": "test deal",
"e54cbe3a434d8e6": 54
},
{
"id": 600,
"title": "test deal",
"e54cbe3a434d8e6": 54
},
],
"additional_data": {
"pagination": {
"start": 0,
"limit": 100,
"more_items_in_collection": false
}
}
}
In my Play 2.2.2 application, using the Scala JSON Reads Combinator, everything works going this way:
implicit val entityReader = Json.reads[Entity]
val futureJson: Future[List[Entity]] = futureResponse.map(
response => (response.json \ "data").validate[List[Entity]].get
The problem now is the key named 'e54cbe3a434d8e6' which I would like to name 'value' in my object:
// This doesn't work, as one might expect
case class Entity(id: Long, title: String, e54cbe3a434d8e6: Long)
// I would like to use 'value' instead of 'e54cbe3a434d8e6'
case class Entity(id: Long, title: String, value: Long)
There is vast information about the combinators here and here but I only want to use a fieldname which is different from the key name in the JSON array. Can some one help me to find a simple way?
I suppose it has got something to do with JSON.writes?!
One simple way without trying to apply transformations on json itself is to define a custom Reads in such a way to handle this:
val json = obj(
"data" -> obj(
"id" -> 600,
"title" -> "test deal",
"e54cbe3a434d8e6" -> 54))
case class Data(id: Long, title: String, value: Int)
val reads = (
(__ \ "id").read[Long] ~
(__ \ "title").read[String] ~
(__ \ "e54cbe3a434d8e6").read[Int] // here you get mapping from your json to Scala case class
)(Data)
def index = Action {
val res = (json \ "data").validate(reads)
println(res) // prints "JsSuccess(Data(600,test deal,54),)"
Ok(json)
}
Another way is to use combinators like this:
... the same json and case class
implicit val generatedReads = reads[Data]
def index = Action {
val res = (json \ "data").validate(
// here we pick value at 'e54cbe3a434d8e6' and put into brand new 'value' branch
__.json.update((__ \ "value").json.copyFrom((__ \ "e54cbe3a434d8e6").json.pick)) andThen
// here we remove 'e54cbe3a434d8e6' branch
(__ \ "e54cbe3a434d8e6").json.prune andThen
// here we validate result with generated reads for our case class
generatedReads)
println(res) // prints "JsSuccess(Data(600,test deal,54),/e54cbe3a434d8e6/e54cbe3a434d8e6)"
Ok(prettyPrint(json))
}
After reading a JSON result from a web service response:
val jsonResult: JsValue = Json.parse(response.body)
Containing content something like:
{
result: [
["Name 1", "Row1 Val1", "Row1 Val2"],
["Name 2", "Row2 Val1", "Row2 Val2"]
]
}
How can I efficiently map the contents of the result array in the JSON with a list (or something similar) like:
val keys = List("Name", "Val1", "Val2")
To get an array of hashmaps?
Something like this ?
This solution is functional and handles None/Failure cases "properly" (by returning a None)
val j = JSON.parseFull( json ).asInstanceOf[ Option[ Map[ String, List[ List[ String ] ] ] ] ]
val res = j.map { m ⇒
val r = m get "result"
r.map { ll ⇒
ll.foldRight( List(): List[ Map[ String, String ] ] ) { ( l, acc ) ⇒
Map( ( "Name" -> l( 0 ) ), ( "Val1" -> l( 1 ) ), ( "Val2" -> l( 2 ) ) ) :: acc
}
}.getOrElse(None)
}.getOrElse(None)
Note 1: I had to put double quotes around result in the JSON String to get the JSON parser to work
Note 2: the code could look nicer using more "monadic" sugar such as for comprehensions or using applicative functors