I have some strange json that I cannot change, and I wish to parse it using
the JsonParsen in lift.
A typical json is like:
{"name":"xxx", "data":{
"data_123456":{"id":"Hello"},
"data_789901":{"id":"Hello"},
"data_987654":{"id":"Hello"},
}}
The issue is that the keys for the data are unknown (data_xxxxx, where the xx:s
are not known).
This is bad json, but I have to live with it.
How am I supposed to setup case-classes in scala to be able to build a proper
structure when the keys here are unknown, but the structure is known?
You can use a Map, and every value can be JValue too, representing unparsed JSON. Example:
case class Id(id: String)
case class Data(name: JValue, data: Map[String, Id])
And then:
json.extract[Data]
res0: Data(JString(xxx),Map(data_123456 -> Id(Hello), data_789901 -> Id(Hello), data_987654 -> Id(Hello)))
Related
I am using Kotlin and I have a service that is getting an object of type Any (this cannot be changed)
The problem with Any is that is an object of 20+ fields and I just need one of them to use it as a filter... therefore I cannot do a simple cast.
So, my object is like: (when I print it)
{messageId=123, userId=32323, address=Some city, phone=111605,type=TYPE1.....
I want to convert it using Kotlinx or Jackson but I cannot convert it first to the expected String format, doing a parseFromString(myObject) will result in an exception as well of a wrong Json format.
I want to convert it to a class like this
#Serializable
private data class UserType(val type: String)
type is the only field I care about.
My convertion is via kotlinx
val format = Json { ignoreUnknownKeys = true }
format.decodeFromString<UserType>(myObject)
I even tried this to see if I can make it in the proper Json format
format.encodeToString(original)
Any idea what I could do here that would be a lightweight solution?
This is my Any type https://kotlinlang.org/api/latest/jvm/stdlib/kotlin/-any/
I have reached a point where my code compiles successfully, yet I have doubts about my solution and am posting this question for that reason.
I have a Node class defined as:
case class Node(id: Long, label: String, parent_id: Option[Long])
The reason I quote/unquote recursive is because technically, I do not store a Node within a Node. Rather, each node has a pointer to its parent and I can say: give me all children of Node id=X.
Here is an example tree, for the sake of visualization. I would like to give the root_node's ID, and obtain the tree's conversion to a Json String:
root_node
|_ node_1
| |_ node_11
| |_ node_111
|_ node_2
|_ node_3
The Json would look like:
{"title": "root_node", "children": [...]}
with the children array containing node_1, 2 and 3 etc... recursively
Here is the Writes Converter for Node:
/** json converter of Node to JSON */
implicit val NodeWrites = new Writes[Node] {
def writes(node: Node) = Json.obj(
"title" -> node.label,
"children" -> Node.getChildrenOf(node.id)
)
}
Quoting Play docs:
The Play JSON API provides implicit Writes for most basic types, such
as Int, Double, String, and Boolean. It also supports Writes for
collections of any type T that a Writes[T] exists.
I need to point out that Node.getChildrenOf(node.id) returns a List of Nodes from the DB. So according to Play's docs, I should be able to convert a List[Node] to Json. It seems that doing this within the Writes converter itself is a bit more troublesome.
Here is the resulting error from running this code:
type mismatch;
found : List[models.Node]
required: play.api.libs.json.Json.JsValueWrapper
Note: implicit value NodeWrites is not applicable here because it comes after the application point and it lacks an explicit result type
I added the "explicit result type" to my Writes converter, here is the result:
/** json converter of Node to JSON */
implicit val NodeWrites: Writes[Node] = new Writes[Node] {
def writes(node: Node) = Json.obj(
"title" -> node.label,
"children" -> Node.getChildrenOf(node.id)
)
}
The code now executes properly, I can visualize the tree on the browser.
Even though this looks to me like the cleanest working solution, IntelliJ still complains about the line:
"children" -> Node.getChildrenOf(node.id)
saying:
Type mismatch: found(String, List[Node]), required (String, Json.JsValueWrapper)
Could it be that IntelliJ's error reporting is not based off of the Scala compiler?
Finally, is the overall approach of the JSON converter terrible?
Thanks and sorry for the long post.
The problem lies in "children" -> Node.getChildrenOf(node.id). Node.getChildrenOf(node.id) returns a List[Node]. Whereas any attribute in the Json.obj expects JsValueWrappers. In this case a JsArray.
Something like this should work:
implicit val writes = new Writes[Node] {
def writes(node: Node) = Json.obj(
"title" -> node.label,
// Note that we have to pass this since the writes hasn't been defined just yet.
"children" -> JsArray(Node.getChildrenOf(node).map(child => Json.toJson(child)(this)))
)
}
This at least compiles, I haven't tested it with any data though.
I have a huge json object and I need to parse it and then write some tests to see if everything goes as expected.
case class User(id: Identification, age:Int, name: String ...)
case class Identification(id: Int, hash: String ....)
... a lot more classes
Now I'm trying to write the tests
val json = parse(Source.fromFile(/path).getLines.mkString("\n"))
import org.json4s.DefaultFormats
implicit val formats = DefaultFormats
So my question is how can i test if the case classes are ok?
I thought maybe I should try to extract for ex. the users and then to check parameter by parameter if they are correct, but I don't thing that is a good way because it is not me who created the json so I'm not interested about the content.
Thanks
This is what I found working with JSON and case classes overt the years the minimum to test.
This three things should be tested always
Serialization with deserialiaztion combined
val example = MyCaseClass()
read[MyCaseClass](write(example)) should Equal example
Checks if a class can be transformed to JSON, read back and still has the same values. This one breaks more often than one would think.
Deserialization: JSON String -> CaseClasses
val exampleAsJSON : String
val exampleAsCaseClass : MyCaseClass
read(exampleAsJSON) shouldEqual exampleAsCaseClass
Checks if JSON still can be deserialized.
Serialization: CaseClasses -> JSON String
val exampleAsJSON : String
val exampleAsCaseClass : MyCaseClass
write(exampleAsCaseClass) shouldEqual exampleAsJSON
Checks if String/ JSON Representation stays stable. Here it is hard to keep the data up to date and often some not nice whitespace changes lead to false alarms.
Additional things to test
Are there optional parameters present? If yes all tests should be done with and without the optional parameters.
What is the fastest way to convert this
{"a":"ab","b":"cd","c":"cd","d":"de","e":"ef","f":"fg"}
into mutable map in scala ? I read this input string from ~500MB file. That is the reason I'm concerned about speed.
If your JSON is as simple as in your example, i.e. a sequence of key/value pairs, where each value is a string. You can do in plain Scala :
myString.substring(1, myString.length - 1)
.split(",")
.map(_.split(":"))
.map { case Array(k, v) => (k.substring(1, k.length-1), v.substring(1, v.length-1))}
.toMap
That looks like a JSON file, as Andrey says. You should consider this answer. It gives some example Scala code. Also, this answer gives some different JSON libraries and their relative merits.
The fastest way to read tree data structures in XML or JSON is by applying streaming API: Jackson Streaming API To Read And Write JSON.
Streaming would split your input into tokens like 'beginning of an object' or 'beginning of an array' and you would need to build a parser for these token, which in some cases is not a trivial task.
Keeping it simple. If reading a json string from file and converting to scala map
import spray.json._
import DefaultJsonProtocol._
val jsonStr = Source.fromFile(jsonFilePath).mkString
val jsonDoc=jsonStr.parseJson
val map_doc=jsonDoc.convertTo[Map[String, JsValue]]
// Get a Map key value
val key_value=map_doc.get("key").get.convertTo[String]
// If nested json, re-map it.
val key_map=map_doc.get("nested_key").get.convertTo[Map[String, JsValue]]
println("Nested Value " + key_map.get("key").get)
I want to convert a scala list of strings, List[String], to an Json object.
For each string in my list I want to add it to my Json object.
So that it would look like something like this:
{
"names":[
{
"Bob",
"Andrea",
"Mike",
"Lisa"
}
]
}
How do I create an json object looking like this, from my list of strings?
To directly answer your question, a very simplistic and hacky way to do it:
val start = """"{"names":[{"""
val end = """}]}"""
val json = mylist.mkString(start, ",", end)
However, what you almost certainly want to do is pick one of the many JSON libraries out there: play-json gets some good comments, as does lift-json. At the worst, you could just grab a simple JSON library for Java and use that.
Since I'm familiar with lift-json, I'll show you how to do it with that library.
import net.liftweb.json.JsonDSL._
import net.liftweb.json.JsonAST._
import net.liftweb.json.Printer._
import net.liftweb.json.JObject
val json: JObject = "names" -> List("Bob", "Andrea", "Mike", "Lisa")
println(json)
println(pretty(render(json)))
The names -> List(...) expression is implicitly converted by the JsonDSL, since I specified that I wanted it to result in a JObject, so now json is the in-memory model of the json data you wanted.
pretty comes from the Printer object, and render comes from the JsonAST object. Combined, they create a String representation of your data, which looks like
{
"names":["Bob","Andrea","Mike","Lisa"]
}
Be sure to check out the lift documentation, where you'll likely find answers to any further questions about lift's json support.