Converting nested lists to json gives superfluous arrays - json

I have some history data, that I want to convert to json. So these are Lists of lists. Their type is List[List[Position]] where Position is a simple case class. I wrote a formatter to help Json.toJson cope. I was expecting an output of exactly one outer array with two inner arrays that contain 3 objects each. What I got instead was this. Note the additional array wrappings.
[[[[{"amount":1.0,"minAmount":2.0,"price":3.0,"volume":4.0},
{"amount":5.0,"minAmount":6.0,"price":7.0,"volume":8.0},
{"amount":9.0,"minAmount":10.0,"price":11.0,"volume":12.0}]],
[[{"amount":0.1,"minAmount":0.2,"price":0.3,"volume":0.4},
{"amount":5.0,"minAmount":6.0,"price":7.0,"volume":8.0},
{"amount":9.0,"minAmount":10.0,"price":11.0,"volume":12.0}]]]]
I don't know where the wrapping arrays come from. Can somebody help me out here? This is a test with the wrapper I am using:
class ApplicationSpec extends Specification {
implicit object PositionFormat extends Format[List[List[Position]]] {
def writes(historyList: List[List[Position]]) : JsValue = {
Json.arr(historyList.map{
o => Json.arr(o.map{ p =>
Json.obj(
"amount" -> JsNumber(p.amount),
"minAmount" -> JsNumber(p.minAmount),
"price" -> JsNumber(p.price),
"volume" -> JsNumber(p.volume)
)
})
})
}
def reads(json: JsValue): JsResult[List[List[Position]]] = ???
}
"Application" should {
"Convert position data to json" in {
val l1 = ListBuffer(new Position(1.0D,2.0D,3.0D,4.0D),
new Position(5.0D,6.0D,7.0D,8.0D),
new Position(9.0D,10.0D,11.0D,12.0D)).toList
val l2 = ListBuffer(new Position(0.1D,0.2D,0.3D,0.4D),
new Position(5.0D,6.0D,7.0D,8.0D),
new Position(9.0D,10.0D,11.0D,12.0D)).toList
val obj = ListBuffer(l1,l2).toList
val json = Json.toJson(obj)
var string: String = json.toString()
println(string)
}
}
}

It seems that Json.arr takes it's arguments and returns a JsValue for a JSON array of them. It looks like you could simply do with Json.toJson:
Here's how arr is meant to be used:
// Json.arr
Json.arr("one", "two")
// Gives you
// play.api.libs.json.JsArray = ["one","two"]
If you instead do:
// vs
val l = List("one", "two")
Json.arr(l)
// Gives you
// play.api.libs.json.JsArray = [["one","two"]]
// ... a nested array, which is what you don't want.
What you need is:
// Json.toJson
Json.toJson(l)
// Gives you:
// play.api.libs.json.JsValue = ["one","two"]

Related

How to convert scala.some to scala.mutable.Map?

i am trying to write code to mask nested json fields..
def maskRecursively(map :mutable.Map[String,Object]):mutable.Map[String,Object] ={
val maskColumns = PII_Data.getPIIData()
for((k,v) <- map){
if(v.isInstanceOf[Map[String,Object]]){
maskRecursively(map.get(k).asInstanceOf[mutable.Map[String,Object]])
}else if(v.isInstanceOf[List[Object]]) {
} else {
if(maskColumns.contains(k)){map+=(k->"*****")}
}
}
map }
calling this method from ..
val mapper = new ObjectMapper()
mapper.registerModule(DefaultScalaModule)
val result = mapper.readValue(jsonStr, classOf[ java.util.Map[String,Object] ])
import scala.collection.JavaConverters._
val myScalaMap = result.asScala
maskRecursively(result.asScala)
i am getting error while trying to iterate a nested json object ..
Cannot cast value of type 'scala.Some' to type 'scala.collection.mutable.Map'
how do i recurse a complex nested json object this way ?
Your mistake was
if(v.isInstanceOf[Map[String,Object]]){
maskRecursively(map.get(k).asInstanceOf[mutable.Map[String,Object]])
There are a few issues:
You check if v is an instance of Map, but then attempt to cast it to mutable.Map. They are technically different types (mutable vs immutable).
You check the type of v, but then apply the cast to map.get(k), which is going to be a different value and type from v. A map's get method returns an Option, hence the error message.
Thanks to type erasure on the JVM, the runtime won't be able to tell the difference between e.g. a Map[String, Object] and a Map[SomethingElse, Whatever] - both will just look like Map at runtime. The compiler should have given you a warning about the isInstanceOf call for this reason.
If you do an isInstanceOf / asInstanceOf combo, make sure the operand is the same each time. You already have v, so you don't need to look it up a second time from the map. And make sure you use the same type on both instanceOf calls.
Fix this by changing it to
if(v.isInstanceOf[mutable.Map[_, _]]){
maskRecursively(v.asInstanceOf[mutable.Map[String,Object]])
After some digging , i was able to solve this..
def maskJson(jsonStr: String): String = {
implicit val formats = org.json4s.DefaultFormats
val mapper = new ObjectMapper()
mapper.registerModule(DefaultScalaModule)
val result = mapper.readValue(jsonStr, classOf[Map[String, Object]])
val maskedJson = maskRecursively(result)
mapper.writeValueAsString(maskedJson)
}
def maskRecursively(map: Map[String, Object]): collection.mutable.Map[String, Object] = {
val mutable = collection.mutable.Map[String, Object]()
val maskColumns = PII_Data.getJsonPIIFields()
for ((k, v) <- map) {
if (v.isInstanceOf[Map[String, Object]]) {
mutable += k -> maskRecursively(v.asInstanceOf[Map[String, Object]])
} else if (v.isInstanceOf[List[Object]]) {
val list = v.asInstanceOf[List[Map[String, Object]]].map(i => maskRecursively(i)).toList
mutable += k -> list
} else {
if (maskColumns.contains(k)) {
mutable += (k -> "*****")
}
else {
mutable += k -> v
}
}
}
mutable
}

Gson deserialization of optional+nullable value

The problem I am trying to solve is perfectly described by the following text got from this link:
For a concrete example of when this could be useful, consider an API that supports partial updates of objects. Using this API, a JSON object would be used to communicate a patch for some long-lived object. Any included property specifies that the corresponding value of the object should be updated, while the values for any omitted properties should remain unchanged. If any of the object’s properties are nullable, then a value of null being sent for a property is fundamentally different than a property that is missing, so these cases must be distinguished.
That post presents a solution but using the kotlinx.serialization library, however, I must use gson library for now.
So I am trying to implement my own solution as I didn't find anything that could suit my use case (please let me know if there is).
data class MyObject(
val fieldOne: OptionalProperty<String> = OptionalProperty.NotPresent,
val fieldTwo: OptionalProperty<String?> = OptionalProperty.NotPresent,
val fieldThree: OptionalProperty<Int> = OptionalProperty.NotPresent
)
fun main() {
val gson = GsonBuilder()
.registerTypeHierarchyAdapter(OptionalProperty::class.java, OptionalPropertyDeserializer())
.create()
val json1 = """{
"fieldOne": "some string",
"fieldTwo": "another string",
"fieldThree": 18
}
"""
println("json1 result object: ${gson.fromJson(json1, MyObject::class.java)}")
val json2 = """{
"fieldOne": "some string",
"fieldThree": 18
}
"""
println("json2 result object: ${gson.fromJson(json2, MyObject::class.java)}")
val json3 = """{
"fieldOne": "some string",
"fieldTwo": null,
"fieldThree": 18
}
"""
println("json3 result object: ${gson.fromJson(json3, MyObject::class.java)}")
}
sealed class OptionalProperty<out T> {
object NotPresent : OptionalProperty<Nothing>()
data class Present<T>(val value: T) : OptionalProperty<T>()
}
class OptionalPropertyDeserializer : JsonDeserializer<OptionalProperty<*>> {
private val gson: Gson = Gson()
override fun deserialize(
json: JsonElement?,
typeOfT: Type?,
context: JsonDeserializationContext?
): OptionalProperty<*> {
println("Inside OptionalPropertyDeserializer.deserialize json:$json")
return when {
// Is it a JsonObject? Bingo!
json?.isJsonObject == true ||
json?.isJsonPrimitive == true-> {
// Let's try to extract the type in order
// to deserialize this object
val parameterizedType = typeOfT as ParameterizedType
// Returns an Present with the value deserialized
return OptionalProperty.Present(
context?.deserialize<Any>(
json,
parameterizedType.actualTypeArguments[0]
)!!
)
}
// Wow, is it an array of objects?
json?.isJsonArray == true -> {
// First, let's try to get the array type
val parameterizedType = typeOfT as ParameterizedType
// check if the array contains a generic type too,
// for example, List<Result<T, E>>
if (parameterizedType.actualTypeArguments[0] is WildcardType) {
// In case of yes, let's try to get the type from the
// wildcard type (*)
val internalListType = (parameterizedType.actualTypeArguments[0] as WildcardType).upperBounds[0] as ParameterizedType
// Deserialize the array with the base type Any
// It will give us an array full of linkedTreeMaps (the json)
val arr = context?.deserialize<Any>(json, parameterizedType.actualTypeArguments[0]) as ArrayList<*>
// Iterate the array and
// this time, try to deserialize each member with the discovered
// wildcard type and create new array with these values
val result = arr.map { linkedTreeMap ->
val jsonElement = gson.toJsonTree(linkedTreeMap as LinkedTreeMap<*, *>).asJsonObject
return#map context.deserialize<Any>(jsonElement, internalListType.actualTypeArguments[0])
}
// Return the result inside the Ok state
return OptionalProperty.Present(result)
} else {
// Fortunately it is a simple list, like Array<String>
// Just get the type as with a JsonObject and return an Ok
return OptionalProperty.Present(
context?.deserialize<Any>(
json,
parameterizedType.actualTypeArguments[0]
)!!
)
}
}
// It is not a JsonObject or JsonArray
// Let's returns the default state NotPresent.
else -> OptionalProperty.NotPresent
}
}
}
I got most of the code for the custom deserializer from here.
This is the output when I run the main function:
Inside OptionalPropertyDeserializer.deserialize json:"some string"
Inside OptionalPropertyDeserializer.deserialize json:"another string"
Inside OptionalPropertyDeserializer.deserialize json:18
json1 result object: MyObject(fieldOne=Present(value=some string), fieldTwo=Present(value=another string), fieldThree=Present(value=18))
Inside OptionalPropertyDeserializer.deserialize json:"some string"
Inside OptionalPropertyDeserializer.deserialize json:18
json2 result object: MyObject(fieldOne=Present(value=some string), fieldTwo=my.package.OptionalProperty$NotPresent#573fd745, fieldThree=Present(value=18))
Inside OptionalPropertyDeserializer.deserialize json:"some string"
Inside OptionalPropertyDeserializer.deserialize json:18
json3 result object: MyObject(fieldOne=Present(value=some string), fieldTwo=null, fieldThree=Present(value=18))
I am testing the different options for the fieldTwo and it is almost fully working, with the exception of the 3rd json, where I would expect that fieldTwo should be fieldTwo=Present(value=null) instead of fieldTwo=null.
And I see that in this situation, the custom deserializer is not even called for fieldTwo.
Can anyone spot what I am missing here? Any tip would be very appreciated!
I ended giving up of gson and move to moshi.
I implemented this behavior based on the solution presented in this comment.

Deserializing 2d array in kotlinx.serialization

I am quite new to Kotlin but and I have successfully used Kotlin serialization on many cases - works out of the box even for nested classes, mutableLists etc. However I struggle with two dimensional arrays.
My class:
import kotlinx.serialization.*
#Serializable
data class Thing(val name:String)
#Serializable
data class Array2D(val width:Int, val height:Int,
var arrayContents:Array<Array<Thing?>> = Array(1){Array(1){null} }){
init{
arrayContents = Array(width){Array(height){null} }
}
}
And when doing this:
val a = Array2D(2, 2)
a.arrayContents[0][0] = Thing("T0")
a.arrayContents[0][1] = Thing("T1")
a.arrayContents[1][0] = Thing("T2")
a.arrayContents[1][1] = Thing("T3")
val json = Json {
allowStructuredMapKeys = true
}
val jsonString = json.encodeToString(Array2D.serializer(), a)
assertEquals(
"""
{"width":2,"height":2,"arrayContents":[[{"name":"T0"},{"name":"T1"}],[{"name":"T2"},{"name":"T3"}]]}
""".trim(),
jsonString
) // encoding is OK
val b = json.decodeFromString(deserializer = Array2D.serializer(), jsonString)
// this fails to reproduce "a" and stops at first array level
// b.arrayContents = {Thing[2][]} (array of nulls) instead of {Thing[2][2]} (array of array of Thing)
If it can encode the class to String it should decode it as well, right? Or am I missing something here? Maybe I should use custom serializer but there are not many examples that fit my case. One example is https://github.com/Kotlin/kotlinx.serialization/issues/357 but it is only one level of array.
Thanks for any help :)
Serialization/deserialization for arrays (including multi-dimensional) works out of the box.
Unexpected behavior is related to init section of your data class. It's executed after deserialization happens and overwrites data parsed from JSON.
It also happens when you create instance of Array2D manually:
val x = Array2D(1, 1, Array(1) { Array(1) { Thing("0") } })
println(x.arrayContents[0][0]) //will print null
You just need to replace init block with secondary constructor (default value for arrayContents is redundant, by the way), and you may declare arrayContents as val now:
#Serializable
data class Array2D(val width: Int, val height: Int, val arrayContents: Array<Array<Thing?>>) {
constructor(width: Int, height: Int) : this(width, height, Array(width) { Array(height) { null } })
}

Appending values in a list and then sending as a JSON object

var jsonElements = List[String]()
val params = Map("host"->host)
for((key,value)<-mapSql){
val map = Map("metric"->key,"timestamp"->new Date().getTime,"value"->value,"tags"->params)
jsonElements=JsonUtility.toJSONString(map) :: jsonElements
}
val entity = new StringEntity(JsonUtility.toJSONString(jsonElements))
println("json elements final list is "+jsonElements)
println("json elements final JSON Obj is "+JsonUtility.toJSONString(jsonElements))
entity.setContentType(new BasicHeader("Content-Type", "application/json"))
val postRequest: HttpPost = new HttpPost(putURL)
postRequest.setEntity(entity)
val postResponse: CloseableHttpResponse = httpclient.execute(postRequest)
I basically need to add values to a list and then send them together in a JSON Array.
However this is introducing unnecessary escape characters "/" in the output which is rendering the post request useless and I am getting an error to the API hit. the following is the response :
json elements final list is List({"metric":"replicationLag","timestamp":1410179907871,"value":0.0,"tags":{"host":"tg-em-db01.nm.xxxx.com"}}, {"metric":"status","timestamp":1410179907824,"value":1,"tags":{"host":"tg-em-db01.nm.xxxxx.com"}})
json elements final JSON Obj is ["{\"metric\":\"replicationLag\",\"timestamp\":1410179907871,\"value\":0.0,\"tags\":{\"host\":\"tg-em-db01.nm.xxxx.com\"}}","{\"metric\":\"status\",\"timestamp\":1410179907824,\"value\":1,\"tags\":{\"host\":\"tg-em-db01.nm.xxxxx.com\"}}"]
I can replace and remove all the escape characters by the replaceAll function but I do not want to do that. is there a better way to append objects to an already existing JSON object and then change it to an array ( which i can easily do by new JsonArray(List(JsonObj)) ) so that i dont get any escape characters anywhere.
Something like this :
val params = Map("host"->host)
var map = Map[String,Any]()
for((key,value)<-mapSql){
map ++= Map("metric"->key,"timestamp"->new Date().getTime,"value"->value,"tags"->params)
}
val entity = new StringEntity(JsonUtility.toJSONString(List(map)))
println("json elements final list is "+map)
println("json elements final JSON Obj is "+JsonUtility.toJSONString(List(map)))
is giving me this as an ouput :
json elements final list is Map(metric -> replicationLag, timestamp -> 1410180939983, value -> 0.0, tags -> Map(host -> tg-em-db01.nm.xxxx.com))
json elements final JSON Obj is [{"metric":"replicationLag","timestamp":1410180939983,"value":0.0,"tags":{"host":"tg-em-db01.nm.xxxxx.com"}}]
But I need something like this :
[ {"metric":blah blah} , {"metric":blah blah} ]
Is there a way to append to maps such that the same key values are not clubbed ?
Thanks in advancE!
var jsonElements = List[Map[String, Any]]()
val params = Map("host" -> host)
for ((key, value) <- mapSql) {
val map = Map("metric" -> key, "timestamp" -> new Date().getTime, "value" -> value, "tags" -> params)
jsonElements = map :: jsonElements
}
val entity = new StringEntity(JsonUtility.toJSONString(jsonElements))
entity.setContentType(new BasicHeader("Content-Type", "application/json"))
val postRequest: HttpPost = new HttpPost(putURL)
postRequest.setEntity(entity)
val postResponse: CloseableHttpResponse = httpclient.execute(postRequest)
object JsonUtility {
def toJSONString(obj:Any):String = {
compact(JsonAST.render(decompose(obj)))
}
}

Grails JSON array

I'm converting a list of Foo objects to a JSON string. I need to parse the JSON string back into a list of Foos. However in the following example, parsing gives me a list of JSONObjects instead of Foos.
Example
List list = [new Foo("first"), new Foo("second")]
def jsonString = (list as JSON).toString()
List parsedList = JSON.parse(jsonString) as List
println parsedList[0].getClass() // org.codehaus.groovy.grails.web.json.JSONObject
How can I parse it into Foos instead?
Thanks in advance.
I had a look at the API docs for JSON and there doesn't appear to be any way to parse to a JSON string to a specific type of object.
So you'll just have to write the code yourself to convert each JSONObject to a Foo. Something like this should work:
import grails.converters.JSON
import org.codehaus.groovy.grails.web.json.*
class Foo {
def name
Foo(name) {
this.name = name
}
String toString() {
name
}
}
List list = [new Foo("first"), new Foo("second")]
def jsonString = (list as JSON).toString()
List parsedList = JSON.parse(jsonString)
// Convert from a list of JSONObject to a list of Foo
def foos = parsedList.collect {JSONObject jsonObject ->
new Foo(name: jsonObject.get("name"))
}
A more general solution would be to add a new static parse method such as the following to the JSON metaClass, that tries to parse the JSON string to a List of objects of a particular type:
import grails.converters.JSON
import org.codehaus.groovy.grails.web.json.*
class Foo {
def name
Foo(name) {
this.name = name
}
String toString() {
name
}
}
List list = [new Foo("first"), new Foo("second")]
def jsonString = (list as JSON).toString()
List parsedList = JSON.parse(jsonString)
// Define the new method
JSON.metaClass.static.parse = {String json, Class clazz ->
List jsonObjs = JSON.parse(json)
jsonObjs.collect {JSONObject jsonObj ->
// If the user hasn't provided a targetClass read the 'class' proprerty in the JSON to figure out which type to convert to
def targetClass = clazz ?: jsonObj.get('class') as Class
def targetInstance = targetClass.newInstance()
// Set the properties of targetInstance
jsonObj.entrySet().each {entry ->
if (entry.key != "class") {
targetInstance."$entry.key" = entry.value
}
}
targetInstance
}
}
// Try the new parse method
List<Foo> foos = JSON.parse(jsonString, Foo)
// Confirm it worked
assert foos.every {Foo foo -> foo.class == Foo && foo.name in ['first', 'second'] }
You can try out the code above in the groovy console. A few warnings
I have only performed very limited testing on the code above
There are two JSON classes in the latest Grails release, I'm assuming you're using the one that is not deprecated
If you are doing this in a Grails controller, and Foo IS indeed a domain object, don't forget that armed with your JSON map, you can also do:
List list = [new Foo("first"), new Foo("second")]
def jsonString = (list as JSON).toString()
List parsedList = JSON.parse(jsonString) as List
Foo foo = new Foo()
bindData(foo, parsedList[0]);
I've taken this code and extended it to work with nested structures. It relies on a 'class' attribute existing in the JSON. If there's a better way by now in Grails please let me know.
// The default JSON parser just creates generic JSON objects. If there are nested
// JSON arrays they are not converted to theirs types but are left as JSON objects
// This converts nested JSON structures into their types.
// IT RELIES ON A PROPERTY 'class' that must exist in the JSON tags
JSON.metaClass.static.parseJSONToTyped = {def jsonObjects ->
def typedObjects = jsonObjects.collect {JSONObject jsonObject ->
if(!jsonObject.has("class")){
throw new Exception("JSON parsing failed due to the 'class' attribute missing: " + jsonObject)
}
def targetClass = grailsApplication.classLoader.loadClass(jsonObject.get("class"))
def targetInstance = targetClass.newInstance()
// Set the properties of targetInstance
jsonObject.entrySet().each {entry ->
// If the entry is an array then recurse
if(entry.value instanceof org.codehaus.groovy.grails.web.json.JSONArray){
def typedSubObjects = parseJSONToTyped(entry.value)
targetInstance."$entry.key" = typedSubObjects
}
else if (entry.key != "class") {
targetInstance."$entry.key" = entry.value
}
}
targetInstance
}
return typedObjects
}
As of Grails 2.5, this is possible:
Period test = new Period()
test.periodText = 'test'
String j = test as JSON
def p = JSON.parse(j)
test = p.asType(Period)
println(test.periodText)
Output:
test
I am unsure of when it became an option.