I am using Kotlin and I have a service that is getting an object of type Any (this cannot be changed)
The problem with Any is that is an object of 20+ fields and I just need one of them to use it as a filter... therefore I cannot do a simple cast.
So, my object is like: (when I print it)
{messageId=123, userId=32323, address=Some city, phone=111605,type=TYPE1.....
I want to convert it using Kotlinx or Jackson but I cannot convert it first to the expected String format, doing a parseFromString(myObject) will result in an exception as well of a wrong Json format.
I want to convert it to a class like this
#Serializable
private data class UserType(val type: String)
type is the only field I care about.
My convertion is via kotlinx
val format = Json { ignoreUnknownKeys = true }
format.decodeFromString<UserType>(myObject)
I even tried this to see if I can make it in the proper Json format
format.encodeToString(original)
Any idea what I could do here that would be a lightweight solution?
This is my Any type https://kotlinlang.org/api/latest/jvm/stdlib/kotlin/-any/
I am trying to parse the response given by a HTTP endpoint using json4s in scala. The Json returned could have many number of fields (They are all documented and defined, but there are a lot of them, and they are subject to change.) I don't need to reference many of these fields, only pass them on to some other service to deal with.
I want to take the fields I need, and deserialise the rest to a map. The class needs to be serialised correctly also.
e.g. JSON response from endpoint:
{
"name": "value",
"unknown_field": "unknown",
"unknown_array": ["one", "two", "three"],
...
...
}
e.g. case class used in code:
case class TestResponse(name: String, otherFields: Map[String, Any])
Is there a simple solution for this?
I have made an attempt to implement a custom Serialiser for this, but have not had much luck as yet. Seems like this would be a common enough requirement. Is there a way to do this OOTB with json4s?
Cheers
Current attempt at customer serialiser:
private object TestResponseDeserializer extends CustomSerializer[TestResponse](_ => ( {
case JObject(JField("name_one", JString(name)) :: rest) => TestType1Response(name, rest.toMap)
case JObject(JField("name_two", JString(name)) :: rest) => TestType2Response(name, rest.toMap)
}, {
case testType1: TestType1Response=>
JObject(JField("name_one", JString(testType1.name)))
case testType2: TestType2Response=> JObject(JField("name_two", JString(testType2.name)))
}))
I was able to solve this using the custom serialiser in the original question. It didn't work for me due to an unrelated issue.
My app supports protobuf and JSON serialzation. For JSON serialization I use com.trueaccord.scalapb.json.JsonFormat, my dtos are generated from proto definitions.
The com.trueaccord serializer wraps option types to JSON objects which is causing issues for some clients so I want to be able to support org.json4s without braking the existing clients.
I would like to be able to pick a serializer based on a custom http header called JFORMAT. The idea is that if this header is sent I will use json4s otherwise I will use the trueaccord serializer.
I managed to create a Unmarshaller which can pick a request serializer based on a header value:
Unmarshaller.withMaterializer[HttpRequest, T](_ => implicit mat => {
case request: HttpRequest =>
val entity = request.entity
entity.dataBytes.runFold(ByteString.empty)(_ ++ _).map(data => {
entity.contentType match {
case `applicationJsonContentType` =>
val jsFormat = {
val header = request.headers.find(h => h.name() == jsonFormatHeaderName)
if (header.isEmpty) "1.0" else header.get.value()
}
val charBuffer = Unmarshaller.bestUnmarshallingCharsetFor(entity)
val jsonText = data.decodeString(charBuffer.nioCharset().name())
val dto = if(jsFormat == "2.0") {
write[T](value)(formats) // New Formatter
} else {
JsonFormat.fromJsonString[T](jsonText) // Old Formatter
}
dto
case `protobufContentType` =>
companion.parseFrom(CodedInputStream.newInstance(data.asByteBuffer)) // Proto Formatter
case _ =>
throw UnsupportedContentTypeException(applicationJsonContentType, protobufContentType)
}
})
I want to do the same with my Marshaller which I use with Marshaller.oneOf and the JSON handling one looks like:
Marshaller.withFixedContentType(contentType) { value =>
val jsonText = JsonSerializer.toJsonString[T](value)
HttpEntity(contentType, jsonText)
}
Is there a way to construct a Mashaller which is aware of the request http headers? The Akka HTTP docs don't have any examples and I cannot make sense of the PredefinedToRequestMarshallers.
Do I need to combine multiple marshallers somehow or can I append some metadata to a context during the request serialization I can use later in the Marshaller? I want to avoid appending meta to my dto if possible or using a custom content type like application/vnd.api+json
There are lots of other useful info I could use from the request when I format the response like Accept-Encoding, custom headers like unique request id to create a correlation id, I could add JSONP support by reading the callback query parmeter, etc.
To clarify: I need a solution to use the Mashaller, subclass of it or a custom version created by a factory method or maybe multiple Marshallers chained together. Marshaller.withFixedContentType already using the Accept header so there must be a way. I added added bounty to reward a solution to a specific challenge. I am ware of hacks and workarounds and I asked the question because I need a clean solution solving a specific scenario.
Custom Marshallers section mentions Marshaller.oneOf overloaded methods, that seems to be what you want:
Helper for creating a "super-marshaller" from a number of
"sub-marshallers". Content-negotiation determines, which
"sub-marshaller" eventually gets to do the job.
The Marshaller companion object has many methods that receive a Seq[HttpHeader]. You can look into their implementations as well.
I don't have the time to look into the source code myself, but if this is not enough to put you on the right path, let me know.
Edit:
How about?
get {
optionalHeaderValueByName("JFORMAT") { format =>
complete {
format match {
case Some(f) => "Complete with json4s"
case _ => "Complete with trueaccord"
}
}
}
}
I have an object structure like so:
class Message {
static mapWith="mongo"
static embedded = ['to', 'author', 'comments', 'tags']
ObjectId id
Set<ObjectId> to
Author author
String text
List<Comment> comments
Set<String> tags
Date postedOn
Date lastEditOn
}
class Comment {
Author author
String text
int thumbsUp = 0
int thumbsDown = 0
Date postedOn
Date lastEditOn
}
And the following code for serialization to JSON
render Message.findStreamFor( session.user, groups, 0, 20 ) as JSON
However, none of the embedded collections are being serialized. They are just missing. I've tried adding the following to my Config.groovy to make it deeply serialize by default:
grails.converters.json.default.deep=true
But that doesn't seem to change anything. I've seen the objects are populated from MongoDB in the debugger, but it just doesn't make it to the JSON serialized output. How can I fix this?
UPDATE
Ok I've figured out a bit more by debugging the code. Inside the DefaultGrailsDomainClass.getPersistentProperties() it doesn't return the collections as properties when called. And JSON serializer never visits them. On line 103 of DomainClassMarshaller is the call to getPersistentProperties which isn't returning all properties.
GrailsDomainClassProperty[] properties = domainClass.getPersistentProperties();
Seems like this is a bug! How has no one else ever found this?
You could to use GSON plugin. It didn't help me in a similar problem but it may help you.
This plugin was written to overcome nested object deserialization problem in standard Grails JSON converter, but it may also be better at serializing them.
I'm trying to build a restful API and I'm struggling on how to serialize JSON data to a HTTP query string.
There are a number of mandatory and optional arguments that need to be passed in the request, e.g (represented as a JSON object below):
{
"-columns" : [
"name",
"column"
],
"-where" : {
"-or" : {
"customer_id" : 1,
"services" : "schedule"
}
},
"-limit" : 5,
"return" : "table"
}
I need to support a various number of different clients so I'm looking for a standardized way to convert this json object to a query string. Is there one, and how does it look?
Another alternative is to allow users to just pass along the json object in a message body, but I read that I should avoid it (HTTP GET with request body).
Any thoughts?
Edit for clarification:
Listing how some different languages encodes the given json object above:
jQuery using $.param: -columns[]=name&-columns[]=column&-where[-or][customer_id]=1&-where[-or][services]=schedule&-limit=5&return=column
PHP using http_build_query: -columns[0]=name&-columns[1]=column&-where[-or][customer_id]=1&-where[-or][services]=schedule&-limit=5&return=column
Perl using URI::query_form: -columns=name&-columns=column&-where=HASH(0x59d6eb8)&-limit=5&return=column
Perl using complex_to_query: -columns:0=name&-columns:1=column&-limit=5&-where.-or.customer_id=1&-where.-or.services=schedule&return=column
jQuery and PHP is very similar. Perl using complex_to_query is also pretty similar to them. But none look exactly the same.
URL-encode (https://en.wikipedia.org/wiki/Percent-encoding) your JSON text and put it into a single query string parameter. for example, if you want to pass {"val": 1}:
mysite.com/path?json=%7B%22val%22%3A%201%7D
Note that if your JSON gets too long then you will run into a URL length limitation problem. In which case I would use POST with a body (yes, I know, sending a POST when you want to fetch something is not "pure" and does not fit well into the REST paradigm, but neither is your domain specific JSON-based query language).
There is no single standard for JSON to query string serialization, so I made a comparison of some JSON serializers and the results are as follows:
JSON: {"_id":"5973782bdb9a930533b05cb2","isActive":true,"balance":"$1,446.35","age":32,"name":"Logan Keller","email":"logankeller#artiq.com","phone":"+1 (952) 533-2258","friends":[{"id":0,"name":"Colon Salazar"},{"id":1,"name":"French Mcneil"},{"id":2,"name":"Carol Martin"}],"favoriteFruit":"banana"}
Rison: (_id:'5973782bdb9a930533b05cb2',age:32,balance:'$1,446.35',email:'logankeller#artiq.com',favoriteFruit:banana,friends:!((id:0,name:'Colon Salazar'),(id:1,name:'French Mcneil'),(id:2,name:'Carol Martin')),isActive:!t,name:'Logan Keller',phone:'+1 (952) 533-2258')
O-Rison: _id:'5973782bdb9a930533b05cb2',age:32,balance:'$1,446.35',email:'logankeller#artiq.com',favoriteFruit:banana,friends:!((id:0,name:'Colon Salazar'),(id:1,name:'French Mcneil'),(id:2,name:'Carol Martin')),isActive:!t,name:'Logan Keller',phone:'+1 (952) 533-2258'
JSURL: ~(_id~'5973782bdb9a930533b05cb2~isActive~true~balance~'!1*2c446.35~age~32~name~'Logan*20Keller~email~'logankeller*40artiq.com~phone~'*2b1*20*28952*29*20533-2258~friends~(~(id~0~name~'Colon*20Salazar)~(id~1~name~'French*20Mcneil)~(id~2~name~'Carol*20Martin))~favoriteFruit~'banana)
QS: _id=5973782bdb9a930533b05cb2&isActive=true&balance=$1,446.35&age=32&name=Logan Keller&email=logankeller#artiq.com&phone=+1 (952) 533-2258&friends[0][id]=0&friends[0][name]=Colon Salazar&friends[1][id]=1&friends[1][name]=French Mcneil&friends[2][id]=2&friends[2][name]=Carol Martin&favoriteFruit=banana
URLON: $_id=5973782bdb9a930533b05cb2&isActive:true&balance=$1,446.35&age:32&name=Logan%20Keller&email=logankeller#artiq.com&phone=+1%20(952)%20533-2258&friends#$id:0&name=Colon%20Salazar;&$id:1&name=French%20Mcneil;&$id:2&name=Carol%20Martin;;&favoriteFruit=banana
QS-JSON: isActive=true&balance=%241%2C446.35&age=32&name=Logan+Keller&email=logankeller%40artiq.com&phone=%2B1+(952)+533-2258&friends(0).id=0&friends(0).name=Colon+Salazar&friends(1).id=1&friends(1).name=French+Mcneil&friends(2).id=2&friends(2).name=Carol+Martin&favoriteFruit=banana
The shortest among them is URL Object Notation.
How about you try this sending them as follows:
http://example.com/api/wtf?
[-columns][]=name&
[-columns][]=column&
[-where][-or][customer_id]=1&
[-where][-or][services]=schedule&
[-limit]=5&
[return]=table&
I tried with a REST Client
And on the server side (Ruby with Sinatra) I checked the params, it gives me exactly what you want. :-)
Another option might be node-querystring. It also uses a similar scheme to the ones you've so far listed.
It's available in both npm and bower, which is why I have been using it.
Works well for nested objects.
Passing complex objects as query parameters of a url.
In the example below, obj is the JSON object to pass into query parameters.
Injecting JSON object as query parameters:
value = JSON.stringify(obj);
URLSearchParams to convert a string to an object representing search params. toString to retain string type for appending to url:
queryParams = new URLSearchParams(value).toString();
Pass the query parameters using template literals:
url = `https://some-url.com?key=${queryParams}`;
Now url will contain the JSON object as query parameters under key (user-defined name)
Extracing JSON from url:
This is assuming you have access to the url (either as string or URL object)
url_obj = new URL(url); (only if url is NOT a URL object, otherwise ignore this step)
Extract all query parameters in the url:
queryParams = new URLSearchParams(url_obj.search);
Use the key to extract the specific value:
obj = JSON.parse(queryParams.get('key').slice(0, -1));
slice() is used to extract a tailing = in the query params which is not required.
Here obj will be the same object passed in the query params.
I recommend to try these steps in the web console to understand better.
You can test with JSON examples here: https://json.org/example.html