I'm scraping external sources, mostly JSON. I'm using new JsonSlurper().parse(body) to parse them and I operate on them using constructs like def name = json.user[0].name. These being external, can change without notice, so I want to be able to detect this and log it somehow.
After reading a bit about the MOP I thought that I can change the appropriate methods of the maps and lists to log if the property is missing. I only want to do that the json object and on its properties recursively. The thing is, I don't know how to do that.
Or, is there a better way to accomplish all this?
[EDIT] For example if I get this JSON:
def json = '''{
"owners": [
{
"firstName": "John",
"lastName": "Smith"
},
{
"firstName": "Jane",
"lastName": "Smith"
}
]
}'''
def data = new groovy.json.JsonSlurper().parse(json.bytes)
assert data.owners[0].firstName == 'John'
However, if they change "owners" to "ownerInfo" the above access would throw NPE. What I want is intercept the access and do something (like log it in a special log, or whatever). I can also decide to throw a more specialized exception.
I don't want to catch NullPointerException, because it may be caused by some bug in my code instead of the changed data format. Besides, if they changed "firstName" to "givenName", but kept the "owners" name, I'd just get a null value, not NPE. Ideally I want to detect this case as well.
I also don't want to put a lot of if or evlis operators, if possible.
I actually managed to intercept that for maps:
data.getMetaClass().getProperty = {name -> println ">>> $name"; delegate.get(name)}
assert data.owners // this prints ">>> owners"
I still can't find out how to do that for the list:
def owners = data.owners
owners.getMetaClass().getAt(o -> println "]]] $o"; delegate.getAt(o)}
assert owners[0] // this doesn't print anything
Try this
owners.getMetaClass().getAt = { Integer o -> println "]]] $o"; delegate.get(o)}
I'm only guessing that it got lost because of multiple getAt() methods, so you have to define type. I also delegated to ArrayList's Java get() method since getAt() resulted in recursive calls.
If you want to more control over all methods calls, you could always do this
owners.getMetaClass().invokeMethod = { String methodName, Object[] args ->
if (methodName == "getAt") {
println "]]] $args"
}
return ArrayList.getMetaClass().getMetaMethod(methodName, args).invoke(delegate, args)
}
The short answer is that you can't do this with the given example. The reason is that the owners object is a java.util.ArrayList, and you are calling the get(int index) method on it. The metaClass object is specific to Groovy, and if you have a Java object making a method call to a Java method, it will have no knowledge of the metaClass. Here's a somewhat related question.
The good news is that there is and option, although I'm not sure if it works for your use case. You can create a Groovy wrapper object for this list, so that you can capture method calls.
For example, you could change your code from this
def owners = data.owners
to this
def owners = new GroovyList(data.owners)
and then create this new class
class GroovyList {
private List list
public GroovyList(List list) {
this.list = list
}
public Object getAt(int index) {
println "]]] $index"
list.getAt(index)
}
}
Now when you call
owners[0]
you'll get the output
]]] 0
[firstName:John, lastName:Smith]
Related
I've been playing with Kotlinx.serialisation. I've been trying to find a quick way to use Kotlinx.serialisation to create a plain simple JSON (mostly to send it away), with minimum code clutter.
For a simple string such as:
{"Album": "Foxtrot", "Year": 1972}
I've been doing is something like:
val str:String = Json.stringify(mapOf(
"Album" to JsonPrimitive("Foxtrot"),
"Year" to JsonPrimitive(1972)))
Which is far from being nice. My elements are mostly primitive, so I wish I had something like:
val str:String = Json.stringify(mapOf(
"Album" to "Sergeant Pepper",
"Year" to 1967))
Furthermore, I'd be glad to have a solution with a nested JSON. Something like:
Json.stringify(JsonObject("Movies", JsonArray(
JsonObject("Name" to "Johnny English 3", "Rate" to 8),
JsonObject("Name" to "Grease", "Rate" to 1))))
That would produce:
{
"Movies": [
{
"Name":"Johnny English 3",
"Rate":8
},
{
"Name":"Grease",
"Rate":1
}
]
}
(not necessarily prettified, even better not)
Is there anything like that?
Note: It's important to use a serialiser, and not a direct string such as
"""{"Name":$name, "Val": $year}"""
because it's unsafe to concat strings. Any illegal char might disintegrate the JSON! I don't want to deal with escaping illegal chars :-(
Thanks
Does this set of extension methods give you what you want?
#ImplicitReflectionSerializer
fun Map<*, *>.toJson() = Json.stringify(toJsonObject())
#ImplicitReflectionSerializer
fun Map<*, *>.toJsonObject(): JsonObject = JsonObject(map {
it.key.toString() to it.value.toJsonElement()
}.toMap())
#ImplicitReflectionSerializer
fun Any?.toJsonElement(): JsonElement = when (this) {
null -> JsonNull
is Number -> JsonPrimitive(this)
is String -> JsonPrimitive(this)
is Boolean -> JsonPrimitive(this)
is Map<*, *> -> this.toJsonObject()
is Iterable<*> -> JsonArray(this.map { it.toJsonElement() })
is Array<*> -> JsonArray(this.map { it.toJsonElement() })
else -> JsonPrimitive(this.toString()) // Or throw some "unsupported" exception?
}
This allows you to pass in a Map with various types of keys/values in it, and get back a JSON representation of it. In the map, each value can be a primitive (string, number or boolean), null, another map (representing a child node in the JSON), or an array or collection of any of the above.
You can call it as follows:
val json = mapOf(
"Album" to "Sergeant Pepper",
"Year" to 1967,
"TestNullValue" to null,
"Musicians" to mapOf(
"John" to arrayOf("Guitar", "Vocals"),
"Paul" to arrayOf("Bass", "Guitar", "Vocals"),
"George" to arrayOf("Guitar", "Sitar", "Vocals"),
"Ringo" to arrayOf("Drums")
)
).toJson()
This returns the following JSON, not prettified, as you wanted:
{"Album":"Sergeant Pepper","Year":1967,"TestNullValue":null,"Musicians":{"John":["Guitar","Vocals"],"Paul":["Bass","Guitar","Vocals"],"George":["Guitar","Sitar","Vocals"],"Ringo":["Drums"]}}
You probably also want to add handling for some other types, e.g. dates.
But can I just check that you want to manually build up JSON in code this way rather than creating data classes for all your JSON structures and serializing them that way? I think that is generally the more standard way of handling this kind of stuff. Though maybe your use case does not allow that.
It's also worth noting that the code has to use the ImplicitReflectionSerializer annotation, as it's using reflection to figure out which serializer to use for each bit. This is still experimental functionality which might change in future.
Please pardon me if this is a repeat question. I have been through some of the questions/answers with a similar requirement but somehow got a bit overwhelmed and confused at the same time. My requirement is:
I get a JSON string/object as a request parameter. ( eg: params.timesheetJSON )
I then have to parse/iterate through it.
Here is the JSON that my grails controller will be receiving:
{
"loginName":"user1",
"timesheetList":
[
{
"periodBegin":"2014/10/12",
"periodEnd":"2014/10/18",
"timesheetRows":[
{
"task":"Cleaning",
"description":"cleaning description",
"paycode":"payCode1"
},
{
"task":"painting",
"activityDescription":"painting description",
"paycode":"payCode2"
}
]
}
],
"overallStatus":"SUCCESS"
}
Questions:
How can I retrieve the whole JSON string from the request? Does request.JSON be fine here? If so, will request.JSON.timesheetJSON yield me the actual JSON that I want as a JSONObject?
What is the best way to parse through the JSON object that I got from the request? Is it grails.converters.JSON? Or is there any other easy way of parsing through? Like some API which will return the JSON as a collection of objects by automatically taking care of parsing. Or is programatically parsing through the JSON object the only way?
Like I said, please pardon me if the question is sounding vague. Any good references JSON parsing with grails might also be helpful here.
Edit: There's a change in the way I get the JSON string now. I get the JSON string as a request paramter.
String saveJSON // This holds the above JSON string.
def jsonObject = grails.converters.JSON.parse(saveJSON) // No problem here. Returns a JSONObject. I checked the class type.
def jsonArray = jsonArray.timesheetList // No problem here. Returns a JSONArray. I checked the class type.
println "*** Size of jsonArray1: " + jsonArray1.size() // Returns size 1. It seemed fine as the above JSON string had only one timesheet in timesheetList
def object1 = jsonArray[1] // This throws the JSONException, JSONArray[1] not found. I tried jsonArray.getJSONObject(1) and that throws the same exception.
Basically, I am looking to seamlessly iterate through the JSON string now.
I have wrote some code that explains how this can be done, that you can see below, but to be clear, first the answers to your questions:
Your JSON String as you wrote above will be the contents of your POST payload to the rest controller. Grails will use its data binding mechanism to bind the incomming data to a Command object that your should prepare. It has to have fields corresponding to the parameters in your JSON String (see below). After you bind your command object to your actual domain object, you can get all the data you want, by simply operating on fields and lists
The way to parse thru the JSON object is shown in my example below. The incomming request is esentially a nested map, with can be simply accessed with a dot
Now some code that illustrates how to do it.
In your controller create a method that accepts "YourCommand" object as input parameter:
def yourRestServiceMethod (YourCommand comm){
YourClass yourClass = new YourClass()
comm.bindTo(yourClass)
// do something with yourClass
// println yourClass.timeSheetList
}
The command looks like this:
class YourCommand {
String loginName
List<Map> timesheetList = []
String overallStatus
void bindTo(YourClass yourClass){
yourClass.loginName=loginName
yourClass.overallStatus=overallStatus
timesheetList.each { sheet ->
TimeSheet timeSheet = new TimeSheet()
timeSheet.periodBegin = sheet.periodBegin
timeSheet.periodEnd = sheet.periodEnd
sheet.timesheetRows.each { row ->
TimeSheetRow timeSheetRow = new TimeSheetRow()
timeSheetRow.task = row.task
timeSheetRow.description = row.description
timeSheetRow.paycode = row.paycode
timeSheet.timesheetRows.add(timeSheetRow)
}
yourClass.timeSheetList.add(timeSheet)
}
}
}
Its "bindTo" method is the key piece of logic that understands how to get parameters from the incomming request and map it to a regular object. That object is of type "YourClass" and it looks like this:
class YourClass {
String loginName
Collection<TimeSheet> timeSheetList = []
String overallStatus
}
all other classes that are part of that class:
class TimeSheet {
String periodBegin
String periodEnd
Collection<TimeSheetRow> timesheetRows = []
}
and the last one:
class TimeSheetRow {
String task
String description
String paycode
}
Hope this example is clear enough for you and answers your question
Edit: Extending the answer according to the new requirements
Looking at your new code, I see that you probably did some typos when writting that post
def jsonArray = jsonArray.timesheetList
should be:
def jsonArray = jsonObject.timesheetList
but you obviously have it properly in your code since otherwise it would not work, then the same with that line with "println":
jsonArray1.size()
shuold be:
jsonArray.size()
and the essential fix:
def object1 = jsonArray[1]
shuold be
def object1 = jsonArray[0]
your array is of size==1, the indexing starts with 0. // Can it be that easy? ;)
Then "object1" is again a JSONObject, so you can access the fields with a "." or as a map, for example like this:
object1.get('periodEnd')
I see your example contains errors, which lead you to implement more complex JSON parsing solutions.
I rewrite your sample to the working version. (At least now for Grails 3.x)
String saveJSON // This holds the above JSON string.
def jsonObject = grails.converters.JSON.parse(saveJSON)
println jsonObject.timesheetList // output timesheetList structure
println jsonObject.timesheetList[0].timesheetRows[1] // output second element of timesheetRows array: [paycode:payCode2, task:painting, activityDescription:painting description]
I tried to implement a CSV MediaTypeFormatter for my Web API as described here:
http://www.tugberkugurlu.com/archive/creating-custom-csvmediatypeformatter-in-asp-net-web-api-for-comma-separated-values-csv-format
(I don't want to paste in all the code from there)
But I don't get it to work using the Web API Controller below.
I used Fiddler to call the Web API with: http://myhostname.com/api/csvexport?format=csv
public dynamic Get()
{
var ret = new[] { "CarId", "Make", "Model", "Name" };
return ret;
}
For "type" in the CsvFormatter i get a:
DeclaringMethod = 'type.DeclaringMethod' threw an exception of type 'System.InvalidOperationException'
with
Method may only be called on a Type for which Type.IsGenericParameter is true.
So I might not get the concept of the Formatter right and have a problem with the type?
You are getting this error because Tugberk's formatter only works for models which implement the generic IEnumerable<T> interface. It makes sense, since people generally want CSV formatted data when they are getting a set of results. If you only want one data entity, why would you want it in CVS format?
Your method return type is dynamic, not IEnumerable<T>. You might be able to get it to work by doing something more like this:
public IEnumerable<string> Get()
{
var ret = new[] { "CarId", "Make", "Model", "Name" };
return ret;
}
When my app is fed syntactically incorrect JSON I want to be able to report the error to the user with some useful detail that will allow the problem area to be located.
So in this example j will be None because of the trailing comma after "File1". Is there a way to obtain details of last parse error?
val badSyntax = """
{
"menu1": {
"id": "file1",
"value": "File1",
},
"menu2": {
"id": "file2",
"value": "File2",
}
}"""
val j = JSON.parseFull(badSyntax)
When you get a parse error, use JSON.lastNoSuccess to get the last error. It is of type JSON.NoSuccess of which thare are two subclasses, JSON.Error and JSON.Failure, both containing a msg: String member detailing the error.
Note that JSON.lastNoSuccess is not thread safe (it is a mere global variable) and is now deprecated (bound to disappear in scala 2.11)
UPDATE: Apparently, I was wrong about it not being thread-safe: it was indeed not thread-safe before scala 2.10, but now lastNoSuccess is backed by a thread-local variable (and is thus safe to use in a multi-threaded context).
After seing this, you'd be forgiven to think that as long as you read right after a parsing failure in the same thread as the one that was used to do the parsing (the thread where you called parseFull), then everything will work as expected. Unfortunately, during this refactor they also changed how they use lastNoSuccess internally inside Parsers.phrase (which is called by JSON.parseFull.
See https://github.com/scala/scala/commit/1a4faa92faaf495f75a6dd2c58376f6bb3fbf44c
Since this refactor, lastNoSuccess is reset to None at the end of Parsers.phrase. This is no problem in parsers in general, as lastNoSuccess is used as a temporary value that is returned as the result of Parsers.phrase anyway.
The problem here is that we don't call Parsers.phrase, but JSON.parseFull, which drops any error info (see case None => None inside method JSON.parseRaw at https://github.com/scala/scala/blob/v2.10.0/src/library/scala/util/parsing/json/JSON.scala).
The fact that JSON.parseFull drops any error info could easily be circumvented prior to scala 2.10 by directly reading JSON.lastNoSuccess as I advised before, but now that this value is reset at the end of Parsers.phrase, there is not much you can do to get the error information out of JSON.
Any solution? Yes. What you can do is to create your very own version of JSON that will not drop the error information:
import util.parsing.json._
object MyJSON extends Parser {
def parseRaw(input : String) : Either[NoSuccess, JSONType] = {
phrase(root)(new lexical.Scanner(input)) match {
case Success(result, _) => Right(result)
case ns: NoSuccess => Left(ns)
}
}
def parseFull(input: String): Either[NoSuccess, Any] = {
parseRaw(input).right.map(resolveType)
}
def resolveType(input: Any): Any = input match {
case JSONObject(data) => data.transform {
case (k,v) => resolveType(v)
}
case JSONArray(data) => data.map(resolveType)
case x => x
}
}
I just changed Option to Either as the result type, so that I can return parsing errors as an Left. Some test in the REPL:
scala> MyJSON.parseFull("[1,2,3]")
res11: Either[MyJSON.NoSuccess,Any] = Right(List(1.0, 2.0, 3.0))
scala> MyJSON.parseFull("[1,2,3")
res12: Either[MyJSON.NoSuccess,Any] =
Left([1.7] failure: end of input
[1,2,3
^)
I want every JSON response to post-request to contain a field success. What's the best way to add this field there?
I use code like this to generate JSON responses:
try {
def entity = myService.saveEntity(arg1,arg2)
render entity as JSON //I want to add artificial field 'success = "yes"' here
} catch (ValidationException e) {
render parseErrors(e.errors) as JSON //field 'success = "no"' here
}
I just struggled with this exact issue this week. I wanted to send back a domain class as JSON but at the same time add an 'errorMessage' property that would potentially contain additional information.
Turns out that when using as JSON in grails it sends back a converter object, but its possible to turn that converter instance into a jsonObject using JSON.parse() which we can easily add new values to.
def jsonObject = JSON.parse((entity AS JSON).toString())
jsonObject.put("success", "yes")
render jsonObject as JSON
I think there are a couple of different approaches but this ended up being the easiest for me since I already have custom converters for most of my domain classes and I didn't want to add any other transient properties to my domain object.
Could you return a map containing the success field, and the object wrapped inside a separate variable:
try {
def entity = myService.saveEntity(arg1,arg2)
render [ success:'yes', val:entity ] as JSON
} catch (ValidationException e) {
render [ success:'no', val:parseErrors(e.errors) ] as JSON
}
Not tested it mind...
You can register your own JSON marshaller (at BootStrap.groovy, for example), like:
JSON.registerObjectMarshaller(MyEntity) { MyEntity it ->
return [
someField : it.someField, // you should specify fields for output, or put all '.properties'
success : true // as I understand you always have 'true' for real entity
]
}
where MyEntity is your class you want to use