I want to load & parse a JSON file with scala 2.11.8, in a generic way, like this:
private val objectMapper = new ObjectMapper with ScalaObjectMapper
objectMapper.registerModule(DefaultScalaModule)
def loadFile[T](path: Path): Try[T] = Try(
objectMapper.readValue(Files.readAllBytes(path), classOf[T])
)
Then the goal is to call the loadFile method with only the expected return type.
However this returns me:
class type required but T found
By googling, I found references to erasures, manifests, ClassTag but nothing works. What is the correct solution?
The generic type gets erased, so you need a ClassTag to make it work. This is how you can use them:
def loadFile[T: ClassTag](path: Path): Try[T] = Try(
objectMapper.readValue(
Files.readAllBytes(path),
implicitly[ClassTag[T]].runtimeClass.asInstanceOf[Class[T]])
)
(For some reason, runtimeClass doesn't have the generic type, so you need the cast.)
Related
I'm trying to write a method that will allow Jackson ObjectMapper readValue on a json string to a parameterized object type. Something like this
case class MyObj(field1: String, field2: String)
val objectMapper: ObjectMapper = new ObjectMapper().registerModule(new DefaultScalaModule)
def fromJson[T](jsonString: String, objTyp: T): T = {
objectMapper.readValue(jsonString, classOf[T])
}
val x = fromJson("""{"field1": "something", "field2": "something"}""", MyObj)
This of course returns an error of
class type required but T found
i've looked at this issue Scala classOf for type parameter
but it doesn't seem to help. It seems like this is possible to do somehow. Looking for any help
You have to give it the actual runtime class to parse into, not just a type parameter.
One way to do it is passing the class directly:
def fromJson[T](json: String, clazz: Class[T]) = objectMapper.readValue[T](json, clazz)
val x = fromJson("""...""", classOf[MyObj])
Alternatively, you can use ClassTag, which looks a bit messier in implementation, but kinda prettier at call site:
def fromJson[T : ClassTag](json: String): T = objectMapper.readValue[T](
json,
implicitly[ClassTag[T]].runtimeClass.asInstanceOf[Class[T]]
)
val x = fromJson[MyObj]("""{"field1": "something", "field2": "something"}""")
i've looked at this issue Scala classOf for type parameter but it doesn't seem to help.
In the very first answer there it's written classTag[T].runtimeClass as a replacement of classOf[T]. This should help.
Regarding the signature
def fromJson[T](jsonString: String, objTyp: T): T
You should notice that MyObj has type MyObj.type (companion-object type), not MyObj (case-class type).
Class companion object vs. case class itself
So if you call fromJson("""...""", MyObj) then the types in these two places
def fromJson[...](jsonString: String, objTyp: ???): ???
^^^ ^^^ <--- HERE
can't be the same.
If it's enough for you to call
fromJson("""...""", classOf[MyObj])
or
fromJson[MyObj]("""...""")
(normally it should be enough) then please see #Dima's answer, you should prefer those options, they're easier.
Just in case, if you really want to call like fromJson("""...""", MyObj) then for example you can use the type class ToCompanion (this is more complicated) from
Invoke construcotr based on passed parameter
Get companion object of class by given generic type Scala (answer)
// ToCompanion should be defined in a different subproject
def fromJson[C, T](jsonString: String, objTyp: C)(implicit
toCompanion: ToCompanion.Aux[C, T],
classTag: ClassTag[T]
): T =
objectMapper.readValue(jsonString, classTag.runtimeClass.asInstanceOf[Class[T]])
val x = fromJson("""{"field1": "something", "field2": "something"}""", MyObj)
// MyObj(something,something)
I'm using Jackson in my Scala/Spark program and I've distilled my issue to the simple example
below. My problem is that when my case class has the Option[Int] field (age) set to None
I see reasonable deserialization output (that is: a struct with empty=true). However, when
age is defined, i.e., set to some Int like Some(99), I never see the integer value in the
deserialization output .
Given :
import com.fasterxml.jackson.databind.ObjectMapper
import java.io.ByteArrayOutputStream
import scala.beans.BeanProperty
case class Dog(#BeanProperty name: String, #BeanProperty age: Option[Integer])
object OtherTest extends App {
jsonOut(Dog("rex", None))
jsonOut(Dog("mex", Some(99)))
private def jsonOut(dog: Dog) = {
val mapper = new ObjectMapper();
val stream = new ByteArrayOutputStream();
mapper.writeValue(stream, dog);
System.out.println("result:" + stream.toString());
}
}
My output is as shown below. Any hints/help greatly appreciated !
result:{"name":"rex","age":{"empty":true,"defined":false}}
result:{"name":"mex","age":{"empty":false,"defined":true}}
Update after the Helpful Answer
Here are the dependencies that worked for me:
implementation 'org.scala-lang:scala-library:2.12.2'
implementation "org.apache.spark:spark-sql_2.12:3.1.2"
implementation "org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2"
implementation "org.apache.spark:spark-avro_2.12:3.1.2"
implementation 'com.fasterxml.jackson.module:jackson-module-scala_2.12:2.10.0'
Here is the updated code (with frequent flyer bonus - round trip example):
private def jsonOut(dog: Dog) = {
val mapper = new ObjectMapper()
mapper.registerModule(DefaultScalaModule)
val stream = new ByteArrayOutputStream();
mapper.writeValue(stream, dog);
val serialized = stream.toString()
System.out.println("result:" + serialized);
// verify we can read the serialized thing back to case class:
val recovered = mapper.readValue(serialized, classOf[Dog])
System.out.println("here is what we read back:" + recovered);
}
Here is the resultant output (as expected now ;^) ->
> Task :OtherTest.main()
result:{"name":"rex","age":null}
here is what we read back:Dog(rex,None)
result:{"name":"mex","age":99}
here is what we read back:Dog(mex,Some(99))
You need to add the Jackson module for Scala to make it work with standard Scala data types.
Add this module as your dependency: https://github.com/FasterXML/jackson-module-scala
Follow the readme on how to initialize your ObjectMapper with this module.
I'm trying to serialize/deserialize a Scala class to JSON using Jackson ObjectMapper. The serialization works fine, but I was getting type exceptions trying to read the JSON back in. I fixed most of those by adding appropriate annotations, but it's not working for my Map members... it seems like Jackson is trying to treat the keys in the JSON object as properties in a class instead of keys in a map. (I believe this is different than other questions like this one since they are calling readValue on the map contents directly.)
Here's my ObjectMapper setup:
val mapper = new ObjectMapper() with ScalaObjectMapper
mapper.registerModule(DefaultScalaModule)
Here's what my annotated class and member look like:
#JsonInclude(JsonInclude.Include.NON_DEFAULT)
class MyClass extends Serializable {
#JsonDeserialize(
as = classOf[mutable.HashMap[String, Long]],
keyAs = classOf[java.lang.String],
contentAs = classOf[java.lang.Long]
)
val counts = mutable.Map.empty[String, Long]
}
If I give it some JSON like:
{"counts":{"foo":1,"bar":2}}
And read it with mapper.readValue[MyClass](jsonString)
I get an exception like UnrecognizedPropertyException: Unrecognized field "foo" (class mutable.HashMap), not marked as ignorable.
I tried adding DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES to my mapper configuration but that didn't seem to do anything in this case, and I'm not sure that kind of global setting is desirable.
How do I convince Jackson to treat the strings "foo" and "bar" as keys in the map member field and not as properties in the HashMap class? It seems to have done the right thing automatically writing it out.
Also worth noting: the deserialization appears to work fine in a quick out/in unit test to a temp file or a string variable, but not when I try to run the whole application and it reads the JSON its previously written. I don't know why it seems to work in the test, as far as I know it's making the same readValue call.
I made one simple test like this:
case class TestClass (counts: mutable.HashMap[String, Long])
And I converted it like:
val objectMapper = new ObjectMapper() with ScalaObjectMapper
objectMapper.registerModule(DefaultScalaModule)
val r3 = objectMapper.readValue("{\"counts\":{\"foo\":1,\"bar\":2}}", classOf[TestClass])
And apparently it works for me. Maybe it's something about the version you're using of Jackson, or Scala. Have you tried different versions of Jackson for example?
Try jsoniter-scala and you will enjoy how it can be handy, safely, and efficient to parse and serialize JSON these days with Scala: https://github.com/plokhotnyuk/jsoniter-scala
One of it's crucial features is an ability to generate codecs in compile time and even print their sources. You will have no any runtime magic like reflection or byte code replacement that will affect your code.
My problem was a race condition due to not using chaining in my mapper configuration singleton.
My old code was more like this:
private var mapper: ObjectMapper with ScalaObjectMapper = _
def getMapper: ObjectMapper with ScalaObjectMapper = {
if (mapper == null) {
mapper = new ObjectMapper() with ScalaObjectMapper
mapper.registerModule(DefaultScalaModule)
mapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)
}
mapper
}
As you can see, if one thread initializes the mapper, but hasn't yet disabled unknown properties failure, a second thread could return and use a mapper that hasn't had that flag set yet, which explains why I was seeing the error only some of the time.
The correct code uses chaining so that the mapper singleton is set with all of the configuration:
private var mapper: ObjectMapper = _
def getMapper: ObjectMapper = {
if (mapper == null) {
mapper = new ObjectMapper()
.registerModule(DefaultScalaModule)
.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)
}
mapper
}
(I also removed the experimental ScalaObjectMapper mix-in.)
I seem to be able to use Jackson to make a mapper of Json-String --> scala.collection.Map.
How can I hook up that same mapper to a RestTemplate?
val restTemplate = new RestTemplate()
val module = new OptionModule with MapModule with SeqModule with IteratorModule
val mapper = new ObjectMapper()
mapper.registerModule(module)
// Get some example JSON
val uri = "http://...."
val response:String = restTemplate.getForObject(uri, classOf[String] )
// *** success ***
// Use the mapper directly: String --> scala.collection.Map
val map1 = mapper.readValue(response, classOf[scala.collection.Map[String, Any]])
// Try hooking up the same module to the RestTemplate:
val wrappingConverter = new WrappingHttpMessageConverter()
wrappingConverter.getObjectMapper().registerModule(module)
val list = restTemplate.getMessageConverters()
list.add(wrappingConverter)
restTemplate.setMessageConverters(list)
// *** FAILS ***
// org.springframework.http.converter.HttpMessageNotReadableException: Could not read
// JSON: Can not construct instance of scala.collection.Map, problem: abstract types
// either need to be mapped to concrete types, have custom deserializer, or be
// instantiated with additional type information
val map2 = restTemplate.getForObject(uri, classOf[scala.collection.Map[String, Any]] )
Assumtions
WrappingHttpMessageConverter is a derived class of MappingJackson2HttpMessageConverter, or some class that works like it.
You're using Spring 4.0 (though this answer is probably also true for Spring 3.2)
The problem
The default RestTemplate constructor tries to detect if Jackson is on your classpath, and if it is, adds MappingJackson2HttpMessageConverter to the MessageConverters list. Since it's already on the list, it's going to be used before your WrappingHttpMessageConverter is ever checked.
That default converter doesn't have the Scala module installed. Here's where things get tricky. HttpMessageConverterExtractor tries to ask if the first converter can deserialize the type; currently ObjectMapper returns true for this test (whether it should is a much longer topic, not as clear cut as it might seem). The extractor doesn't handle the idea that one converter could fail, but a later one might succeed (as it would in your case).
Workarounds
You need to make sure that Spring finds an ObjectMapper configured with the Scala module before it tries with any other. You can do this in a number of ways; the most robust is to search the preconfigured converters and update the first you find, adding a new one if you don't find any:
val jacksonConverter = list.asScala.collectFirst {
case p: MappingJackson2HttpMessageConverter => p
}
if (jacksonConverter.isDefined) {
jacksonConverter.get.getObjectMapper.registerModule(module)
}
else {
list.add(new MappingJackson2HttpMessageConverter) // or your derived class if you prefer
}
Other options include adding your custom message converter to the front of the list, or removing the existing Jackson converter before adding your own.
I'm new to Scala and I'm trying to do my first Play 2.0 application. I'm using:
Scala 2.9.3
Play 2.0.4
SalatWithPlay2 1.1
I have this in my conf/routes file:
PUT /newVenue controllers.Application.createNewVenue
I have this in my Application.scala file:
def createNewVenue = Action(parse.json) { request =>
val newVenue = Venue.fromJSON(request.body.toString)
Venue.insert(newVenue)
Ok("New Venue Created")
}
And this is the code for Venue.scala:
import play.api.Play.current
import com.novus.salat._
import com.novus.salat.global._
import com.novus.salat.annotations._
import com.novus.salat.dao._
import com.mongodb.casbah.Imports._
import se.radley.plugin.salat._
object Venue extends ModelCompanion[Venue, ObjectId] {
val collection = mongoCollection("venues")
val dao = new SalatDAO[Venue, ObjectId](collection = collection) {}
}
case class Venue(
#Key("_id") val venueId:ObjectId,
var playlist:Playlist,
var isPlaying:Boolean = false)
To test if this is working I send a PUT request to localhost:9000/newVenue with this JSON as a body:
{"venueId": 3,"playlist":{"playlistId":2,"currentSongPosition":5},"isPlaying":false}
And then I get this error:
[error] application -
! #6d7oco1hf - Internal server error, for request [PUT /newVenue] ->
play.core.ActionInvoker$$anonfun$receive$1$$anon$1: Execution exception [[NoSuchMethodException: model.Venue$.apply$default$1()]]
at play.core.ActionInvoker$$anonfun$receive$1.apply(Invoker.scala:134) [play_2.9.1.jar:2.0.4]
at play.core.ActionInvoker$$anonfun$receive$1.apply(Invoker.scala:115) [play_2.9.1.jar:2.0.4]
at akka.actor.Actor$class.apply(Actor.scala:318) [akka-actor.jar:2.0.2]
at play.core.ActionInvoker.apply(Invoker.scala:113) [play_2.9.1.jar:2.0.4]
at akka.actor.ActorCell.invoke(ActorCell.scala:626) [akka-actor.jar:2.0.2]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:197) [akka-actor.jar:2.0.2]
Caused by: java.lang.NoSuchMethodException: model.Venue$.apply$default$1()
at java.lang.Class.getMethod(Class.java:1605) ~[na:1.6.0_37]
at com.novus.salat.ConcreteGrater.defaultArg(Grater.scala:350) ~[salat-core_2.9.1-1.9.1.jar:1.9.1]
at com.novus.salat.ConcreteGrater.safeDefault(Grater.scala:360) ~[salat-core_2.9.1-1.9.1.jar:1.9.1]
at com.novus.salat.ConcreteGrater$$anonfun$6$$anonfun$apply$7.apply(Grater.scala:319) ~[salat-core_2.9.1-1.9.1.jar:1.9.1]
at com.novus.salat.ConcreteGrater$$anonfun$6$$anonfun$apply$7.apply(Grater.scala:319) ~[salat-core_2.9.1-1.9.1.jar:1.9.1]
at scala.Option.orElse(Option.scala:218) ~[scala-library.jar:0.11.3]
I think the line triggering the error is val newVenue = Venue.fromJSON(request.body.toString).
Does anyone have a clue of what's wrong?
I followed the tutorial in SalatWithPlay2 page and also followed some advices in similar problems), but I had no luck so far.
UPDATE:
I found a workaround, although it's not exactly a solution, but maybe it's helpful for somebody else and for understanding what is the actual solution.
If I remove the annotation #Key, the code of the case class Venue looks like this:
case class Venue(
val venueId:ObjectId,
var playlist:Playlist,
var isPlaying:Boolean = false)
And then I get this other error:
[RuntimeException: in: unexpected OID input class='net.liftweb.json.JsonAST$JInt', value='3']
And if instead of using ObjectId I use Long for example, the code looks like this:
case class Venue(
val venueId:Long,
var playlist:Playlist,
var isPlaying:Boolean = false)
And I get no error!
So the NoSuchMethodException apparently is related to the #Key annotation related to the _id. I also tried to rename venueId to _id and the same NoSuchMethodException error appeared. So the question is: why can't I use the #Key annotation to say which of my attributes maps to the object id of the document in the mongo database?
Besides that, regular Integers cannot be automatically converted to ObjectId instances for some other reason.
Thanks!
Just a shot in the dark, but can you try naming your object class to something else? Maybe to VenueCompanion? It seems it is possible the compiler is getting confused on which object to apply to. IE it is not applying to the case class Venue, that you want. I am curious if the exception is still the same afterwards.
So I found what was the problem.
The attributes of the Json sent in the body of the request must match the names of the attributes of the mongodb's document, not the attributes of the Scala object.
So if my class Venue is defined like this:
case class Venue(
#Key("_id") val venueId:ObjectId,
var playlist:Playlist,
var isPlaying:Boolean = false)
The Json I have to send in the PUT request is:
{"_id": 3,"playlist":{"playlistId":2,"currentSongPosition":5},"isPlaying":false}
And NOT:
{"venueId": 3,"playlist":{"playlistId":2,"currentSongPosition":5},"isPlaying":false}