I`ve found some questions that are very close to my question (e.g. Play Framework / Scala: abstract repository and Json de/serialization) but they didnt solve my problem.
What I`m trying to achieve is an abstraction of my CRUD DAO for the common crud operations.
I build a GenericMongoDaoActor for that:
abstract class GenericMongoDaoActor[T <: Entity: Writes](implicit inj:Injector, implicit val f:Format[T]) extends Actor with Injectable {
protected val db = inject[DefaultDB]
protected val collectionName: String
protected val collection:JSONCollection
//set to None to use fallback
def defaultSelector(obj:T):Option[JsObject] = None
def fallbackSelector(obj:T) = Json.obj("_id" -> Json.obj("$elemMatch" -> obj._id))
protected def find(jsObject: JsObject) = {
val currentSender = sender
val futureOption = collection
.find(jsObject)
.cursor[T](ReadPreference.primaryPreferred)
.headOption
futureOption.mapTo[Option[T]].flatMap {
case Some(pobj) =>
currentSender ! Some(pobj)
Future{pobj}
case None =>
currentSender ! None
Future{None}
}
}
protected def save(obj:T):Unit = {
update(obj, true)
}
protected def update(obj:T):Unit = {
update(obj, false)
}
private def update(obj:T, upsert: Boolean):Unit = {
val selector = defaultSelector(obj) match {
case None => fallbackSelector(obj)
case Some(sel) => sel
}
collection.update(
selector,
obj,
GetLastError.Default,
upsert).onComplete {
case Failure(e) => Logger.error("[" + this.getClass.getName + "] Couldn`t update " + obj.getClass.getName + ": " + Json.prettyPrint(Json.toJson(obj)), e)
case Success(lastError) => //currentSender ! lastError todo: do something with lastError
}
}
def findAll() = {
collection.find(Json.obj()).cursor[T](ReadPreference.primaryPreferred).collect[List]()
}
}
The DAOActor handles Entities that inherit abstract class "Entity":
abstract class Entity {
val _id: BSONObjectID
}
Currently there are 2 classes that inherit Entity..
As you can see my DOAActor is already context bound to look for Writes[T] in scope..
abstract class GenericMongoDaoActor[T <: Entity: Writes]
When I try to build my project like that, it complains that there are no OWrites given to serialize "obj" of type "T" in the update method.
No Json serializer as JsObject found for type T. Try to implement an implicit OWrites or OFormat for this type.
collection.update( <-------------
I couldnt find a way to solve this issue. Please let me know if you can.
I had similar problems when I migrated from an earlier version of ReactiveMongo. What worked for me was sprinkling around some .as[JsObject] conversions in the various calls to the ReactiveMongo API.
So if before I had:
collection.update(
selector,
obj,
...
)
I replaced it with:
collection.update(
selector,
obj.as[JsObject],
...
)
This seemed to be sufficient, although I am supplying the necessary JSON converter(s) in a slightly different way to you; subclasses of my abstract class have to implement an implicit val fmt:Format[T] member. I doubt whether that is important, but it is an approach that seems to be working :-)
You need to use OWrites and OFormat instead of Writes and Format.
I know OWrites extends Writes and OFormat extends Format, but the reactivemongo version you are using is waiting for OWrites and OFormat, not its super types.
Related
I am using Play Framework (Scala) for a micro-service and am using Kafka as the event bus. I have an event consumer that maps to an event class that looks like:
case class MovieEvent[T] (
mediaId: String,
config: T
)
object MovieEvent {
implicit def movieEventFormat[T: Format]: Format[MovieEvent[T]] =
((__ \ "mediaId").format[String] ~
(__ \ "config").format[T]
)(MovieEvent.apply _, unlift(MovieEvent.unapply))
}
object MovieProvider extends SerializableEnumeration {
implicit val providerReads: Reads[MovieProvider.Value] = SerializableEnumeration.jsonReader(MovieProvider)
implicit val providerWrites: Writes[MovieProvider.Value] = SerializableEnumeration.jsonWrites
val Dreamworks, Disney, Paramount = Value
}
The consumer looks like:
class MovieEventConsumer #Inject()(movieService: MovieService
) extends ConsumerRecordProcessor with LazyLogging {
override def process(record: IncomingRecord): Unit = {
val movieEventJson = Json.parse(record.valueString).validate[MovieEvent[DreamworksConfiguration]]
movieEventJson match {
case event: JsSuccess[MovieEvent[DreamworksJobOptions]] => processMovieEvent(event.get)
case er: JsError =>
logger.error("Unrecognized MovieEvent, attempting to parse as MovieUploadEvent: " + JsError.toJson(er).toString())
try {
val data = (Json.parse(record.valueString) \ "upload").as[MovieUploadEvent]
processUploadEvent(data)
} catch {
case er: Exception => logger.error("Unrecognized kafka event", er)
}
}
}
def processMovieEvent[T](event: MovieEvent[T]): Unit = {
logger.debug(s"Received movie event: ${event}")
movieService.createMovieJob(event)
}
def processUploadEvent(event: MovieUploadEvent): Unit = {
logger.debug(s"Received upload event: ${event}")
movieService.addToCollection(event)
}
}
Right now, I can only validate one of the three different MovieEvent configurations (Dreamwork, Disney, and Paramount). I can swap out which one I validate through the code but thats not the point. However, I would like to validate any of the three without having to make additional consumers. I've tried playing with a few different ideas but none of them compile. I'm pretty new to Play and Kafka and wondering if there is a good way to do this.
Thanks in advance!
I am going to assume that the number of possible configurations is finite and all known in compile time (in your example, 3).
One possibility is to make MovieEvent a sealed trait with a generic type T. Here's a minimal example:
case class DreamWorksJobOptions(anOption: String, anotherOption: String)
case class DisneyJobOptions(anOption: String)
sealed trait MovieEvent[T] {
def mediaId: String
def config: T
}
case class DreamWorksEvent(mediaId: String, config: DreamWorksJobOptions) extends MovieEvent[DreamWorksJobOptions]
case class DisneyEvent(mediaId: String, config: DisneyJobOptions) extends MovieEvent[DisneyJobOptions]
def tryParse(jsonString: String): MovieEvent[_] = {
// ... parsing logic goes here
DreamWorksEvent("dw", DreamWorksJobOptions("some option", "another option"))
}
val parseResult = tryParse("asdfasdf")
parseResult match {
case DreamWorksEvent(mediaId, config) => println(mediaId + " : " + config.anOption + " : " + config.anotherOption)
case DisneyEvent(mediaId, config) => println(mediaId + config)
}
which prints out
dw : some option : another option
I omitted the parsing part because I don't have access to Play Json atm. But since you have a sealed hierarchy, you can try each of your options one by one. (And you pretty much have to, since we cannot guarantee statically that DreamWorksEvent doesn't have the same Json structure as DisneyEvent - you need to decide which gets tried first and fallback to parsing the JSON as another type when the first fails to parse).
Now your other code is very generic. To add a new Event type you just need to add another subclass to MovieEvent, as well as making sure your parsing logic handles that new case. The magic here is that you don't have to specify your T when referring to MovieEvent, since you know you have a sealed hierachy and can thus recover T through pattern matching.
I'm Java developer and pretty new to scala.
I'm implementing some rest API that use spray and akka
The API should expose some kind of user CRUD. I'll use only create user in this question...
trait DefaultJsonFormats extends DefaultJsonProtocol with SprayJsonSupport with MetaMarshallers {}
class RegistrationService(registration: ActorRef)
(implicit executionContext: ExecutionContext)
extends Directives with DefaultJsonFormats {
implicit val timeout = Timeout(2.seconds)
implicit val userFormat = jsonFormat3(User)
implicit val registerFormat = jsonFormat1(Register)
implicit val registeredFormat = jsonFormat1(Registered)
val route =
path("register") {
post { handleWith { ru: Register => (registration ? ru).mapTo[Registered] } }
}
//------ Actor
object RegistrationActor {
case class User(id:String, name:String)
case class Register(user: User)
case class Registered(status: String)
case object NotRegistered
}
class RegistrationActor(implDef: String) extends Actor {
def receive: Receive = {
case Register(user)=>
val status=// create user real code with return status
sender ! new Registered(status)
} }
In this approach the json serialization and desiarelization is pretty annoying. For every object I need to deal with API I must define the appropriate format
implicit val userFormat = jsonFormat3(User)
implicit val registerFormat = jsonFormat1(Register)
implicit val registeredFormat = jsonFormat1(Registered)
I would like to avoid such definition and use some general json converter and return a pojo objects, so the conversion will happen under-the-hood
The question is how can I change this code to use by default Gson/Jackson/Spray default converter and avoid definition of the implicit ... jsonFormats?
For every object I need to deal with API I must define the appropriate format
It is normal to do this once, in a "JsonProtocol" class and import that where needed, rather than defining new formats each time:
import MyJsonProtocol._
val route =
path("register") {
post { handleWith { ru: Register => (registration ? ru).mapTo[Registered] } }
how can I change this code to use by default Gson/Jackson/Spray default converter and avoid definition of the implicit ... jsonFormats?
You would need to declare an implicit marshaller from Registered to HttpResponse (or an intermediate value like String) which was backed by Jackson instead of spray-json, then import that marshaller instead of SprayJsonSupport.
Have a look at the implementation of SprayJsonSupport to see how to do this. It's fairly straightforward, if you're comfortable with implicit conversions.
You can also see how this is done in Json4sSupport in Spray -- that trait implements a Marshaller[T, String] for ALL types T. Then, at runtime, the Json4s library will try to serialize the object to JSON.
In this approach the json serialization and desiarelization is pretty annoying
There are two main advantages of spray-jsons approach over Jackson's:
There is no reflection, so it is faster at runtime
This is no runtime determining of JSON formats, so any issues are caught at compile-time
[edit]
So, i got a quick and dirty solution, thanks to Edmondo1984, I don't know if it's the best solution. I don't handle null values with pattern matching at the write function. You can read more details about my problem after this editing. Here is my code now:
object DBNames extends DefaultJsonProtocol {
implicit val pkFormat: JsonFormat[Pk[Int]] = new JsonFormat[Pk[Int]] {
def write(obj: Pk[Int]): JsValue = JsNumber(obj.get)
def read(json: JsValue): Pk[Int] = json.asJsObject.getFields("id") match {
case Seq(JsNumber(id)) => new Pk[Int] { id.toInt }
case _ => throw new DeserializationException("Int expected")
}
}
implicit val nameFormat = jsonFormat2(Name)
jsonFormat2 will implicitly use pkFormat to parse Pk[Int] values.
In my controller class I have this:
def listNames() = Action {
val names = DBNames.findAll()
implicit val writer = DBNames.nameFormat
var json = names.toJson
Ok(json.toString()).as("application/json")
}
I had to get the nameFormat from my model and make it implicit, so bars.toJson could use it to parse the Seq[Name] names.
[/edit]
I'm trying to use Play! Framework with Scala, I'm new to Scala programming and Play Framework, and everything seems nice, but I'm working on this problem during several hours and didn't find a solution.
I have a Case Class:
case class Name (id: Pk[Int], name: String)
And an object to deal with MySql. I created a implicit val nameFormat = jsonFormat2(Name) to deal with JSON.
object DBNames extends DefaultJsonProtocol {
implicit val nameFormat = jsonFormat2(Name)
var parser =
{
get[Pk[Int]]("id") ~
get[String]("name") map {
case id ~ name => Name(id,name)
}
}
def findAll():Seq[Name] =
{
DB.withConnection {
implicit connection =>
SQL("select * from names").as(DBNames.parser *)
}
}
def create(name: Name){
DB.withConnection {
implicit connection =>
SQL("insert into names (name) values ({name})").on(
'name -> name.name
).executeUpdate()
}
}
}
But when I try to compile it, Play! gives me this result:
[error] D:\ProjetosJVM\TaskList\app\models\Names.scala:20: could not find implicit value for evidence parameter of type models.DBNames.JF[anorm.Pk[Int]]
It seems like he couldn't find a way to parse the id value, since it is a Pk[Int] value.
So, by reading this: https://github.com/spray/spray-json I didn't found a way to parse it without creating a complete object parser like they show in the documentation:
object MyJsonProtocol extends DefaultJsonProtocol {
implicit object ColorJsonFormat extends RootJsonFormat[Color] {
def write(c: Color) = JsObject(
"name" -> JsString(c.name),
"red" -> JsNumber(c.red),
"green" -> JsNumber(c.green),
"blue" -> JsNumber(c.blue)
)
def read(value: JsValue) = {
value.asJsObject.getFields("name", "red", "green", "blue") match {
case Seq(JsString(name), JsNumber(red), JsNumber(green), JsNumber(blue)) =>
new Color(name, red.toInt, green.toInt, blue.toInt)
case _ => throw new DeserializationException("Color expected")
}
}
}
}
I have a "big" (actually small) project where I want to make most of things work with Ajax, so I think this is not a good way to do it.
How can I deal with JSON objects in this project, where almost all case classes will have a "JSON parser", without creating large ammounts of code, like the snippet above? And also, how can I make it work with an Seq[Name]?
You don't need to write a complete parser. The compiler says:
[error] D:\ProjetosJVM\TaskList\app\models\Names.scala:20: could not find implicit
value for evidence parameter of type models.DBNames.JF[anorm.Pk[Int]]
The scala compiler is looking for an implicit parameter of type JF[anorm.Pk[Int]] and there is no such an implicit parameter in scope. What is JF[anorm.Pk[Int]]? Well, you need to know the library and I didn't, so I had browsed spray-json source and found out:
trait StandardFormats {
this: AdditionalFormats =>
private[json] type JF[T] = JsonFormat[T] // simple alias for reduced verbosity
so JF[T] is just an alias for JsonFormat[T]. It all make sense: PK[Int] is a class coming from Anorm and spray-json provides out-of-the-box json support for standard types, but does not even know Anorm exists So you have to code your support for Pk[Int] and make it implicit in scope.
You will have code like the following:
object DBNames extends DefaultJsonProtocol {
implicit val pkFormat : JsonFormat[Pk[Int]] = new JsonFormat[Pk[Int]] {
//implementation
}
// rest of your code
}
If you have just started with Scala, you would probably have to read more about implicits and their resolution. I am providing you with a minimal answer: once you have provided the right implementation, your code will compile. I suggest you to refer to the javadoc of anorm.Pk and of JsonFormat to understand how to implement it correctly for your type.
Pk looks like scala.Option and in StandardFormats source code inside spray-json you find the JsonFormat implementation for Option, from which you can copy
I am trying to use spray-json in scala to recognize the choice between Ec2Provider and OpenstackProvider when converting to Json and back.
I would like to be able to give choices in "Provider", and if those choices don't fit the ones available then it should not validate.
My attempt at this can be seen in the following code:
import spray.json._
import DefaultJsonProtocol._
case class Credentials(username: String, password: String)
abstract class Provider
case class Ec2Provider(endpoint: String,credentials: Credentials) extends Provider
case class OpenstackProvider(credentials: Credentials) extends Provider
case class Infrastructure(name: String, provider: Provider, availableInstanceTypes: List[String])
case class InfrastructuresList(infrastructures: List[Infrastructure])
object Infrastructures extends App with DefaultJsonProtocol {
implicit val credFormat = jsonFormat2(Credentials)
implicit val ec2Provider = jsonFormat2(Ec2Provider)
implicit val novaProvider = jsonFormat1(OpenstackProvider)
implicit val infraFormat = jsonFormat3(Infrastructure)
implicit val infrasFormat = jsonFormat1(InfrastructuresList)
println(
InfrastructuresList(
List(
Infrastructure("test", Ec2Provider("nova", Credentials("user","pass")), List("1", "2"))
)
).toJson
)
}
Unfortunately, it fails because it can not find a formatter for Provider abstract class.
test.scala:19: could not find implicit value for evidence parameter of type Infrastructures.JF[Provider]
Anyone have any solution for this?
What you want to do is not available out of the box (i.e. via something like type hints that allow the deserializer to know what concrete class to instantiate), but it's certainly possible with a little leg work. First, the example, using a simplified version of the code you posted above:
case class Credentials(user:String, password:String)
abstract class Provider
case class Ec2Provider(endpoint:String, creds:Credentials) extends Provider
case class OpenstackProvider(creds:Credentials) extends Provider
case class Infrastructure(name:String, provider:Provider)
object MyJsonProtocol extends DefaultJsonProtocol{
implicit object ProviderJsonFormat extends RootJsonFormat[Provider]{
def write(p:Provider) = p match{
case ec2:Ec2Provider => ec2.toJson
case os:OpenstackProvider => os.toJson
}
def read(value:JsValue) = value match{
case obj:JsObject if (obj.fields.size == 2) => value.convertTo[Ec2Provider]
case obj:JsObject => value.convertTo[OpenstackProvider]
}
}
implicit val credFmt = jsonFormat2(Credentials)
implicit val ec2Fmt = jsonFormat2(Ec2Provider)
implicit val openStackFmt = jsonFormat1(OpenstackProvider)
implicit val infraFmt = jsonFormat2(Infrastructure)
}
object PolyTest {
import MyJsonProtocol._
def main(args: Array[String]) {
val infra = List(
Infrastructure("ec2", Ec2Provider("foo", Credentials("me", "pass"))),
Infrastructure("openstack", OpenstackProvider(Credentials("me2", "pass2")))
)
val json = infra.toJson.toString
val infra2 = JsonParser(json).convertTo[List[Infrastructure]]
println(infra == infra2)
}
}
In order to be able to serialize/deserialize instances of the abstract class Provider, I've created a custom formatter where I am supplying operations for reading and writing Provider instances. All I'm doing in these functions though is checking a simple condition (binary here as there are only 2 impls of Provider) to see what type it is and then delegating to logic to handle that type.
For writing, I just need to know which instance type it is which is easy. Reading is a little trickier though. For reading, I'm checking how many properties the object has and since the two impls have different numbers of props, I can differentiate which is which this way. The check I'm making here is very rudimentary, but it shows the point that if you can look at the Json AST and differentiate the types, you can then pick which one to deserialize to. Your actual check can be as simple or as complicated as you like, as long as it's is deterministic in differentiating the types.
I've got following case:
I'd like to serialize Scala case classes that extend parent class with var of type java.util.UUID.
Serialization of this case classes should happen without any configuration of them - no annotations and definition of custom formats. Any serialization hints may be situated in parent class.
I tried sjson, but Reflection based serialization can't serialize UUID types and type based serialization forces me to define formats for every case class.
Which json serialization library would best fit this case?
Here's one solution with Lift JSON.
import java.util.UUID
import net.liftweb.json._
import net.liftweb.json.JsonAST._
import net.liftweb.json.JsonDSL._
import net.liftweb.json.Serialization._
sealed abstract class Parent {
def uuid: UUID
}
case class Foo(uuid: UUID, name: String) extends Parent
object UUIDTest extends Application {
implicit val formats = Serialization.formats(NoTypeHints) + new UUIDSerializer
val f = Foo(UUID.randomUUID, "foo")
val ser = write(f)
println(ser)
val f2 = read[Foo](ser)
assert(f == f2)
// Special serializer for UUID type
class UUIDSerializer extends Serializer[UUID] {
private val Class = classOf[UUID]
def deserialize(implicit format: Formats): PartialFunction[(TypeInfo, JValue), UUID] = {
case (TypeInfo(Class, _), json) => json match {
case JObject(JField("mostSig", JInt(m)) :: JField("leastSig", JInt(l)) :: Nil) =>
new UUID(m.longValue, l.longValue)
case x => throw new MappingException("Can't convert " + x + " to UUID")
}
}
def serialize(implicit format: Formats): PartialFunction[Any, JValue] = {
case x: UUID =>
("mostSig" -> x.getMostSignificantBits) ~ ("leastSig" -> x.getLeastSignificantBits)
}
}
}
It prints:
{"uuid":{"mostSig":-8054689529719995935,"leastSig":-5722404370736228056},"name":"foo"}'
Another solution which uses a custom serializer for Parent type.
sealed abstract class Parent {
var uuid: UUID = UUID.randomUUID
}
case class Foo(name: String) extends Parent
object UUIDTest extends Application {
implicit val formats =
Serialization.formats(NoTypeHints) + new ParentSerializer
val f = Foo("foo")
val ser = write(f)
println(ser)
val f2 = read[Foo](ser)
assert(f == f2)
// Special serializer for Parent type
class ParentSerializer extends Serializer[Parent] {
def deserialize(implicit format: Formats): PartialFunction[(TypeInfo, JValue), Parent] = {
case (t#TypeInfo(cl, _), json) if (classOf[Parent].isAssignableFrom(cl)) =>
val x = Extraction.extract(json, t)(DefaultFormats).asInstanceOf[Parent]
x.uuid = (for {
JField("mostSig", JInt(m)) <- json
JField("leastSig", JInt(l)) <- json
} yield new UUID(m.longValue, l.longValue)).head
x
}
def serialize(implicit format: Formats): PartialFunction[Any, JValue] = {
case x: Parent =>
Extraction.decompose(x)(DefaultFormats) ++
JField("mostSig", x.uuid.getMostSignificantBits) ++
JField("leastSig", x.uuid.getLeastSignificantBits)
}
}
}
If the type is important, you should take a look at YAML.
http://www.google.fr/search?q=java+yaml
It's a subset of json with improved stuff, like variable typing.
You could try jerkson: https://github.com/codahale/jerkson
Its working good for my use, but that is mostly list/map structures. Would not be surprised if it supports your needs though..
Edit: Tried it with the following example (inspired by the lift example in another answer). Seems to work fine.
import java.util.UUID
import com.codahale.jerkson.Json
import org.scalatest.FunSuite
sealed abstract class Parent {
def uuid: UUID
}
case class Foo(uuid: UUID, name: String) extends Parent
class TmpJsonTest extends FunSuite {
test("Json case class serialize") {
val f = Foo(UUID.randomUUID, "foo")
val ser = Json.generate(f)
println(ser)
val f2 = Json.parse[Foo](ser)
assert(f === f2)
}
}
Try the XStream library which includes JSON support. I have used this successfully in a few projects. It has a number of default converters, including one for java.util.UUID. A full list of default converters is located here: http://x-stream.github.io/converters.html.
A brief tutorial on using XStream for JSON reading and writing is located here: http://x-stream.github.io/json-tutorial.html. The tutorial code is written for Java but it should work just the same for Scala since reflection is being used behind the scenes.
Keep in mind that serializing and then deserializing arbitrary graphs of objects is not always possible with this library. In particular, loops in your data cannot be handled, i.e. your data must be a purely hierarchical tree. This is a reasonable limitation given the intentions of the JSON format.
Referenced links:
XStream JSON tutorial: http://x-stream.github.io/json-tutorial.html
XStream default converters: http://x-stream.github.io/converters.html