Comma separated list with Enumerator - json

I've just started working with Scala in my new project (Scala 2.10.3, Play2 2.2.1, Reactivemongo 0.10.0), and encountered a pretty standard use case, which is - stream all the users in MongoDB to the external client. After navigating Enumerator, Enumeratee API I have not found a solid solution for that, and so I solved this in following way:
val users = collection.find(Json.obj()).cursor[User].enumerate(Integer.MAX_VALUE, false)
var first:Boolean = true
val indexedUsers = (users.map(u => {
if(first) {
first = false;
Json.stringify(Json.toJson(u))
} else {
"," + Json.stringify(Json.toJson(u))
}
}))
Which, from my point of view, is a little bit tricky - mainly because I needed to add Json Start Array, Json End Array and comma separators in element list, and I was not able to provide it as a pure Json stream, so I converted it to String steam.
What is a standard solution for that, using reactivemongo in play?

I wrote a helper function which does what you want to achieve:
def intersperse[E](e: E, enum: Enumerator[E]): Enumerator[E] = new Enumerator[E] {
val element = Input.El(e)
override def apply[A](i1: Iteratee[E, A]): Future[Iteratee[E, A]] = {
var iter = i1
val loop: Iteratee[E, Unit] = {
lazy val contStep = Cont(step)
def step(in: Input[E]): Iteratee[E, Unit] = in match {
case Input.Empty ⇒ contStep
case Input.EOF ⇒ Done((), Input.Empty)
case e # Input.El(_) ⇒
iter = Iteratee.flatten(iter.feed(element).flatMap(_.feed(e)))
contStep
}
lazy val contFirst = Cont(firstStep)
def firstStep(in: Input[E]): Iteratee[E, Unit] = in match {
case Input.EOF ⇒ Done((), Input.Empty)
case Input.Empty ⇒
iter = Iteratee.flatten(iter.feed(in))
contFirst
case Input.El(x) ⇒
iter = Iteratee.flatten(iter.feed(in))
contStep
}
contFirst
}
enum(loop).map { _ ⇒ iter }
}
}
Usage:
val prefix = Enumerator("[")
val suffix = Enumerator("]")
val asStrings = Enumeratee.map[User] { u => Json.stringify(Json.toJson(u)) }
val result = prefix >>> intersperse(",", users &> asStrings) >>> suffix
Ok.chunked(result)

Related

Generate dynamic bundle

In my project, I have many different custom Bundle.
They can be completely different.
For example these ones:
class MyBundle extends Bundle {
val v1 = UInt(8.W)
val v2 = Bool()
val v3 = UInt(4.W)
}
class MyBundle2 extends Bundle {
val v4 = UInt(18.W)
val v5 = UInt(2.W)
}
...
Instead of manually creating new Bundle to perform each operation, I want to be able to generate for all of them the corresponding Bundle.
So for MyBundle, I want to do:
// Must be generated for each Bundle
class MyGenBundle extends Bundle {
val v1 = UInt(log2Ceil(8 + 1).W) // width = MyBundle.v1 width + 1
val v2 = UInt(log2Ceil(1 + 1).W) // width = MyBundle.v2 width + 1
val v3 = UInt(log2Ceil(4 + 1).W) // width = MyBundle.v3 width + 1
}
class MyModule extends Module {
...
...
val w_b = Wire(new MyBundle())
val w_gb = Wire(new MyGenBundle())
// Must be automated for each Bundle
w_gb.v1 := PopCount(w_b.v1)
w_gb.v2 := PopCount(w_b.v2)
w_gb.v3 := PopCount(w_b.v3)
}
My goal is to automatically generate MyGenBundle (or similar directly in MyModule) from MyBundle, and perform in MyModule the same operation to all signals.
It also means that I need to dynamically address all signals in each Bundle.
Finally, I think the solution can have the following form:
val w_b = Wire(new MyBundle())
val w_gb = Wire(new AutoGenBundle(new MyBundle())) // AutoGenBundle generates a Bundle from parameter
val v_sig = Seq(v1, v1, v3) // Can be recovered automatically
// from w_b.elements I think
foreach (s <- v_sig) {
w_gb."s" := PopCount(w_b."s") // Here "s" the dynamic name of the signal
} // But how in the real case ?
Is this working technically possible in Chisel/Scala?
If so, do you have any ideas for implementing it?
Basic solution is to use the MixedVec chisel type for GenericBundle
class MyBundle extends Bundle {
val v1 = UInt(8.W)
val v2 = Bool()
val v3 = UInt(4.W)
}
class GenericBundle[T <: Bundle](data: T) extends Bundle {
val v = MixedVec(data.elements.values.map(x => UInt(log2Ceil(x.getWidth + 1).W)).toList)
def popCount(data: T): Unit = {
data.elements.values.zip(v).foreach { case (left, right) =>
right := PopCount(left.asUInt())
}
}
}
class MyModule extends MultiIOModule {
val w_b = IO(Input(new MyBundle))
val w_gb = IO(Output(new GenericBundle(w_b)))
w_gb.popCount(w_b)
}

How to use output from one generator in another generator in Kotest?

Using an example from Clojure's test.check let generator, generate a non-empty list of strings, give that list to another generator to pick a string from, then create a map that contains the string list and the selected string. In Clojure, it looks as follows:
(gen/let [list-of-strings (gen/not-empty (gen/list gen/string))
a-string (gen/element list-of-strings)] ;; use the generated list-of-strings above
{:all-strings list-of-strings
:selected a-string})
Taking io.kotest.property.arbitrary.bind for inspiration, I've tried implementing it as follows, but it doesn't work (Kotlin compiler spitted out "Type inference failed"):
fun <A, B, T: Any> let(genA: Gen<A>, genB: (A) -> Gen<B>, bindFn: (A, B) -> T): Arb<T> {
return arb { rs ->
val iterA = genA.generate(rs).iterator()
generateSequence {
val a = iterA.next()
val iterB = genB(a.value).generate(rs).iterator()
val b = iterB.next()
bindFn(a.value, b.value)
}
}
}
Turns out dropping bindFn parameter solves the problem, but the solution looks a little ugly as it needs to return a Pair:
fun <A, B> let(genA: Gen<A>, genBFn: (A) -> Gen<B>): Arb<Pair<A, B>> {
return arb { rs ->
val iterA = genA.generate(rs).iterator()
generateSequence {
val a = iterA.next().value
// could combine the following to one line, but split for clarity
val genB = genBFn(a)
val iterB = genB.generate(rs).iterator()
Pair(a, iterB.next().value)
}
}
}
Then with the above, using it looks as follows:
class StringTest : StringSpec({
"element is in list" {
val letGen = let(
Arb.list(Arb.string(), range=1..100), // genA
{ xs -> Arb.element(xs) } // genBFn
)
forAll(letGen) { (xs, x) ->
x in xs
}
}
})
Inspire from above solution and wrote a shorter one
fun <A, B> Gen<A>.then(genFn: (A) -> Gen<B>): Arb<Pair<A, B>> =
arbitrary { rs ->
val first = this.generate(rs).first().value
val second = genFn(first).generate(rs).first().value
Pair(first, second)
}
class StringTest : StringSpec({
"element is in list" {
val dependArb =
Arb.list(Arb.string(), range=1..100).then { Arb.element(it) } // genBFn
forAll(dependArb) { (xs, x) ->
x in xs
}
}
})

Play framework - pass parameter to Writes and Reads

Is there a way to pass parameter into a Writes to I will be able to control they way the JsValue is written?
This is how it looks right now:
implicit val myClassWrites = new Writes[MyClass] {
override def writes(l: MyClass): JsValue = Json.obj("a" -> l.a, "b" -> l.b)
}
But I want to do something like this:
implicit val myClassWrites = new Writes[MyClass] (extended: Option[Boolean]) {
override def writes(l: MyClass): JsValue = {
extended match{
case true => //do something
case false => //do something else
}
}
}
Is there an elegant way to achieve this? or something similar?
I managed do implement this need with Reads like this(dropping the implicit):
def myClassReads(c: String) : Reads[MyClass] = (
Reads.pure(c) and
(JsPath \ "a").read[String] and
(JsPath \ "b").read[String]
) (MyClass.apply _)
And then when I want to use the reads (usually in the controller when I want to validate the body of the request) I do:
request.body.validate[MyClass](MyClass.myClassReads("foo")).fold(
errors => //
myClass=> // do domething
)
So the Writes is still a mystery.
You can do this with implicit parameters. See code below:
case class Extended(b: Boolean)
object MyClass {
implicit def MyClassWrites(implicit extended: Extended): Writes[MyClass] = new Writes[MyClass] {
def writes(l: MyClass) =
if (extended.b) JsString("foo")
else JsString("bar")
}
}
And then use like this
implicit var extd = Extended(true)
println(Json.toJson(myClass)) // foo
extd = Extended(false)
println(Json.toJson(myClass)) // bar

Customizing JSON serialization in Play

I'm using renderJSON(Object) to return some objects as JSON values, and it's working fine except for one field. Is there an easy way to add in that one field without having to manually create the whole json template?
Play uses GSON to build the JSON string. If your one field is a specific object type, then you can easily do this by providing a customised serialisation for that type. See the documentation here
http://sites.google.com/site/gson/gson-user-guide#TOC-Custom-Serialization-and-Deserializ
However, if it is an Integer class for example, that you want to work in one way for one, and another way for another, then you may have a little more difficulty.
Example
GsonBuilder gson = new GsonBuilder();
gson.registerTypeAdapter(SpecificClass.class, new MySerializer());
private class MySerializer implements JsonSerializer<DateTime> {
public JsonElement serialize(SpecificClass src, Type typeOfSrc, JsonSerializationContext context) {
String res = "special format of specificClass"
return new JsonPrimitive(res);
}
}
Simply do a
JsonElement elem = new Gson().toJsonTree(yourObject);
JsonObject obj = elem.getAsJsonObject();
obj.remove("xxx");
obj.addProperty("xxx", "what you want");
// other stuff ...
renderJSON(obj.toString());
etc.
After evaluating the play framework we hit a stumbling block and decision choice on serializing JSON for an external API. Allot of articles out there suggest using the Lift framework within play which just seem like extra overhead.After trying some of the frameworks / modules with in the play framework a college and myself decided to write a light weight code block that could cater for our needs.
case class User (
user_id: Int,
user_name: Option[String],
password: Option[String],
salt: Option[String]
) extends Serializable {
def toXml =
<user>
<user_id>{user_id}</user_id>
<user_name>{user_name.getOrElse("")}</user_name>
</user>
override def toJson =
"{" + JSON.key("user_id") + JSON.value(user_id) + ", " + JSON.key("user_name") + JSON.value(user_name) + "}"
}
class Serializable {
def toJson = ""
}
object JSON {
def key(x:String) = value(x) + ": "
def value(x:Any):String = {
x match {
case s:String => "\"" + s + "\""
case y:Some[String] => value(y.getOrElse(""))
case i:Int => value(i.toString)
case s:Serializable => s.toJson
case xs:List[Any] => "[" + xs.map(x => value(x)).reduceLeft(_ + ", " + _) + "]"
}
}
}
def searchUserByName(user_name: String) = {
(for (
u <- Users if u.user_name.like(("%"+user_name+"%").bind)
) yield u.*)
.list
.map(User.tupled(_))
}
def toXml(users:List[User]) = {
<users>
{ users.map(u => u.toXml) }
</users>
}
def toJson(users:List[User]) = {
"[" + users.map(u => u.toJson).reduceLeft(_ + ", " + _) + "]"
}
And from the controller.
// -- http://localhost:9000/api/users/getUser/xml
// -- http://localhost:9000/api/users/getUser/json
def getUser(requestType:String) = {
db withSession{
val user = Users.byUserName("King.Kong")
if(requestType == "xml") {
Xml(user.toXml)
} else {
user.toJson
}
}
}
//--- http://localhost:9000/api/users/searchuser/xml
//--- http://localhost:9000/api/users/searchuser/json
def searchUser(requestType:String) = {
db withSession{
val users = Users.searchUserByName("Doctor.Spoc")
if(requestType == "xml") {
Xml(Users.toXml(users))
} else {
val jsonList = Users.toJson(users)
Json(jsonList)
}
}

restart iterator on exceptions in Scala

I have an iterator (actually a Source.getLines) that's reading an infinite stream of data from a URL. Occasionally the iterator throws a java.io.IOException when there is a connection problem. In such situations, I need to re-connect and re-start the iterator. I want this to be seamless so that the iterator just looks like a normal iterator to the consumer, but underneath is restarting itself as necessary.
For example, I'd like to see the following behavior:
scala> val iter = restartingIterator(() => new Iterator[Int]{
var i = -1
def hasNext = {
if (this.i < 3) {
true
} else {
throw new IOException
}
}
def next = {
this.i += 1
i
}
})
res0: ...
scala> iter.take(6).toList
res1: List[Int] = List(0, 1, 2, 3, 0, 1)
I have a partial solution to this problem, but it will fail on some corner cases (e.g. an IOException on the first item after a restart) and it's pretty ugly:
def restartingIterator[T](getIter: () => Iterator[T]) = new Iterator[T] {
var iter = getIter()
def hasNext = {
try {
iter.hasNext
} catch {
case e: IOException => {
this.iter = getIter()
iter.hasNext
}
}
}
def next = {
try {
iter.next
} catch {
case e: IOException => {
this.iter = getIter()
iter.next
}
}
}
}
I keep feeling like there's a better solution to this, maybe some combination of Iterator.continually and util.control.Exception or something like that, but I couldn't figure one out. Any ideas?
This is fairly close to your version and using scala.util.control.Exception:
def restartingIterator[T](getIter: () => Iterator[T]) = new Iterator[T] {
import util.control.Exception.allCatch
private[this] var i = getIter()
private[this] def replace() = i = getIter()
def hasNext: Boolean = allCatch.opt(i.hasNext).getOrElse{replace(); hasNext}
def next(): T = allCatch.opt(i.next).getOrElse{replace(); next}
}
For some reason this is not tail recursive but it that can be fixed by using a slightly more verbose version:
def restartingIterator2[T](getIter: () => Iterator[T]) = new Iterator[T] {
import util.control.Exception.allCatch
private[this] var i = getIter()
private[this] def replace() = i = getIter()
#annotation.tailrec def hasNext: Boolean = {
val v = allCatch.opt(i.hasNext)
if (v.isDefined) v.get else {replace(); hasNext}
}
#annotation.tailrec def next(): T = {
val v = allCatch.opt(i.next)
if (v.isDefined) v.get else {replace(); next}
}
}
Edit: There is a solution with util.control.Exception and Iterator.continually:
def restartingIterator[T](getIter: () => Iterator[T]) = {
import util.control.Exception.allCatch
var iter = getIter()
def f: T = allCatch.opt(iter.next).getOrElse{iter = getIter(); f}
Iterator.continually { f }
}
There is a better solution, the Iteratee:
http://apocalisp.wordpress.com/2010/10/17/scalaz-tutorial-enumeration-based-io-with-iteratees/
Here is for example an enumerator that restarts on encountering an exception.
def enumReader[A](r: => BufferedReader, it: IterV[String, A]): IO[IterV[String, A]] = {
val tmpReader = r
def loop: IterV[String, A] => IO[IterV[String, A]] = {
case i#Done(_, _) => IO { i }
case Cont(k) => for {
s <- IO { try { val x = tmpReader.readLine; IO(x) }
catch { case e => enumReader(r, it) }}.join
a <- if (s == null) k(EOF) else loop(k(El(s)))
} yield a
}
loop(it)
}
The inner loop advances the Iteratee, but the outer function still holds on to the original. Since Iteratee is a persistent data structure, to restart you just have to call the function again.
I'm passing the Reader by name here so that r is essentially a function that gives you a fresh (restarted) reader. In practise you will want to bracket this more effectively (close the existing reader on exception).
Here's an answer that doesn't work, but feels like it should:
def restartingIterator[T](getIter: () => Iterator[T]): Iterator[T] = {
new Traversable[T] {
def foreach[U](f: T => U): Unit = {
try {
for (item <- getIter()) {
f(item)
}
} catch {
case e: IOException => this.foreach(f)
}
}
}.toIterator
}
I think this very clearly describes the control flow, which is great.
This code will throw a StackOverflowError in Scala 2.8.0 because of a bug in Traversable.toStream, but even after the fix for that bug, this code still won't work for my use case because toIterator calls toStream, which means that it will store all items in memory.
I'd love to be able to define an Iterator by just writing a foreach method, but there doesn't seem to be any easy way to do that.