I have a sequence of case class A (id: UUID, profile: String, data: JsValue)
I'd like to sort the sequence by updated from the JsValue data field
data field looks like this
{
"doors":2,
"color":"Black",
"updated":"2019-09-24T15:59:21+0200",
"username":"John",
"year":2016
}
I tried
sequenceOfA.sortWith(_.data \ "updated" < _.data \ "updated") but that doesn't work because < is not a member of play's JsLookupResult
Casting it to String doesn't work either
sequenceOfA.sortWith((_.data \ "updated").as[String] < (_.data \ "updated").as[String])
What would be the most idiomatic way to do this in Scala?
What you would need is to handle absent field. This can be done via more safe approach with explicitly handling:
sequenceOfA.sortWith { case (left, right) =>
val leftUpdate = (left.data \ "update").validate[String].asOpt
val rightUpdate = (right.data \ "update").validate[String].asOpt
leftUpdate -> rightUpdate match {
case (Some(left), Some(right)) => left < right
case (None, Some(_)) => true // objects with `update` absent field goes first
case (Some(_), None) => false // objects with `update` present field goes after absent field
}
}
Or just invoke get method, which might throw an exception - which is highly unrecommended:
sequenceOfA.sortWith((m \ "updated").as[String].get < (_.data \ "updated").as[String].get)
Hope this helps!
A potential solution could be:
// Input is Seq[A] i.e val input: Seq[A]
// ConvertToTimestamp is a method which takes time as string and returns timestamp.
//This is required since time is in string and also timezones are involved.
//So this method will convert time string to UTC time(in epochs)
// def convertToTimeStamp(time: String): Long
val updated: Seq[Long] = A.map(value => (value.data \ "updated").as[String]).map(x => convertToTimestamp(x)) // fetch value
val pairs = input zip updated // List of ordered tuple
val sortedInput = pairs.sort(sortBy(pair => pair._2)).map(_._1).toSeq // sort using timestamp
The above solution assumes that updated field is not empty. If forsome input it is empty, change the function where we are parsing this field(while computing updated value as shown in above to have a default value based on the requirement(either 0 if you want such records to come first or Duration.Inf if you want those records to come last).
Let me know if it helps!!
Related
I need to render a sorted map by a user-defined type.
SortedMap[X, Seq[Y]]
Json library should render the map as ordered by the X.name
case class X(order: Int, name: String) extends Ordered[X]
Assume I have X(1, "James"), X(2, "Mel"), X(3, "Ashley").
The output map should be
"James" : Seq(Y)
"Mel" : Seq(Y)
"Ashley": Seq(Y)
The inMap is correctly sorted (as viewed by the debugger), but after the rendering, the sorting order(X.order) is lost. Probably due to the toSeq. Any ideas?
implicit val myWrites = new Writes[SortedMap[X, Seq[Y]]] {
def writes(inMap: SortedMap[X, Seq[Y]]): JsValue =
Json.obj(inMap.map {case (s, o) =>
val r: (String, JsValueWrapper) = s.name() -> Json.toJson(o)
r
}.toSeq:_*)
}
So...
I never meet the word "render" used as "convert to"
ordering by key original key in SortedSet is lost after mapping because you change the key type so the result is ordered by a new key type (here: String)
if you want to preserve the order of items in between mapping I would suggest using ListMap
though in your particular case you can do away with Seq of tuples, as at the end of the day, this is what you need to produce
implicit val myWrites: Writes[SortedMap[X, Seq[Y]]] = new Writes[SortedMap[X, Seq[Y]]] {
def writes(inMap: SortedMap[X, Seq[Y]]): JsValue =
Json.obj(inMap.toSeq.map { case (s, o) =>
s.name() -> Json.toJson(o)
}:_*)
}
Consider the code below
object SparkUDFApp {
def main(args: Array[String]) {
val df = ctx.read.json(".../example.json")
df.registerTempTable("example")
val fn = (_: String).length // % 10
ctx.udf.register("len10", fn)
val res0 = ctx sql "SELECT len10('id') FROM example LIMIT 1" map {_ getInt 0} collect
println(res0.head)
}
}
JSON example
{"id":529799371026485248,"text":"Example"}
The code should return a length of the field value from JSON (e.g. 'id' has value 18). But instead of returning '18' it returns '2', which is the length of 'id' I suppose.
So my question is how to rewrite UDF to fix it?
The problem is that you are passing the string id as a literal to your UDF so it is interpreted as one instead of a column (notice that it has 2 letters this is why it returns such number). To solve this just change the way how you formulate the SQL query.
E.g.
val res0 = ctx sql "SELECT len10(id) FROM example LIMIT 1" map {_ getInt 0} collect
// Or alternatively
val len10 = udf(word => word.length)
df.select(len10(df("id")).as("length")).show()
simple question:
How can one use play-json (2.3.x) to sort all JsArrays in some JsValue (recursively)?
my usecase:
consider an app that uses Set[String] internally, and when data is requested,
the output JSON serialize the set as a JSON array. the order is not important.
now, if one wants to write some tests to cover this functionality, since the order of items is not important (it is a set after all. internally, and conceptually), and all I want to check is that everything returned as it should, I may want to compare the response JSON with an "expected" JSON object I create explicitly.
for that exact reason, I want to sort the JSON arrays, and compare the JsValue's.
how would one write such transformer?
EDIT:
I have managed to write a transformer that answers my needs, but it won't sort every JsArry in some JsValue. I'll post it here, since it might be useful for others, but it is not what I was asking for.
val jsonSortTransformer = (__ \ 'fields).json.update(
Reads.JsObjectReads.map{
case JsObject(xs) => JsObject(
xs.map{
case (n,jv) => {
n -> (jv match {
case JsArray(arr) if arr.forall(_.isInstanceOf[JsString]) => JsArray(arr.sortBy(_.as[String]))
case _ => jv
})
}
}
)
}
)
You can use the value property on JsArray to get a Seq[JsValue], then sort arbitrarily, and then recreate a JsArray. For example:
scala> myJsArray
play.api.libs.json.JsArray = ["11","4","5","1","22","2"]
scala> JsArray(myJsArray.value.sortBy(_.as[JsString].value.toInt))
play.api.libs.json.JsArray = ["1","2","4","5","11","22"]
If all you're doing is trying to compare actual and expected values of what you know is a set, you can also just use value on both properties, build a Set and check for equality:
Set(actual.value: _*) == Set(expected.value: _*)
Or sort them both:
val sortedSeq: JsArray => Seq[String] = array => array.value.map(_.toString).sorted
sortedSeq(actual) == sortedSeq(expected)
To recursively sort all the JsArrays in an arbitrary JsValue, it might look something like:
def sortArrays(json: JsValue): JsValue = json match {
case JsObject(obj) => JsObject(obj.toMap.mapValues(sortArrays(_)).toList)
case JsArray(arr) => JsArray(arr.map(sortArrays).sortBy(_.toString))
case other => other
}
scala> myObj
play.api.libs.json.JsValue = {"a":[2,1],"b":[{"c":[3,2]},{"d":[4,3]}],"e":{"f":[5,4]}}
scala> sortArrays(myObj)
play.api.libs.json.JsValue = {"a":[1,2],"b":[{"c":[2,3]},{"d":[3,4]}],"e":{"f":[4,5]}}
I'm afraid that #Ben's answer is quite incorrect.
I would approach this problem by defining an Ordering class for JsValues and then use its comparison method to verify equality (meaning this should actually be an object - not an anonymous class, as shown in the example).
One doesn't have to use Ordering, I just find it a bit more convenient than a simple compareTo method. Of course, one can also define this class/object as implicit.
val jsonOrdering: Ordering[JsValue] = new Ordering[JsValue]() {
override def compare(x: JsValue, y: JsValue): Int = {
x.getClass.getName.compareTo(y.getClass.getName) match {
case 0 =>
(x, y) match {
case (JsNull, JsNull) => 0
case (JsString(valueX), JsString(valueY)) =>
valueX.compareTo(valueY)
case (JsNumber(valueX), JsNumber(valueY)) =>
valueX.compare(valueY)
case (JsBoolean(boolX), JsBoolean(boolY)) =>
boolX.compareTo(boolY)
case (JsArray(elementsX), JsArray(elementsY)) =>
elementsX.size.compareTo(elementsY.size) match {
case 0 =>
elementsX
// .sorted(this) // uncomment if array order DOES NOT matter
.zip(elementsY
// .sorted(this) // uncomment if array order DOES NOT matter
)
.view
.map {
case (elementX, elementY) => compare(elementX, elementY)
}
.find(_ != 0)
.getOrElse(0)
case nonZero => nonZero
}
case (JsObject(fieldsX), JsObject(fieldsY)) =>
fieldsX.size.compareTo(fieldsY.size) match {
case 0 =>
fieldsX.toSeq
.sortBy(_._1)
.zip(fieldsY.toSeq.sortBy(_._1))
.view
.flatMap {
case ((keyX, valueX), (keyY, valueY)) =>
Seq(keyX.compareTo(keyY), compare(valueX, valueY))
}
.find(_ != 0)
.getOrElse(0)
case nonZero => nonZero
}
}
case nonZero => nonZero
}
}
I would perhaps split some parts into private/nested functions (I got lazy this time). Anyway, let's go over this:
Compare the two values' class names, and if they aren't the same then return the comparison between their names.
If the values are of any primitive JSON type, simply return the comparison between them.
If the values are arrays, then:
Compare their sizes, and if they aren't the same then return the comparison between the sizes.
Only if the order of arrays doesn't matter - sort each of the arrays (with the same ordering class; i.e, this is recursive).
Zip the elements of both arrays (so that you get an array of pairs of elements).
Find the first pair that its two elements are not the same, and return their comparison.
If no such pair exists, this means that the arrays are the same (return 0).
If the values are maps (objects):
Compare their sizes, and if they aren't the same then return the comparison between the sizes.
Turn the maps into a sequence of tuples, and sort these sequences by their key (first element of a tuple).
Zip the tuples of both sequences (so that you get an array of pairs of tuples).
Find the first pair that its tuples are not the same, and return their comparison. Compare these tuples in the following manner:
Compare their keys (strings), and return their comparison if they're not the same.
Compare their values (JsValue, thus using the same method recursively), and return their comparison if they're not the same.
Otherwise, they are the same.
If no such pair exists, this means that the maps (objects) are the same (return 0).
Note that although this ordering is consistent and deterministic, it is quite arbitrary and doesn't convey much logical meaning.
I got this similar question but it doesn't help me. (Anorm parse float values). And I can honestly say I didn't understand the solution of that question.
I am getting this complie time error:
could not find implicit value for parameter c: anorm.Column[Float]
at
def getInformation(id: Long): List[(Float, Float, Float)] = {
DB.withConnection { implicit con =>
val query = SQL("select principal,interest,value from myTable where userId={id} and status=true").on("id"->id)
val result = query().map { row =>
Tuple3(row[Float]("principal"), row[Float]("inetrest"), row[Float]("value"))
// ^
}.toList
return result
}
}
Maybe a short review of implicits help you. Let's construct a very basic example:
// some class which will be used as implicit (can be anything)
case class SomeImplicitInformation(maybe: Int, with: Int, data: Int)
// lets assume we have a function that requires an implicit
def functionRequiringImplicit(regularParameters: Int)(implicit imp: SomeImplicitInformation) {
// ...
}
// now if you try to call the function without having an implicit in scope
// you would have to pass it explicitly as second parameter list:
functionRequiringImplicit(0)(SomeImplicitInformation(0,0,0))
// instead you can declare an implicit somewhere in your scope:
implicit val imp = SomeImplicitInformation(0,0,0)
// and now you can call:
functionRequiringImplicit(0)
The error you get simply says that anorm.Column[Float] in not in the scope as implicit. You can solve it by adding it implicitly to your scope or pass it explicitly.
More detailed instructions for you: Since the Column companion object only provides an implicit for rowToDouble you simply have to use the code that is linked in your question. To get it to work put it before your result computation. Later you might want to place it in a val in some enclosing scope.
try this...
def getInformation(id: Long): List[(Float, Float, Float)] = {
DB.withConnection { implicit con =>
val query = SQL("select principal,interest,value from myTable where userId={id} and status=true").on("id"->id)
val result = query().map { row =>
Tuple3(row[Float]("principal").asInstanceOf[Float], row[Float]("inetrest").asInstanceOf[Float], row[Float]("value").asInstanceOf[Float])
}.toList
return result
}
}
implicit def rowToFloat: Column[Float] = Column.nonNull { (value, meta) =>
val MetaDataItem(qualified, nullable, clazz) = meta
value match {
case d: Float => Right(d)
case _ => Left(TypeDoesNotMatch("Cannot convert " + value + ":" + value.asInstanceOf[AnyRef].getClass + " to Float for column " + qualified))
}
}
Some functions can accept what we call implicit parameters. Such parameters can, under certain conditions, be derived from the context. If these parameters can't be found, then you have to specify them by hand. If you expect a parameter to be used as an implicit one, it must have been declared implicit, for instance this way :
implicit val myVal = ...
It can be done in the current block or in an enclosing one (in the class body, for instance, or even sometimes in the imports)
The error you get seems to be related to this feature. You're using a function that needs a parameter of type anorm.Column[Float]. The argument is defined to be implicit so that an implicit value can be used and your code may be more concise. Unfortunately, you don't seem to have such an implicit value in your code, so it fails.
Latest Anorm (included in Play 2.3) provides more numeric conversion (see details at http://applicius-en.tumblr.com/post/87829484643/anorm-whats-new-play-2-3 & in Play migration notes).
If you have missing converter, you can add an issue on Play github project.
Best
I am trying to figure out the issue, and tried different styles that I have read on Scala, but none of them work. My code is:
....
val str = "(and x y)";
def stringParse ( exp: String, pos: Int, expreshHolder: ArrayBuffer[String], follow: Int )
var b = pos; //position of where in the expression String I am currently in
val temp = expreshHolder; //holder of expressions without parens
var arrayCounter = follow; //just counts to make sure an empty spot in the array is there to put in the strings
if(exp(b) == '(') {
b = b + 1;
while(exp(b) == ' '){b = b + 1} //point of this is to just skip any spaces between paren and start of expression type
if(exp(b) == 'a') {
temp(arrayCounter) = exp(b).toString;
b = b+1;
temp(arrayCounter)+exp(b).toString; b = b+1;
temp(arrayCounter) + exp(b).toString; arrayCounter+=1}
temp;
}
}
val hold: ArrayBuffer[String] = stringParse(str, 0, new ArrayBuffer[String], 0);
for(test <- hold) println(test);
My error is:
Driver.scala:35: error: type mismatch;
found : Unit
required: scala.collection.mutable.ArrayBuffer[String]
ho = stringParse(str, 0, ho, 0);
^one error found
When I add an equals sign after the arguments in the method declaration, like so:
def stringParse ( exp: String, pos: Int, expreshHolder: ArrayBuffer[String], follow: Int ) ={....}
It changes it to "Any". I am confused on how this works. Any ideas? Much appreciated.
Here's a more general answer on how one may approach such problems:
It happens sometimes that you write a function and in your head assume it returns type X, but somewhere down the road the compiler disagrees. This almost always happens when the function has just been written, so while the compiler doesn't give you the actual source (it points to the line where your function is called instead) you normally know that your function's return type is the problem.
If you do not see the type problem straight away, there is the simple trick to explicitly type your function. For example, if you thought your function should have returned Int, but somehow the compiler says it found a Unit, it helps to add : Int to your function. This way, you help the compiler to help you, as it will spot the exact place, where a path in your function returns a non-Int value, which is the actual problem you were looking for in the first place.
You have to add the equals sign if you want to return a value. Now, the reason that your function's return value is Any is that you have 2 control paths, each returning a value of a different type - 1 is when the if's condition is met (and the return value will be temp) and the other is when if's condition isn't (and the return value will be b=b+1, or b after it's incremented).
class Test(condition: Boolean) {
def mixed = condition match {
case true => "Hi"
case false => 100
}
def same = condition match {
case true => List(1,2,3)
case false => List(4,5,6)
}
case class Foo(x: Int)
case class Bar(x: Int)
def parent = condition match {
case true => Foo(1)
case false => Bar(1)
}
}
val test = new Test(true)
test.mixed // type: Any
test.same // type List[Int]
test.parent // type is Product, the case class super type
The compiler will do its best to apply the most specific type it can based on the possible set of result types returned from the conditional (match, if/else, fold, etc.).