This code works fine when there are records matching the WHERE clause:
val pinfo = SQL("SELECT * FROM tableName WHERE id={id}").on("id" -> "scala")
pinfo().map { row =>
println("have something")// runs when selected
}
What is going to happen when nothing is selected?
I'd like to print the following when no records are selected from MySQL.
println("nothing is selected")//if no row comes
SQL(...)() returns a Stream[SqlRow] and streams have the isEmpty method:
val pinfo: Stream[SqlRow] = SQL("SELECT * FROM tableName WHERE id={id}").on("id" -> "scala")()
if(!pinfo.isEmpty) pinfo.map { row => println("have something") }
else println("nothing is selected")
Also from the REPL:
scala> 1 #:: 2 #:: empty
res0: scala.collection.immutable.Stream[Int] = Stream(1, ?)
scala> res0.isEmpty
res1: Boolean = false
scala> empty
res2: scala.collection.immutable.Stream[Nothing] = Stream()
scala> res2.isEmpty
res3: Boolean = true
You can also parse it as a Option[T], and then handle the case there is no value within this optional result.
val i: Option[Int] = SQL"SELECT int FROM test".as(scalar[String].singleOpt)
Related
I have a nested JSON dataset that is flattened using Spark scala. Further, when I am trying to flatten it, it is not flattened due to duplicate columns. So, I have to rename it first and then flatten it but, I am unable to do so. Below is my code to flatten the dataset:
def flattenDataframe(df: DataFrame): DataFrame = {
//getting all the fields from schema
val fields = df.schema.fields
val fieldNames = fields.map(x => x.name)
//length shows the number of fields inside dataframe
val length = fields.length
for (i <- 0 to fields.length - 1) {
val field = fields(i)
val fieldtype = field.dataType
val fieldName = field.name
fieldtype match {
case arrayType: ArrayType =>
val fieldName1 = fieldName
val fieldNamesExcludingArray = fieldNames.filter(_ != fieldName1)
val fieldNamesAndExplode = fieldNamesExcludingArray ++ Array(s"explode_outer($fieldName1) as $fieldName1")
//val fieldNamesToSelect = (fieldNamesExcludingArray ++ Array(s"$fieldName1.*"))
val explodedDf = df.selectExpr(fieldNamesAndExplode: _*)
return flattenDataframe(explodedDf)
case structType: StructType =>
val childFieldnames = structType.fieldNames.map(childname => fieldName + "." + childname)
val newfieldNames = fieldNames.filter(_ != fieldName) ++ childFieldnames
val renamedcols = newfieldNames.map(x => (col(x.toString()).as(x.toString().replace(".", "_").replace("$", "_").replace("__", "_").replace(" ", "").replace("-", ""))))
val explodedf = df.select(renamedcols: _*)
return flattenDataframe(explodedf)
case _ =>
}
}
df
}
After flattening the dataset it looks like below:
I have used the below loop for renaming duplicate columns but it is working fine for the JSON elements which are on the same level:
val orgList = df.columns.toList // take the original list
val dupsList = orgList.map(.toLowerCase()).diff(orgList.map(.toLowerCase()).distinct).distinct //take the duplicate columns list in lower case
var i = 1
var newDf: DataFrame = df
for (dups <- dupsList) {
i=1
for (key <- orgList) {
if (key.toLowerCase() == dups) {
newDf = newDf.withColumnRenamed(key, key + "_" + i)
i += 1
//println(i)
}
}
}
How do I rename it before flattening it?
Given mapping having NOT NULL field str with a default value:
case class Tst(id: Option[Int] = None, ii: Int, str: String)
class Tsts(tag: Tag) extends Table[Tst](tag, "tsts") {
def id = column[Option[Int]]("id", O.PrimaryKey, O.AutoInc)
def ii = column[Int]("ii")
def str = column[String]("str", O.Default("ddd"))
def * = (id, ii, str) <> (Tst.tupled, Tst.unapply)
}
How do I insert object specifying the field value if I have it:
Tst(ii = 1, str = "aaa")
and skipping it if I don't:
Tst(ii = 1)
Yes, I know the last statement will not compile.
I tried using Option[String] and other things. It ends up with either inserting null or failing with can't be null error
The compiler depends on you putting default values at the end, like:
scala> case class TST(ii: Int, str: String = "aaa", id: Option[Int] = None)
defined class TST
scala> new TST(3)
res0: TST = TST(3,aaa,None)
Edit: Just realized I didn't answer completely:
scala> new TST(3, id = Some(1))
res1: TST = TST(3,aaa,Some(1))
scala> new TST(3, str = "bbb")
res2: TST = TST(3,bbb,None)
filter (which uses halt inside) terminates other branch even if it has some side-effects:
scala> val p = Process("1","2", "3")
scala> val p1 = p.filter(_ => true).map(_ + "p1").observe(io.stdOutLines)
scala> val p2 = p.filter(_ => false).map(_ + "p2").observe(io.stdOutLines)
scala> (p1 yip p2).run.run
1p1
scala> val p2 = p.filter(_ => true).map(_ + "p1").observe(io.stdOutLines)
scala> (p1 yip p2).run.run
1p1
1p2
2p1
2p2
3p1
3p2
Seems logical as there is no value to be returned to yip after that filter. But what about side-effects, specified with observe?
My current solution is to use flatMap to specify default value:
scala> val p1 = p.map(_ + "p1").flatMap(x => Process.emit(x).observe(io.stdOutLines))
scala> val p2 = p.map(_ + "p2").flatMap(x => Process.emit(""))
scala> (p1 yip p2).run.run
1p1
2p1
3p1
But maybe there is a way to use filter?
P.S. merge combinator executes side-effects for other branch (as it doesn't require value to be returned), but it doesn't wait for other branch if one halts (even if it has side-effects).
To run the effects even after p2 terminates there needs to be clear default behaviour. So there are probably these solutions:
define p2 to supply default value after being terminated
use either wye to get left and rights if we don't really need tuples
perhaps the (1) is closer to question and code will looks like:
val p = Process("1","2", "3")
val p1 = p.filter(_ => true).map(_ + "p1").observe(io.stdOutLines)
val p2 = p.filter(_ => false).map(_ + "p2")
.observe(io.stdOutLines).map(Some(_)) ++ emit(None).repeat
// alternativelly
// val p2 = p.map { v => if (pred(v)) right(v) else left(v) }
// .observeO(o.stdOutLines).flatMap { _.toOption }
// ++ emit(None).repeat
(p1 yip p2).run.run
Actually it should be just something like that:
in.map(emit).flatMap{ p =>
val p1 = p.map(_ + "p1").filter(_ => true).observe(out)
val p2 = p.map(_ + "p2").filter(_ => false).observe(out)
p1 merge p2
}.run.run
It makes all side effects being in order as filter can't get more than one value (produced by emit)
I need to create a dictionary in sml, but I am having extreme difficulty with an insert function.
type dict = string -> int option
As an example, here is the empty dictionary:
val empty : dict = fn key => NONE
Here is my implementation of an insert function:
fun insert (key,value) d = fn d => fn key => value
But this is of the wrong type, what I need is insert : (string*int) -> dict -> dict.
I've searched everything from lazy functions to implementing dictionaries.
Any help or direction would be greatly appreciateds!
If you are still confused on what I am trying to implement, I drafted up what I should expect to get when calling a simple lookup function
fun lookup k d = d k
- val d = insert ("foo",2) (insert ("bar",3) empty);
val d = fn : string -> int option
- lookup2 "foo" d;
val it = SOME 2 : int option
- lookup2 "bar" d;
val it = SOME 3 : int option
- lookup2 "baz" d;
val it = NONE : int option
You can reason on the signature of the function:
val insert = fn: (string * int) -> dict -> dict
When you supply key, value and a dictionary d, you would like to get back a new dictionary d'. Since dict is string -> int option, d' is a function takes a string and returns an int option.
Suppose you supply a string s to that function. There are two cases which could happen: when s is the same as key you return the associated value, otherwise you return a value by looking up d with key s.
Here is a literal translation:
fun insert (key, value) d = fn s => if s = key then SOME value
else d s
Forgive me if this has already been asked elsewhere. I have a Scala syntax question involving function-values and implicit parameters.
I'm comfortable using implicits with Scala's currying feature. For instance if I had a sum function and wanted to make the second argument an implicit:
scala> def sum(a: Int)(implicit b: Int) = a + b
sum: (a: Int)(implicit b: Int)Int
Is there a way to do this using the function-value syntax? Ignoring the implicit for a moment, I typically write curried function-values like this:
scala> val sum2 = (a: Int) => (b: Int) => a + b
sum: (Int) => (Int) => Int = <function1>
However, the function signature in the second approach is much different (the currying is being expressed explicitly). Just adding the implicit keyword to b doesn't make much sense and the compiler complains as well:
scala> val sum2 = (a: Int) => (implicit b: Int) => a + b
<console>:1: error: '=>' expected but ')' found.
val sum2 = (a: Int) => (implicit b: Int) => a + b
^
Furthermore partially-applying sum from the very first approach to get a function-value causes problems as well:
scala> val sumFunction = sum _
<console>:14: error: could not find implicit value for parameter b: Int
val sumFunction = sum _
^
This leads me to believe that functions that have implicit parameters must have said parameters determined when the function-value is created, not when the function-value is applied later on. Is this really the case? Can you ever use an implicit parameter with a function-value?
Thanks for the help!
scala> val sum2 = (a: Int) => {implicit b: Int => a + b}
sum2: (Int) => (Int) => Int = <function1>
This will just make b an implicit value for the scope of the function body, so you can call methods that expect an implicit Int.
I don't think you can have implicit arguments for functions since then it is unclear what the function is. Is it Int => Int or () => Int?
The closest I found is:
scala> case class Foo(implicit b: Int) extends (Int => Int) {def apply(a: Int) = a + b}
defined class Foo
scala> implicit val b = 3
b: Int = 3
scala> Foo()
res22: Foo = <function1>
scala> res22(2)
res23: Int = 5
In this snippet
scala> val sum2 = (a: Int) => (b: Int) => a + b
sum: (Int) => (Int) => Int = <function1>
Note that the precise type of sum2 is Function1[Int, Function1[Int, Int]]. It could also be written as
val sum2 = new Function1[Int, Function1[Int, Int]] {
def apply(a: Int) = new Function1[Int, Int] {
def apply(b: Int) = a + b
}
}
Now, if you try to make b implicit, you get this:
scala> val sum2 = new Function1[Int, Function1[Int, Int]] {
| def apply(a: Int) = new Function1[Int, Int] {
| def apply(implicit b: Int) = a + b
| }
| }
<console>:8: error: object creation impossible, since method apply in trait Function1 of type (v1: Int)Int is not defined
def apply(a: Int) = new Function1[Int, Int] {
^
Or, in other words, Function's interfaces do not have implicit parameters, so anything with an implicit parameter is not a Function.
Try overloading the apply method.
scala> val sum = new Function1[Int, Function1[Int, Int]] {
| def apply(a: Int) = (b: Int) => a + b
| def apply(a: Int)(implicit b: Int) = a + b
|}
sum: java.lang.Object with (Int) => (Int) => Int{def apply(a:Int)(implicit b: Int): Int} = <function1>
scala> sum(2)(3)
res0: Int = 5
scala> implicit val b = 10
b: Int = 10
scala> sum(2)
res1: Int = 12