Spark data frame actions for selected rows [closed] - mysql

Question

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 5 years ago.
Improve this question
I have a data frame like below
+------++-----------------------+
| state| time stamp |
+------+------------------------+
| 0 | Sun Aug 13 10:58:44 |
| 1 | Sun Aug 13 11:59:44 |
| 1 | Sun Aug 13 12:50:43 |
| 1 | Sun Aug 13 13:00:44 |
| 0 | Sun Aug 13 13:58:42 |
| 0 | Sun Aug 13 14:00:41 |
| 0 | Sun Aug 13 14:30:45 |
| 0 | Sun Aug 13 14:58:46 |
| 1 | Sun Aug 13 15:00:47 |
| 0+ | Sun Aug 13 16:00:49 |
+------+------------------------+
I need to select the timestamps only when state changes from 1 to 0,
I need to separate these rows separately
Sun Aug 13 11:59:44
Sun Aug 13 13:58:42
Sun Aug 13 15:00:47
Sun Aug 13 16:00:49
then take the time differences and sum up.
So can some one suggest, what kind of query I should write for this .
I need some result like below
(13:58:42 - 11:59:44) + (16:00:49 - 15:00:47)

Answer 1

Window function should help with your first need. Filter will fulfill you third need. Your third need can be fulfilled by extracting the time from the date-time value.
Given a dataframe as
+-----+-------------------+
|state|timestamp |
+-----+-------------------+
|0 |Sun Aug 13 10:58:44|
|1 |Sun Aug 13 11:59:44|
|1 |Sun Aug 13 12:50:43|
|1 |Sun Aug 13 13:00:44|
|0 |Sun Aug 13 13:58:42|
|0 |Sun Aug 13 14:00:41|
|0 |Sun Aug 13 14:30:45|
|0 |Sun Aug 13 14:58:46|
|1 |Sun Aug 13 15:00:47|
|0 |Sun Aug 13 16:00:49|
+-----+-------------------+
Doing the things I explained above should help. Doing the following should solve your first and second needs.
import org.apache.spark.sql.functions._
df.withColumn("temp", lag("state", 1).over(Window.orderBy("timestamp")))
.withColumn("temp", when(col("temp").isNull, lit(0)).otherwise(col("temp")))
.filter(col("state") =!= col("temp"))
You should have
+-----+-------------------+----+
|state|timestamp |temp|
+-----+-------------------+----+
|1 |Sun Aug 13 11:59:44|0 |
|0 |Sun Aug 13 13:58:42|1 |
|1 |Sun Aug 13 15:00:47|0 |
|0 |Sun Aug 13 16:00:49|1 |
+-----+-------------------+----+
Now regarding you third need, you need to find ways to extract time from timestamp column and do something like below
import org.apache.spark.sql.functions._
df.withColumn("temp", lag("state", 1).over(Window.orderBy("timestamp")))
.withColumn("temp", when(col("temp").isNull, lit(0)).otherwise(col("temp")))
.filter(col("state") =!= col("temp"))
.select(collect_list(col("timestamp")).as("time"))
.withColumn("time", concat_ws(" + ", concat_ws(" - ", $"time"(1), $"time"(0)), concat_ws(" - ", $"time"(3), $"time"(2))))
You should have
+-------------------------------------------------------------------------------------+
|time |
+-------------------------------------------------------------------------------------+
|Sun Aug 13 13:58:42 - Sun Aug 13 11:59:44 + Sun Aug 13 16:00:49 - Sun Aug 13 15:00:47|
+-------------------------------------------------------------------------------------+
I hope the answer is helpful except with extraction of time value from timestamp column