Run spark-shell command in shell script - mysql

Question

#!/bin/sh
spark-shell
import org.apache.spark.sql.SparkSession
val url="jdbc:mysql://localhost:3306/slow_and_tedious"
val prop = new java.util.Properties
prop.setProperty("user",”scalauser”)
prop.setProperty("password","scalauser123")
val people = spark.read.jdbc(url,"sat",prop)
The above commands are used to make a connection between Mysql and Spark using JDBC.
But instead of writing these commands everytime I thought of make a script but when I run the above script it throws this error.

Answer 1

Create scala file named as test.scala with your code like below
import org.apache.spark.sql.SparkSession
val url="jdbc:mysql://localhost:3306/slow_and_tedious"
val prop = new java.util.Properties
prop.setProperty("user",”scalauser”)
prop.setProperty("password","scalauser123")
val people = spark.read.jdbc(url,"sat",prop)
Login to spark-shell using following command.
spark-shell --jars mysql-connector.jar
you can use following command to execute the code which you created above.
scala> :load /path/test.scala
shell script every time it launch sparkContext which takes more time to execute.
If you use above command it will just execute the code which is there in test.scala.
Since sparkContext will be loaded when you logging into spark-shell, time can be saved when you execute script.

Answer 2

Try this,
Write your code in a filex.txt
In your unix shell script include the following
cat filex.txt | spark-shell
Seemingly, you cant push the script in Background (using &)

Answer 3

you can paste your script in a file,then execute
spark-shell < {your file name}