Spark : Exception in thread "main" java.lang.ClassNotFoundException: com.mysql.jdbc.Driver - mysql

Question

I wrote simple program in spark to write a dataframe to table in mySql.
The program is as follows:
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.SparkContext._
import org.apache.spark.rdd._
//import org.apache.spark.rdd.RDD
import org.apache.spark.sql.types._
import org.apache.spark.sql.Row;
import java.util.Properties
import java.sql.{ Connection, DriverManager, SQLException }
object MySQLTrial {
def main(args: Array[String]) {
val sparkConf = new SparkConf().setAppName("AnalyseBusStreaming")
val sc = new SparkContext(sparkConf)
val df = sc.parallelize(Array((1, 234), (2, 1233)))
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._
val prop = new Properties()
prop.put("user", "admin")
prop.put("password", "admin")
val driver = "com.mysql.jdbc.Driver"
Class.forName(driver)
val dfWriter = df.toDF().write.mode("append")
dfWriter.jdbc("jdbc:mysql://127.0.0.1:3306/IOT_DB", "table1", prop)
}
}
The POM file for my project is as follows
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>ggi.bigdata</groupId>
<artifactId>internet_of_things</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.6.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.10</artifactId>
<version>1.6.0</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.38</version>
</dependency>
</dependencies>
</project>
I'm running this program using spark submit (tried on local and yarn mode). I haven't included any jar files explicitly to run this code. I keep getting the error :
Exception in thread "main" java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
What should be done about this ?

Answer 1

This is because your driver isn't present in the uber-jar that you are submitting to the cluster whether it's a standalone cluster or yarn or mesos, etc.
Solution 1 : Since you are using maven, you can use the assembly plugin to build your uber-jar with all the needed dependencies. More information about maven assembly plugin here.
Solution 2 : Provide these dependency libraries on runtime when you submit your application using the --jars option. I advice your to read ore information about advanced dependencies management and submitting applications in the official documentation.
e.g it can look like this :
./bin/spark-submit \
--class <main-class>
--master <master-url> \
--jars /path/to/mysql-connector-java*.jar
I hope this helps !

Answer 2

Eliasah was right. M2Eclipse does create a jar file but it's not a fat/uber jar. If I explicitly install the "maven assembly" plugin on the eclipse, I am able to create a fat jar with the dependency jars included and hence the program runs.