Unable To Read local JSON File using spark submit - json

Question

I am reading JSON file in scala spark using
val df=spark.read.json(properties.getProperty("jsonFilePath"))
This code works when I run from my IDE(Intellij)
But when I try to execute using spark-submit command, it gives the below message
INFO FileScanRDD: Reading File path: file:///Users/path/to/json/file.json, range: 0-8822, partition values: [empty row]
I am not able to process the JSON data due to this. Any idea what could be happening here?
Here is my spark submit command:
${SPARK_HOME_DIR}/bin/spark-submit --master local ${SCALA_JAR_LOC}/<JARNAME>.jar
I tried providing it as part of spark-submit using --files option as well. Need help

Answer 1

spark.read.json supports reading data from a filesystem that's supported by Hadoop. If the JSON is part of the jar that contains your jar you can use the getClass.getResourceAsStream java API to read the json from the classpath of the job.
To read the JSON file from your classpath
def read(file: String): String = {
val stream = getClass.getResourceAsStream(s"/$file")
scala.io.Source.fromInputStream(stream)
.getLines
.toList
.mkString(" ")
.trim
.replaceAll("\\s+", " ")
}
Since you want to read that JSON as a dataframe in your code, you might have to convert the String to an RDD and then to a single record dataframe.
val inputAsRDD = sparkCtxt.parallelize(List(read("/path/within/resources/folder")))
val df = sparkSession.read.json(inputAsRDD)