Read JSON in pySpark with custom schema in GCP Dataproc - json

Question

In GCP Dataproc (with pySpark), I am doing a task i.e. to read JSON file as per custom schema and load it in a Dataframe.
I do have following sample testing JSON:
{"Transactions": [{"schema": "a",
"id": "1",
"app": "testing",
"description": "JSON schema for testing purpose"}]}
I have created following schema:
custom_schema = StructType([
StructField("Transactions",
StructType([
StructField("schema", StringType()),
StructField("id", StringType()),
StructField("app", StringType()),
StructField("description", StringType())
])
)])
Reading JSON as:
df_2 = spark.read.json(json_path, schema = custom_schema)
Getting following results,
Now, I need to check data in Dataframe, When I try to do df_2.show(), it is taking too much time and show as kernel Busy for hours.
I need help, that what I am missing here in code and how can I see the data in dataframe (Tabular format).

Answer 1

I think the problem is with your custom schema definition and the JSON file. The following code and JSON file worked for me:
Code
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType
from pyspark.sql.types import StructField
from pyspark.sql.types import StringType
from pyspark.sql.types import ArrayType
spark = SparkSession \
.builder \
.appName("JSON test") \
.getOrCreate()
custom_schema = StructType([
StructField("schema", StringType(), False),
StructField("id", StringType(), True),
StructField("app", StringType(), True),
StructField("description", StringType(), True)])
df = spark.read.format("json") \
.schema(custom_schema) \
.load("gs://my-bucket/transactions.json")
df.show()
JSON file
The contents of gs://my-bucket/transactions.json is:
{"schema": "a", "id": "1", "app": "foo", "description": "test"}
{"schema": "b", "id": "2", "app": "bar", "description": "test2"}
Output
+------+---+---+-----------+
|schema| id|app|description|
+------+---+---+-----------+
| a| 1|foo| test|
| b| 2|bar| test2|
+------+---+---+-----------+