Printing results of a REST API call with Spark - json

Question

I would appreciate some help on this: I am doing a request to get a json file with data but I cannot print the final result, it says "name 'collapse_columns' is not defined"
This is my code taken from: https://medium.com/geekculture/how-to-execute-a-rest-api-call-on-apache-spark-the-right-way-in-python-4367f2740e78
import requests
import json
from pyspark.sql.functions import udf, col, explode
from pyspark.sql.types import StructType, StructField, IntegerType, StringType, ArrayType
from pyspark.sql import Row
# Declare a function that will execute our REST API call
def executeRestApi(verb, url, headers, body):
#
headers = {
'content-type': "application/json"
}
res = None
# Make API request, get response object back, create dataframe from above schema.
try:
if verb == "get":
res = requests.get(url, data=body, headers=headers)
else:
res = requests.post(url, data=body, headers=headers)
except Exception as e:
return e
if res != None and res.status_code == 200:
return json.loads(res.text)
return None
# Define the response schema and the UDF
schema = StructType([
StructField("Count", IntegerType(), True),
StructField("Message", StringType(), True),
StructField("SearchCriteria", StringType(), True),
StructField("Results", ArrayType(
StructType([
StructField("Make_ID", IntegerType()),
StructField("Make_Name", StringType())
])
))
])
# ensure that the new column, which is used to execute the UDF, will eventually contain data as a structured object rather than plain JSON
udf_executeRestApi = udf(executeRestApi, schema)
# Create the Request DataFrame and Execute
from pyspark.sql import Row
headers = {
'content-type': "application/json"
}
body = json.dumps({
})
RestApiRequestRow = Row("verb", "url", "headers", "body")
request_df = spark.createDataFrame([
RestApiRequestRow("get", "https://vpic.nhtsa.dot.gov/api/vehicles/getallmakes?format=json", headers, body)
])
#Finally we can use withColumn on the Dataframe to execute the UDF and REST API.
result_df = request_df.withColumn("result", udf_executeRestApi(col("verb"), col("url"), col("headers"), col("body")))
#print
df = result_df.select(explode(col("result.Results")).alias("results"))
df.select(collapse_columns(df.schema)).show()
By printing result.df or request_df I get this bellow but how couuld I access the json data?
+----+--------------------+--------------------+----+------+
|verb| url| headers|body|result|
+----+--------------------+--------------------+----+------+
| get|https://vpic.nhts...|[content-type -> ...| {}| [,,,]|
+----+--------------------+--------------------+----+------+
+----+--------------------+--------------------+----+
|verb| url| headers|body|
+----+--------------------+--------------------+----+
| get|https://vpic.nhts...|[content-type -> ...| {}|
+----+--------------------+--------------------+----+
Many thanks!

Answer 1

If you read the description of the sample solution, it says that it uses a user defined function for the collapse, that can by found here:
https://github.com/jamesshocking/collapse-spark-dataframe