PySpark - Referencing a column named "name" in DataFrame - json

Question

I am trying to use PySpark to parse json data. Below is the script.
arrayData = [
{"resource":
{
"id": "123456789",
"name2": "test123"
}
}
]
df = spark.createDataFrame(data=arrayData)
df3 = df.select(df.resource.id, df.resource.name2)
df3.show()
The script works and the output is
+------------+---------------+
|resource[id]|resource[name2]|
+------------+---------------+
| 123456789| test123|
+------------+---------------+
However, after I changed the text "name2" in the variable arrayData to "name", and referenced it in df3 as below,
df3 = df.select(df.resource.id, df.resource.name)
I got the following error
TypeError: Invalid argument, not a string or column: <bound method alias of Column<b'resource'>> of type <class 'method'>. For column literals, use 'lit', 'array', 'struct' or 'create_map' function.
I think the root cause might be that "name" is a reserved word. If so, how can I go around this?

Answer 1

you can use the bracket notation which suresh mentioned. following is the code
df3 = df.select(df.resource.id, df.resource["name"])
df3.show()
+------------+--------------+
|resource[id]|resource[name]|
+------------+--------------+
| 123456789| test123|
+------------+--------------+
if you want only id & name as column name in you dataframe you can use the following
from pyspark.sql import functions as f
df4 = df.select(f.col("resource.id"), f.col("resource.name"))
df4.show()
+---------+-------+
| id| name|
+---------+-------+
|123456789|test123|
+---------+-------+