Pyspark Dropping the header in a dataframe, AttributeError: _jdf - csv

Question

from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
spark = sqlContext.sparkSession
avg_calc = spark.read.csv("quiz2_algo.csv", header= True,inferSchema=True)
header = avg_calc.first()
no_header = avg_calc.subtract(header)
no_header
avg_calc contains 2 columns and I am trying to remove the 1st row from both columns, however I am receiving the following error:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-50-24671d91e691> in <module>()
----> 1 no_header = avg_calc.subtract(header)
C:\spark\spark-2.3.0-bin-hadoop2.7\python\pyspark\sql\dataframe.pyc in subtract(self, other)
1391
1392 """
-> 1393 return DataFrame(getattr(self._jdf, "except")(other._jdf), self.sql_ctx)
1394
1395 #since(1.4)
C:\spark\spark-2.3.0-bin-hadoop2.7\python\pyspark\sql\types.pyc in __getattr__(self, item)
1559 raise AttributeError(item)
1560 except ValueError:
-> 1561 raise AttributeError(item)
1562
1563 def __setattr__(self, key, value):
AttributeError: _jdf
If anyone can help I would appreciate it!
Example of the data: avg_calc.show(5)

Answer 1

first() returns a Row object rather than a DataFrame which is required by subtract. See http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame.first
You could try something like:
avg_calc.subtract(avg_calc.limit(1))
For example:
>>> df = spark.createDataFrame([Row(x=1), Row(x=2)])
>>> print(df.subtract(df.limit(1)).toPandas())
x
0 2
Apply an ordering to you dataframe to ensure the row you would like dropped is in the correct location:
>>> from pyspark.sql import functions as F
>>> df = df.orderBy(F.col('CS202 Quiz#2').desc())
>>> df = df.subtract(df.limit(1))