Trying to convert str object to DataFrame: web scraping with python - json

Question

System: WIN10
IDE: MS VSCode
Language: Python version 3.7.3
Library: pandas version 1.0.1
Data source: https://hoopshype.com/salaries/#hh-tab-team-payroll
Dataset: team payrolls
I am having an issue for some reason when trying to convert a str(table) into a dataframe. I think I am grossly missing something here. The sample code is supplied below and it keeps throwing only the top row of data into the data frame. I think I am missing some other transformation process.
Steps were taken:
searched the net for various tutorials
tried splitting the data first then trying the conversion on it to no luck
Code:
# import Libraries
import pandas as pd
import requests
from bs4 import BeautifulSoup
from tabulate import tabulate
# data visualization
import matplotlib as plt
import seaborn as sns
# setting: For output purposes to show all columns
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', 2000)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)
# parsing the webpage
url = 'https://hoopshype.com/salaries/#hh-tab-team-payroll'
r = requests.get(url)
data = r.text
# create a beautfulsoup object
soup = BeautifulSoup(r.content,'lxml')
soup.prettify
# team salaries by year = tsby
table = soup.find_all('table')[0]
nbatsby = pd.read_html(str(table))
# this is where I am stuck
df = pd.DataFrame(nbatsby)
df.head(100)

Answer 1

pandas has read_html which can scrape an html table an convert it to dataframe directly :
import pandas as pd
import requests
r = requests.get("https://hoopshype.com/salaries/#hh-tab-team-payroll")
data = pd.read_html(r.text, attrs = {'class': 'hh-salaries-ranking-table'})[0]
print(data)
Output:
Unnamed: 0 Team 2019/20 2020/21 2021/22 2022/23 2023/24 2024/25
0 1.0 Portland $XXX,XXX,XXX $XXXX,XXX,XXX $XX,XXX,XXX $XX,XXX,XXX $XX,XXX,XXX $XX,XXX,XXX
1 2.0 Miami $XXX,XXX,XXX $XX,XXX,XXX $XX,XXX,XXX $XX,XXX,XXX $0 $0
................................................................................................................
28 29.0 Indiana $XXX,XXX,XXX $XXX,XXX,XXX $XX,XXX,XXX $XX,XXX,XXX $XX,XXX,XXX $0
29 30.0 New York $XXX,XXX,XXX $XX,XXX,XXX $XX,XXX,XXX $0 $0 $0