BeautifulSoup HTML scraping, how to get row after thead in tbody - html

Question

I'm interest in learning about scraping a website. now I learn how to scraping table on the website. I used BeautifulSoup.
I have a simple HTML table to parse but somehow Beautifulsoup I try to get row in tbody but always get word in "thead" ones. . I'm wondering if anyone would take a look at that and see what's wrong. So I already created the rows object from the HTML table:
<table id="companyTable" class="table table--zebra table-content-page width-block dataTable no-footer" role="grid" aria-describedby="companyTable_info" style="width: 868px;">
<thead>
<tr role="row">
<th class="sorting_disabled" rowspan="1" colspan="1" style="width: 41px;">No</th>
<th class="sorting_disabled" rowspan="1" colspan="1" style="width: 224px;">Kode/Nama Perusahaan</th>
<th class="sorting_disabled" rowspan="1" colspan="1" style="width: 267px;">Nama</th>
<th class="sorting_disabled" rowspan="1" colspan="1" style="width: 187px;">Tanggal Pencatatan</th>
</tr>
</thead>
<tbody>
<tr role="row" class="odd">
<td class="text-center">1</td>
<td class="text-center">AALI</td>
<td>Astra Agro Lestari Tbk</td>
<td>09 Des 1997</td>
</tr>
<tr role="row" class="even">
<td class="text-center">2</td>
<td class="text-center">ABBA</td>
<td>Mahaka Media Tbk</td>
<td>03 Apr 2002</td>
</tr>
I'm really really sorry I've already read and tried this Beautifulsoup HTML table parsing--only able to get the last row? . but still, don't get it.. and get '[ ]' at output.
here's the link that I want to scrape it. : https://www.idx.co.id/perusahaan-tercatat/profil-perusahaan-tercatat/
I want to get this row.
<tr role="row" class="odd">
<td class="text-center">1</td>
<td class="text-center">AALI</td>
<td>Astra Agro Lestari Tbk</td>
<td>09 Des 1997</td>
</tr>
I try to get it but always get word in "thead" ones.
here's my code :
from bs4 import BeautifulSoup as soup
from urllib.request import urlopen as uReq
url = 'https://www.idx.co.id/perusahaan-tercatat/profil-perusahaan-tercatat/'
uClient = uReq(url)
pageHtml = uClient.read()
uClient.close()
pageSoup = soup(pageHtml, "html.parser")
table = pageSoup.findAll('table', id = "companyTable")
table = table[0]
for row in table.findAll('tr'):
for cell in row.findAll('th'):
print(cell.text)

Answer 1

You just need the first tr in the tbody tag. So I'd use this:
first_row = s.find('tbody').find('tr')
Where s is the soup in my case. Here's an example:
>>> html = """<table id="companyTable" class="table table--zebra table-content-page width-block dataTable no-footer" role="grid" aria-describedby="companyTable_info" style="width: 868px;">
... <thead>
... <tr role="row">
... <th class="sorting_disabled" rowspan="1" colspan="1" style="width: 41px;">No</th>
... <th class="sorting_disabled" rowspan="1" colspan="1" style="width: 224px;">Kode/Nama Perusahaan</th>
... <th class="sorting_disabled" rowspan="1" colspan="1" style="width: 267px;">Nama</th>
... <th class="sorting_disabled" rowspan="1" colspan="1" style="width: 187px;">Tanggal Pencatatan</th>
... </tr>
... </thead>
... <tbody>
... <tr role="row" class="odd">
... <td class="text-center">1</td>
... <td class="text-center">AALI</td>
... <td>Astra Agro Lestari Tbk</td>
... <td>09 Des 1997</td>
... </tr>
... <tr role="row" class="even">
... <td class="text-center">2</td>
... <td class="text-center">ABBA</td>
... <td>Mahaka Media Tbk</td>
... <td>03 Apr 2002</td>
... </tr>
... """
>>> s = BeautifulSoup(html)
>>> first_row = s.find('tbody').find('tr')
>>> first_row
<tr class="odd" role="row">
<td class="text-center">1</td>
<td class="text-center">AALI</td>
<td>Astra Agro Lestari Tbk</td>
<td>09 Des 1997</td>
</tr>
It works because find only returns the first element that matches

Answer 2

Solving the problem
If I understood it right, you just want to get the table data from this site. However, inspecting the site and analyzing the requests and responses using the Google Network tools, I just found out that the site is using DataTables and fills the table using JS, with the responses from this request.
In other words, you could just have made
import requests
url = "https://www.idx.co.id/umbraco/Surface/Helper/GetEmiten?emitenType=s"
response = requests.get(url)
print(response.json())
What you should learn from this
Inspecting the page elements and requests/responses in order to know what is the easiest way to get the data. The tool I suggest is the Chrome Devtools, but you may use the browser that fits you the best.