accessing a sub column of another column from wikipedia table using Beautifulsoup - csv

Question

I have a project that requires us to download and read a table from Wikipedia and and use those information for calculation.
wikipedia page is https://en.wikipedia.org/wiki/List_of_United_States_cities_by_crime_rate#Criticism_of_ranking_crime_data
it is required we take the Total Violent crime of each city,state listed ( states are repeated) However all the cells for that columns only have tag to it. it is under one table the question is how would I use beautifulsoup to read that specific column which is under the Violent Crime column
i have scoured the internet and i have landed on many choices from here and other websites but they are not really helping in this specific case But here is the code that i currently have that can take all the values from the table. most variables are holders while i test because i been going at it for a few days
state = soup.find_all('th', limit = 7)
for row in state:
row_data = row.get_text(strip = True, separator = '|').split('|')[0:1]
outfile.write(str(row_data)+ "\n")
umber = soup.find_all('td')
for column in number:
column_data = column.get_text(strip = True, separator = '|').split('|')[0:1]
outfile.write(str(column_data)+ "\n")
i basically want to store those information into a list sort of for later use and then use the links to each cities and get their cords and then reference it to a few cities in Texas for closest to border
We are only allowed to use BeautifulSoup and CSV no Pandas or NumPy
Edit:
the Write out functions are only for testing as well. Its only to see if its grabbing the information of the table correctly. My IDE console cant display all of them so writing it out was the next best thing i could think of

Answer 1

Looks like it's just an issue of creating your lists. You can do this by initializing your list, and then adding your list to that. Or you can append each of the items in your for loop to it. Or you can make that more concise by doing a list comprehension.
The reason you get nothing back, is because you keep overwriting your row_data and column_data in the loop. And it'll write to file however, it's going to put a new line after each, when I'm assuming you'd want to write the whole row, and then do a new line, so I'd also then put your write after the list is created/complete:
Combining the list to a list:
row_data = []
for row in state:
row_data = row_data + row.get_text(strip = True, separator = '|').split('|')[0:1]
outfile.write(str(row_data)+ "\n")
number = soup.find_all('td')
column_data = []
for column in number:
column_data = column_data + column.get_text(strip = True, separator = '|').split('|')[0:1]
outfile.write(str(column_data)+ "\n")
Appending an item/element to a list:
# Initiate and then append to a list
row_data = []
for row in state:
row_data.append(row.text)
outfile.write(str(row_data)+ "\n")
number = soup.find_all('td')
column_data = []
for column in number:
column_data.append(column.text)
outfile.write(str(column_data)+ "\n")
List Comprehension:
#List comprehension
row_data = [ row.text for row in state ]
outfile.write(str(row_data)+ "\n")
column_data = [ column.text for column in number ]
outfile.write(str(column_data)+ "\n")
As far as getting those sub-columns, it's tricky because those aren't child tags. They are however, the next <tr> tag after the <th> tag that you pull, so we could use that.
import bs4
import requests
import csv
url = 'https://en.wikipedia.org/wiki/List_of_United_States_cities_by_crime_rate#Criticism_of_ranking_crime_data'
response = requests.get(url)
soup = bs4.BeautifulSoup(response.text, 'html.parser')
# Only want State and City so limit = 2
headers = soup.find_all('th', limit = 2)
sub_headers = headers[0].findNext('tr')
# Initiate and then append to a list
header_data = []
for data in headers:
header_data.append(data.text.strip())
sub_header_data = []
for data in sub_headers.find_all('th'):
sub_header_data.append(data.text.strip())
# Only want to append the first Total column from the sub_headers
header_data.append(sub_header_data[0])
with open('C:/test.csv', mode='w', newline='') as outfile:
writer = csv.writer(outfile)
writer.writerow(header_data)
table_body = soup.find_all('tbody')[1]
rows = table_body.find_all('tr')
for row in rows:
tds = row.find_all('td', limit = 4)
#Skip the blank rows of data
if tds == []:
continue
tds_data = []
for data in tds:
tds_data.append(data.text.strip())
#Remove the Population number/data
del tds_data[2]
writer.writerow(tds_data)