BeautifulSoup IndexError: list index out of range - html

Question

My code below:
import requests
from bs4 import BeautifulSoup
def investopedia():
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0'}
ticker = 'TSLA'
url = f'https://www.investopedia.com/markets/quote?tvwidgetsymbol={ticker.lower()}'
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, 'lxml')
ip_price = soup.find_all('div', {'class':'tv-symbol-price-quote__value js-symbol-last'})[0].find('span').text
print(ip_price)
investopedia()
The class I used while inspecting element (in html):
<div class="tv-symbol-price-quote__value js-symbol-last"><span>736.27</span></div>
736.27 in "span" is the number I need
Please help out a web scraping beginnger here. Thanks in advance!

Answer 1

You get index out of range error because your code can't find any HTML elements you are looking for right now.
Information you are looking for is kept within an iframe. In order to retrieve the data you want, we have to switch to that iframe. One way to do it is using Selenium.
from selenium import webdriver
def investopedia():
ticker = 'TSLA'
url = f'https://www.investopedia.com/markets/quote?tvwidgetsymbol={ticker.lower()}'
driver = webdriver.Chrome()
driver.get(url)
time.sleep(5) # it takes time to download the webpage
iframe = driver.find_elements_by_css_selector('.tradingview-widget-container > iframe')[0]
driver.switch_to.frame(iframe)
time.sleep(1)
ip_price = driver.find_elements_by_xpath('.//div[#class="tv-symbol-price-quote__value js-symbol-last"]')[0].get_attribute('innerText').strip()
print(ip_price)
investopedia()