Python 3: How to web scrape text from div that contains multiple class values - html

Question

I'm trying to web scrape a website (Here is the link to website), but the div in the page seems to have multiple class attributes which is making me hard to scrape the data. I tried to look for historical questions posted on Stackoverflow, but could not find an answer that I wanted. The below is part of the code I extracted from the website:
<div data-reactid="118">
<div class="ue-ga base_ ue-jk" style="margin-left:-24px;margin-bottom:;" data-reactid="119">
<div style="display: flex; flex-direction: column; width: 100%; padding-left: 24px;" data-reactid="120">
<div class="ue-a3 ue-ap ue-a6 ue-gb ue-ah ue-n ue-f5 ue-ec ue-gc ue-gd ue-ge ue-gf base_ ue-jv ue-gz ue-h0 ue-h1" data-reactid="121">
<div class="ue-a6 ue-bz ue-gb ue-ah ue-gg ue-gh ue-gi" data-reactid="122">
<div class="ue-bn ue-bo ue-cc ue-bq ue-g9 ue-bs" title="Want to extract this part" data-reactid="123">
Want to extract this part
</div>
</div>
</div>
</div>
</div>
</div>
What I want to extract is the text where it states "Want to extract this part". I did think of scraping the data through data-reactid, but different pages have different data-reactid number assigned so wasn't a good idea. I also want to inform that class names are not unique.
Can anyone guide me through this? Much appreciated.

Answer 1

If the classes always remain the same for that specific element on each page you can target it with this selector:
.ue-bn.ue-bo.ue-cc.ue-bq.ue-g9.ue-bs
However, there are many other selectors you could use but it all depends on if they are unique and consistent across pages.

Answer 2

This may help you
from bs4 import BeautifulSoup
html = """<div data-reactid="118">
<div class="ue-ga base_ ue-jk" style="margin-left:-24px;margin-bottom:;" data-reactid="119">
<div style="display: flex; flex-direction: column; width: 100%; padding-left: 24px;" data-reactid="120">
<div class="ue-a3 ue-ap ue-a6 ue-gb ue-ah ue-n ue-f5 ue-ec ue-gc ue-gd ue-ge ue-gf base_ ue-jv ue-gz ue-h0 ue-h1" data-reactid="121">
<div class="ue-a6 ue-bz ue-gb ue-ah ue-gg ue-gh ue-gi" data-reactid="122">
<div class="ue-bn ue-bo ue-cc ue-bq ue-g9 ue-bs" title="Want to extract this part" data-reactid="123">
Want to extract this part
</div>
</div>
</div>
</div>
</div>
</div>"""
soup = BeautifulSoup(html,'html.parser')
tag = soup.find('div', attrs={'class':'ue-bn'})
text = (''.join(tag.stripped_strings))
print (text)

Answer 3

you can use jQuery as below.
$("div[title=Want to extract this part]").text();

Answer 4

Menus:
- all menus to use in loop, css selector: div.base_ h3
- menu by name, xpath: //div[contains(#class,'base_')]//h3[.='Big Mac® Bundles']
Food Cards
- titles, css selector: div[title]
- titles, xpath: //div[./div[#title]]/div[#title]
- prices, xpath: //div[./div[#title]]//span
If you want to loop:
cards = driver.find_elements_by_xpath("//div[./div[#title]]")
for card in cards:
title = card.find_element_by_css_selector("div[title]")
price = card.find_element_by_css_selector("span")
#or using xpath
#title = card.find_element_by_xpath("./div[#title]")
#price = card.find_element_by_xpath(".//span")
Category menu:
- all categories, css selector: a[href*='category']

Answer 5

As per the HTML you have shared to extract the text Want to extract this part as the element is a React element you have to induce WebDriverWait for the element to be visible and you can use either of the following solution:
Using title attribute:
myText = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.base_ div[title]"))).get_attribute("title")
Using innerHTML:
myText = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.base_ div[title]"))).get_attribute("innerHTML")
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC