Working with an in-house developed application, when I go to the web site manually (launch Chrome, type in the URL, etc.) and inspect a particular element, I see that element with the ID attribute as follows:
id="input-145"
When I use Chromedriver and Selenium to run my test, the same element shows up with this ID attribute:
id="input-147"
Has anyone ever seen this before? I'm not sure how to proceed.
I was expecting the ID attribute to remain the same regardless of how the site is accessed.
Some frameworks use dynamic id for input to protect against parsers. I solved this problem using element search by full xpath.
Example:
# xpath: does not work with dynamic id
input = driver.find_element(By.XPATH, "//*[#id='input-147']")
# full xpath: work with dynamic id
input = driver.find_element(By.XPATH, "/html/body/header/div/form/div/input")
Locating by xpath documentation
Related
The XPath selector in Scrapy shell response.xpath('//div[#class="chr-lot-header__bid-details"]//span[#class="chr-lot-header__value-field"] returns an empty list while the same XPath selector selects the right html tag in the "Elements" tab of my Chrome browser.
Here's the website the XPath selector is intended for:
https://www.christies.com/en/lot/lot-5973059
The output I want the XPath selector to produce is "GBP 11,282,500".
I just checked the website you mentioned in your mentioned is getting the required information dynamically loaded, which means it can not be scrapped directly. Because scrapy only scrap statically available data not dynamically loaded data. To scrap dynamically loaded data either you need to mimic real time browser like you can use selenium/playwright and integrate there library inside your scrapy code or you can try to find API calls in network tab, which this required data is being loaded/fetched.
I have the following element that I wish to locate using Robot Framework/Selenium Web Driver:
<div ref="component" class="formio-component formio-component-form formio-component-label-hidden" id="e2cdar9">
If I try using the xpath in chrome's console, it detects it correctly; however, I have found no way of locating it using Robot (Xpath, full xpath, class, I can't use ID since it is randomly generated).
I think it may have something to do with the Ref attribute, since this happens on this element and all elements nested inside (Even elements with simple Name attributes won't be detected by my code), and all of these have that attribute.
Anyone knows what may be happening?
Thanks,
Andres
I am scraping elements from a web page & I can see the element being visible (numeric value) on the web page in a grayed out box ,but when tried to inspect the element I cant find it between the tags. I assumed the URL might be any webservice endpoint & tried to GET from postman but it returned mere HTML code not a JSON response.
In general, we can get values between the tags by finding the element & getting innerText attribute in selenium that too failed as there is no text in between the tags.
I cannot post any URL or responses due to security compliance issues in my org. Please advise any other way I can work around.
Got my answer , tried running JS code "document.querySelector('input[name=assets_tot_mfr]').value;" and ran it through python execute_script. Thanks
Can a website's generated HTML be saved using Canopy? Looking at the documentation under 'Getting Started', I could not find anything related.
You can run arbitrary JavaScript using js, document.documentElement.outerHTML will return the current DOM, so
let html = js "return document.documentElement.outerHTML" |> string
does the trick.
Canopy is a wrapper around Selenium that provides some useful helper functions. But it also provides access to the Selenium IWebElement instances in case you need them, via the element function (halfway down the page; there don't seem to be internal anchors in that page so I couldn't link directly to the function). Then once you have the IWebElement object, your problem becomes similar to this one, where the answer seems to be elem.getAttribute("innerHtml") where elem is the elememt whose content you want (which might even be the html element). Note that the innerHtml attribute is not a standard DOM attribute, so this won't work with all Selenium drivers; it will be dependent on which browser you're running in. But it apparently works on all major Web browsers.
See Get HTML Source of WebElement in Selenium WebDriver using Python for a related question using Python, which has more discussion about whether the innetHtml attribute will work in all browsers. If it doesn't, Canopy also has the js function, which you could leverage to run some Javascript to get the HTML you're looking for -- but if you're having trouble with that, you probably need to ask a Javascript question rather than an F# question.
Firebug shows : <a class="ng-binding" ng-click="goto(menu.MenuInfo.Href)">
FirePath shows : html/body/nav/div[1]/div[1]/ul/li[3]/a
It shows as above when I use Firebug or FirePath to find the web element;
Then I copy it to Selenium IDE Target text and click the find button , But it cannot find the web element.
How can I find the web element and make it run in Selenium IDE to record script?
Automatic XPath detectors are usually not a good choice when trying to figure out the Selenium locator for a specific web element. Especially expressions with numeric indexes (e.g. your li[3]) are likely to change if list/table items are removed, added or resorted.
The best way to locate an element is always by id, as this is always unique (unless you have invalid HTML). But your element doesn't have one, unfortunately.
For <a> elements, it's usually good to use the LinkText for locating the element, provided that a) it's unique and b) your site doesn't have a language toggle functionality, which usually changes all link texts.
Alternatively, you could use the tag name and class via CSS selector:
a.ng-binding
Still, it depends on the structure of your page whether this locator is unique or not. There are no general rules for finding the best locator, just good and bad strategies.