I am experimenting with web-crawling techniques with python and selenium webdriver in win10 and firefox.
I encountered the following typical page with a pop-up box asking for consent to allowing for cookies.
https://irs.thsrc.com.tw/IMINT/
There are only the following two clickable tags in the page with the following xpaths.
//html/body/div[2]/form[1]/div[2]/div[1]/div[1]/div[1]/div[1]/span[1]/a[1]
//html/body/div[2]/form[1]/div[2]/div[1]/div[1]/div[2]/table[1]/tr[1]/td[1]/input[1]
People here suggested that I can first get an element via webdriver and then use the two methods of is_enabled() and is_displayed() to together check if the element is clickable.
However, when I get elements with the following statement:
e = WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, xpath)))
I found that all elements (even those in the shadow) that I got via the above statement are both enabled and displayed.
But in fact in the browser, only the two actionable elements in the pop-up message box were clickable.
All the other elements were in the shadow and cannot be clicked (or actionable).
Can someone tell me how to get the correct information which elements are actually actionable via some API ?
Thank you very much.
Farn
Related
I'm about to finish developing an automated program on Python and the only thing that's stopping me it's a web element that gets immediately hidden when I try to inspect it from an OpenSea page (can't share the link here since it requires to be connected to my wallet):
So far, I have only managed to get the XPATH and the CSS_SELECTOR of this problematic element (div#tippy-298) which is the following:
button_problematic_xpath = driver.find_element(By.XPATH, '//*[#id="tippy-298"]')
button_problematic_css = driver.find_element(By.CSS_SELECTOR, '#tippy-298')
But what I need is to get the XPATH or CSS_SELECTOR of the Web Elements that button_problematic_css contains, more specifically the "Starting" and "Ending" Textboxes :
Also, just in case, the element that's on top of button_problematic_css is called "Duration", here's its XPATH:
button_duration = driver.find_element(By.XPATH, '//*[#id="duration"]').click #after clicked it deploys div#tippy-298 element
I was wondering if there could be any method to force the visibility of this button_problematic_css using the Chrome Console in the Inspection of Web Elements, or by using an specific method from Selenium on Python3.x to inmediately download all of the web elements this page has right after button_problematic_css is visible.
What I have tried so far is to manually inspect the element, by pressing Ctrl + Shift + C and and then clicking the elements contained in button_problematic_css which just ends up hiding this element before even clicking its content, the same happens when trying to display the contents of this element in the DOM of this page, it just dissapears.
I just wish my program could be capable of editing the dates in the "Starting" and "Ending" Textboxes
I guess you need to click the button to show the element. why did I guess? you need to show the url
or u can also try this code:
clickmore=WebDriverWait(driver, 2).until(EC.element_to_be_clickable((By.XPATH, 'blabalabla')))
self.driver.execute_script("arguments[0].click();", clickmore)
button_duration = driver.find_element(By.XPATH, '//*[#id="duration"]').click()
After several days of working, I managed to solve this, it happened that in order to get the XPATH from these particular elements, I had to use the Chrome Console and type $('#duration').click();.
In this case, #duration is a Selector that can be used to get the web element button that allows you to edit the period of time you want to set when clicked.
I guess the general statement would be something like this:
$('#your_css_selector').click();
Now, this way I could managed to look deeper in the DOM without making them hide again.
I'm working on an application that has many links. They all open in the same window, until today. All of a sudden, in all browsers I'm testing in, 3 links in an iframe or object (I've tried both) start opening in a new window. I can't seem to stop this.
An object example follows. The dolnks program generates 3 simple links like the one following the object example and these links open in a new window.
<OBJECT ID='fixed' DATA='dolnks.cgi?str=$params' TARGET='dynamic' NORESIZE></OBJECT>
darea.cgi?str=$dogstr
Can someone help me understand this and how to get these links to open in the same window.
All links that now open in the same window or should be opening in the same window are from
the same domain.
I now have to close the link instead of using the back button.
Thanks,
craigt
I imagine it's because of your TARGET attribute, although it's not using one of the special target values:
target
Where to display the linked URL, as the name for a browsing context (a tab, window, or ). The following keywords have special meanings for where to load the URL:
_self: the current browsing context. (Default)
_blank: usually a new tab, but users can configure browsers to open a new window instead.
_parent: the parent browsing context of the current one. If no parent, behaves as _self.
_top: the topmost browsing context (the "highest" context that’s an ancestor of the current one). If no ancestors, behaves as _self.
From the MDN Docs: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#attributes
Note though that the docs for object do not list a target attribute, so that behavior is apparently undefined, and probably varies depending on the browser, and the plugin displaying the object.
Check to see if it's the items with a target attribute (case does not matter) that are working "wrong", and see if removing that attribute fixes it. If that's not it, next check to see if there are javascript being loaded. Try turning javascript off (hopefully the relevant links are not generated with javascript) and see if that fixes the behavior. If turning javascript off is too heavy handed, you can use the javascript console to see what listeners are attached to the links.
I'm trying to automate the logging in for one of our applications and trying to click on 'Reporting' (as highlighted in yellow). When i inspect it in firefox, it says its an anchor web element (with this version of firefox i'm not able to install firebug/firepath). This page loads immediately after i enter the username/password and click the 'Submit' button on the previous page. The script always gives me error that 'element not visible'. I tried even explicit wait for that element by using the xpath "//ul[#id='mainnav-menu']/li[2]/a" or "//ul[#id='mainnav-menu' and #class='list-group']/li[2]/a". Could you please help to determine why i'm getting this exception all the time. Basically what i'm trying to do is click on this element which will show 2 links once that arrow beside it expands. i need to click on the 1st child element after it shows up.
Please see the HTML code portion for the element from the application in the image below
The error means that element exists (html is loaded), but not visible. It may be out of visible part of the screen or because of java script effects.
Following up on How to rewrite URLs referenced by Javascript code? I'd like to overlay a button on top of someone else's website (e.g. overlay a Paypal button alongside Stackoverflow's bounty button) and have the two <iframe>s scroll together. The button would reside in the top layer. The website would reside in the bottom layer.
I understand that transparent <iframe>s were/are abused for clickjacking but the browser security mechanism seems to block legitimate use-cases. In my case the user is seeing the same button he/she is clicking. It's even possible that this is a browser bug.
Here is what I see under Chrome:
The top <iframe> intercepts all mouse clicks, even for areas that do not contain any components. Meaning, users cannot interact with the bottom layer at all.
If I style the top <iframe> with pointer-events: none the opposite problem occurs: users are able to see the top layer but all mouse clicks go to the bottom layer. Applying pointer-events: auto to child components does not help (clicks still pass through to the bottom layer).
If I size and position the top <iframe> so its area is equal precisely to the button I am trying to overlay, then mouse clicks go to the right layer but the top layer fails to scroll alongside the bottom layer. Meaning, the button always remains in the same absolute position as the bottom layer scrolls.
Is it possible for me to position a button in the top layer so that it always aligns with a certain position in the bottom layer? In the example where I position a Paypal button alongside the Stackoverflow bounty, I expect the Paypal button to scroll off the page as the user scrolls down the question.
https://stackoverflow.com/a/4087397/14731 leads me to believe this is not possible. Is there another way to implement this?
UPDATE: Here is a jsfiddle for you to play with. The test button is found to the right of "NEWS & VIEWS" in the middle of the page.
Update: New approach
Following discussion with Gili below, the requirement for the solution to work across multiple pages made me rethink my solution.
The new approach:
Doesn't require any code changes or specific features on the target site.
Works on every page while the user navigates (as long as they stay on the same domain)
Could be tweaked to inject any HTML/JS into any DOM element within any target page
My solution works as follows:
Send the person you want to demonstrate Widget X to an email with a link to your instructions page
That instructions page contains a bookmarklet which they add to their bookmarks bar
They visit their own site and click your bookmarklet
Your bookmarklet injects javascript into their page
That javascript creates a pop-up window with content that appears to belong to the target domain, since that domain generated the popup
That pop-up window then monitors the DOM on the target browser window (window.opener) and injects our arbitrary HTML whenever the current page doesn't contain our target node ID.
It seems to work well in my testing (perfect in Chrome, haven't tested across all browsers yet) and seems to work on every target site from StackOverflow to Twitter.
Live demo here: How to demo an web widget on a third party site without having access to their code
Sample code below, expanded for readability:
s = "<script type='text/javascript'>setInterval(function() { if(!window.opener.document.getElementById('gctrlPixelator')) {var i=document.createElement('IMG');i.src='//lorempixel.com/400/200/';i.id='gctrlPixelator';i.style.cssText='top:20;right:20;position:absolute;z-index: 9999;';window.opener.document.getElementsByTagName('body')[0].appendChild(i);}},500);</script>";
t = "<div style='text-align: center; font-family: Arial, Helvetica;'><h1 style='font-size: 18px;'>Demo running!</h1> Keep this window open and return to the main site window to continue the demo.</div>";
w = window.open('','name','height=200,width=400');
w.document.write(s);
w.document.write(t);
The above should be converted into a bookmarklet before deploying to an 'instructions' landing page for a potential client.
Original solution
First, sorry for posting this as an answer rather than as a comment. I've been thinking about this for half an hour, and only just realised I need 50 reputation to comment. So, apologies, but I wanted to share...
I agree that it's unlikely there is a cross browser way to do this double iframe trick. I read your other question about javascript URL rewriting and it lead me to an idea: Rather than trying to embed/hijack their site on another URL, how about creating a bookmarklet that allows you to inject your Javascript into their page?
It could work like this:
Direct them to your site where you host the custom bookmarklet link. Ask them to add it to their favorites.
Ask them to go to their own site, then click your bookmarklet in their bookmarks.
This would inject your JS into their page, allowing you to edit the DOM any way you wanted (e.g. changing styles, adding DOM elements, etc.)
Code something like this, converted to a bookmarklet (i.e. wrapped in a function with a javascript: at the start) could do the trick...
var script = document.createElement("script");
script.setAttribute("type", "text/javascript");
script.setAttribute("src", "http://www.example.com/file.js");
document.getElementsByTagName("head")[0].appendChild(script);
I've tested that in Chrome and it seems happy to load the JS cross-domain. The only rule seems to be that the protocols must match (http or https).
Not quite what you asked for, but a possible solution.
My problem is this - when I click the down button on any input control on my form in Chrome a popup window is displayed. I am positive my code does not do it. The fact that it happens only in Chrome makes me think of some misbehaving Chrome plugin/extension.
Anyway, I would like to see the HTML element responsible for this popup. However, I cannot find it in the dev tools and trying to focus on it does not work - the popup closes the moment I click the page.
So, my question - is there a easy way to get hold on this HTML element without clicking the page?
I am using Chrome 23.0.1271.64 with the following extensions (according to the Chrome itself):
Advanced REST client 3.0.30
JSONView 0.0.32
OneClickDownload 1.2 Web
Developer 0.4.1
This looks just like the standard input field suggestion box. I suspect it is not part of the web page. Therefore it won't appear in the DOM and you can't style it, because it is part of the browser not the web page (although it appears over the top of the web page).
You can suppress it by putting the attribute & value autocomplete=off on the form field, although that is from HTML5 and will not work in all browsers just yet. See Is there a W3C valid way to disable autocomplete in a HTML form?