In Chrome, in F12-Network there are lots of network requests, for example for stackoverflow homepage there are:
- stackoverflow.com
- jquery.min.js
- stub.en.js?v=xxxxxxxxx
- stacks.css?v=xxxxxxxxx
- ...
Is it possible to get these requests?
It's not clear from question, but I guess you were to access all these requests via Selenium API. Yes, it became possible with Selenium 4.
Sample below shows catching requests on the google page
ChromeDriver driver = new ChromeDriver(); // driver should be of type ChromeDriver
DevTools chromeDevTools = driver.getDevTools();
chromeDevTools.createSession();
chromeDevTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));
chromeDevTools.addListener(Network.requestWillBeSent(),
req -> {
System.out.println(String.format("Sent %s request to %s",
req.getRequest().getMethod(),
req.getRequest().getUrl()
));
});
driver.get("https://www.google.com");
chromeDevTools.send(Network.disable());
driver.close();
Related
Description/background
I had set up a script which opened a Google site of our company in Google Chrome (not headless) and did some automated work on that page. The login information had to be refreshed occasionally what for I manually logged in. That had been working perfectly the last couple of months until last week. Today I noticed that I get the above mentioned error message as a result of a server response with HTTP status 400 upon entering my Gmail address and clicking the Next button.
Steps to reproduce
Puppeteer version: 2.0.0
Platform / OS version: Windows 10
URLs (if applicable): https://sites.google.com/...
Node.js version: v12.13.0
What steps will reproduce the problem?
Run a Puppeteer script to open a Google Site which requires login.
(async () => {
try {
const browser = await puppeteer.launch({headless: false, userDataDir: "<ProfileDirectory>"});
const pageLogin = await browser.newPage();
await pageLogin.goto('https://sites.google.com/...', {waitUntil: 'networkidle2'});
...
await browser.close();
}
catch (error) {
console.log(error.stacktrace);
}
})();
Manually enter Gmail address and click Next.
Get error message "Sorry, something went wrong there. Try again." as a result of a server response with HTTP status code 400.
Update:
Manually opening Chrome (same userDataDir) and the respective Google site still works as usual.
Recommend to use playwright/puppeteer + Firefox. It seems like google adds something into chrome so they can detect the browser is automated or not
One of the comments on this post mentions that Google tries to block logins with Puppeteer, Selenium etc. this might be why you are getting a 400 error.
One of the recent comments on the aforementioned post, links a gist with some example code that might still work, haven't tried it though.
While I was doing research on Puppeteer for Firefox, I noticed that (1) Puppeteer downloads its own local Google Chrome binaries it is executing and (2) my installed Puppeteer version 2.0.0 was outdated. Meaning, the browser actually used by Puppeteer was probably also outdated. The solution was as easy as to update Puppeteer to the latest version 2.1.1.
In my Selenium-Test (with chromedriver-2.24) I'm trying to access my webpage via basic authentication with the following statement:
WebDriver driver = ...;
driver.get("http://admin:admin#localhost:8080/project/");
But Google Chrome gives me the following warning in the console:
[Deprecation] Subresource requests whose URLs contain embedded credentials (e.g. https://user:pass#host/) are blocked. See https://www.chromestatus.com/feature/5669008342777856 for more details.
In the tagged link is mentioned that the support was dropped:
Drop support for embedded credentials in subresource requests. (removed)
My question now is, is there an other way to basic-authenticate from Selenium?
NOTE: this has not helped: How to Handle HTTP Basic Auth headers in Selenium Webdriver using Java ?
The basic authentication via url is blocked only for sub resources.
So you could still use it on the domain:
driver.get("http://admin:admin#localhost:8080");
driver.get("http://localhost:8080/project");
You could also create a small extension to automatically set the credentials when they are requested:
options = webdriver.ChromeOptions()
options.add_extension(r'C:\dev\credentials.zip')
https://gist.github.com/florentbr/25246cd9337cebc07e2bbb0b9bf0de46
There were some updates in this link as :
Chromium Issue 435547 Drop support for embedded credentials in subresource requests. (removed)
We should block requests for subresources that contain embedded credentials (e.g. "http://ima_user:hunter2#example.com/yay.tiff"). Such resources would be handled as network errors.
However, Basic Authentication functionality still works with Selenium 3.4.0, geckodriver v0.18.0, chromedriver v2.31.488763, Google Chrome 60.x and Mozilla Firefox 53.0 through Selenium-Java bindings.
Here is the example code which tries to open the URL http://the-internet.herokuapp.com/basic_auth with a valid set of credentials and it works.
Firefox:
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
public class BasicAuthentication_FF
{
public static void main(String[] args)
{
System.setProperty("webdriver.gecko.driver", "C:\\Utility\\BrowserDrivers\\geckodriver.exe");
WebDriver driver = new FirefoxDriver();
driver.navigate().to("http://admin:admin#the-internet.herokuapp.com/basic_auth");
}
}
Chrome:
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
public class BasicAuthentication_Chrome
{
public static void main(String[] args)
{
System.setProperty("webdriver.chrome.driver", "C:\\Utility\\BrowserDrivers\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArguments("start-maximized");
options.addArguments("disable-infobars");
options.addArguments("--disable-extensions");
WebDriver driver = new ChromeDriver(options);
driver.navigate().to("http://admin:admin#the-internet.herokuapp.com/basic_auth");
}
}
Florent B.'s approach of calling .get on the URL twice worked for me with a slight modification. In JS:
driver
.get('http://admin:admin#localhost:8080')
.then( () => driver.get('http://localhost:8080') )
working on google chrome 62.0.3202.94 with ChromeDriver 2.33.506092 and the approach seems compatible with firefox 56.0.2 with geckodriver 0.19.1, and phantomjs 2.1.1 all under Debian linux 9.
What I believe is happening is the first call sets up the Authorization header sent by the browser. The second call removes the credentials from the URL and the credentials no longer are applied to subresources. The then synchronizes the two requests ensuring order.
New features for chrome and basic authentication via remote-debug: just for linking it here, so people who are stuck can find a solution for chrome and more: Chrome remote debugging in a seleniumgrid
Selenium 4 supports basic authentication
WebDriver driver = new ChromeDriver();
HasAuthentication authentication = (HasAuthentication) driver;
authentication.register(() -> new UsernameAndPassword("username", "pwd"));
driver.get("your-site.com");
https://www.selenium.dev/blog/2021/a-tour-of-4-authentication/
Such Basic authentication will not be supported directly using the selenium driver.get(URL) method to load a URL prompting for authentication within a JavaScript Popup, I was also stuck here for a long time. It's because of Chrome driver will not allow such authentication techniques after the update 59 (probably). There are still backdoors via Selenium using the JavaScript engine in the browser to load such URLs.
driver.get("https://www.google.com");
JavascriptExecutor jse = (JavascriptExecutor) driver;
URL = "https://username:password#www.example.com";
jse.executeScript("window.open('"+URL+"')");
I need to load URL which has authentication pop up. I used http://username:password#exampl.com format to access the website via selenium webdriver on chrome browser.
But this support seems t be dropped in latest chrome browser, So loading the url is not successful. Browser keeps loading for a long time with no response from server.
Is there any alternate way to overcome authentication pop-up with selenium webdrier ? I know we can use sikui,autoit etc.But i need to run the same code on many platforms - Win/Mac/mobile/tablet.
So please let know how to overcome this.
I am using Selenium 3.4.0 and Chromedriver 2.31, and loading a URL with authentication credentials works fine for me.
As per I know chorme stop URL authentication from chrome 60.x version.
You can try below code or downgrade your chrome version
The Alert Method, authenticateUsing() lets you skip the Http Basic Authentication box.
WebDriverWait wait = new WebDriverWait(driver, 10);
Alert alert = wait.until(ExpectedConditions.alertIsPresent());
alert.authenticateUsing(new UserAndPassword(username, password));
Hope it help you :)
I am trying to write code in Selenium Webdrvier 3.0 + Java 1.8 + Chrome 53 for an application which needs to download and execute a .jnlp file after invoking a get(url). I am not sure whether this could be handled in Selenium webdriver or not?
As I am new to selenium any help or guidance for handling these Windows Pop will be really helpful for me.
Below is the piece of code :
if(browser.contains("CHROME") || browser.equalsIgnoreCase("chrome"))
{
ChromeOptions options = new ChromeOptions();
options.addArguments("--test-type");
options.addArguments("--disable-extensions");
capability = DesiredCapabilities.chrome();
capability.setBrowserName("chrome");
capability.setCapability(ChromeOptions.CAPABILITY, options);
}
capability.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
browserDriver = new RemoteWebDriver(new URL(nodeAddress), capability);
browserDriver.manage().timeouts().pageLoadTimeout(1000, TimeUnit.SECONDS);
browserDriver.manage().timeouts().implicitlyWait(90, TimeUnit.SECONDS);
browserDriver.manage().window().maximize();
browserDriver.get(applicationUrl);
logger.info("WebDriver successfully defined with Session ID:" + browserDriver.getSessionId() + ", Page Title:" + browserDriver.getTitle() + " and URL: " + browserDriver.getCurrentUrl());
Image attached : http://i.stack.imgur.com/esfpk.jpg
The button you are trying to click is part of Chrome's UI and not part of the webpage so Selenium cannot interact with it. From the googling I've done, this is apparently an issue that people have reported as a bug and there is no consistent workaround.
I've found a workaround via Selenium + Python (but similar thing should be feasible in Java too). In Python I just click programmatically on the notification in Chrome UI to allow downloading and installation of the jnlp file (with help of win32api and win32con - I needed it for Win only). See details here
Webscraping is now much easier for me :)
I'm writing some selenium tests for a HTML5 player playing DRM content, the player works fine in Chrome when I test it manually, but nothing is loaded or played in the latest chrome driver if I run my test cases.
Is it because of the drm content isn't authorized to play in chrome driver or something else?
I have no issues running tests for other functions written in selenium.
Any ideas?
Chromedriver launches Chrome with --disable-component-update switch by default, which disables the NaCl (Native Client) support, which is in turn required to load DRM modules (e.g. Widevine Modular DRM).
To get around this, you need to tell the driver not to launch Chrome with this switch, by building the driver with excludeSwitches option, specifying disable-component-update parameter. For example (JS):
var webdriver = require('selenium-webdriver');
var chrome = require("selenium-webdriver/chrome");
var capabilities = new webdriver.Capabilities.chrome();
var chromeOptions = {
'args': ['--user-data-dir=C:/ChromeProfile'], // start with pre-configured Chrome profile
'excludeSwitches': ['disable-component-update'] // stop breaking Native Client support
};
capabilities.set('chromeOptions', chromeOptions);
var driver = new webdriver.Builder().
withCapabilities(capabilities).
build();
driver.get('http://...');
Or using Python bindings:
from selenium import webdriver
def buildDriver():
options = webdriver.ChromeOptions()
options.add_experimental_option('excludeSwitches', ['disable-component-update'])
options.add_argument('--user-data-dir=C:/Temp/ChromeProfile')
return webdriver.Chrome(chrome_options=options)
Hope that helps..
-- ab1
Issue 886: Enabled PNaCl Components in ChromeDriver - Enhancement
If you cannot get #Chainik's answer to work, try this out. It worked for me.
As per https://bugs.chromium.org/p/chromedriver/issues/detail?id=1140 you can work around this issue by doing a few things.
manually start chrome from terminal/command prompt with these command line arguments --
google-chrome --user-data-dir=/path/to/any/custom/directory/home/user/Desktop/Chromedir --profile-directory="Profile 1" --remote-debugging-port=7878
make sure "Profile 1" is already existing in the same --user-data-dir (make usre Profile 1 has necessary chrome://components/ to run Netflix when launched manually)
you can use any free port in place of 7878
verify that http://localhost:7878 is running and returns value.
now connect to the remote-debugging-port=7878 via chromedriver with code below
Verify chrome://components/
I put mine into a .bat file, but you could do the same for a bash script or whatever:
C:/Program Files (x86)/Google/Chrome/Application/chrome.exe --user-data=c:/temp/chromeprofile --profile-directory="Profile 1" --remote-debugging-port=7878
Then set the debugger address in your code to use the browser:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
cr_options = Options()
# This line is where the "magic" happens.
cr_options.add_experimental_option('debuggerAddress','127.0.0.1:7878')
browser = webdriver.Chrome(chrome_options=cr_options)
browser.get('https://www.google.com')
browser.get('chrome://components/')
I'm post a java version of Chainik's answer as a reference for those using Java, please let me know if there's anything wrong.
ChromeOptions options = new ChromeOptions();
List<String> list = new ArrayList<String>();
list.add("disable-component-update");
options.setExperimentalOption("excludeSwitches", list);
options.addArguments("user-data-dir=/Users/myname/Library/Application Support/Google/Chrome/Default");
java.lang.System.setProperty("webdriver.chrome.driver","/usr/bin/chromedriver");
Webdriver driver = new ChromeDriver(options);
Here is an article about chromedriver capabilities and options.
This is late but might help someone else. I was able to get around this and play videos by not using a headless browser.
In Python,
options = Options()
options.headless = False
webdriver.Chrome(executable_path='path/to/chromedriver', options=options)