How to open a website protected by Cloudflare? such as https://coinlist.co,
I just want to open https://coinlist.co using Chromepd with headless = true flag, but Chrome prompts that "Chrome is being controlled by automated test software", and the chrome page blocks at the Cloudflare page. It seems like Cloudflare can detect the Chromedp.
You can try using the github.com/go-rod/stealth package based on the puppeteer stealth plugin like below:
import (
"github.com/chromedp/chromedp"
"github.com/go-rod/stealth"
)
chromedp.Run(
ctx,
chromedp.Evaluate(stealth.JS, nil),
)
If that doesn't work then you may need to use a captcha solver like 2captcha.
Related
I am using puppeteer-extra package with stealth plugin of puppeteer. While using the default puppeteer package, incognito shows up , but while using puppeteer-extra plugin, even while initializing the incognito context, the incognito window doesn't open up. Any idea if its some compatibility issue or someone already came across this problem.
I have tried with args passing "--incognito" mode and also using the context method.
While using --incognito parameter it opens parent window with incognito but while using newPage(), it open a second window which is without incognito flow.
Two approaches I had used
Importing puppeteer extra package:
import puppeteer from 'puppeteer-extra';
import pluginStealth from 'puppeteer-extra-plugin-stealth';
Method 1:
const context = await browser.createIncognitoBrowserContext();
const page = await context.newPage();
Method 2 :
const browser = await puppeteer.launch({args:[--incognito]});
I expect that while using puppeteer-extra package, the behavior should be same as using puppeteer.
The problem
This appears to be caused by a bug in the puppeteer-extra library. When you open a puppeteer instance with puppeteer-extra, the browser instance is hotpatched to better integrate newly opened pages with plugins.
Unfortunately the current implementation of browser._createPageInContext (as of version 2.1.3) doesn't correctly handle which browser context the new page should belong to once it's opened.
The fix
The fix is this pull request.
Specifically, you need to change this line
return async (contextId) => {
to this
return async function (contextId) {
so that arguments on the next line is evaluated correctly
const page = await originalMethod.apply(context, arguments)
In my Selenium-Test (with chromedriver-2.24) I'm trying to access my webpage via basic authentication with the following statement:
WebDriver driver = ...;
driver.get("http://admin:admin#localhost:8080/project/");
But Google Chrome gives me the following warning in the console:
[Deprecation] Subresource requests whose URLs contain embedded credentials (e.g. https://user:pass#host/) are blocked. See https://www.chromestatus.com/feature/5669008342777856 for more details.
In the tagged link is mentioned that the support was dropped:
Drop support for embedded credentials in subresource requests. (removed)
My question now is, is there an other way to basic-authenticate from Selenium?
NOTE: this has not helped: How to Handle HTTP Basic Auth headers in Selenium Webdriver using Java ?
The basic authentication via url is blocked only for sub resources.
So you could still use it on the domain:
driver.get("http://admin:admin#localhost:8080");
driver.get("http://localhost:8080/project");
You could also create a small extension to automatically set the credentials when they are requested:
options = webdriver.ChromeOptions()
options.add_extension(r'C:\dev\credentials.zip')
https://gist.github.com/florentbr/25246cd9337cebc07e2bbb0b9bf0de46
There were some updates in this link as :
Chromium Issue 435547 Drop support for embedded credentials in subresource requests. (removed)
We should block requests for subresources that contain embedded credentials (e.g. "http://ima_user:hunter2#example.com/yay.tiff"). Such resources would be handled as network errors.
However, Basic Authentication functionality still works with Selenium 3.4.0, geckodriver v0.18.0, chromedriver v2.31.488763, Google Chrome 60.x and Mozilla Firefox 53.0 through Selenium-Java bindings.
Here is the example code which tries to open the URL http://the-internet.herokuapp.com/basic_auth with a valid set of credentials and it works.
Firefox:
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
public class BasicAuthentication_FF
{
public static void main(String[] args)
{
System.setProperty("webdriver.gecko.driver", "C:\\Utility\\BrowserDrivers\\geckodriver.exe");
WebDriver driver = new FirefoxDriver();
driver.navigate().to("http://admin:admin#the-internet.herokuapp.com/basic_auth");
}
}
Chrome:
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
public class BasicAuthentication_Chrome
{
public static void main(String[] args)
{
System.setProperty("webdriver.chrome.driver", "C:\\Utility\\BrowserDrivers\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArguments("start-maximized");
options.addArguments("disable-infobars");
options.addArguments("--disable-extensions");
WebDriver driver = new ChromeDriver(options);
driver.navigate().to("http://admin:admin#the-internet.herokuapp.com/basic_auth");
}
}
Florent B.'s approach of calling .get on the URL twice worked for me with a slight modification. In JS:
driver
.get('http://admin:admin#localhost:8080')
.then( () => driver.get('http://localhost:8080') )
working on google chrome 62.0.3202.94 with ChromeDriver 2.33.506092 and the approach seems compatible with firefox 56.0.2 with geckodriver 0.19.1, and phantomjs 2.1.1 all under Debian linux 9.
What I believe is happening is the first call sets up the Authorization header sent by the browser. The second call removes the credentials from the URL and the credentials no longer are applied to subresources. The then synchronizes the two requests ensuring order.
New features for chrome and basic authentication via remote-debug: just for linking it here, so people who are stuck can find a solution for chrome and more: Chrome remote debugging in a seleniumgrid
Selenium 4 supports basic authentication
WebDriver driver = new ChromeDriver();
HasAuthentication authentication = (HasAuthentication) driver;
authentication.register(() -> new UsernameAndPassword("username", "pwd"));
driver.get("your-site.com");
https://www.selenium.dev/blog/2021/a-tour-of-4-authentication/
Such Basic authentication will not be supported directly using the selenium driver.get(URL) method to load a URL prompting for authentication within a JavaScript Popup, I was also stuck here for a long time. It's because of Chrome driver will not allow such authentication techniques after the update 59 (probably). There are still backdoors via Selenium using the JavaScript engine in the browser to load such URLs.
driver.get("https://www.google.com");
JavascriptExecutor jse = (JavascriptExecutor) driver;
URL = "https://username:password#www.example.com";
jse.executeScript("window.open('"+URL+"')");
I need to load URL which has authentication pop up. I used http://username:password#exampl.com format to access the website via selenium webdriver on chrome browser.
But this support seems t be dropped in latest chrome browser, So loading the url is not successful. Browser keeps loading for a long time with no response from server.
Is there any alternate way to overcome authentication pop-up with selenium webdrier ? I know we can use sikui,autoit etc.But i need to run the same code on many platforms - Win/Mac/mobile/tablet.
So please let know how to overcome this.
I am using Selenium 3.4.0 and Chromedriver 2.31, and loading a URL with authentication credentials works fine for me.
As per I know chorme stop URL authentication from chrome 60.x version.
You can try below code or downgrade your chrome version
The Alert Method, authenticateUsing() lets you skip the Http Basic Authentication box.
WebDriverWait wait = new WebDriverWait(driver, 10);
Alert alert = wait.until(ExpectedConditions.alertIsPresent());
alert.authenticateUsing(new UserAndPassword(username, password));
Hope it help you :)
I am writing a mini addon for chrome. In the addon, I have inserted a code to play audio where ajax is success. It doesn't work with websites that use https, but works fine on websites that use http. Can you help me edit it.
manifest.json
mystyle.js
For full resolution of the images:
http://i.stack.imgur.com/x8F5Q.png
http://i.stack.imgur.com/7bX5I.png
At a first glance, there are two issues with your code:
You shouldn't put chrome.tabs.getSelected and chrome.browserAction in mystyle.js. mystyle.js is a content script, which can only access limited chrome api.
However, content scripts have some limitations. They cannot:
Use chrome.* APIs, with the exception of:
extension ( getURL , inIncognitoContext , lastError , onRequest ,
sendRequest )
i18n
runtime ( connect , getManifest , getURL , id , onConnect , onMessage , sendMessage )
storage
Use variables or functions defined by their extension's pages
Use variables or functions defined by web pages or by other content scripts
You can't send http request from a https web page, since it's restricted by SOP and it's a browser behavior. To solve this, you should move your http request logic to background page and add the server url to permissions, see my following answer for more info.
Chrome extension - Disable Blocking of Mixed Content
I'm working with the tvOS beta 3 and trying to do some basic debugging on the tvml/tvjs side of things.
Messages logged via console.log(...) in my js files don't appear in the main Xcode output window.
Is there somewhere else I can find these messages or a setting which needs to be configured?
You should actually use the debug console in Safari. (The developer forum suggests you use Safari 9 and upgrade to El Capitan, both of which I have so haven't been able to test with inferior version)
Open Safari > Develop menu > Simulator
Your app name should appear here once and from there you can use the console.
Give it a few seconds to appear, it's not always instantaneous.
You must give a name to the Bundle Identifier in General/Identity (com.yourcompany.appname) to appear the app in the developers tool.
If you are developing a hybrid application (TVML/TVJS + Swift) with TVMLKitchen you can implement a logging function in Swift and use it in the TVJS code. For my projects I use the following code:
Kitchen.appController.evaluateInJavaScriptContext({context in
let printInJS : #convention(block) (NSString!) -> Void = {
(string : NSString!) -> Void in
print("Log: \(string)\n")
}
context.setObject(unsafeBitCast(printInJS, AnyObject.self), forKeyedSubscript: "printInJS")
})