How to get into the application which use windows authentication,
Hi All am new to puppeteer trying to do some automation and performance testing with puppeteer, so while trying to get into to application and do a sample check am not able to proceed because windows authentication not able to get through please help, i tried below code not working :(
const puppeteer = require('puppeteer');
async function test() {
const proxyUrl = 'URL';
const username = 'Uname';
const password = 'pwd';
const browser = await puppeteer.launch({
args: [`--proxy-server=${proxyUrl}`],
headless: false,
});
// let browser = await puppeteer.launch({ headless: false });
let page = await browser.newPage();
await page.authenticate({ username, password });
await page.goto('URL')
const html = await page.$eval('.ds-tile-container', e => e.innerHTML)
expect(html).not.toBeNull();
await page.pdf({ path: 'hn.pdf', format: 'A4' });
browser.close()
}
test();
I tried below code also
const oldProxyUrl = 'https://siteurl:8080';
const newProxyUrl = await proxyChain.anonymizeProxy(oldProxyUrl);
console.log(newProxyUrl);
const args = [
'--disable-setuid-sandbox',
'--no-sandbox',
'--ignore-certificate-errors',
'--ignore-certificate-errors-spki-list ',
];
const options = {
args,
headless: true,
ignoreHTTPSErrors: true,
};
const browser = await puppeteer.launch(options);
now am getting error like
(node:20520) UnhandledPromiseRejectionWarning: Unhandled promise
rejection (rejection id: 1): Error: Invalid "proxyUrl" option: only
HTTP proxies are currently supported. (node:20520) [DEP0018]
DeprecationWarning: Unhandled promise rejections are deprecated. In
the future, promise rejections that are not handled will terminate the
Node.js process with a non-zero exit code.
Related
Im Trying for the first time the puppeteer-proxy lib,and Im getting this error.
I don know is an error of puppeteer-proxy or of the function await page.setRequestInterception(true) because this guy has the same error as me
Code
const puppeteer = require('puppeteer-extra')
const StealthPlugin = require('puppeteer-extra-plugin-stealth')
const {proxyRequest} = require('puppeteer-proxy')
puppeteer.use(StealthPlugin())
var browser = null
const func = (async () => {
browser = await puppeteer.launch({
headless: false,
executablePath: "chrome-win/chrome.exe"
})
const page = await browser.newPage()
await page.setRequestInterception(true);
page.on('request', async (request) => {
await proxyRequest({
page,
proxyUrl: "https://username:password#174.25.210.207:6286",
request,
});
});
await page.goto("https://www.google.com/")
})();
Error
C:\Users\edina\Documents\Last Developer Projects\Scraping Browser\node_modules\puppeteer-core\lib\cjs\puppeteer\common\Frame.js:238
? new Error(`${response.errorText} at ${url}`)
^
Error: net::ERR_FAILED at https://www.google.com/
at navigate (C:\Users\edina\Documents\Last Developer Projects\Scraping Browser\node_modules\puppeteer-core\lib\cjs\puppeteer\common\Frame.js:238:23)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Frame.goto (C:\Users\edina\Documents\Last Developer Projects\Scraping Browser\node_modules\puppeteer-core\lib\cjs\puppeteer\common\Frame.js:207:21)
at async CDPPage.goto (C:\Users\edina\Documents\Last Developer Projects\Scraping Browser\node_modules\puppeteer-core\lib\cjs\puppeteer\common\Page.js:439:16)
at async C:\Users\edina\Documents\Last Developer Projects\Scraping Browser\test.js:28:5
i use gologin service. gologin is a browser antidetect service where I can fake my browser identity / can manage browser fingerprint.
so I can freely do web-scraping without being detected.
in this case I want to be able to load my extension into that browser using the puppeteer.connect() method.
here's the code:
const puppeteer = require('puppeteer-core');
const GoLogin = require('gologin');
(async () => {
const GL = new GoLogin({
token: 'yU0token',
profile_id: 'yU0Pr0f1leiD',
});
const { status, wsUrl } = await GL.start();
const browser = await puppeteer.connect({
browserWSEndpoint: wsUrl.toString(),
ignoreHTTPSErrors: true,
});
const page = await browser.newPage();
await page.goto('https://myip.link/mini');
console.log(await page.content());
await browser.close();
await GL.stop();
})();
I don't know how. please help me, so i can load my extension using this puppeteer.connect()
Assume your wish is loading chrome-extension into your puppeteer browser.
Find chrome-extension Working Directory Where does Chrome store extensions?
Find your extension ID by go to chrome://extensions/
Sample code:
const puppeteer = require('puppeteer-core');
const MY_EXTENSION_PATH = '~/Library/Application Support/Google/Chrome/Default/Extensions/cdockenadnadldjbbgcallicgledbeoc/0.3.38_0'
async function loadExtension() {
return puppeteer.launch({
headless: 0,
args: [
`--disable-extensions-except=${MY_EXTENSION_PATH}`,
`--load-extension=${MY_EXTENSION_PATH}`,
],
});
}
I am trying to use args in my code to use a proxy service I have. If I remove the args altogether things run fine but if I have them in there I get an error stating: Error: Unable to restart chrome. I checked multiple examples and copied the same to my code but it seems to fail. Any ideas on how to implement this correctly?
Code:
const { Cluster } = require('puppeteer-cluster');
const vanillaPuppeteer = require('puppeteer');
const { addExtra } = require('puppeteer-extra');
const Stealth = require('puppeteer-extra-plugin-stealth')
async function main() {
// Create a custom puppeteer-extra instance using `addExtra`,
// so we could create additional ones with different plugin config.
const puppeteer = addExtra(vanillaPuppeteer)
puppeteer.use(Stealth())
let proxy_server = 'proxy.soax.com:9000';
let user = 'some_user_name';
let pass = 'some_password';
// Launch cluster with puppeteer-extra
const cluster = await Cluster.launch({
puppeteer,
puppeteerOptions: {
headless: false,
args: ['--proxy-server=' + proxy_server,
'--single-process',
'--no-zygote',
'--no-sandbox'],
sameDomainDelay: 1000,
retryDelay: 3000,
workerCreationDelay: 3000},
maxConcurrency: 2,
concurrency: Cluster.CONCURRENCY_CONTEXT,
monitor: false,
skipDuplicateUrls: true
})
// Define task handler
await cluster.task(async ({ page, data: url }) => {
await page.authenticate({
username: user,
password: pass,
});
await page.goto(url)
const { hostname } = new URL(url)
console.log(`checking on ${hostname}`)
await page.screenshot({ path: `${hostname}.png`, fullPage: true })
})
// Queue any number of tasks
cluster.queue('https://whatismyipaddress.com/')
await cluster.idle()
await cluster.close()
console.log(`All done`)
}
main().catch(console.warn)
I played around a bit and discovered by removing the arg --single-process then it works fine.
I bought a proxy server version of socsk5.
In all manuals the same example
const browser = await puppeteer.launch({
headless: true,
ignoreHTTPSErrors: true,
defaultViewport: {...winSize},
args: [
'--proxy-server=socks5://proxyhost:8000',
'--host-resolver-rules="MAP * ~NOTFOUND , EXCLUDE proxyhost"',
],
})
It does not specify a login password for this proxy and it clearly does not work
If you specify this
'--proxy-server=socks5://user:password#proxyhost:8000',
it gives an error
net::ERR_NO_SUPPORTED_PROXIES
I tried with https://github.com/sjitech/proxy-login-automator build a bridge, but it did not work either.
Prompt please
You can use page.authenticate() to provide the credentials for your proxy.
For example:
'use strict';
const puppeteer = require('puppeteer');
(async () => {
const username = 'johndoe';
const password = 'qwerty1';
const browser = await puppeteer.launch({
args: [
'--proxy-server=socks5://proxyhost:8000',
],
});
const page = await browser.newPage();
await page.authenticate({
username,
password,
});
await page.goto('https://www.example.com/');
await browser.close();
})();
I'm new to puppeteer and node, trying to use a proxy with puppeteer in order to collect requests & responses, hopefully also websocket communication, but so far couldn't get anything to work..
I'm trying the following code:
const puppeteer = require('puppeteer');
const httpProxy = require('http-proxy');
const url = require('url');
let runProxy = async ()=> {
// raise a proxy and start collecting req.url/response.statusCode
};
let run = async () => {
await runProxy();
const browser = await puppeteer.launch({
headless: false,
args: ['--start-fullscreen',
'--proxy-server=localhost:8096']
});
page = await browser.newPage();
await page.setViewport({ width: 1920, height: 1080 });
await page.goto('http://www.google.com',
{waitUntil: 'networkidle2', timeout: 120000});
};
run();
I've tried some variation from https://github.com/nodejitsu/node-http-proxy but nothing seems to work for me, some guidance is at need, thanks
try this, use https-proxy-agent or http-proxy-agent to proxy request for per page:
import {Job, Launcher, OnStart, PuppeteerUtil, PuppeteerWorkerFactory} from "../..";
import {Page} from "puppeteer";
class TestTask {
#OnStart({
urls: [
"https://www.google.com",
"https://www.baidu.com",
"https://www.bilibili.com",
],
workerFactory: PuppeteerWorkerFactory
})
async onStart(page: Page, job: Job) {
await PuppeteerUtil.defaultViewPort(page);
await PuppeteerUtil.useProxy(page, "http://127.0.0.1:2007");
await page.goto(job.url);
console.log(await page.evaluate(() => document.title));
}
}
#Launcher({
workplace: __dirname + "/workplace",
tasks: [
TestTask
],
workerFactorys: [
new PuppeteerWorkerFactory({
headless: false,
devtools: true
})
]
})
class App {}