I bought a proxy server version of socsk5.
In all manuals the same example
const browser = await puppeteer.launch({
headless: true,
ignoreHTTPSErrors: true,
defaultViewport: {...winSize},
args: [
'--proxy-server=socks5://proxyhost:8000',
'--host-resolver-rules="MAP * ~NOTFOUND , EXCLUDE proxyhost"',
],
})
It does not specify a login password for this proxy and it clearly does not work
If you specify this
'--proxy-server=socks5://user:password#proxyhost:8000',
it gives an error
net::ERR_NO_SUPPORTED_PROXIES
I tried with https://github.com/sjitech/proxy-login-automator build a bridge, but it did not work either.
Prompt please
You can use page.authenticate() to provide the credentials for your proxy.
For example:
'use strict';
const puppeteer = require('puppeteer');
(async () => {
const username = 'johndoe';
const password = 'qwerty1';
const browser = await puppeteer.launch({
args: [
'--proxy-server=socks5://proxyhost:8000',
],
});
const page = await browser.newPage();
await page.authenticate({
username,
password,
});
await page.goto('https://www.example.com/');
await browser.close();
})();
Related
i use gologin service. gologin is a browser antidetect service where I can fake my browser identity / can manage browser fingerprint.
so I can freely do web-scraping without being detected.
in this case I want to be able to load my extension into that browser using the puppeteer.connect() method.
here's the code:
const puppeteer = require('puppeteer-core');
const GoLogin = require('gologin');
(async () => {
const GL = new GoLogin({
token: 'yU0token',
profile_id: 'yU0Pr0f1leiD',
});
const { status, wsUrl } = await GL.start();
const browser = await puppeteer.connect({
browserWSEndpoint: wsUrl.toString(),
ignoreHTTPSErrors: true,
});
const page = await browser.newPage();
await page.goto('https://myip.link/mini');
console.log(await page.content());
await browser.close();
await GL.stop();
})();
I don't know how. please help me, so i can load my extension using this puppeteer.connect()
Assume your wish is loading chrome-extension into your puppeteer browser.
Find chrome-extension Working Directory Where does Chrome store extensions?
Find your extension ID by go to chrome://extensions/
Sample code:
const puppeteer = require('puppeteer-core');
const MY_EXTENSION_PATH = '~/Library/Application Support/Google/Chrome/Default/Extensions/cdockenadnadldjbbgcallicgledbeoc/0.3.38_0'
async function loadExtension() {
return puppeteer.launch({
headless: 0,
args: [
`--disable-extensions-except=${MY_EXTENSION_PATH}`,
`--load-extension=${MY_EXTENSION_PATH}`,
],
});
}
I'm trying to submit a login form, but all I get is a timeout after 30 seconds.
My code is rather simple and I can't find anything wrong:
const puppeteer = require('puppeteer');
const creds = {
user: "1234",
password: "1234"
};
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.setViewport({width: 1280, height: 800});
await page.goto('https://shop2.idena.de/NewShop/');
await page.type('input[name="FORM_LOGIN"]', creds.user);
await page.type('input[name="FORM_PASSWD"]', creds.password);
await Promise.all([
page.click('button[name="FORM_TYPE"]'),
page.waitForNavigation()
]);
await page.screenshot({path: 'example.png', fullPage: true});
await browser.close();
})();
Any ideas what's going wrong here?
Change the order of the promises a bit, it could be possible, the navigation happens super fast and the waitForNavigation is just waiting for nothing. Or maybe your website loads very slow after clicking the login button.
await Promise.all([
page.waitForNavigation({timeout: 60000}),
page.click('button[name="FORM_TYPE"]'),
]);
If I use your example with headful option, I get this dialog that prevents the page from loading:
So this addition can help (not sure if some dialog emerges with correct credentials):
const puppeteer = require('puppeteer');
const creds = {
user: "1234",
password: "1234"
};
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.setViewport({width: 1280, height: 800});
await page.goto('https://shop2.idena.de/NewShop/');
await page.type('input[name="FORM_LOGIN"]', creds.user);
await page.type('input[name="FORM_PASSWD"]', creds.password);
page.on('dialog', async dialog => {
console.log(dialog.message());
await dialog.accept();
});
await Promise.all([
page.click('button[name="FORM_TYPE"]'),
page.waitForNavigation()
]);
await page.screenshot({path: 'example.png', fullPage: true});
await browser.close();
})();
EDIT: I'm updating my answer since more infomration has been provided in the original question.
The problem is that there's a dialog you need to confirm/dismiss:
Perhaps you didn't see it because the script was too fast. I recommend debugging puppeteer scripts with headless set to false and slowMo to some number greater than 0:
const browser = await puppeteer.launch({ headless: false, slowMo: 200 });
Then you need to get rid of the dialog:
page.on('dialog', async (dialog) => {
await dialog.accept();
});
The whole script that now passes:
const puppeteer = require('puppeteer');
const creds = {
user: "1234",
password: "1234"
};
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.setViewport({width: 1280, height: 800});
await page.goto('https://shop2.idena.de/NewShop/');
await page.type('input[name="FORM_LOGIN"]', creds.user);
await page.type('input[name="FORM_PASSWD"]', creds.password);
page.on('dialog', async (dialog) => {
await dialog.accept();
});
await Promise.all([
page.click('button[name="FORM_TYPE"]'),
page.waitForNavigation()
]);
await page.screenshot({path: 'example.png', fullPage: true});
await browser.close();
})();
Tried below code but getting an error.
Error: net::ERR_SSL_VERSION_OR_CIPHER_MISMATCH at
https://www.xxxxxxsolutions.com/
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({ignoreHTTPSErrors: true, acceptInsecureCerts: true, args: ['--proxy-bypass-list=*', '--disable-gpu', '--disable-dev-shm-usage', '--disable-setuid-sandbox', '--no-first-run', '--no-sandbox', '--no-zygote', '--single-process', '--ignore-certificate-errors', '--ignore-certificate-errors-spki-list', '--enable-features=NetworkService']});
const page = await browser.newPage();
try {
await page.goto('https://www.xxxxxxxsolutions.com/', {waitUntil: 'networkidle2', timeout: 59000});
const cookies = await page._client.send('Network.getAllCookies');
JSON.stringify(cookies, null, 4);
} catch (e) {
console.log(e);
}
await browser.close();
})();
#mujuonly, this is version related issue. Please try the same code above 1.16.0 or latest version 2.0. It's working fine.
How to get into the application which use windows authentication,
Hi All am new to puppeteer trying to do some automation and performance testing with puppeteer, so while trying to get into to application and do a sample check am not able to proceed because windows authentication not able to get through please help, i tried below code not working :(
const puppeteer = require('puppeteer');
async function test() {
const proxyUrl = 'URL';
const username = 'Uname';
const password = 'pwd';
const browser = await puppeteer.launch({
args: [`--proxy-server=${proxyUrl}`],
headless: false,
});
// let browser = await puppeteer.launch({ headless: false });
let page = await browser.newPage();
await page.authenticate({ username, password });
await page.goto('URL')
const html = await page.$eval('.ds-tile-container', e => e.innerHTML)
expect(html).not.toBeNull();
await page.pdf({ path: 'hn.pdf', format: 'A4' });
browser.close()
}
test();
I tried below code also
const oldProxyUrl = 'https://siteurl:8080';
const newProxyUrl = await proxyChain.anonymizeProxy(oldProxyUrl);
console.log(newProxyUrl);
const args = [
'--disable-setuid-sandbox',
'--no-sandbox',
'--ignore-certificate-errors',
'--ignore-certificate-errors-spki-list ',
];
const options = {
args,
headless: true,
ignoreHTTPSErrors: true,
};
const browser = await puppeteer.launch(options);
now am getting error like
(node:20520) UnhandledPromiseRejectionWarning: Unhandled promise
rejection (rejection id: 1): Error: Invalid "proxyUrl" option: only
HTTP proxies are currently supported. (node:20520) [DEP0018]
DeprecationWarning: Unhandled promise rejections are deprecated. In
the future, promise rejections that are not handled will terminate the
Node.js process with a non-zero exit code.
I'm new to puppeteer and node, trying to use a proxy with puppeteer in order to collect requests & responses, hopefully also websocket communication, but so far couldn't get anything to work..
I'm trying the following code:
const puppeteer = require('puppeteer');
const httpProxy = require('http-proxy');
const url = require('url');
let runProxy = async ()=> {
// raise a proxy and start collecting req.url/response.statusCode
};
let run = async () => {
await runProxy();
const browser = await puppeteer.launch({
headless: false,
args: ['--start-fullscreen',
'--proxy-server=localhost:8096']
});
page = await browser.newPage();
await page.setViewport({ width: 1920, height: 1080 });
await page.goto('http://www.google.com',
{waitUntil: 'networkidle2', timeout: 120000});
};
run();
I've tried some variation from https://github.com/nodejitsu/node-http-proxy but nothing seems to work for me, some guidance is at need, thanks
try this, use https-proxy-agent or http-proxy-agent to proxy request for per page:
import {Job, Launcher, OnStart, PuppeteerUtil, PuppeteerWorkerFactory} from "../..";
import {Page} from "puppeteer";
class TestTask {
#OnStart({
urls: [
"https://www.google.com",
"https://www.baidu.com",
"https://www.bilibili.com",
],
workerFactory: PuppeteerWorkerFactory
})
async onStart(page: Page, job: Job) {
await PuppeteerUtil.defaultViewPort(page);
await PuppeteerUtil.useProxy(page, "http://127.0.0.1:2007");
await page.goto(job.url);
console.log(await page.evaluate(() => document.title));
}
}
#Launcher({
workplace: __dirname + "/workplace",
tasks: [
TestTask
],
workerFactorys: [
new PuppeteerWorkerFactory({
headless: false,
devtools: true
})
]
})
class App {}