Make Puppeteer use local profile's cookies - google-chrome

I want to use my local user's profile with Puppeteer. However, it doesn't seem to work.
I launch it with these args.
const browser = await puppeteer.launch({
executablePath: '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
userDataDir: '/Users/me/Library/Application Support/Google/Chrome',
});
When headless, it doesn't use the user's local profile's cookies at all, even though I'd expect it to. When it isn't headless, it can't even open the tab; Puppeteer crashes with
(node:23303) UnhandledPromiseRejectionWarning: Error: Failed to launch chrome!
TROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md
Is there a way to use my local user's profile? I'm using ^1.7.0 and Chrome 70.0.3521.2.

Rather than setting a userDataDir path in the Puppeteer.launch arguments you can use the chrome-cookies-secure NPM package to use cookies for one of your existing Chrome Profiles. This solution does not require Chrome Canary to be installed.
With your macOS keychain authorisation, the package reads the cookies for a given url from your hard-disk and makes them accessible in NodeJS. You can then load them into Puppeteer using the page.setCookie(...) method.
Here's an example:
const chrome = require('chrome-cookies-secure');
const puppeteer = require('puppeteer');
const url = 'https://www.yourUrl.com';
const getCookies = (callback) => {
chrome.getCookies(url, 'puppeteer', function(err, cookies) {
if (err) {
console.log(err, 'error');
return
}
console.log(cookies, 'cookies');
callback(cookies);
}, 'yourProfile') // e.g. 'Profile 2'
}
// find profiles at ~/Library/Application Support/Google/Chrome
getCookies(async (cookies) => {
const browser = await puppeteer.launch({
headless: false
});
const page = await browser.newPage();
await page.setCookie(...cookies);
await page.goto(url);
await page.waitFor(1000);
browser.close()
});

I solved this on MacOS by installing chrome canary, copying my default folder contained in ~/Library/Application Support/Google/Chrome/Default to ~/Library/Application Support/Google/Chrome\ Canary/Default
My working code looks like this:
async function run() {
const browser = await puppeteer.launch({
headless: false,
executablePath: '/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary',
userDataDir: '/Users/radium/Library/Application\ Support/Google/Chrome\ Canary/',
});
}
I was previously using the file path all the way to the Default folder, and truncated it to end with 'Chrome Canary' folder. This fixed everything. I have not tried with regular chrome.

Related

Headful puppeteer does not run on existing chrome profile

I'm getting this error when I try to run puppeteer on my existing chrome profile on headful but it works on headless but I cannot see the tab.
I know it works on the existing chrome because I can get it to open a new tab but then I get the error.
I want to be able to interact with this puppeteer instantiated tab.
C:\Users\Me\node_modules\puppeteer-core\lib\cjs\puppeteer\node\BrowserRunner.js:299
reject(new Error([
^
Error: Failed to launch the browser process!
TROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md
const pptr = require('puppeteer');
(async () =>{
const browser = await pptr.launch({
headless: false,
defaultViewport: {width: 1920,height: 1080},
executablePath: 'C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe',
userDataDir: 'C:\\Users\\Me\\AppData\\Local\\Google\\Chrome\\User Data\\'
})
await browser.newPage()
})()

how to load extension using puppeteer.connect() method

i use gologin service. gologin is a browser antidetect service where I can fake my browser identity / can manage browser fingerprint.
so I can freely do web-scraping without being detected.
in this case I want to be able to load my extension into that browser using the puppeteer.connect() method.
here's the code:
const puppeteer = require('puppeteer-core');
const GoLogin = require('gologin');
(async () => {
const GL = new GoLogin({
token: 'yU0token',
profile_id: 'yU0Pr0f1leiD',
});
const { status, wsUrl } = await GL.start();
const browser = await puppeteer.connect({
browserWSEndpoint: wsUrl.toString(),
ignoreHTTPSErrors: true,
});
const page = await browser.newPage();
await page.goto('https://myip.link/mini');
console.log(await page.content());
await browser.close();
await GL.stop();
})();
I don't know how. please help me, so i can load my extension using this puppeteer.connect()
Assume your wish is loading chrome-extension into your puppeteer browser.
Find chrome-extension Working Directory Where does Chrome store extensions?
Find your extension ID by go to chrome://extensions/
Sample code:
const puppeteer = require('puppeteer-core');
const MY_EXTENSION_PATH = '~/Library/Application Support/Google/Chrome/Default/Extensions/cdockenadnadldjbbgcallicgledbeoc/0.3.38_0'
async function loadExtension() {
return puppeteer.launch({
headless: 0,
args: [
`--disable-extensions-except=${MY_EXTENSION_PATH}`,
`--load-extension=${MY_EXTENSION_PATH}`,
],
});
}

In Puppeteer how to capture Chrome browser log in the console

I'm trying to collect Chrome browser logs: browser-issued warnings such as deprecation and interventions. For example, for site https://uriyaa.wixsite.com/corvid-cli2:
A cookie associated with a cross-site resource at http://wix.com/ was set without the `SameSite` attribute.
A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`.
You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.
I thought the following code would do the trick but it only catches logs generated by the page code.
(async ()=> {
const browser = await puppeteer.launch({dumpio: true});
const page = await browser.newPage();
page.on('console', msg => {
for (let i = 0; i < msg._args.length; ++i)
console.log(`${i}: ${msg._args[i]}`);
});
await page.goto('https://uriyaa.wixsite.com/corvid-cli2', {waitUntil: 'networkidle2', timeout: 20000});
await page.screenshot({path: 'screenshot.png'});
await browser.close();
})();
bellow is not relevant as I thought as reportingobserver does not catch the chrome info on cookies without sameSite:
Reading on the subject led me to https://developers.google.com/web/updates/2018/07/reportingobserver but I'm not sure how to use it, using the example int the browser console didn't work.
I'm not sure in which context the observer code should be used or if the browser need a flag to activate the reporting API. Or if this is the way to got about it.
help is welcomed.
Presumably, the 'console' event only catches console.log() and similar calls from the pages. But it seems you can catch warnings from the browser via CDPSession with Log Domain. Unfortunately, it works for me only with a headful browser:
'use strict';
const puppeteer = require('puppeteer');
(async function main() {
try {
const browser = await puppeteer.launch({ headless: false });
const [page] = await browser.pages();
const cdp = await page.target().createCDPSession();
await cdp.send('Log.enable');
cdp.on('Log.entryAdded', async ({ entry }) => {
console.log(entry);
});
await page.goto('https://uriyaa.wixsite.com/corvid-cli2');
} catch (err) {
console.error(err);
}
})();
And one of the entries:
{
source: 'other',
level: 'warning',
text: 'A cookie associated with a cross-site resource at http://www.wix.com/ was set without the `SameSite` attribute. It has been blocked, as Chrome now only delivers cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.',
timestamp: 1589058118372.802,
url: 'https://uriyaa.wixsite.com/corvid-cli2'
}
When you launch Puppeteer browser, inside the options object, you should set dumpio to true:
await puppeteer.launch({ dumpio: true });
This will basically "pipe browser process stdout and stderr into process.stdout and process.stderr", which means it will redirect browser logs to whatever main process, server, etc. you are running.
You can see this and other launch options you can use when launching Puppeteer in here: https://www.puppeteersharp.com/api/PuppeteerSharp.LaunchOptions.html

Puppeteer-extra allow flash support

I need puppeteer (not in headless mode) to open a page and have flash enabled from the get go.
Meaning no manual downloading or clicking to run flash.
So far i've added puppeteer-extra and its flash plugin as was used in a prior question:
Allowing to run Flash on all sites in Puppeteer
My chrome version is 75.0.3770.142 and my puppeteer dependencies are:
* "puppeteer": "^1.19.0",
* "puppeteer-core": "^1.19.0",
* "puppeteer-extra": "^2.1.3",
* "puppeteer-extra-plugin-flash": "^2.1.3",
* "puppeteer-extra-plugin-user-data-dir": "^2.1.2",
* "puppeteer-extra-plugin-user-preferences": "^2.1.2",
import puppeteer from 'puppeteer';
import PuppeteerCore from 'puppeteer-core';
import PuppeteerExtra from 'puppeteer-extra';
import PuppeteerFlash from 'puppeteer-extra-plugin-flash';
PuppeteerExtra.use(PuppeteerFlash());
(async () => {
const browser = await PuppeteerExtra.launch({
headless: false,
executablePath: '/Applications/Google Chrome.app/Contents/MacOS/Google\ Chrome',
args: [
'--window-size=800,600',
'--enable-webgl',
'--enable-accelerated-2d-canvas',
],
});
const page = await browser.newPage();
await page.setViewport({ width: 800, height: 600 });
await page.goto('http://ultrasounds.com', { waitUntil: 'networkidle2' });
})();
I expected the above code to open the page, download the necessary flash and run the flash content when done.
As it is though, it does the download but still requires a user to click enable flash to make the content run.
I'm wondering if anyone could please let me know if I'm doing anything wrong in the above code, if I've misunderstood something or otherwise?
if you use the localPath chrome app, you needn't the puppeteer-extra-plugin-flash.
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({
executablePath: '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
ignoreHTTPSErrors: true,
headless: false,
});
const page = await browser.newPage();
await page.goto('https://v.qq.com/iframe/preview.html?width=500&height=375&auto=0&vid=a30198lw6j2');
const dimensions = await page.evaluate(() => {
return {
src: document.getElementById('tenvideo_video_player_0').getAttribute('src'),
};
});
console.log('Dimensions:', dimensions);
await browser.close();
})();

How do I implement NTLM authentication using Puppeteer?

I would like to use Puppeteer to automate testing with a site that requires NTLM authentication. It appears the page.authenticate() API only accepts a username and password but no domain. Does anyone have any suggestions or tips?
Until now, the only way that worked for me was using cntlm:
Install cntlm. http://cntlm.sourceforge.net/
Configure cntlm, following the Configuration hints in the page
Run cntlm
Then use it in Puppeteer like this
const puppeteer = require('puppeteer');
async function run() {
const browser = await puppeteer.launch({
args: [
"--proxy-server=localhost:3133" // the cntlm proxy defined in cntlm.conf
]
});
const page = await browser.newPage();
await page.goto('http://example.com');
await page.screenshot({ path: 'screenshots/example.png' });
browser.close();
}
run();