I have a multipage form and am using puppeteer to test. Problem I have is that when I ask to click to the next button it does click through but it doesn't go to the next page of the form. I have tried waitforNavigation but it throws a timeout issue. I have tried to address the timeout issue but to no success.
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.setViewport({ width: 1920, height: 1080 });
await page.goto('https://example.com/form/');
//await page.waitForNavigation({timeout: 10000, waitUntil: "load"}); // The promise resolves after navigation has finished
await page.click('#next_button_1'); // Clicking the link will indirectly cause a navigation
await page.setDefaultNavigationTimeout(0);
await page.waitForNavigation();
await page.screenshot({path: 'example.png',fullPage: true});
// other actions...
await browser.close();
})();
Related
I'm testing a Chrome extensions and in order to setup everything correctly I would like to know the tabId when a new tab is created
const browser = await puppeteer.launch({ ... });
const appPage = await browser.newPage();
await appPage.goto(appUrl, { waitUntil: 'load' });
// TAB ID??????
I checked the documentation of the Page object but there is nothing about a tabId. So my question is, is it possible to extract a tabId with Puppereer?
Recently I am learning puppeteer using their docs and try to scrape some information.
First approach
First I collect a list of url from the mainpage. Second I create a new tab and go those url iterately and collect some data. I doubt when I enter the loop the new tab didn't work as I expect and freezed without giving any data. Eventually I got a error TimeoutError: Navigation timeout of 30000 ms exceeded. Is there any better approach?
(async () => {
const browser = await puppeteer.launch({ headless: true });
const mainpage = await browser.newPage();
console.log('goto main page'.green);
await mainpage.goto(mainURL);
console.log('collecting some url'.green);
const URLS = await mainpage.evaluate(() =>
Array.from(
document.querySelectorAll('.result-actions a'),
(element) => element.href
)
);
if (typeof URLS[0] === 'string') console.log('OK'.green);
console.log('collecting finished'.green);
const newTab= await browser.newPage();
console.log('create new tab'.green);
var data = [];
for (let i = 0, n = URLS.length; i < n; i++) {
//console.log(URLS[i]);
// use this new tab to collect some data then close this tab
// continue this process
await newTab.waitForNavigation();
await newTab.goto(URLS[i]);
await newTab.waitForSelector('.profile-phone-column span a');
console.log('Go each url using new tab'.green);
// collecting data
data.push(collected_data);
// close this tab
await collectNamePage.close();
console.log(data);
}
await mainpage.close();
await browser.close();
console.log('closing browser'.green);
})();
Second approach
This time I want to skip the part where I collect those data using a new tab. Hence I collect my urls using page.$$() and try to iterating using for...of over urls and collect my data using elementHandle.$(selector) but this approach also failed.
I am getting frustrated. Am I doing it wrong way or I didn't understand their documentation?
In your script, you do not need newTab.waitForNavigation(); at all. Usually, this is used when the navigation is caused by some event. When you just use .goto(), the page loading is waited automatically.
Even if you need waitForNavigation(), you usually do not await it before the navigation triggered, otherwise you just get the timeout. You await it with navigation trigger together:
await Promise.all([element.click(), page.waitForNavigation()]);
So try to just delete await newTab.waitForNavigation();.
Also, do not close the new tab in the loop, delete it after the loop.
Edited script:
const puppeteer = require('puppeteer');
const mainURL = 'https://www.psychologytoday.com/us/therapists/illinois/';
(async () => {
const browser = await puppeteer.launch({ headless: false });
const mainpage = await browser.newPage();
console.log('goto main page');
await mainpage.goto(mainURL);
console.log('collecting urls');
const URLS = await mainpage.evaluate(() =>
Array.from(
document.querySelectorAll('.result-actions a'),
(element) => element.href
)
);
if (typeof URLS[0] === 'string') console.log('OK');
console.log('collection finished');
const collectNamePage = await browser.newPage();
console.log('create new tab');
var data = [];
for (let i = 0, totalUrls = URLS.length; i < totalUrls; i++) {
console.log(URLS[i]);
await collectNamePage.goto(URLS[i]);
await collectNamePage.waitForSelector('.profile-phone-column span a');
console.log('create new tab and go there');
// collecting data
const [name, phone] = await collectNamePage.evaluate(
() => [
document.querySelector('.profile-middle .name-title-column h1').innerText,
document.querySelector('.profile-phone-column span a').innerText
]
);
data.push({ name, phone });
}
console.log(data);
await collectNamePage.close();
await mainpage.close();
await browser.close();
console.log('closing browser');
})();
I'm developing an app using Angular js and i'm doing a a questionnaire component which has a lot of nested pages to develop, so instead of doing changes and then spends a lot of time to return to the same page and see the results - i was advised to use Puppeteer to simulate to automate to page navigation ... I've found many examples of Puppeteer that outputs a png file - but none that's simply open a URL...
Here is what i've tried so far...
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('http://cnn.com');
// await page.screenshot({path: 'example.png'});
//await browser.close();
})();
As you can see in my example - it didn't opened a new page...
How to change headless from false to true ?
How to hide browser?
const browser = await puppeteer.launch({headless: false})
const page = await browser.newPage();
await page.goto(LOGIN_URL, { "waitUntil": "networkidle2" });
await page.evaluate((a) => {
$('input[name="username"]').val(a.username)
$('input[name="password"]').val(a.password)
}, {username, password})
// I want to hide the browser
// do something
await browser.close();
It is not possible to hide the browser in puppeteer at runtime — that's because Chromium can be launched either headless or non-headless only.
But during one script you can first run non-headless browser, close it and then open headless:
let browser = await puppeteer.launch({headless : false});
// 1. Enter requisites, log in to a site
// 2. Save cookies
await browser.close();
browser = await puppeteer.launch({headless : true});
// 3. Load cookies
// 4. Go and do headless stuff
Use this:
const browser = await puppeteer.launch({headless: true});
In the following example how do I wait for the pop up window to finish loading?
After clikcing the google icon you get a pop up window to login to gmail, when I try to interact
with the second page it is undefined (as I don't know how to wait for it to fully load.
Any advice?
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({headless: false});
page = await browser.newPage();
await page.goto("https://www.example.com/signin");
await page.waitForSelector(".Icon-google");
await page.click(".Icon-google");
const pages = await browser.pages();
console.log(pages[2].url());
})();
You can wait for a new target to be created.
const browser = await puppeteer.launch({headless: false});
page = await browser.newPage();
await page.goto("https://app.testim.io/#/signin");
await page.waitForSelector(".Icon-google");
const nav = new Promise(res => browser.on('targetcreated', res))
await page.click(".Icon-google");
await nav
const pages = await browser.pages();
console.log(pages.length);//number of pages increases !
console.log(pages.map(page => page.url()));
P.S. first I tried page.waitForNavigation() but it didn't work, probably because it's a popup.
const [newPage] = await Promise.all([
new Promise((resolve) => page.once('popup', resolve)),
page.click('something.that-will-open-the-popup')
]);
await newPage.waitForSelector('.page-is-loaded')