As far as I know both page.goto and page.waitForNavigation accept waitUntil as a parameter, is it just two ways to achieve the same results?
For example:
page.goto(url, {waitUntil: 'domcontentloaded'})
vs:
page.waitForNavigation(url, {waitUntil: 'domcontentloaded'})
On puppeteer version 1.19.0, waitForNavigation does not accept an url. Usually waitForNavigation is used with a click, where clicking might cause a navigation in the browser.
Example:
const [response] = await Promise.all([
page.waitForNavigation(), // The promise resolves after navigation has finished
page.click('a.my-link'), // Clicking the link will indirectly cause a navigation
]);
It accepts options similar to .goto(), but that's all.
So,
page.goto() will go to an url and wait for navigation.
page.waitForNavigation() will only wait for navigation.
Related
A similar question may have been asked, but despite spending a couple of hours I could not find a satisfying answer.
I would like to call Frame.click(). At the time of calling it - I want to make sure that no navigation is pending. I also don't know if the element I am about to click on will result in navigation (it's passed in dynamically).
I tried https://www.npmjs.com/package/pending-xhr-puppeteer, but that seems to no longer be supported. (still uses Request instead of HTTPRequest)
Right now I am resorting to page.WaitForNetworkIdle(). Is this the best I can do?
P.S. It seems page.WaitForNavigation used to return right away if no navigation was taking place (on a year old version of the code). Since I updated to 12.1 it started waiting for the timeout and then throwing an exception if no navigation started within that timeframe.
I think you're confusing navigation and requests/responses. Navigation can be finished but requests/responses can still happened (eg: .aspx requests).
To make sure the navigation has happened, you could wrap it in a promise via Promise.all() method.
await Promise.all([
page.click(`#submit`),
page.waitForNavigation(),
]);
Additionally you could await page.content(); to make sure the full HTML contents of the page, including the doctype.
In the case of an ajax request you can use page.waitForResponse():
await Promise.all([
page.click(`#submit`),
page.waitForResponse(response => response.status() === 200),
]);
I have a button nested in this video player that I am trying to simply click. Any idea how do i select?
await page.waitForSelector('.icon-play');
await page.click('.icon-play');
await page.waitFor(6000);
}
<a tabindex="-1" href="#" role="button" class="icon-play comp largePlayBtn largePlayBtnBorder" aria-label="Play clip" data-order="1" data-plugin-name="largePlayBtn" style="display: block;"></a>
I had the same problem a while back. A nested link inside a image.
This is due to the fact that the element needs to be visually clickable. You can perform a simple javascript click action using the HTMLElement.click() to bypass the puppeteer click action.
page.$eval(`HTMLElementSelector`, element =>
element.click()
);
As you're clicking a link, which implies navigation you would want to englobe that inside a promise. Something like that should do the trick.
await Promise.all([
page.$eval(`HTMLElementSelector`, element =>
element.click()
),
await page.waitForNavigation(),
]);
this answer is to make a clear understanding of why the puppeteer's click doesn't work sometimes.
Puppeteer's API has different semantics from the native browser API.
Puppeteer's page.click() seems like a straightforward wrapper on the browser's native HTMLElement.click(), but it actually operates quite differently.
working of page.click()
when we click using page.click() instead of invoking the click event handler directly on the element as the native HTMLElement.click() does
Puppeteer scrolls the element into view
moves the mouse onto the element
presses one of a few mouse buttons
optionally triggers a delay
then releases the mouse button
You can also trigger multiple clicks. In other words, Puppeteer performs a click like a human would.
That's why when we click page.click() it clicks at (x,y) position of the screen as a human would. So, sometimes we don't see expected results as it doesn't show any error and it shouldn't, we think that page.click() is not working.
So, the easiest solution is to use page.evaluate() and click with native browser API.
Solution
await page.evaluate(() => {
document.querySelector('selector').click();
});
What "navigation" is waitForNavigation waiting for?
The websites navigation? The browsers refresh icon "navigation" to finish spinning?
Or is this just a awkwardly worded method that should be named waitForBrowserToLoad?
But, when I use it, like so:
await this.page.waitForNavigation();
The page DOES finish loading.
And yet it never resolves. I'm not sure why.
What is waitForNavigation waiting for?
async function beforeScrape(page) {
//code gets this far
await this.page.waitForNavigation();
//never resolves
await page.click(".table-header");
}
Documentation says:
[page.waitForNavigation] resolves when the page navigates to a new URL or reloads. It is useful for when you run code which will indirectly cause the page to navigate.
For example you fill a form and the click "Submit" button, after which a new page is shown with results. This way you can wait until the new page loaded:
await Promise.all([
page.waitForNavigation(),
page.click('input[type=submit]'),
]);
If you instruct the script to wait for navigation, but do not cause one, it will just sit there waiting until timing out.
I'm working in an application that requires these types of login where a new window (no tab) pops up and ask you to login, in the same way that twitter or facebook do (or used to do), where it shows where to put your email and password, click "login" and then the window would close and the main window would receive the authentication and keep going.
I can do it in "headed" mode, when I click the authentication button, the window pops up and I get a grip of it with
const newWindow = (await browser.pages())[1]
and then I'd navigate like normal where in my case I only have to click a single button because I'm already logged in the page I'm trying to use for authentication
await newWindow.waitFor("//SomeXpath")
const buttonToClick= (await newWindow.$x("//SomeXpath"))[0]
await buttonToClick.click()
Again, in headed mode it works fine. but in headless mode is like this 2nd page would not open.
If I try:
console.log(await browser.pages())
I only see the main page where I open puppeteer
I've seen people talking about the "target" class, but it seems is for new tabs, and the examples I've found didn't work for me (or probably I wasn't able to understand and properly use them)
I'm afraid that what I'm trying to do is not possible and I'm chasing a ghost.
Thanks
Edited:
A snapshot of the window with the button I have to click
Snapshot of the new window
Also I've tried this, but it didn't work either:
const newPagePromise = new Promise(x => browser.once('targetcreated', target => x(target.page())));
const popup = await newPagePromise;
where popUp would be the new "page". But if I do a console.log(popup) it returns "null"
I'm navigating with Puppeteer around a React website.
Two sample lines of code:
await page.waitForSelector('a.btn-lg[data-target="#loginModal"]');
await page.click('a.btn-lg[data-target="#loginModal"]');
With a sufficient slowMo value, the effects are consistent - the button gets clicked every time.
However, without slowMo, sometimes the button does get clicked, and sometimes it doesn't (a window wired to it doesn't open).
It happens for a lot of elements, not just this one button in particular.
I just started using Puppeteer, and it looks like I'm either misusing the library, or the website somehow screws up my efforts.
Please tell me why sometimes the effects of clicking are visible and sometimes not, and how to remedy it.
UPDATE:
Code such as this does not work either.
await page.evaluate(() => (document.querySelector('span.pum-close') as any).click());
await page.$$eval('span.pum-close', elements =>
elements[0].click()
);