Check for all frames to load - puppeteer

I would like to check that all frames on a page have been loaded. I can't quite figure this out, and part of that is I don't fully understand the frame events (specifically when exactly do frameattached and framenavigated events fire?).
Here is what I am doing right now, but I'm binding to the same page event several times rather than a frame event.
function waitForFrames() {
return Promise.all(page.frames().map((frame) => {
return new Promise(resolve => {
page.on('framenavigated', resolve);
});
})).then(() => {
console.log('Frames loaded');
})
.catch(e => {
console.log(e.message);
});
}
How can I check that all frames are loaded?

The Puppeteer Documentation for the Frame class helps explain the frame events:
class: Frame
At every point of time, page exposes its current frame tree via the page.mainFrame() and frame.childFrames() methods.
Frame object's lifecycle is controlled by three events, dispatched on the page object:
'frameattached' - fired when the frame gets attached to the page. A Frame can be attached to the page only once.
'framenavigated' - fired when the frame commits navigation to a different URL.
'framedetached' - fired when the frame gets detached from the page. A Frame can be detached from the page only once.
You can wait for a frame (by name) using the following function:
const wait_for_frame = (page, frame_name) => {
let fulfill_promise;
const promise = new Promise(x => fulfill_promise = x);
check_frame();
return promise;
const check_frame = () => {
const frame = page.frames().find(current_frame => current_frame.name() === frame_name);
if (frame) {
fulfill_promise(frame);
} else {
page.once('frameattached', check_frame);
}
};
};
// Waiting for frame to become attached:
const frame = await wait_for_frame(page, frame_name);
// Waiting for frame to contain a certain selector:
await frame.waitForSelector(selector);
const button = await frame.$(selector);
button.click();
The above code snippet was influenced by and improved on from: Source.
Finally, you can loop through and await the frames one-by-one.

Using Grant's solution I ran into the problem, that the iFrame I'm looking for sometimes gets detached after pageload and then (re-)attached and navigated. In this case the waitForSelector produces an exception "iFrame got detached". It seems to me that a frame appearing in page.frames() is not guaranteed to actually be attached.
So I changed the wait_for_frame function of his solution to cover this case. This also covers the case when there are multiple iFrames on the page.
const wait_for_frame = (page, frame_name) => {
const check_frame = frame => {
if (frame.name() == frame_name) {
fulfill_promise(frame);
page.off('framenavigated', check_frame);
}
};
let fulfill_promise;
const promise = new Promise(x => fulfill_promise = x);
page.on('framenavigated', check_frame);
return promise;
};
// Waiting for frame to become attached:
const frame = await wait_for_frame(page, frame_name);
// Waiting for frame to contain a certain selector:
await frame.waitForSelector(selector);
const button = await frame.$(selector);
button.click();

Related

How to make sure document is ready before calling getElementById() in Typescript

I have a component that loads some HTML (converted from MD via marked library) and diplays it to the page and if a row is clicked, the document will scroll to the appropriate HTML element with the matching headerID on the page.
import { marked } from 'marked';
const [content, setContent] = React.useState('');
React.useEffect(() => {
if (!source) {
return;
}
getRequest(source)
.then(response => {
if (!response.ok) {
return Promise.reject(response);
}
return response.text().then(faq => {
// sets page to html
setContent(marked(faq));
const nearestHeader = document.getElementById(headerID);
if (nearestHeader) {
// if search result is clicked, jump to result's nearest header
nearestHeader.scrollIntoView(true);
setRowClicked(false);
}
});
})
.catch(e => Dialog.error('Failed to load page.', e));
}, [source, rowClicked]);
However, when I go to test this code the 'nearestHeader' object is always null even after I verified that the headerID matches up with the existing HTML element's ID I want to navigate to. How can I make sure document is ready/loaded before attemping the getElementById call without using extra libraries?
Solved by adding another useEffect call which waits on content of page to be set first. Removed nearestHeader code from the initial useEffect() call that sets content
React.useEffect(() => {
const nearestHeader = document.getElementById(headerID);
if (nearestHeader) {
nearestHeader.scrollIntoView(true);
setRowClicked(false);
}
}, [content]);

Autodesk Forge Viewer - How do I fire an event after model loading is complete?

How do I fire an event after model loading is complete?
I created the "basicSetting" function below.
function basicSetting(){
viewer.setLightPreset(1);
viewer.setQualityLevel(false, false);
viewer.setGhosting(true);
viewer.setGroundShadow(false);
viewer.setGroundReflection(false);
viewer.setEnvMapBackground(false);
viewer.setProgressiveRendering(true);
}
And I applied it to the "onDocumentLoadSuccess" function.
But it didn't work.
Help!
You can use the GEOMETRY_LOADED_EVENT as per this link and use it like this after you have initialised a viewer.
In typescript (using forge-typings)
this.viewer.addEventListener(Autodesk.Viewing.GEOMETRY_LOADED_EVENT, (x) =>
{
basicSetting();
}
Or if you wanna be extra save and make sure the function is never called elsewhere, just remove it and place its contents in the event callback.
this.viewer.addEventListener(Autodesk.Viewing.GEOMETRY_LOADED_EVENT, (x) =>
{
this.viewer.setLightPreset(1);
this.viewer.setQualityLevel(false, false);
this.viewer.setGhosting(true);
this.viewer.setGroundShadow(false);
this.viewer.setGroundReflection(false);
this.viewer.setEnvMapBackground(false);
this.viewer.setProgressiveRendering(true);
}
Not sure in which languages you are developing but it should be pretty simular !
You can use a promise waiting for viewer loading finish like as:
var loadedPromise = new Promise((resolve, reject) => {
var listener = function (event) {
baseViewer.removeEventListener(
Autodesk.Viewing.GEOMETRY_LOADED_EVENT,
listener
);
resolve();
}
baseViewer.addEventListener(
Autodesk.Viewing.GEOMETRY_LOADED_EVENT,
listener
);
});
await loadedPromise;

how to execute a script in every window that gets loaded in puppeteer?

I need to execute a script in every Window object created in Chrome – that is:
tabs opened through puppeteer
links opened by click()ing links in puppeteer
all the popups (e.g. window.open or "_blank")
all the iframes contained in the above
it must be executed without me evaluating it explicitly for that particular Window object...
I checked Chrome's documentation and what I should be using is Page.addScriptToEvaluateOnNewDocument.
However, it doesn't look to be possible to use through puppeteer.
Any idea? Thanks.
This searches for a target in all browser contexts.
An example of finding a target for a page opened
via window.open() or popups:
await page.evaluate(() => window.open('https://www.example.com/'))
const newWindowTarget = await browser.waitForTarget(async target => {
await page.evaluate(() => {
runTheScriptYouLike()
console.log('Hello StackOverflow!')
})
})
via browser.pages() or tabs
This script run evaluation of a script in the second tab:
const pageTab2 = (await browser.pages())[1]
const runScriptOnTab2 = await pageTab2.evaluate(() => {
runTheScriptYouLike()
console.log('Hello StackOverflow!')
})
via page.frames() or iframes
An example of getting eval from an iframe element:
const frame = page.frames().find(frame => frame.name() === 'myframe')
const result = await frame.evaluate(() => {
return Promise.resolve(8 * 7);
});
console.log(result); // prints "56"
Hope this may help you

get post title after Infinite scroll finished

I manage to show all the post on a site where it has load_more button to go to the next page, but something is missing,
I got error of
e Error: Node is either not visible or not an HTMLElement
at ElementHandle._clickablePoint (/Users/minghann/Documents/productnation_scraper/node_modules/puppeteer/lib/ExecutionContext.js:331:13)
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)
Which doesn't happen if I don't load all the post. It's hard to debug because I don't know which post is missing what. Full code as below:
const browser = await puppeteer.launch({
devtools: true
});
const page = await browser.newPage();
await page.goto("https://example.net");
await page.waitForSelector(".load_more_btn");
const load_more_exist = !!(await page.$(".load_more_btn"));
while (load_more_exist > 0) {
await page.click(".load_more_btn");
}
const posts = await page.$$(".post");
let result = [];
for (const post of posts) {
result = [
...result,
{
title: await post.$eval(".post_title a", e => e.innerText)
}
];
}
console.log(result);
browser.close();
There are multiple ways and best way is to combine the following two different ways.
Look for Ajax
Wait for request instead. Whenever you click on Load More, it will do a simple ajax request to ?ajax-request=jnews. We can use .waitForRequest or .waitForResponse for this use case. Here is a working example,
await Promise.all([
page.waitForRequest(response => response.url().includes('?ajax-request=jnews') && response.status() === 200),
page.click(".load_more_btn")
])
Clean DOM and wait for new Element
Refer to these answers here and here.
Basically you can remove the dom elements that you collected, so next time you collect more data, there won't be any duplicates.
So, once you remove all current elements like document.querySelectorAll('.jeg_post'), you can simply do another page.waitFor('.jeg_post') later if you need.

File extension not supported:null ErrorCode:13. when loading multiple models

I'm trying to load 2 models into Autodesk's Forge Viewer.
I'm trying with the following code:
const urn1 = <urn>
const urn2 = <urn>
Autodesk.Viewing.Initializer(
options,
() => {
const viewerDiv = document.getElementById('MyViewerDiv');
viewer = new Autodesk.Viewing.Private.GuiViewer3D(viewerDiv);
this.loadDoc(this.props.urns[1], true);
window.setTimeout(e => {
this.loadDoc(this.props.urns[2], false);
}, 4000);
},
);
loadDoc(urn: string, initializeAndLoad: boolean) {
Autodesk.Viewing.Document.load(urn,
(doc) => {
const viewables = Autodesk.Viewing.Document
.getSubItemsWithProperties(doc.getRootItem(), {'type': 'geometry'}, true);
if (viewables.length === 0) {
return;
}
const initialViewable = viewables[0];
const svfUrl = doc.getViewablePath(initialViewable);
const modelOptions = {
globalOffset: {x: 0, y: 0, z: 0}, // to align the models
sharedPropertyDbPath: doc.getPropertyDbPath(),
};
if (initializeAndLoad) {
viewer.start(svfUrl, modelOptions,
() => {},
() => {console.log('load model error');},
);
} else {
viewer.loadModel(urn, modelOptions,
() => {},
(e) => {
console.warn(e);
});
}
},
() => {}
);
}
The rationale behind the timeout is to load the second model using loadModel after the first model has loaded. I've also tried loading the second model from the viewer.start's onSuccess callback.
No matter what, I get the File extension not supported:null ErrorCode:13. error message (both in the console and in a popup)
I'm pretty sure the message is misleading since both urns have valid SVF derivatives (I can switch between them, whichever one is loaded first displays just fine)
NB I'm using the following version:
'https://developer.api.autodesk.com/modelderivative/v2/viewers/6.2/viewer3D.min.js'
As a side note, I've tried using Autodesk.Viewing.ViewingApplication and selectItem. With this I'm able to load multiple models but I don't seem to be able to set modelOptions (specifically globalOffset) with this approach.
The loadModel method expects a URL with some known file extension (e.g., .svf) but you're calling it with an URN (the base64-encoded identifier of a translated document). That's why it's failing to find the file extension.
Btw. if you want to postpone the loading of the second model after the first one is loaded completely, consider using the geometry-loaded-event instead of a timeout.