im using puppeteer-recorder to record a video from browser activity
this package
https://www.npmjs.com/package/puppeteer-recorder
here is my code
async function check_login()
{
try {
const page = await global_browser.newPage();
await page.setViewport({width: 1000, height: 1100});
record({
browser: global_browser, // Optional: a puppeteer Browser instance,
page: page, // Optional: a puppeteer Page instance,
output: path + 'output.webm',
fps: 60,
frames: 60 * 5, // 5 seconds at 60 fps
prepare: function () {}, // <-- add this line
render: function () {} // <-- add this line
});
await page.goto('http://localhost/home_robot/mock.php', {timeout: 60000})
.catch(function (error) {
throw new Error('TimeoutBrows');
});
await page.close();
}
catch (e) {
console.log(' LOGIN ERROR ---------------------');
console.log(e);
}
}
it works fine but i dont know how to stop recording and so when i get to the end of function i get this error
(node:10000) UnhandledPromiseRejectionWarning: Error: Protocol error (Emulation.setDefaultBackgroundColorOverride): Target closed.
at Promise (C:\wamp64\www\home_robot\node_modules\puppeteer\lib\Connection.js:202:56)
at new Promise (<anonymous>)
at CDPSession.send (C:\wamp64\www\home_robot\node_modules\puppeteer\lib\Connection.js:201:12)
at Page._screenshotTask (C:\wamp64\www\home_robot\node_modules\puppeteer\lib\Page.js:806:26)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:10000) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (
rejection id: 1)
(node:10000) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
and i dont get a video file , i assume its because i close the page without stopping the record
unfortunately there's no documentation and the author doesn't answer the questions
There is no way or need to stop recording with this module, look closely at the frames option of the record method:
frames: 60 * 5, // 5 seconds at 60 fps
After getting 300 frames it will stop by itself.
However for it to work a very important requirement must be met and documentation won't mention it, but you can see it in the source of the module: it uses ffmpeg to make video from screenshots AND that must be in PATH. If not you should provide ffmpeg binary location in options:
const puppeteer = require('puppeteer');
const { record } = require('puppeteer-recorder');
puppeteer.launch({headless : false}).then(async browser => {
const page = await browser.newPage();
await page.goto('https://codepen.io/hexagoncircle/full/joqYEj', {waitUntil : 'networkidle2'});
await record({
ffmpeg: "c:\\ffmpeg\\bin\\ffmpeg.exe" // <-- provide full path to ffmpeg binary
browser: browser, // Optional: a puppeteer Browser instance,
page: page, // Optional: a puppeteer Page instance,
output: 'output.webm',
fps: 24,
frames: 24, // desired seconds of recording multiplied by fps
prepare: function (browser, page) { /* you could click on a button */ },
render: function (browser, page, frame) { /* executed before each capture */ }
});
await browser.close();
});
Result:
Related
I am glad to find the puppeteer cluster. this library made life easy on crawling and automation tasks.tnx to Thomas Dondorf.
according to the author of the puppeteer cluster, when a task finished page will be closed immediately.this is good by the way. but what about some cases that you need to page will be open?
my use case:
I will try to explain briefly:
there is some activity on the page that in the background a socket is involved in for sending some data to the front .this data changes the dome and I need to capture that.
this is my code :
async function runCrawler(){
const links = [
"foo.com/barSome324",
"foo.com/barSome22",
"foo.com/barSome1",
"foo.com/barSome765",
]
const cluster = await Cluster.launch({
concurrency: Cluster.CONCURRENCY_CONTEXT,
workerCreationDelay: 5000,
puppeteerOptions:{args: ['--no-sandbox', '--disable-setuid-sandbox'], headless:false},
maxConcurrency: numCPUs,
});
await cluster.task(async ({ page, data: url }) => {
await crawler(page, url)
});
for(link of links){
await cluster.queue(link);
}
await cluster.idle();
await cluster.close();
}
and this is the crawler logic in page section:
module.exports.crawler = async(page, link)=>{
await page.goto(link, { waitUntil: 'networkidle2' })
await page.waitForTimeout(10000)
await page.waitForSelector('#dbp')
try {
// method to be executed;
setInterval(async()=>{
const tables=await page.evaluate(async()=>{
/// data I need to catch in every 30 seconds
});
},30000)
} catch (error) {
console.log(error)
}
}
I searched And find out in js we can capture DOM changes with mutationObserver .and tried this solution . but did not work either.page will be closed with this error:
UnhandledPromiseRejectionWarning: Error: Protocol error
(Runtime.callFunctionOn): Session closed. Most likely the page has
been closed.
so I have two options here:
1.mutationObserver
2.set interval for every 30 seconds evaluates the page itself.
but they did not suit my needs. so any idea how to overcome this problem?
How can I show progress of a download of a large file from GDrive using the gapi client-side v3 API?
I am using the v3 API, and I've tried to use a Range request in the header, which works, but the download is very slow (below). My ultimate goal is to playback 4K video. GDrive limits playback to 1920x1280. My plan was to download chunks to IndexedDB via v3 API and play from the locally cached data. I have this working using the code below via Range requests, but it is unusably slow. A normal download of the full 438 MB test file directly (e.g. via the GDrive web page) takes about 30-35s on my connection, and, coincidentally, each 1 MB Range requests takes almost exactly the same 30-35s. It feels like the GDrive back-end is reading and sending the full file for each subrange?
I've also tried using XHR and fetch to download the file, which fails. I've been using the webContent link (which typically ends in &export=download) but I cannot get access headers correct. I get either CORS or other odd permission issues. The webContent links work fine in <image> and <video> src tags. I expect this is due to special permission handling or some header information I'm missing that the browser handles specifically for these media tags. My solution must be able to read private (non-public, non-sharable) links, hence the use of the v3 API.
For video files that are smaller than the GDrive limit, I can set up a MediaRecorder and use a <video> element to get the data with progress. Unfortunately, the 1920x1080 limit kills this approach for larger files, where progress feedback is even more important.
This is the client-side gapi Range code, which works, but is unusably slow for large (400 MB - 2 GB) files:
const getRange = (start, end, size, fileId, onProgress) => (
new Promise((resolve, reject) => gapi.client.drive.files.get(
{ fileId, alt: 'media', Range: `bytes=${start}-${end}` },
// { responseType: 'stream' }, Perhaps this fails in the browser?
).then(res => {
if (onProgress) {
const cancel = onProgress({ loaded: end, size, fileId })
if (cancel) {
reject(new Error(`Progress canceled download at range ${start} to ${end} in ${fileId}`))
}
}
return resolve(res.body)
}, err => reject(err)))
)
export const downloadFileId = async (fileId, size, onProgress) => {
const batch = 1024 * 1024
try {
const chunks = []
for (let start = 0; start < size; start += batch) {
const end = Math.min(size, start + batch - 1)
const data = await getRange(start, end, size, fileId, onProgress)
if (!data) throw new Error(`Unable to get range ${start} to ${end} in ${fileId}`)
chunks.push(data)
}
return chunks.join('')
} catch (err) {
return console.error(`Error downloading file: ${err.message}`)
}
}
Authentication works fine for me, and I use other GDrive commands just fine. I'm currently using drives.photos.readonly scope, but I have the same issues even if I use a full write-permission scope.
Tangentially, I'm unable to get a stream when running client-side using gapi (works fine in node on the server-side). This is just weird. If I could get a stream, I think I could use that to get progress. Whenever I add the commented-out line for the responseType: 'stream', I get the following error: The server encountered a temporary error and could not complete your request. Please try again in 30 seconds. That’s all we know. Of course waiting does NOT help, and I can get a successful response if I do not request the stream.
I switched to using XMLHttpRequest directly, rather than the gapi wrapper. Google provides these instructions for using CORS that show how to convert any request from using gapi to a XHR. Then you can attach to the onprogress event (and onload, onerror and others) to get progres.
Here's the drop-in replacement code for the downloadFileId method in the question, with a bunch of debugging scaffolding:
const xhrDownloadFileId = (fileId, onProgress) => new Promise((resolve, reject) => {
const user = gapi.auth2.getAuthInstance().currentUser.get()
const oauthToken = user.getAuthResponse().access_token
const xhr = new XMLHttpRequest()
xhr.open('GET', `https://www.googleapis.com/drive/v3/files/${fileId}?alt=media`)
xhr.setRequestHeader('Authorization', `Bearer ${oauthToken}`)
xhr.responseType = 'blob'
xhr.onloadstart = event => {
console.log(`xhr ${fileId}: on load start`)
const { loaded, total } = event
onProgress({ loaded, size: total })
}
xhr.onprogress = event => {
console.log(`xhr ${fileId}: loaded ${event.loaded} of ${event.total} ${event.lengthComputable ? '' : 'non-'}computable`)
const { loaded, total } = event
onProgress({ loaded, size: total })
}
xhr.onabort = event => {
console.warn(`xhr ${fileId}: download aborted at ${event.loaded} of ${event.total}`)
reject(new Error('Download aborted'))
}
xhr.onerror = event => {
console.error(`xhr ${fileId}: download error at ${event.loaded} of ${event.total}`)
reject(new Error('Error downloading file'))
}
xhr.onload = event => {
console.log(`xhr ${fileId}: download of ${event.total} succeeded`)
const { loaded, total } = event
onProgress({ loaded, size: total })
resolve(xhr.response)
}
xhr.onloadend = event => console.log(`xhr ${fileId}: download of ${event.total} completed`)
xhr.ontimeout = event => {
console.warn(`xhr ${fileId}: download timeout after ${event.loaded} of ${event.total}`)
reject(new Error('Timout downloading file'))
}
xhr.send()
})
I'm trying to record puppeteer to see what happens when i run it on server, as I understand this package does what i want.
https://www.npmjs.com/package/puppeteer-recorder
so here is simplified version of my code
const puppeteer = require('puppeteer');
const { record } = require('puppeteer-recorder');
var path = 'C:\\wamp64\\www\\home_robot\\';
init_puppeteer();
const global_browser ;
async function init_puppeteer() {
global_browser = await puppeteer.launch({headless: false , args: ['--no-sandbox', '--disable-setuid-sandbox']});
check_login()
};
async function check_login()
{
try {
const page = await global_browser.newPage();
await page.setViewport({width: 1000, height: 1100});
await record({
browser: global_browser, // Optional: a puppeteer Browser instance,
page: page, // Optional: a puppeteer Page instance,
output: path + 'output.webm',
fps: 60,
frames: 60 * 5, // 5 seconds at 60 fps
prepare: function () {}, // <-- add this line
render: function () {} // <-- add this line
});
await page.goto('https://www.example.cob', {timeout: 60000})
.catch(function (error) {
throw new Error('TimeoutBrows');
});
await page.close();
}
catch (e) {
console.log(' LOGIN ERROR ---------------------');
console.log(e);
}
}
But I get this error
$ node home.js
(node:7376) UnhandledPromiseRejectionWarning: Error: spawn ffmpeg ENOENT
at Process.ChildProcess._handle.onexit (internal/child_process.js:240:19)
at onErrorNT (internal/child_process.js:415:16)
at process._tickCallback (internal/process/next_tick.js:63:19)
(node:7376) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This
error originated either by throwing inside of an async function without a catch
block, or by rejecting a promise which was not handled with .catch(). (rejection
id: 1)
(node:7376) [DEP0018] DeprecationWarning: Unhandled promise rejections are depre
cated. In the future, promise rejections that are not handled will terminate the
Node.js process with a non-zero exit code.
LOGIN ERROR ---------------------
Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed
at doWrite (_stream_writable.js:406:19)
at writeOrBuffer (_stream_writable.js:394:5)
at Socket.Writable.write (_stream_writable.js:294:11)
at Promise (C:\wamp64\www\home_robot\node_modules\puppeteer-recorder\index.j
s:72:12)
at new Promise (<anonymous>)
at write (C:\wamp64\www\home_robot\node_modules\puppeteer-recorder\index.js:
71:3)
at module.exports.record (C:\wamp64\www\home_robot\node_modules\puppeteer-re
corder\index.js:44:11)
at process._tickCallback (internal/process/next_tick.js:68:7)
i've even ran npm i reinstall ffmpeg --with-libvpx
as it was suggested here
https://github.com/clipisode/puppeteer-recorder/issues/6
but still didnt work .... wha telse do i need to do ?
I know this would be a very late response to your question, but nevertheless.
A years ago, I visited this same stack-overflow thread and I had similar challenge of finding a screen recorder library which does a good job a capturing the video as well as offers an options to manually start and stop the recording.
Finally I wrote one for myself and distributed as NPM library...!!
https://www.npmjs.com/package/puppeteer-screen-recorder
Hope this is helpful...!!
Add two empty functions called prepare and render in the options.
await record({
browser: global_browser, // Optional: a puppeteer Browser instance,
page, // Optional: a puppeteer Page instance,
output: path + 'output.webm',
fps: 60,
frames: 60 * 5, // 5 seconds at 60 fps,
prepare: function () {}, // <-- add this line
render: function () {} // <-- add this line
});
Basically it's missing some default functions and the error is not handled properly.
Also, there's https://www.npmjs.com/package/puppeteer-capture which uses HeadlessExperimental protocol.
I'm currently unsuccessfully trying to make my PWA installable. I have registered a SertviceWorker and linked a manifest as well as I am listening on the beforeInstallPromt event.
My ServiceWorker is listening to any fetch event.
My problem is, that the created beforeInstall banner is just being shown on Chrome desktop but on mobile I get a warning in Chrome inspection tab "Application" in the "Manifest" section:
Installability
Service worker does not have the 'fetch' handler
You can check the message on https://dev.testapp.ga/
window.addEventListener('beforeinstallprompt', (e) => {
// Stash the event so it can be triggered later.
deferredPrompt = e;
mtShowInstallButton();
});
manifest.json
{"name":"TestApp","short_name":"TestApp","start_url":"https://testapp.ga/loginCheck","icons":[{"src":"https://testapp.ga/assets/icons/launcher-ldpi.png","sizes":"36x36","density":0.75},{"src":"https://testapp.ga/assets/icons/launcher-mdpi.png","sizes":"48x48","density":1},{"src":"https://testapp.ga/assets/icons/launcher-hdpi.png","sizes":"72x72","density":1.5},{"src":"https://testapp.ga/assets/icons/launcher-xhdpi.png","sizes":"96x96","density":2},{"src":"https://testapp.ga/assets/icons/launcher-xxhdpi.png","sizes":"144x144","density":3},{"src":"https://testapp.ga/assets/icons/launcher-xxxhdpi.png","sizes":"192x192","density":4},{"src":"https://testapp.ga/assets/icons/launcher-web.png","sizes":"512x512","density":10}],"display":"standalone","background_color":"#ffffff","theme_color":"#0288d1","orientation":"any"}
ServiceWorker:
//This array should NEVER contain any file which doesn't exist. Otherwise no single file can be cached.
var preCache=[
'/favicon.png',
'/favicon.ico',
'/assets/Bears/bear-standard.png',
'/assets/jsInclude/mathjax.js',
'/material.js',
'/main.js',
'functions.js',
'/material.css',
'/materialcolors.css',
'/user.css',
'/translations.json',
'/roboto.css',
'/sw.js',
'/'
];
//Please specify the version off your App. For every new version, any files are being refreched.
var appVersion="v0.2.1";
//Please specify all files which sould never be cached
var noCache=[
'/api/'
];
//On installation of app, all files from preCache are being stored automatically.
self.addEventListener('install', function(event) {
event.waitUntil(
caches.open(appVersion+'-offline').then(function(cache) {
return cache.addAll(preCache).then(function(){
console.log('mtSW: Given files were successfully pre-cached')
});
})
);
});
function shouldCache(url) {
//Checking if url is market as noCache
var isNoCache=noCache.includes(url.substr(8).substr(url.substr(8).indexOf("/")))||noCache.includes((url.substr(8).substr(url.substr(8).indexOf("/"))).substr(0,(url.substr(8).substr(url.substr(8).indexOf("/"))).indexOf("?")));
//Checking of hostname of request != current hostname
var isOtherHost=url.substr(8).substr(0,url.substr(8).indexOf("/"))!=location.hostname&&url.substr(7).substr(0,url.substr(7).indexOf("/"))!=location.hostname;
return((url.substr(0,4)=="http"||url.substr(0,3)=="ftp") && isNoCache==false && isOtherHost==false);
}
//If any fetch fails, it will look for the request in the cache and serve it from there first
self.addEventListener('fetch', function(event) {
//Trying to answer with "online" version if fails, using cache.
event.respondWith(
fetch(event.request).then(function (response) {
if(shouldCache(response.url)) {
console.log('mtSW: Adding file to cache: '+response.url);
caches.open(appVersion+'-offline').then(function(cache) {
cache.add(new Request(response.url));
});
}
return(response);
}).catch(function(error) {
console.log( 'mtSW: Error fetching. Serving content from cache: ' + error );
//Check to see if you have it in the cache
//Return response
//If not in the cache, then return error page
return caches.open(appVersion+'-offline').then(function (cache) {
return cache.match(event.request).then(function (matching) {
var report = !matching || matching.status == 404?Promise.reject('no-match'): matching;
return report
});
});
})
);
})
I checked the mtShowInstallButton function. It's fully working on desktop.
What does this mean? On the Desktop, I never got this warning, just when using a handheld device/emulator.
Fetch function is used to fetch JSon manifest file. Try reading google docs again.
For adding PWA in Mobile you need manifest file to be fetched which is fetched using service-worker using fetch function.
Here is the code :
fetch('examples/example.json')
.then(function(response) {
// Do stuff with the response
})
.catch(function(error) {
console.log('Looks like there was a problem: \n', error);
});
for more about fetch and manifest try this.
I would like to check that all frames on a page have been loaded. I can't quite figure this out, and part of that is I don't fully understand the frame events (specifically when exactly do frameattached and framenavigated events fire?).
Here is what I am doing right now, but I'm binding to the same page event several times rather than a frame event.
function waitForFrames() {
return Promise.all(page.frames().map((frame) => {
return new Promise(resolve => {
page.on('framenavigated', resolve);
});
})).then(() => {
console.log('Frames loaded');
})
.catch(e => {
console.log(e.message);
});
}
How can I check that all frames are loaded?
The Puppeteer Documentation for the Frame class helps explain the frame events:
class: Frame
At every point of time, page exposes its current frame tree via the page.mainFrame() and frame.childFrames() methods.
Frame object's lifecycle is controlled by three events, dispatched on the page object:
'frameattached' - fired when the frame gets attached to the page. A Frame can be attached to the page only once.
'framenavigated' - fired when the frame commits navigation to a different URL.
'framedetached' - fired when the frame gets detached from the page. A Frame can be detached from the page only once.
You can wait for a frame (by name) using the following function:
const wait_for_frame = (page, frame_name) => {
let fulfill_promise;
const promise = new Promise(x => fulfill_promise = x);
check_frame();
return promise;
const check_frame = () => {
const frame = page.frames().find(current_frame => current_frame.name() === frame_name);
if (frame) {
fulfill_promise(frame);
} else {
page.once('frameattached', check_frame);
}
};
};
// Waiting for frame to become attached:
const frame = await wait_for_frame(page, frame_name);
// Waiting for frame to contain a certain selector:
await frame.waitForSelector(selector);
const button = await frame.$(selector);
button.click();
The above code snippet was influenced by and improved on from: Source.
Finally, you can loop through and await the frames one-by-one.
Using Grant's solution I ran into the problem, that the iFrame I'm looking for sometimes gets detached after pageload and then (re-)attached and navigated. In this case the waitForSelector produces an exception "iFrame got detached". It seems to me that a frame appearing in page.frames() is not guaranteed to actually be attached.
So I changed the wait_for_frame function of his solution to cover this case. This also covers the case when there are multiple iFrames on the page.
const wait_for_frame = (page, frame_name) => {
const check_frame = frame => {
if (frame.name() == frame_name) {
fulfill_promise(frame);
page.off('framenavigated', check_frame);
}
};
let fulfill_promise;
const promise = new Promise(x => fulfill_promise = x);
page.on('framenavigated', check_frame);
return promise;
};
// Waiting for frame to become attached:
const frame = await wait_for_frame(page, frame_name);
// Waiting for frame to contain a certain selector:
await frame.waitForSelector(selector);
const button = await frame.$(selector);
button.click();