HTML5 video and canvas CPU optimization - html

Question

I am making an app with HTML5 video along with a canvas drawing on top of it (640x480 px). The objective was to record the canvas drawing along with the encoded video to produce it as a single video. I was able to do all these using FFMPEG. But the problem I am facing is, when the HTML5 video is running it takes around 50% of my CPU. Since drawing on canvas is also demanding CPU, the browser freezes after certain time and the CPU usage for that tab on chrome is showing continously > 100. I have tried to optimize html5 canvas rendering. But nothing helped. Is there a way to run this video with much less CPU usage? or any other optimizations possible?

Answer 1

There is not very much you can if the hardware need to use the CPU to decode and display the video. The keyword is compromise.
A few things can be done though that removes additional barriers. These must be considered general tips though:
Efficient looping
Make sure to use requestAnimationFrame() to invoke your loop in case you aren't.
setTimeout()/setInterval() are relatively performance-heavy and cannot sync properly to the monitor refresh rate.
Reduce update load
Also if you're not already doing this: the canvas is usually updated at 60 FPS while a video is rarely above 30/29.97 FPS (in Europe 25 FPS). This means you can skip every second frame update and still show the video at optimal rate. Use a toggle to achieve this.
Video at 25 FPS will be re-synced to 30 FPS (monitors typically runs at 60 Hz even for European models which are electronically re-synced internally, which also means the browser need to deal with drop/double-frames etc. internally - nothing we can do here).
// draw video at 30 FPS
var toggle = false;
(function loop() {
toggle = !toggle;
if (toggle) { /* draw frame here */ }
requestAnimationFrame(loop);
})();
Avoid scaling of the canvas element
Make sure canvas size and CSS size is the exact same. Or put simple: don't use CSS to set the size of the canvas at all.
Disable alpha channel composition
You can disable alpha composition of the canvas in some browsers to get a little more speed. Consumer-video never come with an alpha-channel.
// disable alpha-composition of the canvas element where supported
var context = canvas.getContext("2d", {alpha: false});
Tweak settings at encoding stage
Make sure to encode the video using a balance between size and decompression load. The more a video is compressed the more need to be reconstructed, at a cost. Encode with different encoder settings to find a balance that works in your scenario.
Also consider aspects such as color-depth i.e. 16 vs 24 bit.
The H264 codec is preferable as it has wide support in various display interface hardware.
Reduce video FPS
If the content of the video allows, f.ex. there is little movement or changes, encode using 15 FPS instead of 30 FPS. If so, also use a MODULO instead of a toggle (as shown above) where you can skip 3 frames and update canvas only at the 4.:
// draw video at 15 FPS
var modToggle = 0;
(function loop() {
if (modToggle++ % 4 === 0) { /* draw frame here */ }
requestAnimationFrame(loop);
})();
Encode video at smaller source size
Encode the video at a slightly smaller size dividable by 8 (in this case I would even suggest half size 320x240 - experiment!). Then draw using the scale parameters of the drawImage() method:
context.drawImage(video, 0, 0, video.videoWidth, video.videoHeight, 0, 0, 640, 480);
This help reduce the amount of data needed to be loaded and decoded but will of course reduce the quality. How it turns out depends again on the content.
You can also turn off interpolation using imageSmoothingEnabled set to false on the context (note: the property need a prefix in some browsers). For this you may not want to reduce the size as much as 50% but only slightly (something like 600x420 in this case).
Note: even if you "reduce" the frame rate the canvas is still redrawn at 60 FPS, but since it doesn't do any actual work on the intermediate frames it's still off-loading the CPU/GPU giving you a less tight performance budget over-all.
Hope this gives some input.