I'm writing an application which records PC's screen in realtime and encodes it with Media Foundation h264 codec. The quality of a resulting video stays high if a picture on the screen doesn't have many dynamic changes. If a picture has many dynamic changes (fast scrolling of a web page, for example) then video quality becomes very low. This sounds like a low bitrate problem, but reference OSX recording application works fine with the same settings.
Codec configuration:
25 fps
1364 x 768 resolution
baseline profile
1.2 MBit bitrate
To maximize quality, I configured codec with the following parameters:
CODECAPI_AVEncCommonRateControlMode = eAVEncCommonRateControlMode_Quality
CODECAPI_AVEncCommonQuality = 100
CODECAPI_AVEncAdaptiveMode = eAVEncAdaptiveMode_FrameRate
CODECAPI_AVEncMPVGOPSize = 10
Unfortunately, this doesn't help much. The only setting that has the real effect is bitrate.
An example video, that demonstrates the problem:
https://www.dropbox.com/s/b26odaeyaygxigo/10-22-2016_15.08.17.mp4?dl=1
you need to set bit rate and quality by this step:
VARIANT controlModeVar;
InitVariantFromUInt32(eAVEncCommonRateControlMode_GlobalVBR,
&controlModeVar);
hr = CodeApi->SetValue(&CODECAPI_AVEncCommonRateControlMode,
&controlModeVar);
VARIANT quality;
InitVariantFromUInt32(BitRate, &quality);
hr = CodeApi->SetValue(&CODECAPI_AVEncCommonMeanBitRate,
&quality);
InitVariantFromUInt32(eAVEncCommonRateControlMode_Quality,
&controlModeVar);
hr = CodeApi->SetValue(&CODECAPI_AVEncCommonRateControlMode,
&controlModeVar);
InitVariantFromUInt32(H264QualityLevel, &quality);
hr = CodeApi->SetValue(&CODECAPI_AVEncCommonQuality, &quality);
the min os require windows8.
Related
I have a working RL model and set up that produce a video for me - however becuase the model is reasonably good, the videos are very short (reach a desitination therfore better = shorter)
Is there a way to drop the frame rate of the video output? I know it can be done with a gif. And that it can be done with ffmpeg but I can't workout how to pass it down.
I've dropped the fps in my environment from 50>10 expecting the video to be 5 times as long but that didn't work.
Save me stackoverflow you're my only hope. (appart from posting on github)
When you say that you dropped the fps from 50 to 10, I assume you changed env.metadata.video.frames_per_second, which initializes to 50 (for gym versions less than 0.22.0):
{
'render.modes': ['human', 'rgb_array'],
'video.frames_per_second': 50
}
Starting with gym 0.22.0, VideoRecorder class gets the frames_per_sec to write the video with:
env.metadata.get("render_fps", 30)
Setting env.metadata["render_fps"] = 4 should result in a slowed down video.
I am using janus to build my webrtc SFU server. I need the chrome browser to send the video resolution from a start to a fixed value and remain unchanged during the transfer. Where should I set it?
I tried setting the degradationPreference in the js code, but it didn't work, the resolution will still change, it seems that chrome does not support this parameter.
var senderList = config.pc.getSenders();
var sender = config.pc.getSenders().find(function(s) {return s.track.kind == "video"});
if(sender) {
var parameters = sender.getParameters();
parameters.degradationPreference = "maintain-resolution";
sender.setParameters(parameters);
}
Image1 with varying resolution
Image2 with varying resolution
I looked at frameHeightSend/frameWidthSend in chrome://webrtc-internals, hoping it will keep the same value from the start, but now it grows slowly at startup and will fluctuate during subsequent transfers.
I found a message that sets the constant resolution in IOS, which is set when the screen is shared, and whether there are similar settings in chrome.
I'm writing a screen capture application that uses the UWP Screen Capture API.
As a result, I get a callback each frame with a ID3D11Texture2D with the target screen or application's image, which I want to use as input to MediaFoundation's SinkWriter to create an h264 in mp4 container file.
There are two issues I am facing:
The texture is not CPU readable and attempts to map it fail
The texture has padding (image stride > pixel width * pixel format size), I assume
To solve them, my approach has been to:
Use ID3D11DeviceContext::CopyResource to copy the original texture into a new one created with D3D11_USAGE_STAGING and D3D11_CPU_ACCESS_READ flags
Since that texture too has padding, create a IMFMediaBuffer buffer wrapping it with MFCreateDXGISurfaceBuffer cast it to IMF2DBuffer and use IMF2DBuffer::ContiguousCopyTo to another IMFMediaBuffer that I create with MFCreateMemoryBuffer
So I'm basically copying around each and every frame two times, once on the GPU and once on the CPU, which works, but seems way inefficient.
What is a better way of doing this? Is MediaFoundation smart enough to deal with input frames that have padding?
The inefficiency comes from your attempt to Map rather than use the texture as video encoder input. MF H.264 encoder, which is in most cases a hardware encoder, can take video memory backed textures as direct input and this is what you want to do (setting up encoder respectively - see D3D/DXGI device managers).
Padding does not apply to texture frames. In case of padding in frames with traditional system memory data Media Foundation primitives are normally capable to process data with padding: Image Stride and MF_MT_DEFAULT_STRIDE, and other Video Format Attributes.
I am making an app with HTML5 video along with a canvas drawing on top of it (640x480 px). The objective was to record the canvas drawing along with the encoded video to produce it as a single video. I was able to do all these using FFMPEG. But the problem I am facing is, when the HTML5 video is running it takes around 50% of my CPU. Since drawing on canvas is also demanding CPU, the browser freezes after certain time and the CPU usage for that tab on chrome is showing continously > 100. I have tried to optimize html5 canvas rendering. But nothing helped. Is there a way to run this video with much less CPU usage? or any other optimizations possible?
There is not very much you can if the hardware need to use the CPU to decode and display the video. The keyword is compromise.
A few things can be done though that removes additional barriers. These must be considered general tips though:
Efficient looping
Make sure to use requestAnimationFrame() to invoke your loop in case you aren't.
setTimeout()/setInterval() are relatively performance-heavy and cannot sync properly to the monitor refresh rate.
Reduce update load
Also if you're not already doing this: the canvas is usually updated at 60 FPS while a video is rarely above 30/29.97 FPS (in Europe 25 FPS). This means you can skip every second frame update and still show the video at optimal rate. Use a toggle to achieve this.
Video at 25 FPS will be re-synced to 30 FPS (monitors typically runs at 60 Hz even for European models which are electronically re-synced internally, which also means the browser need to deal with drop/double-frames etc. internally - nothing we can do here).
// draw video at 30 FPS
var toggle = false;
(function loop() {
toggle = !toggle;
if (toggle) { /* draw frame here */ }
requestAnimationFrame(loop);
})();
Avoid scaling of the canvas element
Make sure canvas size and CSS size is the exact same. Or put simple: don't use CSS to set the size of the canvas at all.
Disable alpha channel composition
You can disable alpha composition of the canvas in some browsers to get a little more speed. Consumer-video never come with an alpha-channel.
// disable alpha-composition of the canvas element where supported
var context = canvas.getContext("2d", {alpha: false});
Tweak settings at encoding stage
Make sure to encode the video using a balance between size and decompression load. The more a video is compressed the more need to be reconstructed, at a cost. Encode with different encoder settings to find a balance that works in your scenario.
Also consider aspects such as color-depth i.e. 16 vs 24 bit.
The H264 codec is preferable as it has wide support in various display interface hardware.
Reduce video FPS
If the content of the video allows, f.ex. there is little movement or changes, encode using 15 FPS instead of 30 FPS. If so, also use a MODULO instead of a toggle (as shown above) where you can skip 3 frames and update canvas only at the 4.:
// draw video at 15 FPS
var modToggle = 0;
(function loop() {
if (modToggle++ % 4 === 0) { /* draw frame here */ }
requestAnimationFrame(loop);
})();
Encode video at smaller source size
Encode the video at a slightly smaller size dividable by 8 (in this case I would even suggest half size 320x240 - experiment!). Then draw using the scale parameters of the drawImage() method:
context.drawImage(video, 0, 0, video.videoWidth, video.videoHeight, 0, 0, 640, 480);
This help reduce the amount of data needed to be loaded and decoded but will of course reduce the quality. How it turns out depends again on the content.
You can also turn off interpolation using imageSmoothingEnabled set to false on the context (note: the property need a prefix in some browsers). For this you may not want to reduce the size as much as 50% but only slightly (something like 600x420 in this case).
Note: even if you "reduce" the frame rate the canvas is still redrawn at 60 FPS, but since it doesn't do any actual work on the intermediate frames it's still off-loading the CPU/GPU giving you a less tight performance budget over-all.
Hope this gives some input.
I've just been able to setup a Cocos2d-x for a multiresolution environment. Ie supporting all/most of the available iPhone screen sizes.
My next step is to scrap the individual files (bunch of pngs) and use sprite atlases (aka sprite sheets) instead.
How does this work in Cocos2D-x? Does anyone have a working sample setup?
Once again, Google and existing documentation (well, I don't think that exists) fail me.
For cocos2d-x v3.5...
The cocos2d-x test classes are always a great place to look to see how things can be done with cocos2d-x.
Check out the AppDelegate.cpp in:
Cocos2d-x/cocos2d-x-3.5/tests/cpp-tests/Classes/AppDelegate.cpp.
In that class the window size is tested to see which set of assets should be loaded. That is part of the issue, deciding which assets to load.
For iOS retina, you can check the size of the screen and then set the content scale factor to either 1.0 for standard resolution or 2.0 for retina resolution.
Unfortunately, cocos2d-x doesn't seem to ever detect certain screen sizes and call Director::getInstanct()->setContentScaleFactor(2.0) for you, so I think we have to do this our self.
I have not tested the impact of setting contentScaleFactor on non-apple devices yet...
Check out the code below. Try running it in your AppDelegate::applicationDidFinishLaunching() method to get an idea of how cocos2d-x sees screen pixels and points; the result is not exactly what I expected. The output below is for an iPhone 5 device.
director->setContentScaleFactor(2);
Size winSizePoints = director->getWinSize();
Size winSizePixels = director->getWinSizeInPixels();
Size visibleSize = director->getVisibleSize();
CCLOG("Content scale factor set to 2.0.");
CCLOG("winSizePoints:%.2f,%.2f", winSizePoints.width, winSizePoints.height);
CCLOG("winSizePixels:%.2f,%.2f", winSizePixels.width, winSizePixels.height);
CCLOG("visibleSize:%.2f,%.2f", visibleSize.width, visibleSize.height);
director->setContentScaleFactor(1);
winSizePoints = director->getWinSize();
winSizePixels = director->getWinSizeInPixels();
visibleSize = director->getVisibleSize();
CCLOG("Content scale factor set to 1.0.");
CCLOG("winSizePoints:%.2f,%.2f", winSizePoints.width, winSizePoints.height);
CCLOG("winSizePixels:%.2f,%.2f", winSizePixels.width, winSizePixels.height);
CCLOG("visibleSize:%.2f,%.2f", visibleSize.width, visibleSize.height);
Output of above code:
Content scale factor set to 2.0.
winSizePoints:1136.00,640.00
winSizePixels:2272.00,1280.00
visibleSize:1136.00,640.00
Content scale factor set to 1.0.
winSizePoints:1136.00,640.00
winSizePixels:1136.00,640.00
visibleSize:1136.00,640.00
So it seems we have to check the the screen size and then choose assets.
One option is to then use code like this to decide which assets to load based on if the screen is retina or not. You could use a similar approach to also loading different size background images to deal with different aspect ratios (more aspect ratios below).
float scaleFactor = CCDirector::getInstance()->getContentScaleFactor();
if (scaleFactor > 1.99)
{
SpriteFrameCache::getInstance()->addSpriteFramesWithFile("spriteSheet-hd.plist", "spriteSheet-hd.png");
}
else
{
SpriteFrameCache::getInstance()->addSpriteFramesWithFile("spriteSheet.png", "spriteSheet.plist");
}
For sprite sheets, I highly recommend Texture Packer. Awesome tool that can create SD and HD sheets and .plist files for you with the press of one button.
If you publish to multiple platforms, you will also want to use the macros to detect the device type and use that information in deciding which assets to load.
e.g.
if (CC_TARGET_PLATFORM == CC_PLATFORM_WIN32)
elif CC_TARGET_PLATFORM == CC_PLATFORM_WP8
elif CC_TARGET_PLATFORM == CC_PLATFORM_ANDROID
elif CC_TARGET_PLATFORM == CC_PLATFORM_IOS
elif CC_TARGET_PLATFORM == CC_PLATFORM_MAC)
endif
Another thing to consider is that on low power CPU and RAM devices, loading large assets might make your frames per second (FPS) go to crap. Or worse, the device might not be able to load the asset or could crash the game. So be sure to test/profile on least common denominator devices that you plan to support.
Aspect Ration is another important issue (landscape width:height).
iPad aspect ratio is 4:3
iPhone 4 is 3:2
iPhone 5 & 6 are 16:9
Android throws in a bunch more width:height ratios.
WP8.1 is 15:9... not sure if others.
The point is that you probably don't want to stretch your backgrounds to get them to fill the screen, or have black bars on the edges where assets are not tall or wide enough.
To deal with this I create background assets that are wider than they need to be going off the right side of the screen. That content looks good but is not vital to convey the meaning of the background. That extra content can be clipped on devices that have narrower aspect ratios, i.e. ipad, and is visible on wider aspect ratio devices, i.e. iPhone 6.
The background images are then scaled up/down to match the screenSize height and retain their aspect ratio.