Can't set the frame rate when recording a video with VecVideoRecorder - reinforcement-learning

I have a working RL model and set up that produce a video for me - however becuase the model is reasonably good, the videos are very short (reach a desitination therfore better = shorter)
Is there a way to drop the frame rate of the video output? I know it can be done with a gif. And that it can be done with ffmpeg but I can't workout how to pass it down.
I've dropped the fps in my environment from 50>10 expecting the video to be 5 times as long but that didn't work.
Save me stackoverflow you're my only hope. (appart from posting on github)

When you say that you dropped the fps from 50 to 10, I assume you changed env.metadata.video.frames_per_second, which initializes to 50 (for gym versions less than 0.22.0):
{
'render.modes': ['human', 'rgb_array'],
'video.frames_per_second': 50
}
Starting with gym 0.22.0, VideoRecorder class gets the frames_per_sec to write the video with:
env.metadata.get("render_fps", 30)
Setting env.metadata["render_fps"] = 4 should result in a slowed down video.

Related

Pygame: How to Create a Music Object

So, first of all, my overall goal is to play a different background music depending on where the player is on the map. I want one track to fade out and the other to fade in when the player reaches a certain point.
Here are my problems:
I've created a "Fader" class, with the help of Fade Between Two Music Tracks in-progress in Pygame. This allows me to fade tracks in and out, but it doesn't specify how to adapt the code to use pygame.mixer.music instead of pygame.mixer.Sound.
If I play both tracks as a Sound, on different Channels, the program slows wayyyyyy down - rendering it unusable. Research has shown that this is because music is played one bit at a time, while sounds are loaded in all at once and then played - which obviously takes quite a bit of processing power.
However, I can't seem to play them as music - and here we come to the heart of my problem. This is my Fader class written to play the music as a Sound:
class Fader(object):
def __init__(self, filename):
self.sound = pygame.mixer.Sound(filename)
# the speed at which it will fade
self.increment = 0.5
# the volume to which it will fade
self.next_vol = 1
def fade_to(self, new_vol):
self.next_vol = new_vol
curr_volume = self.get_volume()
if self.next_vol > curr_volume:
self.set_volume(curr_volume + self.increment)
elif self.next_vol < curr_volume:
self.sound.set_volume(curr_volume - self.increment)
This doesn't work, for reasons stated above. However, I cannot for the life of me achieve the same effect with music. The code written to play music as Music is the same except for line 3, which is now:
self.sound = pygame.mixer.music(filename)
I then get an error message saying "pygame.mixer.music is not callable." I understand that to mean that I'm not creating an object with that line, so I tried:
self.sound = pygame.mixer.music.load(filename)
...as I understood that to be the line that creates the Music object. However, here I get "NoneType object has no attribute 'play'" when I try to play the track.
I can't seem to figure this out, no one else seems to have this problem, and - go figure - the pygame website is down, so I can't look at the docs.
according to the documentation on https://www.pygame.org/docs/ref/music.html#pygame.mixer.music.load
The pygame.mixer.music.load(filename) function loads the music stream into the mixer , and returns None
so to play the sound, replace
self.sound = pygame.mixer.music.load(filename)
with pygame.mixer.music.load(filename)
followed by pygame.mixer.music.play(filename)

How to step frame-by-frame without skipping any, with HLS playback h.264 mpeg-ts across browsers?

I don't seem to be able to get to certain frames like 00:00:04:24 with
// javascript
videojs("player").currentTime(4.96); // for 25 fps video
And this is happening at every second of any video with closed GOP length 25.
All I wanted to achieve was to step frame-by-frame (without skipping any) through the video when my users feel the need to.
Things I'm using:
videojs 5.11.9 / 5.19.1
videojs-contrib-hls 4.0.3 / 5.4.1
videojs-framebyframe
Chrome 57.0.2987.133 (64-bit) / Firefox 51.0.1 (32-bit)
Windows 7 (64-bit)
Details (Chrome)
I have the timecode overlaid on the video I'm working with. With luck, I could pause the video at the 'missing' frames, meaning the frames are there. For example, 00:00:03:24. If I call currentTime() now
videojs("player").currentTime(); // returns **4.002128**
Then if I call
videojs("player").currentTime(4.002128);
That brings me to 00:00:04:00. The same goes with all of these calls which stayed at 00:00:04:00.
videojs("player").currentTime(4); // should be **00:00:03:23**
videojs("player").currentTime(4.04); // should be **00:00:03:24**
videojs("player").currentTime(4.08); // should be **00:00:04:00**, correct
But this
videojs("player").currentTime(3.999999999999999);
will bring me to 00:00:03:23, effectively skipping 00:00:03:24 entirely.
Curiously enough, when I changed the GOP length to 50, the same thing happens only every 2 seconds. Meaning I could jump to 00:00:02:24, 00:00:04:24, but still not 00:00:03:24.
The layout of a closed GOP length 25 (GOP M=2, N=25) looks like this for every second:
IBPBPBPBPBPBPBPBPBPBPBPBP
and for a closed GOP length 50 (GOP M=2, N=50) looks like this for every second:
IBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPP
Which brings me to suspect the P-frame at the end of each GOP is acting up.
With limited knowledge of how truly GOP works myself, I searched for other possible layout of the I/P/B frames. Then I saw open GOP. Using ffmpeg I re-encode the video with -flags -cgop as below
open GOP M=5, N=25
IBBBBPBBBBPBBBBPBBBBPBBBB
or open GOP M=2, N=50
IBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPBPB
Both of these open GOP videos have no problem jumping to any frame. With one other problem...
Problem with the open GOP 'solution'
The HLS playlist, instead of only downloading the required/playing portion of the video, now tries to download one clip after another right from the start. Maybe because now the B-frame at the end of each GOP asks for the next GOP?
Searching for more
While typing out this question, I searched for more and came across this post at bugs.chromium.org. People in that post suggested 2 things:
to re-encode the video w/o B-frames for Chrome
IE11, Safari8, Firefox34 back then seek frames accurately
Here goes testing the first suggestion.
closed GOP M=1, N=25
IPPPPPPPPPPPPPPPPPPPPPPPP
and closed GOP M=1, N=50
IPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPP
Both worked! (not so fast, doesn't work on Firefox)
Details (Firefox)
Remember the second suggestion from that post at bugs.chromium.org? At first, when I tried Firefox with any GOP length 25 video, they were all good (at seeking frame accurately). But it has similar problem as Chrome when dealing with GOP length 50 (both when M=1 and M=2) and GOP length 25 (M=1).
videojs("player").currentTime(1.96); // seeks to **00:00:01:24** correctly
videojs("player").currentTime(2.04); // seeks to **00:00:02:01** correctly
but this
videojs("player").currentTime(2); // should be **00:00:02:00**
gave me 00:00:02:00 or 00:00:02:01 randomly (biased towards the latter, while repeatedly calling only this line).
One easier way to spot this is to continuously step 25 frames ahead with videojs-framebyframe. You would get 00:00:02:01, 00:00:03:01, 00:00:04:01, and then suddenly a 00:00:05:00. Stop there and step 1-frame back and forth and you might not be able to get back to 00:00:05:00 every time.
edited portion starts
More searching data before the questions
Came across this other question. It's almost two years old. With a still working downloadable sample video of variable GOP length.
Its layout (first few GOPs)
IBPBPBPBPPPPPPPPPP
IPPP
IPPBPBPBPPBPPPPPPPP
IBPPPPPBPP
IPBPPPP
IBPPPPPPBPPBPPBPBPPPPBPBP
Here's what I use to show the layout of frame types
// command prompt or bash
ffprobe -show_frames -select_streams 0 video.ts | awk -f iframe.awk
and here's the small awk program (iframe.awk)
BEGIN {
FS="=";i="";
}
/pict_type/{
if (match($2, "I")) {
print i;
i=$2;
} else {
i=i$2;
}
}
END {
print i;
}
edited portion ends
Questions
Sorry for the long post, and thanks for reading it through.
Does this mean there are things needed to be fixed? (bugs?)
Did videojs play a part in creating this bug?
Is writing our own player going to help?
With current browsers/scripts releases, is it possible to seek frame-by-frame with HTML5 videojs HLS player across browsers (well, Chrome and Firefox at least) with one format?
(added question) Any agrees or dis-agrees, at all?
(added question) Is this a bad question as a whole? Too long?
Appreciate any sort of help and/or pointers to solve the problem! Thanks!

Embedded Sounds cut off early

I have a combined Flash Builder/Flash Pro project. Because of the hassles involving in maintaining sound assets on the timeline, my sounds are all embedded into Class files, like:
[Embed (source="/mp3/Welcome_01_V.mp3", mimeType="audio/mpeg")]
private static const WELCOME_1:Class;
These files are then referenced by the base Classes for the symbols that need them, embedded for Actionscript on Frame 10 (because the second frame label is on Frame 10 to give space for you to read the first one).
What I'm finding is that a few of these Sounds don't play all the way through, but the SoundChannel dispatches the "soundComplete" event, and its final position matches the Sound's length.
All sounds are converted from wav to mp3 at 44Hz / 16 kbps. I faked out the compiler to avoid a reference to Flex by including a dummy SoundAsset that extends Sound.
I don't know what other steps to take to debug this. Is there a way to figure out whether the problem is on the compile side or on the run side?
Updated
More things I have tried:
Looked at the Size report: The nonworking sounds were smaller in
their embedded form than the source mp3
Got rid of my own BitmapAsset and let Flash link in the Flex Framework and do whatever that does (definitely worse)
Dropped the encoding from 44 kHz to 22 kHz (no improvement or worse)
Dropped the bit rate to 8kbps (the lowest dbPowerAmp, the tool I use, supports). This usually helps somewhat, but I still usually use a word or two from the end of the file
Dropped both parameters in the encoding. This helped a few that just dropping the bit rate didn't, but not all files. Plus it sounds tinny.
Thanks!
For Flash audio, I recommend importing the sound assets into a FLA using wav files if you have the high quality source wavs. Otherwise, you can consider converting your mp3 into a wav as well. Then set the FLA export settings to the quality you want and Flash will convert your wavs into its own format at the quality you set with hopefully less issues.
Once you do that, you can export the sound symbol for actionscript in your library and set a class name just like how you would embed it.
One other trick I use is I have one FLA just for sound assets which can be used to store as big waves as I want. And when I export that, it becomes a small SWF file which I can then embed in my main application. That way, I never have Flash reconvert the wavs to the swf every single time I export the swf. Instead it just copies the swf data which is much faster as well.
[Embed(source="Audio/Sfx.swf", symbol="WELCOME_1_WAV")]
private static const WELCOME_1:Class;
If you are having audio cut off issues in Flash Pro, you may want to check your frame rate.
I had an issue with sounds cutting off (in Flash pro CC 2014). My issue turned out to be related to the frame rate being set to 25 rather than the default 24. I have been using 25 to resolve an issue unrelated to anything in this project, so my solution was to change the FPS to 24, which invoked the necessity to move all of the synchronized animations to re-align with the corresponding audio.
Why long(ish) audio tracks get cut off at the end when the frame rate is at 25 regardless of using proper keyframing is a mystery. This solved the symptoms however, so if you are having audio cut off issues in Flash Pro, you may want to check your frame rate.
My symptoms were specifically when an audio clip was particularly long, and deep into the time line.
What worked for me: I opened the audio files in an audio editor and added a few seconds of silence to the end.
Good luck! - J.Hall

How to slow down the Webcam reproduction

What I need to do is easy: the purpose of this test is show down the speed of my webcam when the camera capture a white pixel so:
1/ I create a Camera
this.cam = Camera.getCamera();
this.velocidad = 24; // I set up the fps in 24
this.cam.setMode(ancho,alto,velocidad);
vid = new Video(640,480);
vid.width = ancho;
vid.height = alto;
vid.attachCamera(cam);
addChild(vid);
2/ So now, when the pixel is recognized I need to change the current speed of the camera to 12 in order to slow down the user speed
I've tried with this code but the camera is frozen and nothing change.. I don't know if I have to delete the current instance of the camera and set up again with the disire fps
cam.setMode(640,480,12);
See Camera.setMode() to request a different frame rate, but note that what's available will depend on the camera.
I think this is not possible with just a config setting, property or method.
The possible solution would be to capture the cam and store its frames as bitmaps in and play (or render) them in sequence.
If this is not a commercial project, you can use this: http://code.google.com/p/flvrecorder/ to record and than load the video to play as you want.
Edit:
Here are some more links, since I can't code something for you now:
Post-processing captured video in AS3, creating slow motion
playing slow motion, fast forward , rewind in a video player in flash video player
Also, you can search Google for "as3 video slow" and it will give you more reference material and some examples.

Netstream and step() or seek()?

I'm on an AS3 project, playing a video (H264). I want, for some special reasons, to go to a certain position.
a) I try it with NetStream.seek(). There it only goes to keyframes. In my current setting, this means, i can find a position every 1 second. (for a better resolution, i'd have to encode the movie with as many keyframes as possible, aka every frame a keyframe)
this is definetly not my favourite way, because I don't want to reencode all the vids.
b) I try it with NetStream.step(). This should give me the opportunity to step slowly from frame to frame. But in the documentation it says:
This method is available only when data is streaming from Flash Media Server 3.5.3 or higher and when NetStream.inBufferSeek is true.
http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flash/net/NetStream.html#step()
Does this mean, it is not possible with Air for Desktop? When I try it, nothing works.
Any suggestions, how to solve this problem?
Greetings & Thank you!
Nicolas
Flash video can only be advanced by seconds unless you have Flash Media Server hosting your video. Technically, that means that you can have it working as intended in Air, however, the video would have to be streaming (silly adobe...).
You have two options:
1) Import the footage as a movieclip. The Flash IDE has a wizard for this, and if you're developing exclusively in non-FlashIDE environment, you can convert and export as an external asset such as an SWF or SWC. This would then be embedded or runtime loaded into your app giving you access to the per-frame steppable methods of MovieClip. This, however, does come with some audio syncing issues (iirc). Also, scrubbing backwards is not an MC's forté.
2) Write your own video object that loads an image sequence and displays each frame in order. You'd have to setup your own audio syncing abilities, but it might be the most direct solution apart from FLVComponent or NetStream.
I've noticed that flash player 9 scrubs nice and smooth but in players 10+ I get this no scrub problem.
My fix, was to limit frequency the calls to the seek function to <= 200ms. This fixed scrubbing but is much less smooth as player 9. Perhaps because of the "Flash video can only be advanced by seconds" limitation? I used a timer to tigger the function that calls seek() for the video.
private var scrubInterval:Timer = new Timer(200);
private function videoScrubberTouch():void {
_ns.pause();
var bounds:Rectangle = new Rectangle(0,0,340,0);
scrubInterval.addEventListener(TimerEvent.TIMER, scrubTimeline);
scrubInterval.start();
videoThumb.startDrag(false, bounds);
}
private function scrubTimeline(e:TimerEvent):void {
var amt:Number = Math.floor((videoThumb.x / 340) * duration);
trace("SCRUB duration: "+duration+" videoThumb.x: "+videoThumb.x+" amt "+amt);
_ns.seek(amt);
}
Please check this Demo link (or get the SWF file to test outside of browser via desktop Flash Player).
Note: Demo requires FLV with H.264 video codec and AAC or MP3 audio codec.
The source code for that is here: Github link
In the above demo there is (bytes-based) seeking and frame by frame stepping. The functions you want to study mainly are:
Append_SEEK ( position amount ) - This will got to the specified position in bytes and search for the nearest available keyframe.
get_frame_TAG - This will extract a tag holding one frame of data. Audio can be in frames too but lets assume you have video-only. That function is your opportunity to adjust timestamps. When it's run it will also append the tag (so each "get_frame_TAG" is also a "frame step").
For example : You have a 25fps video, you want the third-frame at 4 seconds into playback...
1000 milisecs / 25 fps = 40 units for each timestamp. So 4000 ms == 4 secs + add the 40 x 3rd frame == an expected timestamp of 4120.
So getting that frame means... First find a keyframe. Then step through each frame checking the timestamps that represent a frame you want. If it isnt then change it to the same as most recent keyframe timestamp (this forces Flash to fast-forward through the frames to keep things in sync as it assumes the frame [with smaller than expected timestamp] should have been played by that time). You can "hide" the video object during this process if you don't like the look of fast-forwarding.