I'm trying to create a Windows Store application that allows the UI to be captured and overlaid on video coming from a web camera.
I'm using the MediaCapture class to manage the video capture process. I've created a MFT (based on the Grayscale sample) that allows me to accomplish this in a basic manner. This MFT has been added as an effect to the Record Stream of the MediaCapture class, and I'm able to create a video file with the UI overlaid on the camera video easily enough. (Easily is a relative term)
The problem that I've hit is that the overlay from the MFT is showing up in the preview stream, which is also being displayed on screen. So the UI is being displayed normally, and also in the video stream. This is a bad result, as I don't want the effect applied to the preview stream and don't want the user to see the UI in the video preview, only in the resulting recording.
Is there a way to make the MediaCapture class use the effect only on the record stream, and not the preview stream?
If there is not an easy way to do this, can this be implemented by creating a custom sink? The MediaCapture could record to the custom sink, and the custom sink would add the overlay and save to video?
With some cameras (USB webcams in particular), record/preview/photo all come from the same video stream. So applying an effect to one applies the effect to all. Whether video streams are identical or independent is given by MediaCapture.MediaCaptureSettings.VideoDeviceCharacteristic.
So in your case, using a custom sink seems the way to go. IMFSinkWriter can be used to encode frames after adding the overlay.
For reference, code snippet adding an effect to preview+record for any type of camera (effectively the opposite of what you are trying to do):
MediaCapture capture = ...;
await capture.AddEffectAsync(MediaStreamType.VideoPreview, "Extensions.MyEffect", null);
// If preview and record are different streams, also add the effect there
if ((capture.MediaCaptureSettings.VideoDeviceCharacteristic != VideoDeviceCharacteristic.AllStreamsIdentical) &&
(capture.MediaCaptureSettings.VideoDeviceCharacteristic != VideoDeviceCharacteristic.PreviewRecordStreamsIdentical))
{
await capture.AddEffectAsync(MediaStreamType.VideoRecord, "Extensions.MyEffect", null);
}
Related
I have a app where I play different code-generated sounds. I place these sounds in a AudioBufferSourceNode.
I allow the the user to choose what output device to play the sound through, so I use a MediaStreamAudioDestinationNode with its stream used as the source for an Audio Element. This way when the user chooses an audio output to play the sound to, I set the Sink Id of the Audio element to the requested audio output.
So I have AudioBufferSourceNode -> some Audio Graph (gain nodes, etc) -> MediaStreamAudioDestinationNode -> Audio element.
When I Play the first sound, it sound fine. But when I create a new source and connect it to the same MediaStreamAudioDestinationNode, the sound is played with the wrong pitch.
I created a Fiddle that shows the problem.
Is this a bug, or am I doing something wrong?
The problem was identified based on the OP Chrome Ticket.
It seems to come from the lack of sync between AudioElement and its source AudioNode (AudioBufferSourceNode, OscillatorNode, etc.) when you pause the source and play it back again.
The solution is to always call AudioElement.pause() and AudioElement.start() alongside your source stop and start.
https://jsfiddle.net/k1r7o0xj/3/
It's possible to dynamically change your graph layout by using .connect() and .disconnect(), even when audio is playing or sent through a stream (which could even be streamed over WebRTC).
I couldn't find a reference in the spec, so I'm pretty sure this is taken for granted.
For example, if you have two AudioBufferSourceNodes bufferSource1 and bufferSource2, and a MediaStreamAudioDestinationNode streamDestination:
bufferSource1.connect(streamDestination);
//do some other things here, and after some time, switch to bufferSource2:
//(streamDestination doesn't need to be explicitly specified here)
bufferSource1.disconnect(streamDestination);
bufferSource2.connect(streamDestination);
Example in action.
Edit 1:
Proper implementation:
According to the Editors Draft on the Audio Output API, it is planned/will be possible to choose a custom audio output device for the AudioContext as well (by means of new AudioContext({ sinkId: requestedSinkId });). I couldn't find any info on the progress, and even found a related discussion which the asker apparently read already. According to this and (many) other references, it doesn't seem te be an easy task, but it's planned for WA V1.
Edit:
That section has been removed from the API Draft, but you can still find it in an older version.
Current workaround:
I played around with your workaround (using a MediaStreamAudioDestinationNode and Audio object), and it seems to be related to nothing being connected. I modified my example to toggle a single buffer (similar to your example but with an AudioBufferSourceNode), and observed a similar frequency drop. However, when using a GainNode inbetween and setting it's gain.value to either 0 or 1, the frequency drops disappeared (this isn't gonna be the solution if you want to create and connect new AudioBuffers dynamically).
I am trying to build an audio recorder app similar to iOS7 built in one and looking for guidance on what controls to use for the recording app. I understand I will be using a tableview for the list of previous recordings and a UIView for the top recording view and on tapping record adjust the table view and move down the black recording view.
How should I implement the endless horizontal scrolling view? Should I use a collection view and keep adding elements to the model array as the time increments. Also what should I use for the timer. Is there something like setInterval for Objective C like in Javascript that I can use to keep updating the UI at regular time interval?
If someone also knows of a cocoa pod or sample code that would be greatly appreciated.
For recording the simplest audiorecorder is AVAudioRecorder. Here is a simple implementation of an audio recording app: https://github.com/calmez/Recorder. AVAudioRecorder has simple metering methods where you can read volume output of the channels
Honestly though, Apple would probably use CoreAudio to get the audio because it is more optimized. Novocaine is a good core audio engine that could get you started https://github.com/alexbw/novocaine
For rendering the waveform, I would guess that Apple probably uses OpenGL. I don't see how to do it easily and efficiently otherwise. You could draw them using the standard drawing APIs for UIView like this project does (https://github.com/fulldecent/FDWaveformView) but I don't see this animating well.
For the timer, there is NSTimer
I want to write a Windows Store App that can capture video (without any sound) and take pictures. Imagine a digital camera: you can preview the picture on the screen of your device before pushing the button which takes the pic.
The problem I'm facing now is the fact that the Windows.Media.Capture namespace has only classes for objects that capture video with sound (CameraCaptureUI, MediaCapture). I'm not troubled by the objects' capabilities, but by the fact that I will have to include in the manifest of the app the Microphone capability and it does not make sense for the app to use it. I need a class that uses only the Webcam capability.
Any ideas?
I found the answer and I thought I should share it. I'm sorry for answering my own question, but here goes:
One can specify in the settings of the MediaCapture object, when initializing it, that it will use only the Video part:
var mediaCaptureMgr = new MediaCapture();
var captureSettings = new MediaCaptureInitializationSettings();
captureSettings.StreamingCaptureMode = StreamingCaptureMode.Video;
await mediaCaptureMgr.InitializeAsync(captureSettings);
RTFM!
I've been Googling around a bit for an answer and haven't found a definitive one either way: is it possible to play a video using an HTML5 canvas, and also allow the user to draw on this video? The use case, for some context, is to play a video on infinite loop so the user can draw multiple boxes over specific areas to indicate regions of interest.
As a bonus (:P), if I can figure out how to do this on its own, any hints as to how this could be done within Drupal? I'm already looking at the Canvas Field module, but if you have any hints on this point too (though the first one is the priority), that'd be awesome!
You can draw html5 video elements onto a canvas. The drawImage method accepts a video element in the first parameter just like an image element. This will take the current "frame" of the video element and render it onto the canvas. To get fluid playback of the video you will need to draw the video to the canvas repeatedly.
You can then draw on the canvas normally, making sure you redraw everything after each update of the video frame.
Here is a demo of video on canvas
here is a in-depth look into video and the canvas
I recently received this request from a client to provide this feature, and it must be CMS-friendly. The technique involves three big ideas
a drawing function
repeatedly calling upon the same drawing function
using requestAnimationFrame to paint the next frame
Assuming you have a video element already, you'd take the following steps
Hide the video element
Create a canvas element whose height/width match the video element, store this somewhere
Get the context of the canvas element with `canvas.getContext('2d') and also store that somewhere
Create a drawing function
In that drawing function, you would use canvas.drawImage(src, x, y) where src is the edited version of the current frame of the video;
In that drawing function, use recursion to call itself again
I can give you two examples of this being done (and usable for content management systems)
The first is here: https://jsfiddle.net/yywL381w/19/
A company called SDL makes a tool called Media Manager that hosts videos. What you see is a jQuery plugin that takes its parameters from a data-* , makes a request from the Media Manager Rest API, creates a video, and adds effects based entirely on data* attributes. That plugin could easily be tweaked to work with videos called from other sources. You can look at the repo for it for more details on usage.
Another example is here: http://codepen.io/paceaux/pen/egLOeR
That is not a jQuery plugin; it's an ES6 class instead. You can create an image/video and apply a cropping effect with this:
let imageModule = new ImageCanvasModule(module);
imageModule.createCanvas();
imageModule.drawOnCanvas();
imageModule.hideOriginal();
You'll observe, in the ImageCanvasModule class, this method:
drawFrame () {
if (this.isVideo && this.media.paused) return false;
let x = 0;
let width = this.media.offsetWidth;
let y = 0;
this.imageFrames[this.module.dataset.imageFrame](this.backContext);
this.backContext.drawImage(this.media, x, y, width, this.canvas.height);
this.context.drawImage(this.backCanvas, 0, 0);
if (this.isVideo) {
window.requestAnimationFrame(()=>{
this.drawFrame();
});
}
}
The class has created a second canvas, to use for drawing. That canvas isn't visible, it's just their to save the browser some heartache.
The "manipulation" that is content manageable is this.imageFrames[this.module.dataset.imageFrame](this.backContext);
The "frame" is an attribute stored on the image/video (Which could be output by a template in the CMS). This gets the name of the imageFrame, and runs it as a matching function. It also sends in the context (so I can toggle between drawing on the back canvas or main canvas if needed)
then this.backContext.drawImage(this.media, x, y, width, this.canvas.height); draws the image on the back context.
Finally, this appears on the main canvas with this.context.drawImage(this.backCanvas, 0, 0); where I take the backcanvas, and draw it on to the main canvas. So the canvas that's visible has the least amount of manipulations possible.
And at the end, because this is a video, we want to draw a new frame. So we have the function call itself:
if (this.isVideo) {
window.requestAnimationFrame(()=>{
this.drawFrame();
});
This whole setup allows us to use the CMS to output data-* attributes containing the type of frame the user wants to be drawn around the image. the JavaScript then produces a canvasified version of that image or video. Sample markup might look like:
<video muted loop autoplay data-image-frame="wedgeTop">
Is there any control (To use in a c# project) to record user activities and then encoding it to a video file?
as i understand a video file is a collection of static image frames. You can use this tool's api from http://www.vcskicks.com/ScreenShotTaker.zip, which essentially shows you the mechanism of capturing the screen. So what you may have to do is, automatically capture images in a loop according to a set frame rate and then export it to a video. I'm not sure of how to create a media stream, but you have the library to do the screen capture.