I use MediaCodec to decode H.264/H.265 video streams. I don't think the code of thousands of lines is relevant to my question, so please allow me to avoid pasting the code here. Let me emphasize that the decoding works flawlessly. I am asking this question primarily out of curiosity, not for solving a problem.
MediaCodec.BufferInfo bi = new MediaCodec.BufferInfo();
...
int iOutputBufferIndex = myMediaCodec.dequeueOutputBuffer(bi, TIMEOUT_USEC);
logd("debug", "Output buffer size: " + bi.size);
The official document says that MediaCodec.BufferInfo.size is "The amount of data (in bytes) in the buffer." I have tested three video streams decoded by the code. MediaCodec.BufferInfo.size is 1 for an H.265 video and 8 for two H.264 video streams. These numbers do not look like "The amount of data (in bytes) in the buffer.". My understanding is that the buffer has decoded video frame.
Could anyone shed some light on this (i.e. the exact meaning of MediaCodec.BufferInfo.size) ?
Related
I try to get familiar with libav in order to process a raw H.264 stream from a GenICam supporting camera.
I'd like to receive the raw data via the GenICam provided interfaces (API), and then forward that data into libav in order to produce a container file that then is streamed to a playing device like VLC or (later) to an own implemented display.
So far, I played around with the GenICam sample code, which transferres the raw H.264 data into a "sample.h264" file. This file, I have put through the command line tool ffmpeg, in order to produce an mp4 container file that I can open and watch in VLC
command: ffmpeg -i "sample.h264" -c:v copy -f mp4 "out.mp4"
Currently, I dig through examples and documentations for each H.264, ffmpeg, libav and video processing in general. I have to admit, as total beginner, it confuses me a lot.
I'm at the point where I think I have found the according libav functions that would help my undertaking:
I think, basically, I need the functions avcodec_send_packet() and avcodec_receive_packet() (since avcodec_decode_video2() is deprecated).
Before that, I set up an avCodedContext structure and open (or combine?!?) it with the H.264 codec (AV_CODEC_ID_H264).
So far, my code looks like this (omitting error checking and other stuff):
...
AVCodecContext* avCodecContext = nullptr;
AVCodec *avCodec = nullptr;
AVPacket *avPacket = av_packet_alloc();
AVFrame *avFrame = nullptr;
...
avCodec = avcodec_find_decoder(AV_CODEC_ID_H264);
avCodecContext = avcodec_alloc_context3(avCodec);
avcodec_open2 ( avCodecContext, avCodec, NULL );
av_init_packet(avPacket);
...
while(receivingRawDataFromCamera)
{
...
// receive raw data via GenICam
DSGetBufferInfo<void*>(hDS, sBuffer.BufferHandle, BUFFER_INFO_BASE, NULL, pPtr)
// libav action
avPacket->data =static_cast<uint8_t*>(pPtr);
avErr = avcodec_send_packet(avCodecContext, avPacket);
avFrame = av_frame_alloc();
avErr = avcodec_receive_frame( avCodecContext, avFrame);
// pack frame in container? (not implemented yet)
..
}
The result of the code above is, that both calls to send_packet() and receive_frame() return error codes (-22 and -11), which I'm not able to decrypt via av_strerror() (it only says, these are error codes 22 and 11).
Edit: Maybe as an additional information for those who wonder if
avPacket->data = static_cast<uint8_t*>(pPtr);
is a valid operation...
After the very first call to this operation, the content of avPacket->data is
{0x0, 0x0, 0x0, 0x1, 0x67, 0x64, 0x0, 0x28, 0xad, 0x84, 0x5,
0x45, 0x62, 0xb8, 0xac, 0x54, 0x74, 0x20, 0x2a, 0x2b, 0x15, 0xc5,
0x62}
which somehow looks as something to be expected becaus of the NAL marker and number in the beginning?
I don't know, since I'm really a total beginner....
The question now is, am I on the right path? What is missing and what do the codes 22 and 11 mean?
The next question would be, what to do afterwards, in order to get a container that I can stream (realtime) to a player?
Thanks in advance,
Maik
At least for the initally asked question I found the solution for myself:
In order to get rid of the errors on calling the functions
avcodec_send_packet(avCodecContext, avPacket);
...
avcodec_receive_frame( avCodecContext, avFrame);
I had to manually fill some parameters of 'avCodecContext' and 'avPacket':
avCodecContext->bit_rate = 8000000;
avCodecContext->width = 1920;
avCodecContext->height = 1080;
avCodecContext->time_base.num = 1;
avCodecContext->time_base.den = 25;
...
avPacket->data = static_cast<uint8_t*>(pPtr);
avPacket->size = datasize;
avPacket->pts = frameid;
whereas 'datasize' and 'frameid' are received via GenICam, and may not be the appropriate parameters for the fields, but at least I do not get any errors anymore.
Since this answers my initial question on how I get the raw data into the structures of libav, I think, the question is answered.
The discussion and suggestions with/from Vencat in the commenst section lead to additional questions I have, but which should be discussed in a new question, I guess.
So I have seen a lot of topics on FFMPeg and it's a great tool I learnt about today, but I have spent the day perfecting the command and now am a little stuck with the NodeJS part.
In essence the command does the following: take input from a Mac OSX webcam, and then stream it to a web-socket. Now I looked at a lot of the NodeJS libraries but I couldn't find one that did what I need; or did not understand how to. Here is an example of the command that I am using:
ffmpeg -f avfoundation -framerate 30 -video_size 640x480 -pix_fmt uyvy422 -i "0:1" -f mpegts -codec:v mpeg1video -s 640x480 -b:v 1000k -bf 0 http://localhost:8081/stream
This does everything I need for the streaming side of things, but I wish to call it via NodeJS, and then be able to monitor the log, and parse the data that comes back for example:
frame= 4852 fps= 30 q=6.8 size= 30506kB time=00:02:41.74 bitrate=1545.1kbits/s speed= 1x \r
and use it to get a JSON array back for me to output to a webpage.
Now all I am doing is working on ways of actually parsing the data, and I have looked at lots of other answers for things like this, but I can't seem to split/replace/regex it. I can't get anything but a long string from it.
Here is the code I am using (NodeJS):
var ffmpeg = require('child_process').spawn('/usr/local/Cellar/ffmpeg/3.3.1/bin/ffmpeg', ['-f', 'avfoundation', '-framerate', '30', '-video_size', '640x480', '-pix_fmt', 'uyvy422', '-i', '0:1', '-f', 'mpegts', '-codec:v', 'mpeg1video', '-s', '640x480', '-b:v', '1000k', '-bf', '0', 'http://localhost:8081/test']);
ffmpeg.on('error', function (err) {
console.log(err);
});
ffmpeg.on('close', function (code) {
console.log('ffmpeg exited with code ' + code);
});
ffmpeg.stderr.on('data', function (data) {
// console.log('stderr: ' + data);
var tData = data.toString('utf8');
// var a = tData.split('[\\s\\xA0]+');
var a = tData.split('\n');
console.log(a);
});
ffmpeg.stdout.on('data', function (data) {
var frame = new Buffer(data).toString('base64');
// console.log(frame);
});
I have tried splitting with new lines, carridge return, spaces, tabs, but I just can't seem to get a basic array of bits, that I can work with.
Another thing to note, is you will notice the log comes back via stderr, I have seen this online and apparently it does it for a lot of people? So I am not sure what the deal is with that? but the code is is the sdterr callback.
Any help is very appreciated as I am truly confused on what I am doing wrong.
Thanks.
An update on this, I worked with one of the guys off the IRC channel: #ffmpeg on FreeNode. The answer was to send the output via pipe to stdout.
For example I appended the following to the FFMpeg command:
-progress pipe:1
The progress flag is used to give an output every second with information about the stream, so this is pretty much everything you get from the stderr stream every second, but piped to the stdout stream in a format that I can parse. Below is taken from the documentation.
-progress url (global) Send program-friendly progress information to url. Progress information is written approximately every second and at the end of the encoding process. It is made of "key=value" lines. key consists of only alphanumeric characters. The last key of a sequence of progress information is always "progress".
Here is an example of the code I used to parse the stream information:
ffmpeg.stdout.on('data', function (data) {
var tLines = data.toString().split('\n');
var progress = {};
for (var i = 0; i < tLines.length; i++) {
var item = tLines[i].split('=');
if (typeof item[0] != 'undefined' && typeof item[1] != 'undefined') {
progress[item[0]] = item[1];
}
}
// The 'progress' variable contains a key value array of the data
console.log(progress);
});
Thanks to all that commented!
In the spirit of not reinventing the wheel, you might want to try using fluent-ffmpeg. It dispatches a progress event with a number of useful fields
'progress': transcoding progress information
The progress event is emitted every time ffmpeg reports progress
information. It is emitted with an object argument with the following
keys:
frames: total processed frame count
currentFps: framerate at which FFmpeg is currently processing
currentKbps: throughput at which FFmpeg is currently processing
targetSize: current size of the target file in kilobytes
timemark: the timestamp of the current frame in seconds
percent: an estimation of the progress percentage
If you're curious about how they do this, you can read the source, starting from here and here
Ffmpeg uses stderr to output log info because stdout is used for piping the output to other processes. The stuff in stderr is actually just debug information, and not the actual output of the process.
BONUS ROUND
I've seen some hacky video players that use websockets to stream videos, but that approach has a number of issues with it. I'm not going to go over those, but I will explain why I think you should use hls.js.
Support is pretty good; basically works everywhere except old IE. It uses MSE to upgrade the standard video element, so you don't have to wrestle with building a custom player.
Here are the docs for the hls format flag
Here's some code that I'm using to stream from an IPTV box to a web page.
this.ffmpeg = new FFFmpeg()
this.ffmpeg.input(request(this.http_stream))
.videoCodec('copy')
.audioCodec('copy')
.outputOptions([
'-f hls',
'-hls_list_size 6',
'-hls_flags delete_segments'
])
.output( path.join(this.out_dir, 'video.m3u8') )
.run()
It generates a .m3u8 manifest file along with segmented mpeg-ts video files. All you need to do after that is load the m3u8 file into the hls.js player and you have a live stream!
If you're going to re-encode the stream, you will probably see some low fps and glitchiness. I'm lucky since my source stream is already encoded as mpeg-ts.
I use a MagicalRecord, and am having a little trouble using it.
A server sends me a JSON, and I need it as quickly as possible ro map to the existing NSManagedObject and give it to the block.
NSManagedObjectContext *localContext = [NSManagedObjectContext MR_contextForCurrentThread];
[Review MR_importFromObject:dictionary inContext:localContext];
[localContext MR_saveOnlySelfAndWait];
And after:
[[CacheOperation sharedOperation]saveBestRateProductByDict:reviewDict];
Review *review = [Review MR_findFirstByAttribute:#"id" withValue:[reviewDict objectForKey:#"id"]];
But if I have many objects, it takes a lot of time.
How I can map NSDictionary to NSManagedObject without writing to CoreData?
I guess MR_importFromObject check for the existence of the object to have an insert or update behavior.
That's great for most cases. (and it was made for the 90% http://www.cimgf.com/2012/05/29/importing-data-made-easy/ )
But you are in the 10% (me too, if it can be of any support)
This behavior mean that there is a request to find the object, and a request to update it. Multiple by the number of objects, that can be huge.
You can refer to a good apple doc (the part Implementing Find-or-Create Efficiently ) :
https://developer.apple.com/library/ios/documentation/Cocoa/Conceptual/CoreData/Articles/cdImporting.html
One trick is to make only one request for all the objects you want to update, and one request to update them all. It's worse in memory usage but better in I/O, that should speed you up.
We went another way using TMCache and storing raw JSON for objects changing frequently.
Hope this help.
Hi I am working on WebAudio API . I read HTML5 Web Audio API, porting from javax.sound and getting distortion link but not getting goodquality as in java API.I am getting PCM data from server in signed bytes . Then I have to changed this into it 16 bit format . for changing I am using ( firstbyte<<8 | secondbyte )
but I am not able to get good quality of sound . is there any problem in conversion or any other way to do for getting good quality of sound ?
The Web Audio API uses 32-bit signed floats from -1 to 1, so that's what I'm going to (hopefully) show you how to do, rather than 16-bit as you mentioned in the question.
Assuming your array of samples is called samples and are stored as 2's compliment from -128 to 127, I think this should work:
var floats = new Float32Array(samples.length);
samples.forEach(function( sample, i ) {
floats[i] = sample < 0 ? sample / 0x80 : sample / 0x7F;
});
Then you can do something like this:
var ac = new webkitAudioContext()
, ab = ac.createBuffer(1, floats.length, ac.sampleRate)
, bs = ac.createBufferSource();
ab.getChannelData(0).set(floats);
bs.buffer = ab;
bs.connect(ac.destination);
bs.start(0);
In your experience, in Fortran 90, what is the best way to store large arrays in output files? Previously, I had been trying to write large arrays to ASCII text files. For example, I would do something like this (thanks to the recommendation at the bottom of the page In Fortran 90, what is a good way to write an array to a text file, row-wise?):
PROGRAM testing1
IMPLICIT NONE
INTEGER :: i, j, k
INTEGER, DIMENSION(4,10) :: a
k=1
DO i=1,4
DO j=1,10
a(i,j)=k
k=k+1
END DO
END DO
OPEN(UNIT=12, FILE="output.txt", ACTION="WRITE", STATUS="REPLACE")
DO i=1,4
DO j=1,10
WRITE(12, "(i2,x)", ADVANCE="NO") a(i,j)
END DO
WRITE(12, *)
END DO
CLOSE(UNIT=12)
END PROGRAM testing1
This works, but as pointed out by the topmost reply at In Fortran 90, what is a good way to write an array to a text file, row-wise?, writing large arrays to text files is very slow and creates files that are somewhat larger in size than is necessary. The poster there recommended instead writing to an unformatted Fortran binary, using something like:
PROGRAM testing2
IMPLICIT NONE
INTEGER :: i, j, k
INTEGER, DIMENSION(4,10) :: a
k=1
DO i=1,4
DO j=1,10
a(i,j)=k
k=k+1
END DO
END DO
OPEN(UNIT=13, FILE="output.dat", ACTION="WRITE", STATUS="REPLACE", &
FORM="UNFORMATTED")
WRITE(13) a
CLOSE(UNIT=13)
END PROGRAM testing2
This seems to work, and is indeed much faster and results in smaller file sizes, as promised by the reply here. However, what do I do if I would like to be able to later work with the data stored in Fortran binary (e.g., output.dat above) and analyze its contents? For example, what if I want to open the array stored in the binary in a program such as Microsoft Excel?
When I mentioned matlab in my previous post, the reply suggested that I open the binary as a hexadecimal file and figure out and extract the records from there. But, I am nervous that I am getting into deep water since I have no prior experience in hexadecimal sleuthing. When I asked on the matlab board (here: http://www.mathworks.com/matlabcentral/answers/12639-advice-on-reading-an-unformatted-fortran-binary-file-into-matlab) about reading Fortran files into matlab, the person there suggested that using Fortran stream might be easy. But is Fortran stream (i.e., using the directive ACCESS="STREAM" in the OPEN command) likely to be similar in time and file size to the ASCII text file that I created in my first example above?
Or, do you know if there is any other software that can automatically read Fortran binaries into some sort of human readable form? (Or, do you know of any good tutorials on either hexadecimal sleuthing or Fortran stream?)
Thank you very much for your time.
Stream is a choice independent of the choice of formatted / unformatted -- one is "access", the other "format" The default for Fortran I/O is record oriented access. The typical approach of a Fortran compiler for records (at least unformatted) to write a 4-byte record length before and after each record. (The "after" is to make reading backwards easier.) Using a hex edit you could verify these extra data items that I described and skip them in MatLab. But they are not part of the language standard and are not portable and are certainly not obvious in other languages. If you select stream and unformatted you will just get the raw sequence of bytes corresponding to your data items -- no extra data items to worry about in the other language! In my experience this output tends to be fairly easy to read in other languages (not tried in MatLab). If this is a small & simple project with portability of the files to other computers not an issue, I would probably use this approach (stream & unformatted) rather than a file format specification such as HDF5 or FITS. I'd write the array as write (13) a, as in your final example. Depending on the other language, you might have to transpose the dimensions. If this is a major and long-lived project with portability a concern, then a portable and standard file interface is worth considering.
I don't know whether any of these formats can be read from Excel. More research.... You might have to write a program to read the binary file of whatever format and output a file in a format that Excel understands.
(converting comment into an answer for posterity)
Are you specifically trying to get information into Matlab? If you are, I highly recommend HDF5. This is the portable binary format you have been looking for.
For converting a Fortran binary to HDF5, you're going to have to read in the original Fortran binary and then write out the same data to an HDF5 file. If you have the Fortran source, this should be pretty easy. Allocate your arrays, make sure you read the arrays in the same order as you wrote them and then write out your new shiny HDF5 file.
The HDF5 group has tutorials with examples in C and Fortran. There is likely an example very close to what you're trying to do. When you build HDF5, make sure to manually enable Fortran support. It is disabled by default.
%In MATLAB
fid=fopen('YOUR_FILE.direct','r'); %Fortran Direct ACCESS
frewind(fid);
tbb=ones(367,45203);
for i =1:367
temp=fread(fid,[45203],'single');
tbb(i,:)=temp;
end
fclose(fid)