FFmpeg-基础组件-AVFrame

本章主要介绍FFmpeg基础组件AVFrame.

文章目录 1.结构体成员2.成员函数AVFrame Host内存的获取 av_frame_get_bufferAVFrame device内存获取av_hwframe_get_buffer（）

1.结构体成员

我们把所有的代码先粘贴上来，在后边一个一个解释。

typedef struct AVFrame { #define AV_NUM_DATA_POINTERS 8 /** * pointer to the picture/channel planes. * This might be different from the first allocated byte. For video, * it could even point to the end of the image data. * * All pointers in data and extended_data must point into one of the * AVBufferRef in buf or extended_buf. * * Some decoders access areas outside 0,0 - width,height, please * see avcodec_align_dimensions2(). Some filters and swscale can read * up to 16 bytes beyond the planes, if these filters are to be used, * then 16 extra bytes must be allocated. * * NOTE: Pointers not needed by the format MUST be set to NULL. * * @attention In case of video, the data[] pointers can point to the * end of image data in order to reverse line order, when used in * combination with negative values in the linesize[] array. */ uint8_t *data[AV_NUM_DATA_POINTERS]; /** * For video, a positive or negative value, which is typically indicating * the size in bytes of each picture line, but it can also be: * - the negative byte size of lines for vertical flipping * (with data[n] pointing to the end of the data * - a positive or negative multiple of the byte size as for accessing * even and odd fields of a frame (possibly flipped) * * For audio, only linesize[0] may be set. For planar audio, each channel * plane must be the same size. * * For video the linesizes should be multiples of the CPUs alignment * preference, this is 16 or 32 for modern desktop CPUs. * Some code requires such alignment other code can be slower without * correct alignment, for yet other it makes no difference. * * @note The linesize may be larger than the size of usable data -- there * may be extra padding present for performance reasons. * * @attention In case of video, line size values can be negative to achieve * a vertically inverted iteration over image lines. */ int linesize[AV_NUM_DATA_POINTERS]; /** * pointers to the data planes/channels. * * For video, this should simply point to data[]. * * For planar audio, each channel has a separate data pointer, and * linesize[0] contains the size of each channel buffer. * For packed audio, there is just one data pointer, and linesize[0] * contains the total size of the buffer for all channels. * * Note: Both data and extended_data should always be set in a valid frame, * but for planar audio with more channels that can fit in data, * extended_data must be used in order to access all channels. */ uint8_t **extended_data; /** * @name Video dimensions * Video frames only. The coded dimensions (in pixels) of the video frame, * i.e. the size of the rectangle that contains some well-defined values. * * @note The part of the frame intended for display/presentation is further * restricted by the @ref cropping "Cropping rectangle". * @{ */ int width, height; /** * @} */ /** * number of audio samples (per channel) described by this frame */ int nb_samples; /** * format of the frame, -1 if unknown or unset * Values correspond to enum AVPixelFormat for video frames, * enum AVSampleFormat for audio) */ int format; /** * 1 -> keyframe, 0-> not */ int key_frame; /** * Picture type of the frame. */ enum AVPictureType pict_type; /** * Sample aspect ratio for the video frame, 0/1 if unknown/unspecified. */ AVRational sample_aspect_ratio; /** * Presentation timestamp in time_base units (time when frame should be shown to user). */ int64_t pts; /** * DTS copied from the AVPacket that triggered returning this frame. (if frame threading isn't used) * This is also the Presentation time of this AVFrame calculated from * only AVPacket.dts values without pts values. */ int64_t pkt_dts; /** * Time base for the timestamps in this frame. * In the future, this field may be set on frames output by decoders or * filters, but its value will be by default ignored on input to encoders * or filters. */ AVRational time_base; /** * picture number in bitstream order */ int coded_picture_number; /** * picture number in display order */ int display_picture_number; /** * quality (between 1 (good) and FF_LAMBDA_MAX (bad)) */ int quality; /** * for some private data of the user */ void *opaque; /** * When decoding, this signals how much the picture must be delayed. * extra_delay = repeat_pict / (2*fps) */ int repeat_pict; /** * The content of the picture is interlaced. */ int interlaced_frame; /** * If the content is interlaced, is top field displayed first. */ int top_field_first; /** * Tell user application that palette has changed from previous frame. */ int palette_has_changed; /** * reordered opaque 64 bits (generally an integer or a double precision float * PTS but can be anything). * The user sets AVCodecContext.reordered_opaque to represent the input at * that time, * the decoder reorders values as needed and sets AVFrame.reordered_opaque * to exactly one of the values provided by the user through AVCodecContext.reordered_opaque */ int64_t reordered_opaque; /** * Sample rate of the audio data. */ int sample_rate; /** * Channel layout of the audio data. */ uint64_t channel_layout; /** * AVBuffer references backing the data for this frame. All the pointers in * data and extended_data must point inside one of the buffers in buf or * extended_buf. This array must be filled contiguously -- if buf[i] is * non-NULL then buf[j] must also be non-NULL for all j < i. * * There may be at most one AVBuffer per data plane, so for video this array * always contains all the references. For planar audio with more than * AV_NUM_DATA_POINTERS channels, there may be more buffers than can fit in * this array. Then the extra AVBufferRef pointers are stored in the * extended_buf array. */ AVBufferRef *buf[AV_NUM_DATA_POINTERS]; /** * For planar audio which requires more than AV_NUM_DATA_POINTERS * AVBufferRef pointers, this array will hold all the references which * cannot fit into AVFrame.buf. * * Note that this is different from AVFrame.extended_data, which always * contains all the pointers. This array only contains the extra pointers, * which cannot fit into AVFrame.buf. * * This array is always allocated using av_malloc() by whoever constructs * the frame. It is freed in av_frame_unref(). */ AVBufferRef **extended_buf; /** * Number of elements in extended_buf. */ int nb_extended_buf; AVFrameSideData **side_data; int nb_side_data; /** * @defgroup lavu_frame_flags AV_FRAME_FLAGS * @ingroup lavu_frame * Flags describing additional frame properties. * * @{ */ /** * The frame data may be corrupted, e.g. due to decoding errors. */ #define AV_FRAME_FLAG_CORRUPT (1 << 0) /** * A flag to mark the frames which need to be decoded, but shouldn't be output. */ #define AV_FRAME_FLAG_DISCARD (1 << 2) /** * @} */ /** * Frame flags, a combination of @ref lavu_frame_flags */ int flags; /** * MPEG vs JPEG YUV range. * - encoding: Set by user * - decoding: Set by libavcodec */ enum AVColorRange color_range; enum AVColorPrimaries color_primaries; enum AVColorTransferCharacteristic color_trc; /** * YUV colorspace type. * - encoding: Set by user * - decoding: Set by libavcodec */ enum AVColorSpace colorspace; enum AVChromaLocation chroma_location; /** * frame timestamp estimated using various heuristics, in stream time base * - encoding: unused * - decoding: set by libavcodec, read by user. */ int64_t best_effort_timestamp; /** * reordered pos from the last AVPacket that has been input into the decoder * - encoding: unused * - decoding: Read by user. */ int64_t pkt_pos; /** * duration of the corresponding packet, expressed in * AVStream->time_base units, 0 if unknown. * - encoding: unused * - decoding: Read by user. */ int64_t pkt_duration; /** * metadata. * - encoding: Set by user. * - decoding: Set by libavcodec. */ AVDictionary *metadata; /** * decode error flags of the frame, set to a combination of * FF_DECODE_ERROR_xxx flags if the decoder produced a frame, but there * were errors during the decoding. * - encoding: unused * - decoding: set by libavcodec, read by user. */ int decode_error_flags; #define FF_DECODE_ERROR_INVALID_BITSTREAM 1 #define FF_DECODE_ERROR_MISSING_REFERENCE 2 #define FF_DECODE_ERROR_CONCEALMENT_ACTIVE 4 #define FF_DECODE_ERROR_DECODE_SLICES 8 /** * number of audio channels, only used for audio. * - encoding: unused * - decoding: Read by user. */ int channels; /** * size of the corresponding packet containing the compressed * frame. * It is set to a negative value if unknown. * - encoding: unused * - decoding: set by libavcodec, read by user. */ int pkt_size; /** * For hwaccel-format frames, this should be a reference to the * AVHWFramesContext describing the frame. */ AVBufferRef *hw_frames_ctx; /** * AVBufferRef for free use by the API user. FFmpeg will never check the * contents of the buffer ref. FFmpeg calls av_buffer_unref() on it when * the frame is unreferenced. av_frame_copy_props() calls create a new * reference with av_buffer_ref() for the target frame's opaque_ref field. * * This is unrelated to the opaque field, although it serves a similar * purpose. */ AVBufferRef *opaque_ref; /** * @anchor cropping * @name Cropping * Video frames only. The number of pixels to discard from the the * top/bottom/left/right border of the frame to obtain the sub-rectangle of * the frame intended for presentation. * @{ */ size_t crop_top; size_t crop_bottom; size_t crop_left; size_t crop_right; /** * @} */ /** * AVBufferRef for internal use by a single libav* library. * Must not be used to transfer data between libraries. * Has to be NULL when ownership of the frame leaves the respective library. * * Code outside the FFmpeg libs should never check or change the contents of the buffer ref. * * FFmpeg calls av_buffer_unref() on it when the frame is unreferenced. * av_frame_copy_props() calls create a new reference with av_buffer_ref() * for the target frame's private_ref field. */ AVBufferRef *private_ref; } AVFrame;

AVFrame中核心成员，我们常用的就是下面几个

typedef struct AVFrame { ... uint8_t *data[AV_NUM_DATA_POINTERS] int linesize[AV_NUM_DATA_POINTERS]; uint8_t **extended_data;uint8_t **extended_data; int width, height; int format; int key_frame; int64_t pts; int64_t pkt_dts; AVRational time_base; AVBufferRef *buf[AV_NUM_DATA_POINTERS]; ... }

这个结构体主要保存解码后的YUV数据，特别注意的是AVBufferRef *buf[AV_NUM_DATA_POINTERS];这个成员，因为它用来进行分配内存的释放。 其实在很多地方用到了AVBufferRef ，原理就是其中内部使用了一个引用，当引用为1的时候，就把内部指向的memory释放掉。

/** * A reference counted buffer type. It is opaque and is meant to be used through * references (AVBufferRef). */ typedef struct AVBuffer AVBuffer; /** * A reference to a data buffer. * * The size of this struct is not a part of the public ABI and it is not meant * to be allocated directly. */ typedef struct AVBufferRef { AVBuffer *buffer; /** * The data buffer. It is considered writable if and only if * this is the only reference to the buffer, in which case * av_buffer_is_writable() returns 1. */ uint8_t *data; /** * Size of data in bytes. */ size_t size; } AVBufferRef;

其中data就是yuv数据，size就是数据大小，我们看到里面的AVBuffer ，其详细结构如下，其中refcount就是引用数量，free就是释放函数

struct AVBuffer { uint8_t *data; /**< data described by this buffer */ size_t size; /**< size of data in bytes */ /** * number of existing AVBufferRef instances referring to this buffer */ atomic_uint refcount; /** * a callback for freeing the data */ void (*free)(void *opaque, uint8_t *data); /** * an opaque pointer, to be used by the freeing callback */ void *opaque; /** * A combination of AV_BUFFER_FLAG_* */ int flags; /** * A combination of BUFFER_FLAG_* */ int flags_internal; };

关于这一块智能引用，其提供了一些函数族

/** * Allocate an AVBuffer of the given size using av_malloc(). * * @return an AVBufferRef of given size or NULL when out of memory */ AVBufferRef *av_buffer_alloc(size_t size); /** * Same as av_buffer_alloc(), except the returned buffer will be initialized * to zero. */ AVBufferRef *av_buffer_allocz(size_t size); /** * Create an AVBuffer from an existing array. * * If this function is successful, data is owned by the AVBuffer. The caller may * only access data through the returned AVBufferRef and references derived from * it. * If this function fails, data is left untouched. * @param data data array * @param size size of data in bytes * @param free a callback for freeing this buffer's data * @param opaque parameter to be got for processing or passed to free * @param flags a combination of AV_BUFFER_FLAG_* * * @return an AVBufferRef referring to data on success, NULL on failure. */ AVBufferRef *av_buffer_create(uint8_t *data, size_t size, void (*free)(void *opaque, uint8_t *data), void *opaque, int flags); /** * Default free callback, which calls av_free() on the buffer data. * This function is meant to be passed to av_buffer_create(), not called * directly. */ void av_buffer_default_free(void *opaque, uint8_t *data); /** * Create a new reference to an AVBuffer. * * @return a new AVBufferRef referring to the same AVBuffer as buf or NULL on * failure. */ AVBufferRef *av_buffer_ref(const AVBufferRef *buf); /** * Free a given reference and automatically free the buffer if there are no more * references to it. * * @param buf the reference to be freed. The pointer is set to NULL on return. */ void av_buffer_unref(AVBufferRef **buf); /** * @return 1 if the caller may write to the data referred to by buf (which is * true if and only if buf is the only reference to the underlying AVBuffer). * Return 0 otherwise. * A positive answer is valid until av_buffer_ref() is called on buf. */ int av_buffer_is_writable(const AVBufferRef *buf); /** * @return the opaque parameter set by av_buffer_create. */ void *av_buffer_get_opaque(const AVBufferRef *buf); int av_buffer_get_ref_count(const AVBufferRef *buf);

这些函数族用户是用不到的，一般在ffmpeg内部模块中使用到。比如AVFrame函数族中。

其余根据字面意思很好理解。其中linesize表示每个plane的步长。 比如YUV420的数，其数据长度大小为：

if (frame->pixel_format == TOPSCODEC_PIX_FMT_I420 || frame->pixel_format == TOPSCODEC_PIX_FMT_NV12 || frame->pixel_format == TOPSCODEC_PIX_FMT_NV21 || frame->pixel_format == TOPSCODEC_PIX_FMT_P010 || frame->pixel_format == TOPSCODEC_PIX_FMT_P010LE) { int Y = frame->linesize[0] * frame->height; int U = frame->linesize[1] * ((frame->height + 1) / 2); int V = frame->linesize[2] * ((frame->height + 1) / 2); }

也就是说yuv数据的宽度是多大。 其实这个并不用手工来计算，ffmpeg框架提供了专用的函数来处理：

int av_image_fill_plane_sizes(size_t sizes[4], enum AVPixelFormat pix_fmt, int height, const ptrdiff_t linesizes[4])

这个函数用来计算每个plane的长度是多少，通过传入参数我们很容易看到height.原理和我们上面一样。

下面这个函数比较特殊，可以计算linesizes，也就是每个通道的stride.

int av_image_fill_linesizes(int linesizes[4], enum AVPixelFormat pix_fmt, int width)

这个函数专门通过width和pix_fmt来计算linesizes[4]，这样就不用我们通过手动来去计算了。

还有另外一个函数用来计算每个planes数据长度

该类函数族ffmpeg提供了很多，具体在imgutils.h中

/** * Compute the size of an image line with format pix_fmt and width * width for the plane plane. * * @return the computed size in bytes */ int av_image_get_linesize(enum AVPixelFormat pix_fmt, int width, int plane); /** * Fill plane linesizes for an image with pixel format pix_fmt and * width width. * * @param linesizes array to be filled with the linesize for each plane * @return >= 0 in case of success, a negative error code otherwise */ int av_image_fill_linesizes(int linesizes[4], enum AVPixelFormat pix_fmt, int width); /** * Fill plane sizes for an image with pixel format pix_fmt and height height. * * @param size the array to be filled with the size of each image plane * @param linesizes the array containing the linesize for each * plane, should be filled by av_image_fill_linesizes() * @return >= 0 in case of success, a negative error code otherwise * * @note The linesize parameters have the type ptrdiff_t here, while they are * int for av_image_fill_linesizes(). */ int av_image_fill_plane_sizes(size_t size[4], enum AVPixelFormat pix_fmt, int height, const ptrdiff_t linesizes[4]); /** * Fill plane data pointers for an image with pixel format pix_fmt and * height height. * * @param data pointers array to be filled with the pointer for each image plane * @param ptr the pointer to a buffer which will contain the image * @param linesizes the array containing the linesize for each * plane, should be filled by av_image_fill_linesizes() * @return the size in bytes required for the image buffer, a negative * error code in case of failure */ int av_image_fill_pointers(uint8_t *data[4], enum AVPixelFormat pix_fmt, int height, uint8_t *ptr, const int linesizes[4]); /** * Allocate an image with size w and h and pixel format pix_fmt, and * fill pointers and linesizes accordingly. * The allocated image buffer has to be freed by using * av_freep(&pointers[0]). * * @param align the value to use for buffer size alignment * @return the size in bytes required for the image buffer, a negative * error code in case of failure */ int av_image_alloc(uint8_t *pointers[4], int linesizes[4], int w, int h, enum AVPixelFormat pix_fmt, int align); /** * Copy image plane from src to dst. * That is, copy "height" number of lines of "bytewidth" bytes each. * The first byte of each successive line is separated by *_linesize * bytes. * * bytewidth must be contained by both absolute values of dst_linesize * and src_linesize, otherwise the function behavior is undefined. * * @param dst_linesize linesize for the image plane in dst * @param src_linesize linesize for the image plane in src */ void av_image_copy_plane(uint8_t *dst, int dst_linesize, const uint8_t *src, int src_linesize, int bytewidth, int height); /** * Copy image in src_data to dst_data. * * @param dst_linesizes linesizes for the image in dst_data * @param src_linesizes linesizes for the image in src_data */ void av_image_copy(uint8_t *dst_data[4], int dst_linesizes[4], const uint8_t *src_data[4], const int src_linesizes[4], enum AVPixelFormat pix_fmt, int width, int height); /** * Setup the data pointers and linesizes based on the specified image * parameters and the provided array. * * The fields of the given image are filled in by using the src * address which points to the image data buffer. Depending on the * specified pixel format, one or multiple image data pointers and * line sizes will be set. If a planar format is specified, several * pointers will be set pointing to the different picture planes and * the line sizes of the different planes will be stored in the * lines_sizes array. Call with src == NULL to get the required * size for the src buffer. * * To allocate the buffer and fill in the dst_data and dst_linesize in * one call, use av_image_alloc(). * * @param dst_data data pointers to be filled in * @param dst_linesize linesizes for the image in dst_data to be filled in * @param src buffer which will contain or contains the actual image data, can be NULL * @param pix_fmt the pixel format of the image * @param width the width of the image in pixels * @param height the height of the image in pixels * @param align the value used in src for linesize alignment * @return the size in bytes required for src, a negative error code * in case of failure */ int av_image_fill_arrays(uint8_t *dst_data[4], int dst_linesize[4], const uint8_t *src, enum AVPixelFormat pix_fmt, int width, int height, int align); /** * Return the size in bytes of the amount of data required to store an * image with the given parameters. * * @param pix_fmt the pixel format of the image * @param width the width of the image in pixels * @param height the height of the image in pixels * @param align the assumed linesize alignment * @return the buffer size in bytes, a negative error code in case of failure */ int av_image_get_buffer_size(enum AVPixelFormat pix_fmt, int width, int height, int align); /** * Copy image data from an image into a buffer. * * av_image_get_buffer_size() can be used to compute the required size * for the buffer to fill. * * @param dst a buffer into which picture data will be copied * @param dst_size the size in bytes of dst * @param src_data pointers containing the source image data * @param src_linesize linesizes for the image in src_data * @param pix_fmt the pixel format of the source image * @param width the width of the source image in pixels * @param height the height of the source image in pixels * @param align the assumed linesize alignment for dst * @return the number of bytes written to dst, or a negative value * (error code) on error */ int av_image_copy_to_buffer(uint8_t *dst, int dst_size, const uint8_t * const src_data[4], const int src_linesize[4], enum AVPixelFormat pix_fmt, int width, int height, int align); /** * Check if the given dimension of an image is valid, meaning that all * bytes of the image can be addressed with a signed int. * * @param w the width of the picture * @param h the height of the picture * @param log_offset the offset to sum to the log level for logging with log_ctx * @param log_ctx the parent logging context, it may be NULL * @return >= 0 if valid, a negative error code otherwise */ int av_image_check_size(unsigned int w, unsigned int h, int log_offset, void *log_ctx); /** * Check if the given dimension of an image is valid, meaning that all * bytes of a plane of an image with the specified pix_fmt can be addressed * with a signed int. * * @param w the width of the picture * @param h the height of the picture * @param max_pixels the maximum number of pixels the user wants to accept * @param pix_fmt the pixel format, can be AV_PIX_FMT_NONE if unknown. * @param log_offset the offset to sum to the log level for logging with log_ctx * @param log_ctx the parent logging context, it may be NULL * @return >= 0 if valid, a negative error code otherwise */ int av_image_check_size2(unsigned int w, unsigned int h, int64_t max_pixels, enum AVPixelFormat pix_fmt, int log_offset, void *log_ctx); 2.成员函数 const char *av_get_colorspace_name(enum AVColorSpace val); AVFrame *av_frame_alloc(void); void av_frame_free(AVFrame **frame); int av_frame_ref(AVFrame *dst, const AVFrame *src); AVFrame *av_frame_clone(const AVFrame *src); void av_frame_move_ref(AVFrame *dst, AVFrame *src); int av_frame_get_buffer(AVFrame *frame, int align); int av_frame_is_writable(AVFrame *frame); int av_frame_make_writable(AVFrame *frame); int av_frame_copy(AVFrame *dst, const AVFrame *src); int av_frame_copy_props(AVFrame *dst, const AVFrame *src); AVFrameSideData *av_frame_new_side_data(AVFrame *frame, enum AVFrameSideDataType type, size_t size); AVFrameSideData *av_frame_new_side_data_from_buf(AVFrame *frame, enum AVFrameSideDataType type, AVBufferRef *buf); void av_frame_remove_side_data(AVFrame *frame, enum AVFrameSideDataType type); const char *av_frame_side_data_name(enum AVFrameSideDataType type);

上面函数核心就是操作AVFrame。

AVFrame Host内存的获取 av_frame_get_buffer

ffmpeg提供了一个比较特殊的接口

/** * Allocate new buffer(s) for audio or video data. * * The following fields must be set on frame before calling this function: * - format (pixel format for video, sample format for audio) * - width and height for video * - nb_samples and channel_layout for audio * * This function will fill AVFrame.data and AVFrame.buf arrays and, if * necessary, allocate and fill AVFrame.extended_data and AVFrame.extended_buf. * For planar formats, one buffer will be allocated for each plane. * * @warning: if frame already has been allocated, calling this function will * leak memory. In addition, undefined behavior can occur in certain * cases. * * @param frame frame in which to store the new buffers. * @param align Required buffer size alignment. If equal to 0, alignment will be * chosen automatically for the current CPU. It is highly * recommended to pass 0 here unless you know what you are doing. * * @return 0 on success, a negative AVERROR on error. */ int av_frame_get_buffer(AVFrame *frame, int align)

这个函数是用来从ffmpeg内存池中为frame中获取buf，从上面的注释可以看到，如果想要获取buf，必须要设置

/* * The following fields must be set on frame before calling this function: * - format (pixel format for video, sample format for audio) * - width and height for video * - nb_samples and channel_layout for audio * /

查看详细的函数代码：

int av_frame_get_buffer(AVFrame *frame, int align) { if (frame->format < 0) return AVERROR(EINVAL); if (frame->width > 0 && frame->height > 0) return get_video_buffer(frame, align); else if (frame->nb_samples > 0 && (frame->channel_layout || frame->channels > 0)) return get_audio_buffer(frame, align); return AVERROR(EINVAL); }

详细的调用流程如下：

AVFrame device内存获取av_hwframe_get_buffer（）

既然有host的内存获取，那么就有device内存的获取

/** * Allocate a new frame attached to the given AVHWFramesContext. * * @param hwframe_ctx a reference to an AVHWFramesContext * @param frame an empty (freshly allocated or unreffed) frame to be filled with * newly allocated buffers. * @param flags currently unused, should be set to zero * @return 0 on success, a negative AVERROR code on failure */ int av_hwframe_get_buffer(AVBufferRef *hwframe_ctx, AVFrame *frame, int flags);

其调用流程如下，注意分支代码，这里是继承了父类后，通过mmap去map到父类的内存上去。