本文主要將含有編碼的H.264,H.265視頻流文件解碼爲原始視頻數據,解碼後便可渲染到屏幕或用做其餘用途.git
正如咱們所知,編碼數據僅用於傳輸,沒法直接渲染到屏幕上,因此這裏利用蘋果原生框架VideoToolbox解析文件中的編碼的視頻流,並將壓縮視頻數據(h264/h265)解碼爲指定格式(yuv,RGB)的視頻原始數據,以渲染到屏幕上.github
注意: 本例主要爲解碼,須要藉助FFmpeg搭建模塊,視頻解析模塊,渲染模塊,這些模塊在下面閱讀前提皆有連接可直接訪問.緩存
整體思想即將FFmpeg parse到的數據裝到CMBlockBuffer
中,將extra data分離出的vps,sps,pps裝到CMVideoFormatDesc
中,將計算好的時間戳裝到CMTime
中,最後便可拼成完成的CMSampleBuffer
以用來提供給解碼器.bash
FFmpeg parse流程網絡
avformat_alloc_context
avformat_open_input
avformat_find_stream_info
formatContext->streams[i]->codecpar->codec_type == (isVideoStream ? AVMEDIA_TYPE_VIDEO : AVMEDIA_TYPE_AUDIO)
m_formatContext->streams[m_audioStreamIndex]
av_read_frame
av_bitstream_filter_filter
VideoToolbox decode流程session
CMVideoFormatDescriptionCreateFromH264ParameterSets
,CMVideoFormatDescriptionCreateFromHEVCParameterSets
裝載vps,sps,pps等NALU header信息.VTDecompressionSessionCreate
CMBlockBufferRef
裝載解碼前數據,再將其轉爲CMSampleBufferRef
以提供給解碼器.VTDecompressionSessionDecodeFrame
CVImageBufferRef
即爲解碼後的數據,可轉爲CMSampleBufferRef
傳出.初始化preview數據結構
解碼後的視頻數據將渲染到該預覽層架構
- (void)viewDidLoad {
[super viewDidLoad];
[self setupUI];
}
- (void)setupUI {
self.previewView = [[XDXPreviewView alloc] initWithFrame:self.view.frame];
[self.view addSubview:self.previewView];
[self.view bringSubviewToFront:self.startBtn];
}
複製代碼
- (void)startDecodeByVTSessionWithIsH265Data:(BOOL)isH265 {
NSString *path = [[NSBundle mainBundle] pathForResource:isH265 ? @"testh265" : @"testh264" ofType:@"MOV"];
XDXAVParseHandler *parseHandler = [[XDXAVParseHandler alloc] initWithPath:path];
XDXVideoDecoder *decoder = [[XDXVideoDecoder alloc] init];
decoder.delegate = self;
[parseHandler startParseWithCompletionHandler:^(BOOL isVideoFrame, BOOL isFinish, struct XDXParseVideoDataInfo *videoInfo, struct XDXParseAudioDataInfo *audioInfo) {
if (isFinish) {
[decoder stopDecoder];
return;
}
if (isVideoFrame) {
[decoder startDecodeVideoData:videoInfo];
}
}];
}
複製代碼
注意: 若是數據中含有B幀則須要作一個重排序才能渲染,本例提供兩個文件,一個不含B幀的h264類型文件,一個含B幀的h265類型文件.框架
- (void)getVideoDecodeDataCallback:(CMSampleBufferRef)sampleBuffer {
if (self.isH265File) {
// Note : the first frame not need to sort.
if (self.isDecodeFirstFrame) {
self.isDecodeFirstFrame = NO;
CVPixelBufferRef pix = CMSampleBufferGetImageBuffer(sampleBuffer);
[self.previewView displayPixelBuffer:pix];
}
XDXSortFrameHandler *sortHandler = [[XDXSortFrameHandler alloc] init];
sortHandler.delegate = self;
[sortHandler addDataToLinkList:sampleBuffer];
}else {
CVPixelBufferRef pix = CMSampleBufferGetImageBuffer(sampleBuffer);
[self.previewView displayPixelBuffer:pix];
}
}
- (void)getSortedVideoNode:(CMSampleBufferRef)sampleBuffer {
int64_t pts = (int64_t)(CMTimeGetSeconds(CMSampleBufferGetPresentationTimeStamp(sampleBuffer)) * 1000);
static int64_t lastpts = 0;
NSLog(@"Test marigin - %lld",pts - lastpts);
lastpts = pts;
[self.previewView displayPixelBuffer:CMSampleBufferGetImageBuffer(sampleBuffer)];
}
複製代碼
使用FFmpeg parse的數據裝在XDXParseVideoDataInfo
結構體中,結構體定義以下,parse模塊可在上文連接中學習,本節只將解碼模塊.異步
struct XDXParseVideoDataInfo {
uint8_t *data;
int dataSize;
uint8_t *extraData;
int extraDataSize;
Float64 pts;
Float64 time_base;
int videoRotate;
int fps;
CMSampleTimingInfo timingInfo;
XDXVideoEncodeFormat videoFormat;
};
複製代碼
經過緩存當前extra data能夠將當前獲取的extra data與上一次的進行對比,若是改變須要從新建立解碼器,若是沒有改變則解碼器可複用.(此代碼尤爲適用於網絡流中的視頻流,由於視頻流可能會改變)
uint8_t *extraData = videoInfo->extraData;
int size = videoInfo->extraDataSize;
BOOL isNeedUpdate = [self isNeedUpdateExtraDataWithNewExtraData:extraData
newSize:size
lastData:&_lastExtraData
lastSize:&_lastExtraDataSize];
......
- (BOOL)isNeedUpdateExtraDataWithNewExtraData:(uint8_t *)newData newSize:(int)newSize lastData:(uint8_t **)lastData lastSize:(int *)lastSize {
BOOL isNeedUpdate = NO;
if (*lastSize == 0) {
isNeedUpdate = YES;
}else {
if (*lastSize != newSize) {
isNeedUpdate = YES;
}else {
if (memcmp(newData, *lastData, newSize) != 0) {
isNeedUpdate = YES;
}
}
}
if (isNeedUpdate) {
[self destoryDecoder];
*lastData = (uint8_t *)malloc(newSize);
memcpy(*lastData, newData, newSize);
*lastSize = newSize;
}
return isNeedUpdate;
}
複製代碼
建立解碼器必需要有NALU Header中的一些關鍵信息,如vps,sps,pps,以用來組成一個CMVideoFormatDesc
描述視頻信息的數據結構,如上圖
注意: h264碼流須要sps,pps, h265碼流則須要vps,sps,pps
首先肯定start code的位置,經過比較前四個字節是否爲00 00 00 01
便可. 對於h264的數據,start code以後緊接着的是sps,pps, 對於h265的數據則是vps,sps,pps
經過sps索引與pps索引值能夠肯定sps長度,其餘相似,注意,碼流結構中均以4個字節的start code做爲分界符,因此須要減去對應長度.
對於h264類型數據將數據&
上0x1F
能夠肯定NALU header的類型,對於h265類型數據,將數據&
上0x4F
能夠肯定NALU header的類型,這源於h264,h265的碼流結構,若是不懂請參考文章最上方閱讀前提中碼流結構相關文章.
獲得對應類型的數據與大小後,將其賦給全局變量,便可供後面使用.
if (isNeedUpdate) {
log4cplus_error(kModuleName, "%s: update extra data",__func__);
[self getNALUInfoWithVideoFormat:videoInfo->videoFormat
extraData:extraData
extraDataSize:size
decoderInfo:&_decoderInfo];
}
......
- (void)getNALUInfoWithVideoFormat:(XDXVideoEncodeFormat)videoFormat extraData:(uint8_t *)extraData extraDataSize:(int)extraDataSize decoderInfo:(XDXDecoderInfo *)decoderInfo {
uint8_t *data = extraData;
int size = extraDataSize;
int startCodeVPSIndex = 0;
int startCodeSPSIndex = 0;
int startCodeFPPSIndex = 0;
int startCodeRPPSIndex = 0;
int nalu_type = 0;
for (int i = 0; i < size; i ++) {
if (i >= 3) {
if (data[i] == 0x01 && data[i - 1] == 0x00 && data[i - 2] == 0x00 && data[i - 3] == 0x00) {
if (videoFormat == XDXH264EncodeFormat) {
if (startCodeSPSIndex == 0) {
startCodeSPSIndex = i;
}
if (i > startCodeSPSIndex) {
startCodeFPPSIndex = i;
}
}else if (videoFormat == XDXH265EncodeFormat) {
if (startCodeVPSIndex == 0) {
startCodeVPSIndex = i;
continue;
}
if (i > startCodeVPSIndex && startCodeSPSIndex == 0) {
startCodeSPSIndex = i;
continue;
}
if (i > startCodeSPSIndex && startCodeFPPSIndex == 0) {
startCodeFPPSIndex = i;
continue;
}
if (i > startCodeFPPSIndex && startCodeRPPSIndex == 0) {
startCodeRPPSIndex = i;
}
}
}
}
}
int spsSize = startCodeFPPSIndex - startCodeSPSIndex - 4;
decoderInfo->sps_size = spsSize;
if (videoFormat == XDXH264EncodeFormat) {
int f_ppsSize = size - (startCodeFPPSIndex + 1);
decoderInfo->f_pps_size = f_ppsSize;
nalu_type = ((uint8_t)data[startCodeSPSIndex + 1] & 0x1F);
if (nalu_type == 0x07) {
uint8_t *sps = &data[startCodeSPSIndex + 1];
[self copyDataWithOriginDataRef:&decoderInfo->sps newData:sps size:spsSize];
}
nalu_type = ((uint8_t)data[startCodeFPPSIndex + 1] & 0x1F);
if (nalu_type == 0x08) {
uint8_t *pps = &data[startCodeFPPSIndex + 1];
[self copyDataWithOriginDataRef:&decoderInfo->f_pps newData:pps size:f_ppsSize];
}
} else {
int vpsSize = startCodeSPSIndex - startCodeVPSIndex - 4;
decoderInfo->vps_size = vpsSize;
int f_ppsSize = startCodeRPPSIndex - startCodeFPPSIndex - 4;
decoderInfo->f_pps_size = f_ppsSize;
nalu_type = ((uint8_t) data[startCodeVPSIndex + 1] & 0x4F);
if (nalu_type == 0x40) {
uint8_t *vps = &data[startCodeVPSIndex + 1];
[self copyDataWithOriginDataRef:&decoderInfo->vps newData:vps size:vpsSize];
}
nalu_type = ((uint8_t) data[startCodeSPSIndex + 1] & 0x4F);
if (nalu_type == 0x42) {
uint8_t *sps = &data[startCodeSPSIndex + 1];
[self copyDataWithOriginDataRef:&decoderInfo->sps newData:sps size:spsSize];
}
nalu_type = ((uint8_t) data[startCodeFPPSIndex + 1] & 0x4F);
if (nalu_type == 0x44) {
uint8_t *pps = &data[startCodeFPPSIndex + 1];
[self copyDataWithOriginDataRef:&decoderInfo->f_pps newData:pps size:f_ppsSize];
}
if (startCodeRPPSIndex == 0) {
return;
}
int r_ppsSize = size - (startCodeRPPSIndex + 1);
decoderInfo->r_pps_size = r_ppsSize;
nalu_type = ((uint8_t) data[startCodeRPPSIndex + 1] & 0x4F);
if (nalu_type == 0x44) {
uint8_t *pps = &data[startCodeRPPSIndex + 1];
[self copyDataWithOriginDataRef:&decoderInfo->r_pps newData:pps size:r_ppsSize];
}
}
}
- (void)copyDataWithOriginDataRef:(uint8_t **)originDataRef newData:(uint8_t *)newData size:(int)size {
if (*originDataRef) {
free(*originDataRef);
*originDataRef = NULL;
}
*originDataRef = (uint8_t *)malloc(size);
memcpy(*originDataRef, newData, size);
}
複製代碼
根據編碼數據類型肯定使用h264解碼器仍是h265解碼器,如上圖咱們可得知,咱們須要將數據拼成一個CMSampleBuffer類型以傳給解碼器解碼.
CMVideoFormatDescriptionRef
經過(vps)sps,pps信息組成CMVideoFormatDescriptionRef
. 這裏須要注意的是, h265編碼數據有的碼流數據中含有兩個pps, 因此在拼裝時須要判斷以肯定參數數量.
經過指定kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
將視頻數據類型設置爲yuv 420sp, 如需其餘格式可自行更改適配.
經過上面提供的全部信息,便可調用VTDecompressionSessionCreate
生成解碼器上下文對象.
// create decoder
if (!_decoderSession) {
_decoderSession = [self createDecoderWithVideoInfo:videoInfo
videoDescRef:&_decoderFormatDescription
videoFormat:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
lock:_decoder_lock
callback:VideoDecoderCallback
decoderInfo:_decoderInfo];
}
- (VTDecompressionSessionRef)createDecoderWithVideoInfo:(XDXParseVideoDataInfo *)videoInfo videoDescRef:(CMVideoFormatDescriptionRef *)videoDescRef videoFormat:(OSType)videoFormat lock:(pthread_mutex_t)lock callback:(VTDecompressionOutputCallback)callback decoderInfo:(XDXDecoderInfo)decoderInfo {
pthread_mutex_lock(&lock);
OSStatus status;
if (videoInfo->videoFormat == XDXH264EncodeFormat) {
const uint8_t *const parameterSetPointers[2] = {decoderInfo.sps, decoderInfo.f_pps};
const size_t parameterSetSizes[2] = {static_cast<size_t>(decoderInfo.sps_size), static_cast<size_t>(decoderInfo.f_pps_size)};
status = CMVideoFormatDescriptionCreateFromH264ParameterSets(kCFAllocatorDefault,
2,
parameterSetPointers,
parameterSetSizes,
4,
videoDescRef);
}else if (videoInfo->videoFormat == XDXH265EncodeFormat) {
if (decoderInfo.r_pps_size == 0) {
const uint8_t *const parameterSetPointers[3] = {decoderInfo.vps, decoderInfo.sps, decoderInfo.f_pps};
const size_t parameterSetSizes[3] = {static_cast<size_t>(decoderInfo.vps_size), static_cast<size_t>(decoderInfo.sps_size), static_cast<size_t>(decoderInfo.f_pps_size)};
if (@available(iOS 11.0, *)) {
status = CMVideoFormatDescriptionCreateFromHEVCParameterSets(kCFAllocatorDefault,
3,
parameterSetPointers,
parameterSetSizes,
4,
NULL,
videoDescRef);
} else {
status = -1;
log4cplus_error(kModuleName, "%s: System version is too low!",__func__);
}
} else {
const uint8_t *const parameterSetPointers[4] = {decoderInfo.vps, decoderInfo.sps, decoderInfo.f_pps, decoderInfo.r_pps};
const size_t parameterSetSizes[4] = {static_cast<size_t>(decoderInfo.vps_size), static_cast<size_t>(decoderInfo.sps_size), static_cast<size_t>(decoderInfo.f_pps_size), static_cast<size_t>(decoderInfo.r_pps_size)};
if (@available(iOS 11.0, *)) {
status = CMVideoFormatDescriptionCreateFromHEVCParameterSets(kCFAllocatorDefault,
4,
parameterSetPointers,
parameterSetSizes,
4,
NULL,
videoDescRef);
} else {
status = -1;
log4cplus_error(kModuleName, "%s: System version is too low!",__func__);
}
}
}else {
status = -1;
}
if (status != noErr) {
log4cplus_error(kModuleName, "%s: NALU header error !",__func__);
pthread_mutex_unlock(&lock);
[self destoryDecoder];
return NULL;
}
uint32_t pixelFormatType = videoFormat;
const void *keys[] = {kCVPixelBufferPixelFormatTypeKey};
const void *values[] = {CFNumberCreate(NULL, kCFNumberSInt32Type, &pixelFormatType)};
CFDictionaryRef attrs = CFDictionaryCreate(NULL, keys, values, 1, NULL, NULL);
VTDecompressionOutputCallbackRecord callBackRecord;
callBackRecord.decompressionOutputCallback = callback;
callBackRecord.decompressionOutputRefCon = (__bridge void *)self;
VTDecompressionSessionRef session;
status = VTDecompressionSessionCreate(kCFAllocatorDefault,
*videoDescRef,
NULL,
attrs,
&callBackRecord,
&session);
CFRelease(attrs);
pthread_mutex_unlock(&lock);
if (status != noErr) {
log4cplus_error(kModuleName, "%s: Create decoder failed",__func__);
[self destoryDecoder];
return NULL;
}
return session;
}
複製代碼
XDXDecodeVideoInfo
結構體中,以便後續擴展使用.typedef struct {
CVPixelBufferRef outputPixelbuffer;
int rotate;
Float64 pts;
int fps;
int source_index;
} XDXDecodeVideoInfo;
複製代碼
CMBlockBufferRef
中.CMBlockBufferRef
生成CMSampleBufferRef
經過VTDecompressionSessionDecodeFrame
函數便可完成解碼一幀視頻數據.第三個參數能夠指定解碼採用同步或異步方式.
// start decode
[self startDecode:videoInfo
session:_decoderSession
lock:_decoder_lock];
......
- (void)startDecode:(XDXParseVideoDataInfo *)videoInfo session:(VTDecompressionSessionRef)session lock:(pthread_mutex_t)lock {
pthread_mutex_lock(&lock);
uint8_t *data = videoInfo->data;
int size = videoInfo->dataSize;
int rotate = videoInfo->videoRotate;
CMSampleTimingInfo timingInfo = videoInfo->timingInfo;
uint8_t *tempData = (uint8_t *)malloc(size);
memcpy(tempData, data, size);
XDXDecodeVideoInfo *sourceRef = (XDXDecodeVideoInfo *)malloc(sizeof(XDXParseVideoDataInfo));
sourceRef->outputPixelbuffer = NULL;
sourceRef->rotate = rotate;
sourceRef->pts = videoInfo->pts;
sourceRef->fps = videoInfo->fps;
CMBlockBufferRef blockBuffer;
OSStatus status = CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault,
(void *)tempData,
size,
kCFAllocatorNull,
NULL,
0,
size,
0,
&blockBuffer);
if (status == kCMBlockBufferNoErr) {
CMSampleBufferRef sampleBuffer = NULL;
const size_t sampleSizeArray[] = { static_cast<size_t>(size) };
status = CMSampleBufferCreateReady(kCFAllocatorDefault,
blockBuffer,
_decoderFormatDescription,
1,
1,
&timingInfo,
1,
sampleSizeArray,
&sampleBuffer);
if (status == kCMBlockBufferNoErr && sampleBuffer) {
VTDecodeFrameFlags flags = kVTDecodeFrame_EnableAsynchronousDecompression;
VTDecodeInfoFlags flagOut = 0;
OSStatus decodeStatus = VTDecompressionSessionDecodeFrame(session,
sampleBuffer,
flags,
sourceRef,
&flagOut);
if(decodeStatus == kVTInvalidSessionErr) {
pthread_mutex_unlock(&lock);
[self destoryDecoder];
if (blockBuffer)
CFRelease(blockBuffer);
free(tempData);
tempData = NULL;
CFRelease(sampleBuffer);
return;
}
CFRelease(sampleBuffer);
}
}
if (blockBuffer) {
CFRelease(blockBuffer);
}
free(tempData);
tempData = NULL;
pthread_mutex_unlock(&lock);
}
複製代碼
解碼後的數據可在回調函數中獲取.這裏須要將解碼後的數據CVImageBufferRef
轉爲CMSampleBufferRef
.而後經過代理傳出.
#pragma mark - Callback
static void VideoDecoderCallback(void *decompressionOutputRefCon, void *sourceFrameRefCon, OSStatus status, VTDecodeInfoFlags infoFlags, CVImageBufferRef pixelBuffer, CMTime presentationTimeStamp, CMTime presentationDuration) {
XDXDecodeVideoInfo *sourceRef = (XDXDecodeVideoInfo *)sourceFrameRefCon;
if (pixelBuffer == NULL) {
log4cplus_error(kModuleName, "%s: pixelbuffer is NULL status = %d",__func__,status);
if (sourceRef) {
free(sourceRef);
}
return;
}
XDXVideoDecoder *decoder = (__bridge XDXVideoDecoder *)decompressionOutputRefCon;
CMSampleTimingInfo sampleTime = {
.presentationTimeStamp = presentationTimeStamp,
.decodeTimeStamp = presentationTimeStamp
};
CMSampleBufferRef samplebuffer = [decoder createSampleBufferFromPixelbuffer:pixelBuffer
videoRotate:sourceRef->rotate
timingInfo:sampleTime];
if (samplebuffer) {
if ([decoder.delegate respondsToSelector:@selector(getVideoDecodeDataCallback:)]) {
[decoder.delegate getVideoDecodeDataCallback:samplebuffer];
}
CFRelease(samplebuffer);
}
if (sourceRef) {
free(sourceRef);
}
}
- (CMSampleBufferRef)createSampleBufferFromPixelbuffer:(CVImageBufferRef)pixelBuffer videoRotate:(int)videoRotate timingInfo:(CMSampleTimingInfo)timingInfo {
if (!pixelBuffer) {
return NULL;
}
CVPixelBufferRef final_pixelbuffer = pixelBuffer;
CMSampleBufferRef samplebuffer = NULL;
CMVideoFormatDescriptionRef videoInfo = NULL;
OSStatus status = CMVideoFormatDescriptionCreateForImageBuffer(kCFAllocatorDefault, final_pixelbuffer, &videoInfo);
status = CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault, final_pixelbuffer, true, NULL, NULL, videoInfo, &timingInfo, &samplebuffer);
if (videoInfo != NULL) {
CFRelease(videoInfo);
}
if (samplebuffer == NULL || status != noErr) {
return NULL;
}
return samplebuffer;
}
複製代碼
用完後記得銷燬,以便下次使用.
if (_decoderSession) {
VTDecompressionSessionWaitForAsynchronousFrames(_decoderSession);
VTDecompressionSessionInvalidate(_decoderSession);
CFRelease(_decoderSession);
_decoderSession = NULL;
}
if (_decoderFormatDescription) {
CFRelease(_decoderFormatDescription);
_decoderFormatDescription = NULL;
}
複製代碼
注意,若是視頻文件或視頻流中含有B幀,則渲染時須要對視頻幀作一個重排序,本文重點講解碼,排序將在後面文章中更新,代碼中以實現,如需瞭解請下載Demo.