iOS渲染-將視頻原始數據(RGB,YUV)渲染到屏幕上
需求
在作如美顏,濾鏡等功能時,咱們不能使用相機原生的AVCaptureVideoPreviewLayer,而是須要經過其餘方式將視頻原始幀數據如RGB,NV12等等渲染到iOS界面上.ios
實現原理
利用OpenGL完成高效的渲染功能.本例中僅提供簡單流程講解,具體每行代碼含義可在開源庫中查詢.git
注意:github
- 系統的
AVCaptureVideoPreviewLayer僅能直接渲染從相機採集到的數據,當作美顏等功能時沒法使用. - 本例僅實現了RGB與NV12兩種類型視頻數據的渲染,其餘可根據需求自行添加.
閱讀前提
- 音視頻基礎
- OpenGL基礎
代碼地址 : iOS視頻渲染
掘金地址 : iOS視頻渲染
簡書地址 : iOS視頻渲染
博客地址 : iOS視頻渲染
具體步驟
1. 建立EAGLContext上下文對象
- 建立OpenGL預覽層
CAEAGLLayer *eaglLayer = (CAEAGLLayer *)self.layer;
eaglLayer.opaque = YES;
eaglLayer.drawableProperties = @{kEAGLDrawablePropertyRetainedBacking : [NSNumber numberWithBool:NO],
kEAGLDrawablePropertyColorFormat : kEAGLColorFormatRGBA8};
複製代碼
- 建立OpenGL上下文對象
EAGLContext *context = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2];
[EAGLContext setCurrentContext:context];
複製代碼
- 設置上下文渲染緩衝區
- (void)setupBuffersWithContext:(EAGLContext *)context width:(int *)width height:(int *)height colorBufferHandle:(GLuint *)colorBufferHandle frameBufferHandle:(GLuint *)frameBufferHandle {
glDisable(GL_DEPTH_TEST);
glEnableVertexAttribArray(ATTRIB_VERTEX);
glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, GL_FALSE, 2 * sizeof(GLfloat), 0);
glEnableVertexAttribArray(ATTRIB_TEXCOORD);
glVertexAttribPointer(ATTRIB_TEXCOORD, 2, GL_FLOAT, GL_FALSE, 2 * sizeof(GLfloat), 0);
glGenFramebuffers(1, frameBufferHandle);
glBindFramebuffer(GL_FRAMEBUFFER, *frameBufferHandle);
glGenRenderbuffers(1, colorBufferHandle);
glBindRenderbuffer(GL_RENDERBUFFER, *colorBufferHandle);
[context renderbufferStorage:GL_RENDERBUFFER fromDrawable:(CAEAGLLayer *)self.layer];
glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_WIDTH , width);
glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_HEIGHT, height);
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, *colorBufferHandle);
}
複製代碼
- 加載着色器
着色器本Demo中只添加了NV12格式與RGB格式兩種原始視頻數據.緩存
- (void)loadShaderWithBufferType:(XDXPixelBufferType)type {
GLuint vertShader, fragShader;
NSURL *vertShaderURL, *fragShaderURL;
NSString *shaderName;
GLuint program;
program = glCreateProgram();
if (type == XDXPixelBufferTypeNV12) {
shaderName = @"XDXPreviewNV12Shader";
_nv12Program = program;
} else if (type == XDXPixelBufferTypeRGB) {
shaderName = @"XDXPreviewRGBShader";
_rgbProgram = program;
}
vertShaderURL = [[NSBundle mainBundle] URLForResource:shaderName withExtension:@"vsh"];
if (![self compileShader:&vertShader type:GL_VERTEX_SHADER URL:vertShaderURL]) {
log4cplus_error(kModuleName, "Failed to compile vertex shader");
return;
}
fragShaderURL = [[NSBundle mainBundle] URLForResource:shaderName withExtension:@"fsh"];
if (![self compileShader:&fragShader type:GL_FRAGMENT_SHADER URL:fragShaderURL]) {
log4cplus_error(kModuleName, "Failed to compile fragment shader");
return;
}
glAttachShader(program, vertShader);
glAttachShader(program, fragShader);
glBindAttribLocation(program, ATTRIB_VERTEX , "position");
glBindAttribLocation(program, ATTRIB_TEXCOORD, "inputTextureCoordinate");
if (![self linkProgram:program]) {
if (vertShader) {
glDeleteShader(vertShader);
vertShader = 0;
}
if (fragShader) {
glDeleteShader(fragShader);
fragShader = 0;
}
if (program) {
glDeleteProgram(program);
program = 0;
}
return;
}
if (type == XDXPixelBufferTypeNV12) {
uniforms[UNIFORM_Y] = glGetUniformLocation(program , "luminanceTexture");
uniforms[UNIFORM_UV] = glGetUniformLocation(program, "chrominanceTexture");
uniforms[UNIFORM_COLOR_CONVERSION_MATRIX] = glGetUniformLocation(program, "colorConversionMatrix");
} else if (type == XDXPixelBufferTypeRGB) {
_displayInputTextureUniform = glGetUniformLocation(program, "inputImageTexture");
}
if (vertShader) {
glDetachShader(program, vertShader);
glDeleteShader(vertShader);
}
if (fragShader) {
glDetachShader(program, fragShader);
glDeleteShader(fragShader);
}
}
- (BOOL)compileShader:(GLuint *)shader type:(GLenum)type URL:(NSURL *)URL {
NSError *error;
NSString *sourceString = [[NSString alloc] initWithContentsOfURL:URL
encoding:NSUTF8StringEncoding
error:&error];
if (sourceString == nil) {
log4cplus_error(kModuleName, "Failed to load vertex shader: %s", [error localizedDescription].UTF8String);
return NO;
}
GLint status;
const GLchar *source;
source = (GLchar *)[sourceString UTF8String];
*shader = glCreateShader(type);
glShaderSource(*shader, 1, &source, NULL);
glCompileShader(*shader);
glGetShaderiv(*shader, GL_COMPILE_STATUS, &status);
if (status == 0) {
glDeleteShader(*shader);
return NO;
}
return YES;
}
- (BOOL)linkProgram:(GLuint)prog {
GLint status;
glLinkProgram(prog);
glGetProgramiv(prog, GL_LINK_STATUS, &status);
if (status == 0) {
return NO;
}
return YES;
}
複製代碼
- 建立視頻紋理緩存區
if (!*videoTextureCache) {
CVReturn err = CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, context, NULL, videoTextureCache);
if (err != noErr)
log4cplus_error(kModuleName, "Error at CVOpenGLESTextureCacheCreate %d",err);
}
複製代碼
2. 將pixelBuffer渲染到屏幕
- 開始渲染前先清空緩存數據
- (void)cleanUpTextures {
if (_lumaTexture) {
CFRelease(_lumaTexture);
_lumaTexture = NULL;
}
if (_chromaTexture) {
CFRelease(_chromaTexture);
_chromaTexture = NULL;
}
if (_renderTexture) {
CFRelease(_renderTexture);
_renderTexture = NULL;
}
CVOpenGLESTextureCacheFlush(_videoTextureCache, 0);
}
複製代碼
- 根據pixelBuffer格式肯定視頻數據類型
XDXPixelBufferType bufferType;
if (CVPixelBufferGetPixelFormatType(pixelBuffer) == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange || CVPixelBufferGetPixelFormatType(pixelBuffer) == kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange) {
bufferType = XDXPixelBufferTypeNV12;
} else if (CVPixelBufferGetPixelFormatType(pixelBuffer) == kCVPixelFormatType_32BGRA) {
bufferType = XDXPixelBufferTypeRGB;
}else {
log4cplus_error(kModuleName, "Not support current format.");
return;
}
複製代碼
- 經過當前的pixelBuffer對象建立
CVOpenGLESTexture對象
CVOpenGLESTextureRef lumaTexture,chromaTexture,renderTexture;
if (bufferType == XDXPixelBufferTypeNV12) {
// Y
glActiveTexture(GL_TEXTURE0);
error = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
videoTextureCache,
pixelBuffer,
NULL,
GL_TEXTURE_2D,
GL_LUMINANCE,
frameWidth,
frameHeight,
GL_LUMINANCE,
GL_UNSIGNED_BYTE,
0,
&lumaTexture);
if (error) {
log4cplus_error(kModuleName, "Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", error);
}else {
_lumaTexture = lumaTexture;
}
glBindTexture(CVOpenGLESTextureGetTarget(lumaTexture), CVOpenGLESTextureGetName(lumaTexture));
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
// UV
glActiveTexture(GL_TEXTURE1);
error = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
videoTextureCache,
pixelBuffer,
NULL,
GL_TEXTURE_2D,
GL_LUMINANCE_ALPHA,
frameWidth / 2,
frameHeight / 2,
GL_LUMINANCE_ALPHA,
GL_UNSIGNED_BYTE,
1,
&chromaTexture);
if (error) {
log4cplus_error(kModuleName, "Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", error);
}else {
_chromaTexture = chromaTexture;
}
glBindTexture(CVOpenGLESTextureGetTarget(chromaTexture), CVOpenGLESTextureGetName(chromaTexture));
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
} else if (bufferType == XDXPixelBufferTypeRGB) {
// RGB
glActiveTexture(GL_TEXTURE0);
error = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
videoTextureCache,
pixelBuffer,
NULL,
GL_TEXTURE_2D,
GL_RGBA,
frameWidth,
frameHeight,
GL_BGRA,
GL_UNSIGNED_BYTE,
0,
&renderTexture);
if (error) {
log4cplus_error(kModuleName, "Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", error);
}else {
_renderTexture = renderTexture;
}
glBindTexture(CVOpenGLESTextureGetTarget(renderTexture), CVOpenGLESTextureGetName(renderTexture));
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
}
複製代碼
- 選擇OpenGL程序
glBindFramebuffer(GL_FRAMEBUFFER, frameBufferHandle);
glViewport(0, 0, backingWidth, backingHeight);
glClearColor(0.1f, 0.0f, 0.0f, 1.0f);
glClear(GL_COLOR_BUFFER_BIT);
if (bufferType == XDXPixelBufferTypeNV12) {
if (self.lastBufferType != bufferType) {
glUseProgram(nv12Program);
glUniform1i(uniforms[UNIFORM_Y], 0);
glUniform1i(uniforms[UNIFORM_UV], 1);
glUniformMatrix3fv(uniforms[UNIFORM_COLOR_CONVERSION_MATRIX], 1, GL_FALSE, preferredConversion);
}
} else if (bufferType == XDXPixelBufferTypeRGB) {
if (self.lastBufferType != bufferType) {
glUseProgram(rgbProgram);
glUniform1i(displayInputTextureUniform, 0);
}
}
複製代碼
- 計算非全屏尺寸
渲染的畫面多是全屏,多是留黑邊.bash
static CGSize normalizedSamplingSize;
if (self.lastFullScreen != self.isFullScreen || self.pixelbufferWidth != frameWidth || self.pixelbufferHeight != frameHeight
|| normalizedSamplingSize.width == 0 || normalizedSamplingSize.height == 0 || self.screenWidth != [UIScreen mainScreen].bounds.size.width) {
normalizedSamplingSize = [self getNormalizedSamplingSize:CGSizeMake(frameWidth, frameHeight)];
self.lastFullScreen = self.isFullScreen;
self.pixelbufferWidth = frameWidth;
self.pixelbufferHeight = frameHeight;
self.screenWidth = [UIScreen mainScreen].bounds.size.width;
quadVertexData[0] = -1 * normalizedSamplingSize.width;
quadVertexData[1] = -1 * normalizedSamplingSize.height;
quadVertexData[2] = normalizedSamplingSize.width;
quadVertexData[3] = -1 * normalizedSamplingSize.height;
quadVertexData[4] = -1 * normalizedSamplingSize.width;
quadVertexData[5] = normalizedSamplingSize.height;
quadVertexData[6] = normalizedSamplingSize.width;
quadVertexData[7] = normalizedSamplingSize.height;
}
複製代碼
- 渲染畫面
glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, quadVertexData);
glEnableVertexAttribArray(ATTRIB_VERTEX);
glVertexAttribPointer(ATTRIB_TEXCOORD, 2, GL_FLOAT, 0, 0, quadTextureData);
glEnableVertexAttribArray(ATTRIB_TEXCOORD);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
glBindRenderbuffer(GL_RENDERBUFFER, colorBufferHandle);
if ([EAGLContext currentContext] == context) {
[context presentRenderbuffer:GL_RENDERBUFFER];
}
複製代碼
相關文章
- 1. 使用D3D渲染YUV視頻數據
- 2. js渲染yuv或rgb數據
- 3. OpenGLES渲染YUV數據
- 4. iOS下 WebRTC 視頻渲染
- 5. iOS離屏渲染
- 6. iOS 離屏渲染
- 7. opengl將畫面渲染到屏幕的指定區域和多屏渲染
- 8. 在iOS上使用OpenGL ES渲染YUV
- 9. 將 libVLC 視頻渲染到 QWidget 中
- 10. iOS中OpenGL是如何將幀緩衝(frameBuffer)渲染到屏幕上的? OpenGL iOS端渲染流程
- 更多相關文章...
- • 屏幕分辨率 統計 - 瀏覽器信息
- • 高屏幕分辨率 統計 - 瀏覽器信息
- • Flink 數據傳輸及反壓詳解
- • TiDB 在摩拜單車在線數據業務的應用和實踐
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
