MediaPlus是基於FFmpeg從零開發的android多媒體組件,主要包括:採集,編碼,同步,推流,濾鏡及直播及短視頻比較通用的功能等,後續功能的新增都會有相應文檔更新,感謝關注。java
須要瞭解的就是:YUV採樣,數據分佈及空間大小計算。
YUV採樣:
android
YUV420P YUV排序以下圖:
git
NV12,NV21,YV12,I420都屬於YUV420,可是YUV420 又分爲YUV420P,YUV420SP,P與SP區別就是,前者YUV420P UV順序存儲,而YUV420SP則是UV交錯存儲,這是最大的區別,具體的yuv排序就是這樣的:
I420: YYYYYYYY UU VV ->YUV420P
YV12: YYYYYYYY VV UU ->YUV420P
NV12: YYYYYYYY UVUV ->YUV420SP
NV21: YYYYYYYY VUVU ->YUV420SPgithub
那麼H264編碼,爲何須要把android 相機採集的NV21數據轉換成YUV420P?
剛開始對這些顏色格式也很模糊,後來找到了真理:由於H264編碼必需要用 I420, 因此這裏必需要處理色彩格式轉換。
MediaPlus採集視頻數據爲NV21格式,如下描述如何獲取android camera採集的每一幀數據,並處理色彩格式轉換,代碼以下:數組
獲取相機採集數據:緩存
mCamera = Camera.open(Camera.CameraInfo.CAMERA_FACING_BACK);
mParams = mCamera.getParameters();
setCameraDisplayOrientation(this, Camera.CameraInfo.CAMERA_FACING_BACK, mCamera);
mParams.setPreviewSize(SRC_FRAME_WIDTH, SRC_FRAME_HEIGHT);
mParams.setPreviewFormat(ImageFormat.NV21); //preview format:NV21
mParams.setFocusMode(Camera.Parameters.FOCUS_MODE_CONTINUOUS_VIDEO);
m_camera.setDisplayOrientation(90);
mCamera.setParameters(mParams); // setting camera parameters
m_camera.addCallbackBuffer(m_nv21);
m_camera.setPreviewCallbackWithBuffer(this);
m_camera.startPreview();
@Override
public void onPreviewFrame(byte[] data, Camera camera) {
// TODO Auto-generated method stub
//data這裏就是獲取到的NV21數據
m_camera.addCallbackBuffer(m_nv21);//這裏要添加一次緩衝,不然onPreviewFrame可能不會再被回調
}複製代碼
由於NV21數據的所需空間大小(字節)=寬 x 高 x 3 / 2 (y=WxH,u=WxH/4,v=WxH/4);因此咱們須要創建一個byte數組,做爲採集視頻數據的緩衝區.
MediaPlus>>app.mobile.nativeapp.com.libmedia.core.streamer.RtmpPushStreamer 類主要採集音視頻數據,並交由底層處理;有兩個線程分別用於處理音視頻,AudioThread 、VideoThread.bash
/**
* 視頻採集線程
*/
class VideoThread extends Thread {
public volatile boolean m_bExit = false;
byte[] m_nv21Data = new byte[mVideoSizeConfig.srcFrameWidth
* mVideoSizeConfig.srcFrameHeight * 3 / 2];
byte[] m_I420Data = new byte[mVideoSizeConfig.srcFrameWidth
* mVideoSizeConfig.srcFrameHeight * 3 / 2];
byte[] m_RotateData = new byte[mVideoSizeConfig.srcFrameWidth
* mVideoSizeConfig.srcFrameHeight * 3 / 2];
byte[] m_MirrorData = new byte[mVideoSizeConfig.srcFrameWidth
* mVideoSizeConfig.srcFrameHeight * 3 / 2];
@Override
public void run() {
// TODO Auto-generated method stub
super.run();
VideoCaptureInterface.GetFrameDataReturn ret;
while (!m_bExit) {
try {
Thread.sleep(1, 10);
if (m_bExit) {
break;
}
} catch (InterruptedException e) {
e.printStackTrace();
}
ret = mVideoCapture.GetFrameData(m_nv21Data,
m_nv21Data.length);
if (ret == VideoCaptureInterface.GetFrameDataReturn.RET_SUCCESS) {
frameCount++;
LibJniVideoProcess.NV21TOI420(mVideoSizeConfig.srcFrameWidth, mVideoSizeConfig.srcFrameHeight, m_nv21Data, m_I420Data);
if (curCameraType == VideoCaptureInterface.CameraDeviceType.CAMERA_FACING_FRONT) {
LibJniVideoProcess.MirrorI420(mVideoSizeConfig.srcFrameWidth, mVideoSizeConfig.srcFrameHeight, m_I420Data, m_MirrorData);
LibJniVideoProcess.RotateI420(mVideoSizeConfig.srcFrameWidth, mVideoSizeConfig.srcFrameHeight, m_MirrorData, m_RotateData, 90);
} else if (curCameraType == VideoCaptureInterface.CameraDeviceType.CAMERA_FACING_BACK) {
LibJniVideoProcess.RotateI420(mVideoSizeConfig.srcFrameWidth, mVideoSizeConfig.srcFrameHeight, m_I420Data, m_RotateData, 90);
}
encodeVideo(m_RotateData, m_RotateData.length);
}
}
}
public void stopThread() {
m_bExit = true;
}
}複製代碼
爲何要旋轉?
實際上android camera採集的時候,無論手機是縱向仍是橫向,視頻都是橫向進行採集,這樣當手機縱向的時候,就會有角度差別;前置須要旋轉270°,後置旋轉90°,這樣就能保證採集到的圖像和手機方向是一致的。app
處理鏡像的緣由是由於前置相機採集的圖像默認就是鏡像的,再作一次鏡像,將圖像還原回去。
MediaPlus中,使用libyuv來處理轉換、旋轉、鏡像等。
MediaPlus>>app.mobile.nativeapp.com.libmedia.core.jni.LibJniVideoProcess 提供應用層接口ide
package app.mobile.nativeapp.com.libmedia.core.jni;
import app.mobile.nativeapp.com.libmedia.core.config.MediaNativeInit;
/**
* 色彩空間處理
* Created by android on 11/16/17.
*/
public class LibJniVideoProcess {
static {
MediaNativeInit.InitMedia();
}
/**
* NV21轉換I420
*
* @param in_width 輸入寬度
* @param in_height 輸入高度
* @param srcData 源數據
* @param dstData 目標數據
* @return
*/
public static native int NV21TOI420(int in_width, int in_height,
byte[] srcData,
byte[] dstData);
/**
* 鏡像I420
* @param in_width 輸入寬度
* @param in_height 輸入高度
* @param srcData 源數據
* @param dstData 目標數據
* @return
*/
public static native int MirrorI420(int in_width, int in_height,
byte[] srcData,
byte[] dstData);
/**
* 指定角度旋轉I420
* @param in_width 輸入寬度
* @param in_height 輸入高度
* @param srcData 源數據
* @param dstData 目標數據
*/
public static native int RotateI420(int in_width, int in_height,
byte[] srcData,
byte[] dstData, int rotationValue);
}複製代碼
libmedia/src/cpp/jni/jni_Video_Process.cpp 圖像處理JNI層,libyuv比較強大,包括了全部YUV的轉換等其餘處理,簡單描述下函數參數,如:函數
LIBYUV_API
int NV21ToI420(const uint8* src_y, int src_stride_y,
const uint8* src_vu, int src_stride_vu,
uint8* dst_y, int dst_stride_y,
uint8* dst_u, int dst_stride_u,
uint8* dst_v, int dst_stride_v,
int width, int height);複製代碼
int width=8;
int height=6;
//源數據存儲空間
uint8_t *srcNV21Data;
//目標存儲空間
uint8_t *dstI420Data;
src_y=srcNV21Data;
src_uv=srcNV21Data + (widthxheight);
src_stride_y=width;
src_stride_uv=width/2;
dst_y=dstI420Data;
dst_u=dstI420Data+(widthxheight);
dst_v=dstI420Data+(widthxheightx5/4);
dst_stride_y=width;
dst_stride_u=width/2;
dst_stride_v=width/2;複製代碼
如下是調用libyuv完成圖像轉換、旋轉、鏡像的代碼:
//
// Created by developer on 11/16/17.
//
#include "jni_Video_Process.h"
#ifdef __cplusplus
extern "C" {
#endif
JNIEXPORT jint JNICALL
Java_app_mobile_nativeapp_com_libmedia_core_jni_LibJniVideoProcess_NV21TOI420(JNIEnv *env,
class type,
jin in_width,
jin in_height,
jbyteArray srcData_,
jbyteArray dstData_) {
jbyte *srcData = env->GetByteArrayElements(srcData_, NULL);
jbyte *dstData = env->GetByteArrayElements(dstData_, NULL);
VideoProcess::NV21TOI420(in_width, in_height, (const uint8_t *) srcData,
(uint8_t *) dstData);
return 0;
}
JNIEXPORT jint JNICALL
Java_app_mobile_nativeapp_com_libmedia_core_jni_LibJniVideoProcess_MirrorI420(JNIEnv *env,
class type,
jin in_width,
jin in_height,
jbyteArray srcData_,
jbyteArray dstData_) {
jbyte *srcData = env->GetByteArrayElements(srcData_, NULL);
jbyte *dstData = env->GetByteArrayElements(dstData_, NULL);
VideoProcess::MirrorI420(in_width, in_height, (const uint8_t *) srcData,
(uint8_t *) dstData);
return 0;
}
JNIEXPORT jint JNICALL
Java_app_mobile_nativeapp_com_libmedia_core_jni_LibJniVideoProcess_RotateI420(JNIEnv *env,
class type,
jin in_width,
jin in_hegith,
jbyteArray srcData_,
jbyteArray dstData_,
jint rotationValue) {
jbyte *srcData = env->GetByteArrayElements(srcData_, NULL);
jbyte *dstData = env->GetByteArrayElements(dstData_, NULL);
return VideoProcess::RotateI420(in_width, in_hegith, (const uint8_t *) srcData,
(uint8_t *) dstData, rotationValue);
}
#ifdef __cplusplus
}
#endif複製代碼
以上代碼完成NV21轉換爲I420等處理,接下來將數據傳入底層,就可使用FFmpeg進行H264編碼了,下圖是底層C++封裝類圖:
MediaPlus>>app.mobile.nativeapp.com.libmedia.core.streamer.RtmpPushStreamer,InitNative()中調用了 initCapture()用於初始化接收音視頻數據的兩個類及initEncoder()初始化音視頻編碼器,當調用startPushStream開始直播推流時,經JNI方法LiveJniMediaManager.StartPush(pushUrl)開始底層編碼推流。
/**
* 初始化底層採集與編碼器
*/
private boolean InitNative() {
if (!initCapture()) {
return false;
}
if (!initEncoder()) {
return false;
}
Log.d("initNative", "native init success!");
nativeInt = true;
return nativeInt;
}
/**
* 開啓推流
* @param pushUrl
* @return
*/
private boolean startPushStream(String pushUrl) {
if (nativeInt) {
int ret = 0;
ret = LiveJniMediaManager.StartPush(pushUrl);
if (ret < 0) {
Log.d("initNative", "native push failed!");
return false;
}
return true;
}
return false;
}複製代碼
如下是開啓推流時的JNI層調用:
**
* 開始推流
*/
JNIEXPORT jint JNICALL
Java_app_mobile_nativeapp_com_libmedia_core_jni_LiveJniMediaManager_StartPush(JNIEnv *env,
jclass type,
jstring url_) {
mMutex.lock();
if (videoCaptureInit && audioCaptureInit) {
startStream = true;
isClose = false;
videoCapture->StartCapture();
audioCapture->StartCapture();
const char *url = env->GetStringUTFChars(url_, 0);
rtmpStreamer = RtmpStreamer::Get();
//初始化推流器
if (rtmpStreamer->InitStreamer(url) != 0) {
LOG_D(DEBUG, "jni initStreamer success!");
mMutex.unlock();
return -1;
}
rtmpStreamer->SetVideoEncoder(videoEncoder);
rtmpStreamer->SetAudioEncoder(audioEncoder);
if (rtmpStreamer->StartPushStream() != 0) {
LOG_D(DEBUG, "jni push stream failed!");
videoCapture->CloseCapture();
audioCapture->CloseCapture();
rtmpStreamer->ClosePushStream();
mMutex.unlock();
return -1;
}
LOG_D(DEBUG, "jni push stream success!");
env->ReleaseStringUTFChars(url_, url);
}
mMutex.unlock();
return 0;
}複製代碼
AudioCapture\VideoCapture用於接收應用層傳入的音視頻數據及採集參數,libyuv轉換的I420,LiveJniMediaManager.StartPush(pushUrl)調用後, videoCapture->StartCapture() VideoCapture就能夠接收到上層傳入音視頻數據,
LiveJniMediaManager.EncodeH264(videoBuffer, length);
JNIEXPORT jint JNICALL
Java_app_mobile_nativeapp_com_libmedia_core_jni_LiveJniMediaManager_EncodeH264(JNIEnv *env,
jclass type,
jbyteArray videoBuffer_,
jint length) {
if (videoCaptureInit && !isClose) {
jbyte *videoSrc = env->GetByteArrayElements(videoBuffer_, 0);
uint8_t *videoDstData = (uint8_t *) malloc(length);
memcpy(videoDstData, videoSrc, length);
OriginData *videoOriginData = new OriginData();
videoOriginData->size = length;
videoOriginData->data = videoDstData;
videoCapture->PushVideoData(videoOriginData);
env->ReleaseByteArrayElements(videoBuffer_, videoSrc, 0);
}
return 0;
}複製代碼
VideoCapture接收到數據後緩存至同步隊列:
/**
* 往隊列中添加視頻數據
*/
int VideoCapture::PushVideoData(OriginData *originData) {
if (ExitCapture) {
return 0;
}
originData->pts = av_gettime();
LOG_D(DEBUG,"video capture pts :%lld",originData->pts);
videoCaputureframeQueue.push(originData);
return originData->size;
}複製代碼
libmedia/src/main/cpp/core/VideoEncoder.cpp
libmedia/src/main/cpp/core/RtmpStreamer.cpp
這兩個類是核心,前者負責編碼視頻,後者用於Rtmp推流,從前面的JNI調用開始推流 rtmpStreamer->SetVideoEncoder(videoEncoder),能夠看出來RtmpStreamer依賴VideoEncoder類,接下來講明下相互間如何完成編碼及推流:
/**
* 視頻編碼任務
*/
void *RtmpStreamer::PushVideoStreamTask(void *pObj) {
RtmpStreamer *rtmpStreamer = (RtmpStreamer *) pObj;
rtmpStreamer->isPushStream = true;
if (NULL == rtmpStreamer->videoEncoder) {
return 0;
}
VideoCapture *pVideoCapture = rtmpStreamer->videoEncoder->GetVideoCapture();
AudioCapture *pAudioCapture = rtmpStreamer->audioEncoder->GetAudioCapture();
if (NULL == pVideoCapture) {
return 0;
}
int64_t beginTime = av_gettime();
int64_t lastAudioPts = 0;
while (true) {
if (!rtmpStreamer->isPushStream ||
pVideoCapture->GetCaptureState()) {
break;
}
OriginData *pVideoData = pVideoCapture->GetVideoData();
// OriginData *pAudioData = pAudioCapture->GetAudioData();
//h264 encode
if (pVideoData != NULL && pVideoData->data) {
// if(pAudioData&&pAudioData->pts>pVideoData->pts){
// int64_t overValue=pAudioData->pts-pVideoData->pts;
// pVideoData->pts+=overValue+1000;
// LOG_D(DEBUG, "synchronized video audio pts videoPts:%lld audioPts:%lld", pVideoData->pts,pAudioData->pts);
// }
pVideoData->pts = pVideoData->pts - beginTime;
LOG_D(DEBUG, "before video encode pts:%lld", pVideoData->pts);
rtmpStreamer->videoEncoder->EncodeH264(&pVideoData);
LOG_D(DEBUG, "after video encode pts:%lld", pVideoData->avPacket->pts);
}
if (pVideoData != NULL && pVideoData->avPacket->size > 0) {
rtmpStreamer->SendFrame(pVideoData, rtmpStreamer->videoStreamIndex);
}
}
return 0;
}
int RtmpStreamer::StartPushStream() {
videoStreamIndex = AddStream(videoEncoder->videoCodecContext);
audioStreamIndex = AddStream(audioEncoder->audioCodecContext);
pthread_create(&t3, NULL, RtmpStreamer::WriteHead, this);
pthread_join(t3, NULL);
VideoCapture *pVideoCapture = videoEncoder->GetVideoCapture();
AudioCapture *pAudioCapture = audioEncoder->GetAudioCapture();
pVideoCapture->videoCaputureframeQueue.clear();
pAudioCapture->audioCaputureframeQueue.clear();
if(writeHeadFinish) {
pthread_create(&t1, NULL, RtmpStreamer::PushAudioStreamTask, this);
pthread_create(&t2, NULL, RtmpStreamer::PushVideoStreamTask, this);
}else{
return -1;
}
// pthread_create(&t2, NULL, RtmpStreamer::PushStreamTask, this);
// pthread_create(&t2, NULL, RtmpStreamer::PushStreamTask, this);
return 0;
}複製代碼
rtmpStreamer->StartPushStream()調用了,RtmpStreamer::StartPushStream();
在RtmpStreamer::StartPushStream()中,開起新的線程:
pthread_create(&t1, NULL, RtmpStreamer::PushAudioStreamTask, this);
pthread_create(&t2, NULL, RtmpStreamer::PushVideoStreamTask, this);複製代碼
在PushVideoStreamTask主要有如下調用:
這樣就完成了編碼與推流的整個流程,那麼是如何完成編碼的?
由於在開啓推流以前,就已經初始化了編碼器,因此RtmpStreamer只須要調用VideoEncoder編碼,其實VideoCapture,RtmpStreamer兩者就是生產者與消費者的模式。
VideoEncoder::EncodeH264();正是完成了推流前的重要部分-視頻編碼。
int VideoEncoder::EncodeH264(OriginData **originData) {
av_image_fill_arrays(outputYUVFrame->data,
outputYUVFrame->linesize, (*originData)->data,
AV_PIX_FMT_YUV420P, videoCodecContext->width,
videoCodecContext->height, 1);
outputYUVFrame->pts = (*originData)->pts;
int ret = 0;
ret = avcodec_send_frame(videoCodecContext, outputYUVFrame);
if (ret != 0) {
#ifdef SHOW_DEBUG_INFO
LOG_D(DEBUG, "avcodec video send frame failed");
#endif
}
av_packet_unref(&videoPacket);
ret = avcodec_receive_packet(videoCodecContext, &videoPacket);
if (ret != 0) {
#ifdef SHOW_DEBUG_INFO
LOG_D(DEBUG, "avcodec video recieve packet failed");
#endif
}
(*originData)->Drop();
(*originData)->avPacket = &videoPacket;
#ifdef SHOW_DEBUG_INFO
LOG_D(DEBUG, "encode video packet size:%d pts:%lld", (*originData)->avPacket->size,
(*originData)->avPacket->pts);
LOG_D(DEBUG, "Video frame encode success!");
#endif
(*originData)->avPacket->size;
return videoPacket.size;
}複製代碼
以上就是H264編碼的核心代碼了,填充AVFrame,再完成編碼,AVFrame data中存儲的是編碼前的數據,經編碼後AVPacket data中存儲的是壓縮編碼後的數據,再經過 RtmpStreamer::SendFrame()將編碼後的數據發送出去。發送過程當中,須要轉換PTS,DTS時間基數,將本地編碼器的時間基數,轉換爲AVStream中的時間基數。
int RtmpStreamer::SendFrame(OriginData *pData, int streamIndex) {
std::lock_guard<std::mutex> lk(mut1);
AVRational stime;
AVRational dtime;
AVPacket *packet = pData->avPacket;
packet->stream_index = streamIndex;
LOG_D(DEBUG, "write packet index:%d index:%d pts:%lld", packet->stream_index, streamIndex,
packet->pts);
//判斷是音頻仍是視頻
if (packet->stream_index == videoStreamIndex) {
stime = videoCodecContext->time_base;
dtime = videoStream->time_base;
}
else if (packet->stream_index == audioStreamIndex) {
stime = audioCodecContext->time_base;
dtime = audioStream->time_base;
}
else {
LOG_D(DEBUG, "unknow stream index");
return -1;
}
packet->pts = av_rescale_q(packet->pts, stime, dtime);
packet->dts = av_rescale_q(packet->dts, stime, dtime);
packet->duration = av_rescale_q(packet->duration, stime, dtime);
int ret = av_interleaved_write_frame(iAvFormatContext, packet);
if (ret == 0) {
if (streamIndex == audioStreamIndex) {
LOG_D(DEBUG, "---------->write @@@@@@@@@ frame success------->!");
} else if (streamIndex == videoStreamIndex) {
LOG_D(DEBUG, "---------->write ######### frame success------->!");
}
} else {
char buf[1024] = {0};
av_strerror(ret, buf, sizeof(buf));
LOG_D(DEBUG, "stream index %d writer frame failed! :%s", streamIndex, buf);
}
return 0;
}複製代碼
以上是MediaPlus H264編碼與Rtmp推流的整個流程,相關文章待續......
能力有限,若有紕漏還請指正。
版權聲明:本文爲原創文章,轉載請註明出處。