Android基於RTMP視頻流的人臉識別（上篇）

時間 2020-07-06

原文原文鏈接

相信你們都對直播不會陌生，直播的技術也愈來愈成熟了，目前有這樣的一個技術，當彈幕飄到主播的臉上的時候，彈幕會自動消失，出了人臉範圍內，就繼續顯示出來。這個原理很是的簡單，其實就是人臉識別，將人臉識別範圍內的彈幕全都隱藏。提及來容易作起來難，本文將分如下幾點講述如何實現RTMP視頻流的人臉識別。html

方案選擇
ffmpeg的接入
ffmpeg的數據解析
OpenGL的數據繪製
人臉跟蹤以及人臉框的繪製

1、方案的選擇

筆者一開始想直接使用別人封裝好的播放器，輸入地址就能播放。接入後發現，確實接入和使用都很簡單，也可以顯示，可是有一個很致命的問題，就是沒有提供獲取裸數據的接口，於是沒辦法進行人臉識別，後面我就轉用了ffmpeg。固然若是隻是想在設備上播放RTMP流，bilibili的ijkplayer的框架是徹底沒有問題的，接入和使用都很簡單下面是他們的地址。java

https://github.com/Bilibili/i...

解析方案已經選擇完畢，接下來就是繪製和人臉識別，繪製我採用OpenGL。緣由是以前有本身封裝過一個自定義surfaceView，直接拿來用就能夠了。人臉識別引擎我選擇虹軟的引擎，緣由有二，一是使用起來比較簡單，虹軟的demo寫的不錯，不少東西能夠直接抄過來；二是免費，其餘公司的引擎我也用過，都是有試用期限，我不喜歡有期限的東西，並且虹軟的引擎效果也不錯，固然是個人首選。android

2、ffmpeg的接入

1.目錄結構

在src/main目錄下新建cpp以及jniLibs目錄，並將ffmpeg庫放入，以下圖所示。
git

2.CMakeLists

首先咱們在src/main/cpp目錄下新建兩個文件，CMakeLists.txt，rtmpplayer-lib。CMake用於庫間文件的管理與構建，rtmpplayer-lib是放咱們解析數據流的jni代碼用的。github

CMakeLists.txt多線程

# For more information about using CMake with Android Studio, read the
# documentation: https://d.android.com/studio/projects/add-native-code.html

# Sets the minimum version of CMake required to build the native library.

cmake_minimum_required(VERSION 3.4.1)

# Creates and names a library, sets it as either STATIC
# or SHARED, and provides the relative paths to its source code.
# You can define multiple libraries, and CMake builds them for you.
# Gradle automatically packages shared libraries with your APK.

add_library( # Sets the name of the library.
        rtmpplayer-lib

        # Sets the library as a shared library.
        SHARED

        # Provides a relative path to your source file(s).
        rtmpplayer-lib.cpp)
include_directories(ffmpeg)

# Searches for a specified prebuilt library and stores the path as a
# variable. Because CMake includes system libraries in the search path by
# default, you only need to specify the name of the public NDK library
# you want to add. CMake verifies that the library exists before
# completing its build.

find_library( # Sets the name of the path variable.
        log-lib

        # Specifies the name of the NDK library that
        # you want CMake to locate.
        log)

# Specifies libraries CMake should link to your target library. You
# can link multiple libraries, such as libraries you define in this
# build script, prebuilt third-party libraries, or system libraries.

target_link_libraries( # Specifies the target library.
        rtmpplayer-lib

        # Links the target library to the log library
        # included in the NDK.
        ${log-lib}
        android
        ${PROJECT_SOURCE_DIR}/../jniLibs/${CMAKE_ANDROID_ARCH_ABI}/libavcodec.so
        ${PROJECT_SOURCE_DIR}/../jniLibs/${CMAKE_ANDROID_ARCH_ABI}/libavdevice.so
        ${PROJECT_SOURCE_DIR}/../jniLibs/${CMAKE_ANDROID_ARCH_ABI}/libavfilter.so
        ${PROJECT_SOURCE_DIR}/../jniLibs/${CMAKE_ANDROID_ARCH_ABI}/libavformat.so
        ${PROJECT_SOURCE_DIR}/../jniLibs/${CMAKE_ANDROID_ARCH_ABI}/libavutil.so
        ${PROJECT_SOURCE_DIR}/../jniLibs/${CMAKE_ANDROID_ARCH_ABI}/libpostproc.so
        ${PROJECT_SOURCE_DIR}/../jniLibs/${CMAKE_ANDROID_ARCH_ABI}/libswresample.so
        ${PROJECT_SOURCE_DIR}/../jniLibs/${CMAKE_ANDROID_ARCH_ABI}/libswscale.so
        )

3.build.gradle

咱們須要在指定上面咱們的CMake文件的位置，以及指定構建的架構。架構

android{
      defaultConfig {
        ...
        ...
        externalNativeBuild {
            cmake {
                abiFilters "armeabi-v7a"
            }
        }
        ndk {
            abiFilters 'armeabi-v7a'    //只生成armv7的so
        }
    }
    externalNativeBuild {
        cmake {
        //path即爲上面CMakeLists的地址
            path "src/main/cpp/CMakeLists.txt"
            version "3.10.2"
        }
    }
}

4.完成構建

在上述的步驟都完成後，咱們就能夠構建了，點擊build下的refresh linked C++ prject，再點擊右側Gradle/other/externalNativeBuildDebug，等待構建完成後就能夠在build/intermediates/cmake下就能夠看到本身構建成功的so庫了，若是能看到libnative-lib.so那麼恭喜你，ffmpeg接入就算完成了。框架

3、ffmpeg的數據解析

1.JNI數據流解析

上面提到了native-lib.cpp，咱們要在這個文件內編寫解析RTMP數據流的Jni代碼。ide

#include <jni.h>
#include <string>
#include <android/log.h>
#include <fstream>

#define LOGE(FORMAT, ...) __android_log_print(ANDROID_LOG_ERROR, "player", FORMAT, ##__VA_ARGS__);
#define LOGI(FORMAT, ...) __android_log_print(ANDROID_LOG_INFO, "player", FORMAT, ##__VA_ARGS__);

extern "C" {
#include "libavformat/avformat.h"
#include "libavcodec/avcodec.h"
#include "libswscale/swscale.h"
#include "libavutil/imgutils.h"
#include "libavdevice/avdevice.h"
}


static AVPacket *pPacket;
static AVFrame *pAvFrame, *pFrameNv21;
static AVCodecContext *pCodecCtx;
struct SwsContext *pImgConvertCtx;
static AVFormatContext *pFormatCtx;
uint8_t *v_out_buffer;
jobject frameCallback = NULL;
bool stop;
extern "C"
JNIEXPORT jint JNICALL
Java_com_example_rtmpplaydemo_RtmpPlayer_nativePrepare(JNIEnv *env, jobject, jstring url) {
    // 初始化
#if LIBAVCODEC_VERSION_INT < AV_VERSION_INT(55, 28, 1)
#define av_frame_alloc  avcodec_alloc_frame
#endif
    if (frameCallback == NULL) {
        return -1;
    }
    //申請空間
    pAvFrame = av_frame_alloc();
    pFrameNv21 = av_frame_alloc();
    const char* temporary = env->GetStringUTFChars(url,NULL);
    char input_str[500] = {0};
    strcpy(input_str,temporary);
    env->ReleaseStringUTFChars(url,temporary);

    //註冊庫中全部可用的文件格式和編碼器
    avcodec_register_all();
    av_register_all();
    avformat_network_init();
    avdevice_register_all();

    pFormatCtx = avformat_alloc_context();
    int openInputCode = avformat_open_input(&pFormatCtx, input_str, NULL, NULL);
    LOGI("openInputCode = %d", openInputCode);
    if (openInputCode < 0)
        return -1;
    avformat_find_stream_info(pFormatCtx, NULL);

    int videoIndex = -1;
    //遍歷各個流，找到第一個視頻流,並記錄該流的編碼信息
    for (unsigned int i = 0; i < pFormatCtx->nb_streams; i++)
    {
        if (pFormatCtx->streams[i]->codec->codec_type == AVMEDIA_TYPE_VIDEO) {
            //這裏獲取到的videoindex的結果爲1.
            videoIndex = i;
            break;
        }
    }
    if (videoIndex == -1) {
        return -1;
    }
    pCodecCtx = pFormatCtx->streams[videoIndex]->codec;
    AVCodec *pCodec = avcodec_find_decoder(pCodecCtx->codec_id);
    avcodec_open2(pCodecCtx, pCodec, NULL);

    int width = pCodecCtx->width;
    int height = pCodecCtx->height;
    LOGI("width = %d , height = %d", width, height);
    int numBytes = av_image_get_buffer_size(AV_PIX_FMT_NV21, width, height, 1);
    v_out_buffer = (uint8_t *) av_malloc(numBytes * sizeof(uint8_t));
    av_image_fill_arrays(pFrameNv21->data, pFrameNv21->linesize, v_out_buffer, AV_PIX_FMT_NV21,
                         width,
                         height, 1);
    pImgConvertCtx = sws_getContext(
            pCodecCtx->width,             //原始寬度
            pCodecCtx->height,            //原始高度
            pCodecCtx->pix_fmt,           //原始格式
            pCodecCtx->width,             //目標寬度
            pCodecCtx->height,            //目標高度
            AV_PIX_FMT_NV21,               //目標格式
            SWS_FAST_BILINEAR,                    //選擇哪一種方式來進行尺寸的改變,關於這個參數,能夠參考:http://www.cnblogs.com/mmix2009/p/3585524.html
            NULL,
            NULL,
            NULL);
    pPacket = (AVPacket *) av_malloc(sizeof(AVPacket));
    //onPrepared 回調
    jclass clazz = env->GetObjectClass(frameCallback);
    jmethodID onPreparedId = env->GetMethodID(clazz, "onPrepared", "(II)V");
    env->CallVoidMethod(frameCallback, onPreparedId, width, height);
    env->DeleteLocalRef(clazz);
    return videoIndex;
}

extern "C"
JNIEXPORT void JNICALL
Java_com_example_rtmpplaydemo_RtmpPlayer_nativeStop(JNIEnv *env, jobject) {
    //中止播放
    stop = true;
    if (frameCallback == NULL) {
        return;
    }
    jclass clazz = env->GetObjectClass(frameCallback);
    jmethodID onPlayFinishedId = env->GetMethodID(clazz, "onPlayFinished", "()V");
    //發送onPlayFinished 回調
    env->CallVoidMethod(frameCallback, onPlayFinishedId);
    env->DeleteLocalRef(clazz);
    //釋放資源
    sws_freeContext(pImgConvertCtx);
    av_free(pPacket);
    av_free(pFrameNv21);
    avcodec_close(pCodecCtx);
    avformat_close_input(&pFormatCtx);
}

extern "C"
JNIEXPORT void JNICALL
Java_com_example_rtmpplaydemo_RtmpPlayer_nativeSetCallback(JNIEnv *env, jobject,
                                                           jobject callback) {
    //設置回調
    if (frameCallback != NULL) {
        env->DeleteGlobalRef(frameCallback);
        frameCallback = NULL;
    }
    frameCallback = (env)->NewGlobalRef(callback);
}

extern "C"
JNIEXPORT void JNICALL
Java_com_example_rtmpplaydemo_RtmpPlayer_nativeStart(JNIEnv *env, jobject) {
    //開始播放
    stop = false;
    if (frameCallback == NULL) {
        return;
    }
    // 讀取數據包
    int count = 0;
    while (!stop) {
        if (av_read_frame(pFormatCtx, pPacket) >= 0) {
            //解碼
            int gotPicCount = 0;
            int decode_video2_size = avcodec_decode_video2(pCodecCtx, pAvFrame, &gotPicCount,
                                                           pPacket);
            LOGI("decode_video2_size = %d , gotPicCount = %d", decode_video2_size, gotPicCount);
            LOGI("pAvFrame->linesize  %d  %d %d", pAvFrame->linesize[0], pAvFrame->linesize[1],
                 pCodecCtx->height);
            if (gotPicCount != 0) {
                count++;
                sws_scale(
                        pImgConvertCtx,
                        (const uint8_t *const *) pAvFrame->data,
                        pAvFrame->linesize,
                        0,
                        pCodecCtx->height,
                        pFrameNv21->data,
                        pFrameNv21->linesize);
                //獲取數據大小 寬高等數據
                int dataSize = pCodecCtx->height * (pAvFrame->linesize[0] + pAvFrame->linesize[1]);
                LOGI("pAvFrame->linesize  %d  %d %d %d", pAvFrame->linesize[0],
                     pAvFrame->linesize[1], pCodecCtx->height, dataSize);
                jbyteArray data = env->NewByteArray(dataSize);
                env->SetByteArrayRegion(data, 0, dataSize,
                                        reinterpret_cast<const jbyte *>(v_out_buffer));
                // onFrameAvailable 回調
                jclass clazz = env->GetObjectClass(frameCallback);
                jmethodID onFrameAvailableId = env->GetMethodID(clazz, "onFrameAvailable", "([B)V");
                env->CallVoidMethod(frameCallback, onFrameAvailableId, data);
                env->DeleteLocalRef(clazz);
                env->DeleteLocalRef(data);

            }
        }
        av_packet_unref(pPacket);
    }
}

2.Java層數據回調

在上面的jni操做完成後，咱們已經得到了解析完成的裸數據，接下來只要將裸數據傳到java層，咱們也就算是大功告成了，這裏咱們用回調來實現。post

//Rtmp回調
public interface PlayCallback {
    //數據準備回調
    void onPrepared(int width, int height);
    //數據回調
    void onFrameAvailable(byte[] data);
    //播放結束回調
    void onPlayFinished();
}

接着咱們只須要將這個回調傳入native，再經過jni將解析好的數據傳給java便可。

RtmpPlayer.getInstance().nativeSetCallback(new PlayCallback() {
    @Override
    public void onPrepared(int width, int height) {
        //start 循環調運會阻塞主線程 須要在子線程裏運行
        RtmpPlayer.getInstance().nativeStart();
    }

    @Override
    public void onFrameAvailable(byte[] data) {
       //得到裸數據，裸數據的格式爲NV21
        Log.i(TAG, "onFrameAvailable: " + Arrays.hashCode(data));
        surfaceView.refreshFrameNV21(data);
    }

    @Override
    public void onPlayFinished() {
         //播放結束的回調
    }
});
//數據準備
int code = RtmpPlayer.getInstance().prepare("rtmp://58.200.131.2:1935/livetv/hunantv");
if (code == -1) {
    //code爲-1則證實rtmp的prepare有問題
    Toast.makeText(MainActivity.this, "prepare Error", Toast.LENGTH_LONG).show();
}

onFrameAvailable獲得的data就是咱們須要的NV21格式的數據了，下圖是我播放湖南衛視獲得的數據回調，從hashCode上來看，每次的數據回調都不同，能夠認爲數據是實時刷新的。

3.Java層與JNI的交互

新建了RtmpPlayer單例類作爲Jni與java層交互的通道。

public class RtmpPlayer {

    private static volatile RtmpPlayer mInstance;
    private static final int PREPARE_ERROR = -1;

    private RtmpPlayer() {
    }

    //雙重鎖定防止多線程操做致使的建立多個實例
    public static RtmpPlayer getInstance() {
        if (mInstance == null) {
            synchronized (RtmpPlayer.class) {
                if (mInstance == null) {
                    mInstance = new RtmpPlayer();
                }
            }
        }
        return mInstance;
    }

    //數據準備操做
    public int prepare(String url) {
        if(nativePrepare(url) == PREPARE_ERROR){
            Log.i("rtmpPlayer", "PREPARE_ERROR ");
        }
        return nativePrepare(url);
    }

    //加載庫
    static {
        System.loadLibrary("rtmpplayer-lib");
    }

    //數據準備
    private native int nativePrepare(String url);
    //開始播放
    public native void nativeStart();
    //設置回調
    public native void nativeSetCallback(PlayCallback playCallback);
    //中止播放
    public native void nativeStop();
}

4、小結

至此爲止咱們已經得到NV21的裸數據，因爲時間有限，文章須要寫的內容比較多，所以須要分爲上下兩篇進行講述。在下篇內我會講述如何經過OpenGL將咱們得到的NV21數據繪製上去，以及如何經過NV21裸數據進行人臉識別，並繪製人臉框。這也是爲什麼咱們費盡心機想要獲得NV21裸數據的緣由。上篇的ffmpeg接入若有問題，能夠在下篇最後的附錄查看我上傳的demo，參考demo可能上手的更容易些。