Android Camera原理之CameraDeviceCallbacks回調模塊

時間 2019-11-06

標籤 android camera 原理 cameradevicecallbacks 模塊欄目 Android 简体版

原文原文鏈接

在講解《Android Camera原理之openCamera模塊(二)》一文的時候提到了CameraDeviceCallbacks回調，當時沒有詳細展開，本文咱們詳細展開講解一下。
CameraDeviceCallbacks生成過程：
《Android Camera進程間通訊類總結》中2.ICameraDeviceCallbacks.aidl詳細總結了CameraDeviceCallbacks的生成過程。
frameworks/av/camera/ndk/impl/ACameraDevice.h中回調接口以下。這個CameraDeviceCallbacks是openCamera的時候設置到camera service端的，後續HAL層有camera響應的話會調用ACameraDevice.h中的ServiceCallback接口來實現回調。咱們須要從流程上搞清楚這些回調是在什麼場景下觸發的，明白了這些，才真正明白camera capture的流程。java

// Callbacks from camera service
    class ServiceCallback : public hardware::camera2::BnCameraDeviceCallbacks {
      public:
        explicit ServiceCallback(CameraDevice* device) : mDevice(device) {}
        binder::Status onDeviceError(int32_t errorCode,
                           const CaptureResultExtras& resultExtras) override;
        binder::Status onDeviceIdle() override;
        binder::Status onCaptureStarted(const CaptureResultExtras& resultExtras,
                              int64_t timestamp) override;
        binder::Status onResultReceived(const CameraMetadata& metadata,
                              const CaptureResultExtras& resultExtras,
                              const std::vector<PhysicalCaptureResultInfo>& physicalResultInfos) override;
        binder::Status onPrepared(int streamId) override;
        binder::Status onRequestQueueEmpty() override;
        binder::Status onRepeatingRequestError(int64_t lastFrameNumber,
                int32_t stoppedSequenceId) override;
      private:
        const wp<CameraDevice> mDevice;
    };

1.onDeviceError

onDeviceError回調流程.jpgsession

connectHelper
在openCamera執行的時候會檢查當前camera device是否正常。

binderDied
在camera service 死亡的時候，死亡回調中會通知上層當前的camera device可能有問題。

還有一個調用的地方是Camera3Device::notifyError--->從HAL傳遞上來的，關於當前device是否正常的信息，若是device存在問題執行回調--->listener->notifyError(errorCode, resultExtras);ide

void Camera3Device::notifyError(const camera3_error_msg_t &msg,
        sp<NotificationListener> listener) {
    ATRACE_CALL();
    // Map camera HAL error codes to ICameraDeviceCallback error codes
    // Index into this with the HAL error code
    static const int32_t halErrorMap[CAMERA3_MSG_NUM_ERRORS] = {
        // 0 = Unused error code
        hardware::camera2::ICameraDeviceCallbacks::ERROR_CAMERA_INVALID_ERROR,
        // 1 = CAMERA3_MSG_ERROR_DEVICE
        hardware::camera2::ICameraDeviceCallbacks::ERROR_CAMERA_DEVICE,
        // 2 = CAMERA3_MSG_ERROR_REQUEST
        hardware::camera2::ICameraDeviceCallbacks::ERROR_CAMERA_REQUEST,
        // 3 = CAMERA3_MSG_ERROR_RESULT
        hardware::camera2::ICameraDeviceCallbacks::ERROR_CAMERA_RESULT,
        // 4 = CAMERA3_MSG_ERROR_BUFFER
        hardware::camera2::ICameraDeviceCallbacks::ERROR_CAMERA_BUFFER
    };

    int32_t errorCode =
            ((msg.error_code >= 0) &&
                    (msg.error_code < CAMERA3_MSG_NUM_ERRORS)) ?
            halErrorMap[msg.error_code] :
            hardware::camera2::ICameraDeviceCallbacks::ERROR_CAMERA_INVALID_ERROR;

    int streamId = 0;
    if (msg.error_stream != NULL) {
        Camera3Stream *stream =
                Camera3Stream::cast(msg.error_stream);
        streamId = stream->getId();
    }
    ALOGV("Camera %s: %s: HAL error, frame %d, stream %d: %d",
            mId.string(), __FUNCTION__, msg.frame_number,
            streamId, msg.error_code);

    CaptureResultExtras resultExtras;
    switch (errorCode) {
        case hardware::camera2::ICameraDeviceCallbacks::ERROR_CAMERA_DEVICE:
            // SET_ERR calls notifyError
            SET_ERR("Camera HAL reported serious device error");
            break;
        case hardware::camera2::ICameraDeviceCallbacks::ERROR_CAMERA_REQUEST:
        case hardware::camera2::ICameraDeviceCallbacks::ERROR_CAMERA_RESULT:
        case hardware::camera2::ICameraDeviceCallbacks::ERROR_CAMERA_BUFFER:
            {
                Mutex::Autolock l(mInFlightLock);
                ssize_t idx = mInFlightMap.indexOfKey(msg.frame_number);
                if (idx >= 0) {
                    InFlightRequest &r = mInFlightMap.editValueAt(idx);
                    r.requestStatus = msg.error_code;
                    resultExtras = r.resultExtras;
                    if (hardware::camera2::ICameraDeviceCallbacks::ERROR_CAMERA_RESULT == errorCode
                            ||  hardware::camera2::ICameraDeviceCallbacks::ERROR_CAMERA_REQUEST ==
                            errorCode) {
                        r.skipResultMetadata = true;
                    }
                    if (hardware::camera2::ICameraDeviceCallbacks::ERROR_CAMERA_RESULT ==
                            errorCode) {
                        // In case of missing result check whether the buffers
                        // returned. If they returned, then remove inflight
                        // request.
                        removeInFlightRequestIfReadyLocked(idx);
                    }
                } else {
                    resultExtras.frameNumber = msg.frame_number;
                    ALOGE("Camera %s: %s: cannot find in-flight request on "
                            "frame %" PRId64 " error", mId.string(), __FUNCTION__,
                            resultExtras.frameNumber);
                }
            }
            resultExtras.errorStreamId = streamId;
            if (listener != NULL) {
                listener->notifyError(errorCode, resultExtras);
            } else {
                ALOGE("Camera %s: %s: no listener available", mId.string(), __FUNCTION__);
            }
            break;
        default:
            // SET_ERR calls notifyError
            SET_ERR("Unknown error message from HAL: %d", msg.error_code);
            break;
    }
}

2.onDeviceIdle

onDeviceIdle回調流程.jpg函數

openCamera執行以後開始調用到onDeviceIdle回調。這兒的調用過程須要講講Camera3Device::initializeCommonLockedoop

status_t Camera3Device::initializeCommonLocked() {
    /** Start up status tracker thread */
    mStatusTracker = new StatusTracker(this);
    status_t res = mStatusTracker->run(String8::format("C3Dev-%s-Status", mId.string()).string());
    if (res != OK) {
        SET_ERR_L("Unable to start status tracking thread: %s (%d)",
                strerror(-res), res);
        mInterface->close();
        mStatusTracker.clear();
        return res;
    }
//......
}

StatusTracker是C++中定義的線程，和java層有點相似，線程執行run會自動調用到threadLoop()，這應該很好理解。this

class StatusTracker: public Thread {
  public:
    explicit StatusTracker(wp<Camera3Device> parent);
    ~StatusTracker();
    virtual void requestExit();
  protected:
    virtual bool threadLoop();
}

接下來這個threadLoop()會一直執行，若是在執行過程當中發現當前的mStateTransitions[表示設備狀態]處於IDLE，這時候回調到上層。spa

// Notify parent for all intermediate transitions
    if (mStateTransitions.size() > 0 && parent.get()) {
        for (size_t i = 0; i < mStateTransitions.size(); i++) {
            bool idle = (mStateTransitions[i] == IDLE);
            ALOGV("Camera device is now %s", idle ? "idle" : "active");
            parent->notifyStatus(idle);
        }
    }

3.onCaptureStarted

這個回調函數表示camera device已經準備好，能夠開始調用camera獲取capture frame數據了。回調的開始是camera HAL層獲取底層camera device driver通知表示當前device已經準備好。通知的函數在Camera3Device::notify線程

Camera3Device::notify
底層msg通知當前camera 開門準備好了，能夠隨時準備拍照了。

void Camera3Device::notify(const camera3_notify_msg *msg) {
    ATRACE_CALL();
    sp<NotificationListener> listener;
    {
        Mutex::Autolock l(mOutputLock);
        listener = mListener.promote();
    }

    if (msg == NULL) {
        SET_ERR("HAL sent NULL notify message!");
        return;
    }

    switch (msg->type) {
        case CAMERA3_MSG_ERROR: {
            notifyError(msg->message.error, listener);
            break;
        }
        case CAMERA3_MSG_SHUTTER: {
            notifyShutter(msg->message.shutter, listener);
            break;
        }
        default:
            SET_ERR("Unknown notify message from HAL: %d",
                    msg->type);
    }
}

Camera3Device::notifyShutter
---> CameraDeviceClient::notifyShutter
此時回調到上層通知開發者當前的device shutter準備好了。

void CameraDeviceClient::notifyShutter(const CaptureResultExtras& resultExtras,
        nsecs_t timestamp) {
    // Thread safe. Don't bother locking.
    sp<hardware::camera2::ICameraDeviceCallbacks> remoteCb = getRemoteCallback();
    if (remoteCb != 0) {
        remoteCb->onCaptureStarted(resultExtras, timestamp);
    }
    Camera2ClientBase::notifyShutter(resultExtras, timestamp);
}

4.onResultReceived

預覽的時候這個函數很重要，表示抓取的幀數據不斷地返回，camera device正在不斷消耗capture frame。code

CameraDeviceClient::initializeImpl--->CameraDeviceClient初始化執行的時候會啓動FrameProcessorBase線程。
FrameProcessorBase繼承一個線程，執行run以後，它的threadLoop開始啓動。

template<typename TProviderPtr>
status_t CameraDeviceClient::initializeImpl(TProviderPtr providerPtr, const String8& monitorTags) {
//......
    String8 threadName;
    mFrameProcessor = new FrameProcessorBase(mDevice);
    threadName = String8::format("CDU-%s-FrameProc", mCameraIdStr.string());
    mFrameProcessor->run(threadName.string());

    mFrameProcessor->registerListener(FRAME_PROCESSOR_LISTENER_MIN_ID,
                                      FRAME_PROCESSOR_LISTENER_MAX_ID,
                                      /*listener*/this,
                                      /*sendPartials*/true);
//......
}

FrameProcessorBase::threadLoop
此時會不斷到Camera HAL層取請求，是否有新的frame數據，若是有的話，處理capture frame數據。

bool FrameProcessorBase::threadLoop() {
    status_t res;

    sp<CameraDeviceBase> device;
    {
        device = mDevice.promote();
        if (device == 0) return false;
    }

    res = device->waitForNextFrame(kWaitDuration);
    if (res == OK) {
        processNewFrames(device);
    } else if (res != TIMED_OUT) {
        ALOGE("FrameProcessorBase: Error waiting for new "
                "frames: %s (%d)", strerror(-res), res);
    }

    return true;
}

FrameProcessorBase::processNewFrames
---> FrameProcessorBase::processSingleFrame
--->FrameProcessorBase::processListeners
--->CameraDeviceClient::onResultAvailable
經過CameraDeviceClient::onResultAvailable函數回調到上層，而後在回調到CameraCaptureSession.CaptureCallback--->onCaptureProgressed告知開發者當前正在不斷捕獲capture frame數據。

/** Device-related methods */
void CameraDeviceClient::onResultAvailable(const CaptureResult& result) {
    ATRACE_CALL();
    ALOGV("%s", __FUNCTION__);

    // Thread-safe. No lock necessary.
    sp<hardware::camera2::ICameraDeviceCallbacks> remoteCb = mRemoteCallback;
    if (remoteCb != NULL) {
        remoteCb->onResultReceived(result.mMetadata, result.mResultExtras,
                result.mPhysicalMetadatas);
    }
}

5.onPrepared

CameraCaptureSession中有兩個接口：這個函數主要是用來給surface預分配內存，可是爲了加速預覽顯示的速度。orm

public abstract void prepare(@NonNull Surface surface) throws CameraAccessException;

    public abstract void prepare(int maxCount, @NonNull Surface surface)
            throws CameraAccessException;

而CameraDeviceCallbacks->onPrepared回調就是在執行CameraCaptureSession->prepare以後回調執行的。若是向HAL申請camera device 內存分配成功，纔會觸發這個回調。

6.onRequestQueueEmpty

onRequestQueueEmpty回調是當前camera device的非預覽流隊列爲空，開始準備capture 下一張圖片了，只是一箇中間狀態。這兒不過重要，我列個調用流程就好了。

Camera3Device::RequestThread::threadLoop

bool Camera3Device::RequestThread::threadLoop() {
//......
    // Wait for the next batch of requests.
    waitForNextRequestBatch();
//......
}

Camera3Device::RequestThread::waitForNextRequestBatch

void Camera3Device::RequestThread::waitForNextRequestBatch() {
//......
    NextRequest nextRequest;
    nextRequest.captureRequest = waitForNextRequestLocked();
    if (nextRequest.captureRequest == nullptr) {
        return;
    }

    nextRequest.halRequest = camera3_capture_request_t();
    nextRequest.submitted = false;
    mNextRequests.add(nextRequest);

    // Wait for additional requests
    const size_t batchSize = nextRequest.captureRequest->mBatchSize;

    for (size_t i = 1; i < batchSize; i++) {
        NextRequest additionalRequest;
        additionalRequest.captureRequest = waitForNextRequestLocked();
        if (additionalRequest.captureRequest == nullptr) {
            break;
        }

        additionalRequest.halRequest = camera3_capture_request_t();
        additionalRequest.submitted = false;
        mNextRequests.add(additionalRequest);
    }
//......
}

Camera3Device::RequestThread::waitForNextRequestLocked

sp<Camera3Device::CaptureRequest>
        Camera3Device::RequestThread::waitForNextRequestLocked() {
//......
    if (nextRequest == NULL) {
        // Don't have a repeating request already in hand, so queue
        // must have an entry now.
        RequestList::iterator firstRequest =
                mRequestQueue.begin();
        nextRequest = *firstRequest;
        mRequestQueue.erase(firstRequest);
        if (mRequestQueue.empty() && !nextRequest->mRepeating) {
            sp<NotificationListener> listener = mListener.promote();
            if (listener != NULL) {
                listener->notifyRequestQueueEmpty();
            }
        }
    }
//......
}

CameraDeviceClient::notifyRequestQueueEmpty這兒會直接回調到上層。

void CameraDeviceClient::notifyRequestQueueEmpty() {
    // Thread safe. Don't bother locking.
    sp<hardware::camera2::ICameraDeviceCallbacks> remoteCb = getRemoteCallback();
    if (remoteCb != 0) {
        remoteCb->onRequestQueueEmpty();
    }
}

7.onRepeatingRequestError

Camera3Device.h中定義了一個RequestThread線程，用來管理capture request和HAL device之間的鏈接。

/**
     * Thread for managing capture request submission to HAL device.
     */
    class RequestThread : public Thread {
//......
      protected:
        virtual bool threadLoop();
    }

RequestThread啓動的地方在Camera3Device::initializeCommonLocked中。

/** Start up request queue thread */
    mRequestThread = new RequestThread(this, mStatusTracker, mInterface, sessionParamKeys);
    res = mRequestThread->run(String8::format("C3Dev-%s-ReqQueue", mId.string()).string());
    if (res != OK) {
        SET_ERR_L("Unable to start request queue thread: %s (%d)",
                strerror(-res), res);
        mInterface->close();
        mRequestThread.clear();
        return res;
    }

線程核心的執行邏輯都在threadLoop函數中--->bool Camera3Device::RequestThread::threadLoop()

bool Camera3Device::RequestThread::threadLoop() {
//......
    // Prepare a batch of HAL requests and output buffers.
    res = prepareHalRequests();
    if (res == TIMED_OUT) {
        // Not a fatal error if getting output buffers time out.
        cleanUpFailedRequests(/*sendRequestError*/ true);
        // Check if any stream is abandoned.
        checkAndStopRepeatingRequest();
        return true;
    } else if (res != OK) {
        cleanUpFailedRequests(/*sendRequestError*/ false);
        return false;
    }
//......
}

請求HAL 層的device，若是請求超時，此時的預覽是沒法進行下去的。執行checkAndStopRepeatingRequest();

void Camera3Device::RequestThread::checkAndStopRepeatingRequest() {
//......
    if (listener != NULL && surfaceAbandoned) {
        listener->notifyRepeatingRequestError(lastFrameNumber);
    }
}

最終調用到CameraDeviceClient::notifyRepeatingRequestError，而後直接回調到上層，通知開發者當前的capture request請求失敗了。

void CameraDeviceClient::notifyRepeatingRequestError(long lastFrameNumber) {
    sp<hardware::camera2::ICameraDeviceCallbacks> remoteCb = getRemoteCallback();

    if (remoteCb != 0) {
        remoteCb->onRepeatingRequestError(lastFrameNumber, mStreamingRequestId);
    }

    Mutex::Autolock idLock(mStreamingRequestIdLock);
    mStreamingRequestId = REQUEST_ID_NONE;
}