【融雲分析】iOS 基於實時音視頻 SDK 實現屏幕共享功能

Replaykit 介紹

在以前的 iOS 版本中，iOS 開發者只能拿到編碼後的數據，拿不到原始的 PCM 和 YUV，到 iOS 10 以後，開發者能夠拿到原始數據，可是隻能錄製 App 內的內容，若是切到後臺，將中止錄製，直到 iOS 11，蘋果對屏幕共享進行了升級並開放了權限，既能夠拿到原始數據，又能夠錄製整個系統，如下咱們重點來講 iOS 11 以後的屏幕共享功能。

系統屏幕共享

- (void)initMode_1 {
    self.systemBroadcastPickerView = [[RPSystemBroadcastPickerView alloc] initWithFrame:CGRectMake(0, 64, ScreenWidth, 80)];
    self.systemBroadcastPickerView.preferredExtension = @"cn.rongcloud.replaytest.Recoder";
    self.systemBroadcastPickerView.backgroundColor = [UIColor colorWithRed:53.0/255.0 green:129.0/255.0 blue:242.0/255.0 alpha:1.0];
    self.systemBroadcastPickerView.showsMicrophoneButton = NO;
    [self.view addSubview:self.systemBroadcastPickerView];
}

在 iOS 11 建立一個 Extension 以後，調用上面的代碼就能夠開啓屏幕共享了，而後系統會爲咱們生成一個 SampleHandler 的類，在這個方法中，蘋果會根據 RPSampleBufferType 上報不一樣類型的數據。

- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType

那怎麼經過融雲的 RongRTCLib 將屏幕共享數據發送出去呢？

1. 基於 Socket 的逼格玩法

1.1. Replaykit 框架啓動和建立 Socket

//
//  ViewController.m
//  Socket_Replykit
//
//  Created by Sun on 2020/5/19.
//  Copyright © 2020 RongCloud. All rights reserved.
//

#import "ViewController.h"
#import <ReplayKit/ReplayKit.h>
#import "RongRTCServerSocket.h"

@interface ViewController ()<RongRTCServerSocketProtocol>

@property (nonatomic, strong) RPSystemBroadcastPickerView *systemBroadcastPickerView;
/**
 server socket
 */
@property(nonatomic , strong)RongRTCServerSocket *serverSocket;

@end

@implementation ViewController

- (void)viewDidLoad {
    [super viewDidLoad];
    self.view.backgroundColor = [UIColor whiteColor];
    // Do any additional setup after loading the view.
    [self.serverSocket createServerSocket];

    self.systemBroadcastPickerView = [[RPSystemBroadcastPickerView alloc] initWithFrame:CGRectMake(0, 64, [UIScreen mainScreen].bounds.size.width, 80)];
    self.systemBroadcastPickerView.preferredExtension = @"cn.rongcloud.sealrtc.RongRTCRP";
    self.systemBroadcastPickerView.backgroundColor = [UIColor colorWithRed:53.0/255.0 green:129.0/255.0 blue:242.0/255.0 alpha:1.0];
    self.systemBroadcastPickerView.showsMicrophoneButton = NO;
    [self.view addSubview:self.systemBroadcastPickerView];
}

- (RongRTCServerSocket *)serverSocket {
    if (!_serverSocket) {
        RongRTCServerSocket *socket = [[RongRTCServerSocket alloc] init];
        socket.delegate = self;

        _serverSocket = socket;
    }
    return _serverSocket;
}

- (void)didProcessSampleBuffer:(CMSampleBufferRef)sampleBuffer {
    // 這裏拿到了最終的數據，好比最後可使用融雲的音視頻SDK RTCLib 進行傳輸就能夠了
}

@end

其中，包括了建立 Server Socket 的步驟，咱們把主 App 當作 Server，而後屏幕共享 Extension 當作 Client ，經過 Socket 向咱們的主 APP 發送數據。

在 Extension 裏面，咱們拿到 ReplayKit 框架上報的屏幕視頻數據後：

//
//  SampleHandler.m
//  SocketReply
//
//  Created by Sun on 2020/5/19.
//  Copyright © 2020 RongCloud. All rights reserved.
//

#import "SampleHandler.h"
#import "RongRTCClientSocket.h"
@interface SampleHandler()

/**
 Client Socket
 */
@property (nonatomic, strong) RongRTCClientSocket *clientSocket;

@end

@implementation SampleHandler

- (void)broadcastStartedWithSetupInfo:(NSDictionary<NSString *,NSObject *> *)setupInfo {
    // User has requested to start the broadcast. Setup info from the UI extension can be supplied but optional.
    self.clientSocket = [[RongRTCClientSocket alloc] init];
    [self.clientSocket createCliectSocket];
}

- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType {
    switch (sampleBufferType) {
        case RPSampleBufferTypeVideo:
            // Handle video sample buffer
            [self sendData:sampleBuffer];
            break;
        case RPSampleBufferTypeAudioApp:
            // Handle audio sample buffer for app audio
            break;
        case RPSampleBufferTypeAudioMic:
            // Handle audio sample buffer for mic audio
            break;
        default:
            break;
    }
}

- (void)sendData:(CMSampleBufferRef)sampleBuffer {
    [self.clientSocket encodeBuffer:sampleBuffer];
}

@end

可見 ，這裏咱們建立了一個 Client Socket，而後拿到屏幕共享的視頻 sampleBuffer 以後，經過 Socket 發給咱們的主 App，這就是屏幕共享的流程。

1.2 Local Socket 的使用

//
//  RongRTCSocket.m
//  SealRTC
//
//  Created by Sun on 2020/5/7.
//  Copyright © 2020 RongCloud. All rights reserved.
//

#import "RongRTCSocket.h"
#import <arpa/inet.h>
#import <netdb.h>
#import <sys/types.h>
#import <sys/socket.h>
#import <ifaddrs.h>
#import "RongRTCThread.h"

@interface RongRTCSocket()

/**
 receive thread
 */
@property (nonatomic, strong) RongRTCThread *receiveThread;

@end

@implementation RongRTCSocket
- (int)createSocket {
    int socket = socket(AF_INET, SOCK_STREAM, 0);
    self.socket = socket;
    if (self.socket == -1) {
        close(self.socket);
        NSLog(@"socket error : %d", self.socket);
    }

    self.receiveThread = [[RongRTCThread alloc] init];
    [self.receiveThread run];
    return socket;
}

- (void)setSendBuffer {
    int optVal = 1024 * 1024 * 2;
    int optLen = sizeof(int);
    int res = setsockopt(self.socket, SOL_SOCKET, SO_SNDBUF, (char *)&optVal,optLen);
    NSLog(@"set send buffer:%d", res);
}

- (void)setReceiveBuffer {
    int optVal = 1024 * 1024 * 2;
    int optLen = sizeof(int);
    int res = setsockopt(self.socket, SOL_SOCKET, SO_RCVBUF, (char*)&optVal,optLen );
    NSLog(@"set send buffer:%d",res);
}

- (void)setSendingTimeout {
    struct timeval timeout = {10,0};
    int res = setsockopt(self.socket, SOL_SOCKET, SO_SNDTIMEO, (char *)&timeout, sizeof(int));
    NSLog(@"set send timeout:%d", res);
}

- (void)setReceiveTimeout {
    struct timeval timeout = {10, 0};
    int  res = setsockopt(self.socket, SOL_SOCKET, SO_RCVTIMEO, (char *)&timeout, sizeof(int));
    NSLog(@"set send timeout:%d", res);
}

- (BOOL)connect {
    NSString *serverHost = [self ip];
    struct hostent *server = gethostbyname([serverHost UTF8String]);
    if (server == NULL) {
        close(self.socket);
        NSLog(@"get host error");
        return NO;
    }

    struct in_addr *remoteAddr = (struct in_addr *)server->h_addr_list[0];
    struct sockaddr_in addr;
    addr.sin_family = AF_INET;
    addr.sin_addr = *remoteAddr;
    addr.sin_port = htons(CONNECTPORT);
    int res = connect(self.socket, (struct sockaddr *) &addr, sizeof(addr));
    if (res == -1) {
        close(self.socket);
        NSLog(@"connect error");
        return NO;
    }

    NSLog(@"socket connect to server success");
    return YES;
}

- (BOOL)bind {
    struct sockaddr_in client;
    client.sin_family = AF_INET;
    NSString *ipStr = [self ip];
    if (ipStr.length <= 0) {
        return NO;
    }

    const char *ip = [ipStr cStringUsingEncoding:NSASCIIStringEncoding];
    client.sin_addr.s_addr = inet_addr(ip);
    client.sin_port = htons(CONNECTPORT);
    int bd = bind(self.socket, (struct sockaddr *) &client, sizeof(client));
    if (bd == -1) {
        close(self.socket);
        NSLog(@"bind error: %d", bd);
        return NO;
    }
    return YES;
}

- (BOOL)listen {
    int ls = listen(self.socket, 128);
    if (ls == -1) {
        close(self.socket);
        NSLog(@"listen error: %d", ls);
        return NO;
    }
    return YES;
}

- (void)receive {
    dispatch_async(dispatch_get_global_queue(0, 0), ^{
        [self receiveData];
    });
}

- (NSString *)ip {
    NSString *ip = nil;
    struct ifaddrs *addrs = NULL;
    struct ifaddrs *tmpAddrs = NULL;
    BOOL res = getifaddrs(&addrs);
    if (res == 0) {
        tmpAddrs = addrs;
        while (tmpAddrs != NULL) {
            if (tmpAddrs->ifa_addr->sa_family == AF_INET) {
                // Check if interface is en0 which is the wifi connection on the iPhone
                NSLog(@"%@", [NSString stringWithUTF8String:inet_ntoa(((struct sockaddr_in *)tmpAddrs->ifa_addr)->sin_addr)]);
                if ([[NSString stringWithUTF8String:tmpAddrs->ifa_name] isEqualToString:@"en0"]) {
                    // Get NSString from C String
                    ip = [NSString stringWithUTF8String:inet_ntoa(((struct sockaddr_in *)tmpAddrs->ifa_addr)->sin_addr)];
                }
            }
            tmpAddrs = tmpAddrs->ifa_next;
        }
    }

    // Free memory
    freeifaddrs(addrs);
    NSLog(@"%@",ip);
    return ip;
}

- (void)close {
    int res = close(self.socket);
    NSLog(@"shut down: %d", res);
}

- (void)receiveData {
}

- (void)dealloc {
    [self.receiveThread stop];
}
@end

首先建立了一個 Socket 的父類，而後用 Server Socket 和 Client Socket 分別繼承類來實現連接、綁定等操做。能夠看到有些數據能夠設置，有些則不用，這裏不是核心，核心是怎樣收發數據。

1.3 發送屏幕共享數據

//
//  RongRTCClientSocket.m
//  SealRTC
//
//  Created by Sun on 2020/5/7.
//  Copyright © 2020 RongCloud. All rights reserved.
//

#import "RongRTCClientSocket.h"
#import <arpa/inet.h>
#import <netdb.h>
#import <sys/types.h>
#import <sys/socket.h>
#import <ifaddrs.h>
#import "RongRTCThread.h"
#import "RongRTCSocketHeader.h"
#import "RongRTCVideoEncoder.h"

@interface RongRTCClientSocket() <RongRTCCodecProtocol> {
    pthread_mutex_t lock;
}

/**
 video encoder
 */
@property (nonatomic, strong) RongRTCVideoEncoder *encoder;

/**
 encode queue
 */
@property (nonatomic, strong) dispatch_queue_t encodeQueue;

@end

@implementation RongRTCClientSocket

- (BOOL)createClientSocket {
    if ([self createSocket] == -1) {
        return NO;
    }

    BOOL isC = [self connect];
    [self setSendBuffer];
    [self setSendingTimeout];

    if (isC) {
        _encodeQueue = dispatch_queue_create("cn.rongcloud.encodequeue", NULL);
        [self createVideoEncoder];
        return YES;
    } else {
        return NO;
    }
}

- (void)createVideoEncoder {
    self.encoder = [[RongRTCVideoEncoder alloc] init];
    self.encoder.delegate = self;

    RongRTCVideoEncoderSettings *settings = [[RongRTCVideoEncoderSettings alloc] init];
    settings.width = 720;
    settings.height = 1280;
    settings.startBitrate = 300;
    settings.maxFramerate = 30;
    settings.minBitrate = 1000;
    [self.encoder configWithSettings:settings onQueue:_encodeQueue];
}

- (void)clientSend:(NSData *)data {
    //data length
    NSUInteger dataLength = data.length;

    // data header struct
    DataHeader dataH;
    memset((void *)&dataH, 0, sizeof(dataH));

    // pre
    PreHeader preH;
    memset((void *)&preH, 0, sizeof(preH));
    preH.pre[0] = '&';
    preH.dataLength = dataLength;

    dataH.preH = preH;

    // buffer
    int headerlength = sizeof(dataH);
    int totalLength = dataLength + headerlength;

    // srcbuffer
    Byte *src = (Byte *)[data bytes];

    // send buffer
    char *buffer = (char *)malloc(totalLength * sizeof(char));
    memcpy(buffer, &dataH, headerlength);
    memcpy(buffer + headerlength, src, dataLength);

    // tosend
    [self sendBytes:buffer length:totalLength];
    free(buffer);
}

- (void)encodeBuffer:(CMSampleBufferRef)sampleBuffer {
    [self.encoder encode:sampleBuffer];
}

- (void)sendBytes:(char *)bytes length:(int)length {
    LOCK(self->lock);
    int hasSendLength = 0;

    while (hasSendLength < length) {
        // connect socket success
        if (self.socket > 0) {
            // send
            int sendRes = send(self.socket, bytes, length - hasSendLength, 0);
            if (sendRes == -1 || sendRes == 0) {
                UNLOCK(self->lock);
                NSLog(@"send buffer error");
                [self close];
                break;
            }

            hasSendLength += sendRes;
            bytes += sendRes;
        } else {
            NSLog(@"client socket connect error");
            UNLOCK(self->lock);
        }
    }
    UNLOCK(self->lock); 
}

- (void)spsData:(NSData *)sps ppsData:(NSData *)pps {
    [self clientSend:sps];
    [self clientSend:pps];
}

- (void)naluData:(NSData *)naluData {
    [self clientSend:naluData];
}

- (void)deallo c{
    NSLog(@"dealoc cliect socket");
}

@end

這裏的核心思想是拿到屏幕共享的數據以後，先進行壓縮，當壓縮完成後會經過回調上報給當前類。既而經過 clientSend 方法，發給主 App。發給主 App 的數據中自定義了一個頭部，頭部添加了一個前綴和一個每次發送字節的長度，當接收端收到數據包後解析便可。

- (void)clientSend:(NSData *)data {
    //data length
    NSUInteger dataLength = data.length;

    // data header struct
    DataHeader dataH;
    memset((void *)&dataH, 0, sizeof(dataH));

    // pre
    PreHeader preH;
    memset((void *)&preH, 0, sizeof(preH));
    preH.pre[0] = '&';
    preH.dataLength = dataLength;

    dataH.preH = preH;

    // buffer
    int headerlength = sizeof(dataH);
    int totalLength = dataLength + headerlength;

    // srcbuffer
    Byte *src = (Byte *)[data bytes];

    // send buffer
    char *buffer = (char *)malloc(totalLength * sizeof(char));
    memcpy(buffer, &dataH, headerlength);
    memcpy(buffer + headerlength, src, dataLength);

    // to send
    [self sendBytes:buffer length:totalLength];
    free(buffer);
}

1.4 接收屏幕共享數據

//
//  RongRTCServerSocket.m
//  SealRTC
//
//  Created by Sun on 2020/5/7.
//  Copyright © 2020 RongCloud. All rights reserved.
//

#import "RongRTCServerSocket.h"
#import <arpa/inet.h>
#import <netdb.h>
#import <sys/types.h>
#import <sys/socket.h>
#import <ifaddrs.h>
#import <UIKit/UIKit.h>
#import "RongRTCThread.h"
#import "RongRTCSocketHeader.h"
#import "RongRTCVideoDecoder.h"

@interface RongRTCServerSocket() <RongRTCCodecProtocol>
{
    pthread_mutex_t lock;
    int _frameTime;
    CMTime _lastPresentationTime;
    Float64 _currentMediaTime;
    Float64 _currentVideoTime;
    dispatch_queue_t _frameQueue;
}

@property (nonatomic, assign) int acceptSocket;

/**
 data length
 */
@property (nonatomic, assign) NSUInteger dataLength;

/**
 timeData
 */
@property (nonatomic, strong) NSData *timeData;

/**
 decoder queue
 */
@property (nonatomic, strong) dispatch_queue_t decoderQueue;

/**
 decoder
 */
@property (nonatomic, strong) RongRTCVideoDecoder *decoder;

@end

@implementation RongRTCServerSocket

- (BOOL)createServerSocket {
    if ([self createSocket] == -1) {
        return NO;
    }

    [self setReceiveBuffer];
    [self setReceiveTimeout];
    BOOL isB = [self bind];
    BOOL isL = [self listen];

    if (isB && isL) {
        _decoderQueue = dispatch_queue_create("cn.rongcloud.decoderQueue", NULL);
        _frameTime = 0;
        [self createDecoder];
        [self receive];
        return YES;
    } else {
        return NO;
    }
}

- (void)createDecoder {
    self.decoder = [[RongRTCVideoDecoder alloc] init];
    self.decoder.delegate = self;
    RongRTCVideoEncoderSettings *settings = [[RongRTCVideoEncoderSettings alloc] init];
    settings.width = 720;
    settings.height = 1280;
    settings.startBitrate = 300;
    settings.maxFramerate = 30;
    settings.minBitrate = 1000;
    [self.decoder configWithSettings:settings onQueue:_decoderQueue];
}

- (void)receiveData {
    struct sockaddr_in rest;
    socklen_t rest_size = sizeof(struct sockaddr_in);
    self.acceptSocket = accept(self.socket, (struct sockaddr *) &rest, &rest_size);
    while (self.acceptSocket != -1) {
        DataHeader dataH;
        memset(&dataH, 0, sizeof(dataH));

        if (![self receiveData:(char *)&dataH length:sizeof(dataH)]) {
            continue;
        }

        PreHeader preH = dataH.preH;
        char pre = preH.pre[0];
        if (pre == '&') {
            // rongcloud socket
            NSUInteger dataLenght = preH.dataLength;
            char *buff = (char *)malloc(sizeof(char) * dataLenght);
            if ([self receiveData:(char *)buff length:dataLenght]) {
                NSData *data = [NSData dataWithBytes:buff length:dataLenght];
                [self.decoder decode:data];
                free(buff);
            }
        } else {
            NSLog(@"pre is not &");
            return;
        }
    }
}

- (BOOL)receiveData:(char *)data length:(NSUInteger)length {
    LOCK(lock);
    int receiveLength = 0;
    while (receiveLength < length) {
        ssize_t res = recv(self.acceptSocket, data, length - receiveLength, 0);
        if (res == -1 || res == 0) {
            UNLOCK(lock);
            NSLog(@"receive data error");
            break;
        }

        receiveLength += res;
        data += res;
    }

    UNLOCK(lock);
    return YES;
}

- (void)didGetDecodeBuffer:(CVPixelBufferRef)pixelBuffer {
    _frameTime += 1000;
    CMTime pts = CMTimeMake(_frameTime, 1000);
    CMSampleBufferRef sampleBuffer = [RongRTCBufferUtil sampleBufferFromPixbuffer:pixelBuffer time:pts];
    // Check to see if there is a problem with the decoded data. If the image appears, you are right.
    UIImage *image = [RongRTCBufferUtil imageFromBuffer:sampleBuffer];
    [self.delegate didProcessSampleBuffer:sampleBuffer];
    CFRelease(sampleBuffer);
}

- (void)close {
    int res = close(self.acceptSocket);
    self.acceptSocket = -1;
    NSLog(@"shut down server: %d", res);
    [super close];
}

- (void)dealloc {
    NSLog(@"dealoc server socket");
}

@end

主 App 經過 Socket 會持續收到數據包，再將數據包進行解碼，將解碼後的數據經過代理 didGetDecodeBuffer 代理方法回調給 App 層。App 層就能夠經過融雲 RongRTCLib 的發送自定義流方法將視頻數據發送到對端。

1.5 VideotoolBox 硬編碼

//
//  RongRTCVideoEncoder.m
//  SealRTC
//
//  Created by Sun on 2020/5/13.
//  Copyright © 2020 RongCloud. All rights reserved.
//

#import "RongRTCVideoEncoder.h"

#import "helpers.h"

@interface RongRTCVideoEncoder() {
    VTCompressionSessionRef _compressionSession;
    int _frameTime;
}

/**
 settings
 */
@property (nonatomic, strong) RongRTCVideoEncoderSettings *settings;

/**
 callback queue
 */
@property (nonatomic , strong ) dispatch_queue_t callbackQueue;

- (void)sendSpsAndPPSWithSampleBuffer:(CMSampleBufferRef)sampleBuffer;
- (void)sendNaluData:(CMSampleBufferRef)sampleBuffer;

@end

void compressionOutputCallback(void *encoder,
                               void *params,
                               OSStatus status,
                               VTEncodeInfoFlags infoFlags,
                               CMSampleBufferRef sampleBuffer) {
    RongRTCVideoEncoder *videoEncoder = (__bridge RongRTCVideoEncoder *)encoder;
    if (status != noErr) {
        return;
    }

    if (infoFlags & kVTEncodeInfo_FrameDropped) {
        return;
    }

    BOOL isKeyFrame = NO;
    CFArrayRef attachments = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, 0);

    if (attachments != nullptr && CFArrayGetCount(attachments)) {
        CFDictionaryRef attachment = static_cast<CFDictionaryRef>(CFArrayGetValueAtIndex(attachments, 0)) ;
        isKeyFrame = !CFDictionaryContainsKey(attachment, kCMSampleAttachmentKey_NotSync);
    }

    CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(sampleBuffer);
    CMBlockBufferRef contiguous_buffer = nullptr;

    if (!CMBlockBufferIsRangeContiguous(block_buffer, 0, 0)) {
        status = CMBlockBufferCreateContiguous(nullptr, block_buffer, nullptr, nullptr, 0, 0, 0, &contiguous_buffer);
        if (status != noErr) {
            return;
        }
    } else {
        contiguous_buffer = block_buffer;
        CFRetain(contiguous_buffer);
        block_buffer = nullptr;
    }

    size_t block_buffer_size = CMBlockBufferGetDataLength(contiguous_buffer);
    if (isKeyFrame) {
        [videoEncoder sendSpsAndPPSWithSampleBuffer:sampleBuffer];
    }

    if (contiguous_buffer) {
        CFRelease(contiguous_buffer);
    }

    [videoEncoder sendNaluData:sampleBuffer];
}

@implementation RongRTCVideoEncoder

@synthesize settings = _settings;
@synthesize callbackQueue = _callbackQueue;

- (BOOL)configWithSettings:(RongRTCVideoEncoderSettings *)settings onQueue:(nonnull dispatch_queue_t)queue {
    self.settings = settings;
    if (queue) {
        _callbackQueue = queue;
    } else {
        _callbackQueue = dispatch_get_main_queue();
    }

    if ([self resetCompressionSession:settings]) {
        _frameTime = 0;
        return YES;
    } else {
        return NO;
    }
}

- (BOOL)resetCompressionSession:(RongRTCVideoEncoderSettings *)settings {
    [self destroyCompressionSession];
    OSStatus status = VTCompressionSessionCreate(nullptr, settings.width, settings.height, kCMVideoCodecType_H264, nullptr, nullptr, nullptr, compressionOutputCallback, (__bridge void * _Nullable)(self), &_compressionSession);
    if (status != noErr) {
        return NO;
    }

    [self configureCompressionSession:settings];
    return YES;
}

- (void)configureCompressionSession:(RongRTCVideoEncoderSettings *)settings {
    if (_compressionSession) {
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_RealTime, true);
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_ProfileLevel, kVTProfileLevel_H264_Baseline_AutoLevel);
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_AllowFrameReordering, false);

        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_MaxKeyFrameInterval, 10);
        uint32_t targetBps = settings.startBitrate * 1000;
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_AverageBitRate, targetBps);
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_ExpectedFrameRate, settings.maxFramerate);
        int bitRate = settings.width * settings.height * 3 * 4 * 4;
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_AverageBitRate, bitRate);
        int bitRateLimit = settings.width * settings.height * 3 * 4;
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_DataRateLimits, bitRateLimit);
    }
}

- (void)encode:(CMSampleBufferRef)sampleBuffer {
    CVImageBufferRef imageBuffer = (CVImageBufferRef)CMSampleBufferGetImageBuffer(sampleBuffer);
    CMTime pts = CMTimeMake(self->_frameTime++, 1000);
    VTEncodeInfoFlags flags;
    OSStatus res = VTCompressionSessionEncodeFrame(self->_compressionSession,
                                                   imageBuffer,
                                                   pts,
                                                   kCMTimeInvalid,
                                                   NULL, NULL, &flags);

    if (res != noErr) {
        NSLog(@"encode frame error:%d", (int)res);
        VTCompressionSessionInvalidate(self->_compressionSession);
        CFRelease(self->_compressionSession);
        self->_compressionSession = NULL;
        return;
    }
}

- (void)sendSpsAndPPSWithSampleBuffer:(CMSampleBufferRef)sampleBuffer {
    CMFormatDescriptionRef format = CMSampleBufferGetFormatDescription(sampleBuffer);
    const uint8_t *sps ;
    const uint8_t *pps;
    size_t spsSize ,ppsSize , spsCount,ppsCount;
    OSStatus spsStatus = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 0, &sps, &spsSize, &spsCount, NULL);
    OSStatus ppsStatus = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 1, &pps, &ppsSize, &ppsCount, NULL);
    if (spsStatus == noErr && ppsStatus == noErr) {
        const char bytes[] = "\x00\x00\x00\x01";
        size_t length = (sizeof bytes) - 1;

        NSMutableData *spsData = [NSMutableData dataWithCapacity:4+ spsSize];
        NSMutableData *ppsData  = [NSMutableData dataWithCapacity:4 + ppsSize];
        [spsData appendBytes:bytes length:length];
        [spsData appendBytes:sps length:spsSize];

        [ppsData appendBytes:bytes length:length];
        [ppsData appendBytes:pps length:ppsSize];
        if (self && self.callbackQueue) {
            dispatch_async(self.callbackQueue, ^{
                if (self.delegate && [self.delegate respondsToSelector:@selector(spsData:ppsData:)]) {
                    [self.delegate spsData:spsData ppsData:ppsData];
                }
            });
        }
    } else {
        NSLog(@"sps status:%@, pps status:%@", @(spsStatus), @(ppsStatus));
    }
}

- (void)sendNaluData:(CMSampleBufferRef)sampleBuffer {
    size_t totalLength = 0;
    size_t lengthAtOffset=0;
    char *dataPointer;
    CMBlockBufferRef blockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
    OSStatus status1 = CMBlockBufferGetDataPointer(blockBuffer, 0, &lengthAtOffset, &totalLength, &dataPointer);

    if (status1 != noErr) {
        NSLog(@"video encoder error, status = %d", (int)status1);
        return;
    }

    static const int h264HeaderLength = 4;
    size_t bufferOffset = 0;

    while (bufferOffset < totalLength - h264HeaderLength) {
        uint32_t naluLength = 0;
        memcpy(&naluLength, dataPointer + bufferOffset, h264HeaderLength);
        naluLength = CFSwapInt32BigToHost(naluLength);

        const char bytes[] = "\x00\x00\x00\x01";
        NSMutableData *naluData = [NSMutableData dataWithCapacity:4 + naluLength];
        [naluData appendBytes:bytes length:4];
        [naluData appendBytes:dataPointer + bufferOffset + h264HeaderLength length:naluLength];

        dispatch_async(self.callbackQueue, ^{
            if (self.delegate && [self.delegate respondsToSelector:@selector(naluData:)]) {
                [self.delegate naluData:naluData];
            }
        });

        bufferOffset += naluLength + h264HeaderLength;
    }
}

- (void)destroyCompressionSession {
    if (_compressionSession) {
        VTCompressionSessionInvalidate(_compressionSession);
        CFRelease(_compressionSession);
        _compressionSession = nullptr;
    }
}

- (void)dealloc {
    if (_compressionSession) {
        VTCompressionSessionCompleteFrames(_compressionSession, kCMTimeInvalid);
        VTCompressionSessionInvalidate(_compressionSession);
        CFRelease(_compressionSession);
        _compressionSession = NULL;
    }
}
@end

1.6 VideotoolBox 解碼

//
//  RongRTCVideoDecoder.m
//  SealRTC
//
//  Created by Sun on 2020/5/14.
//  Copyright © 2020 RongCloud. All rights reserved.
//

#import "RongRTCVideoDecoder.h"
#import <UIKit/UIKit.h>
#import "helpers.h"

@interface RongRTCVideoDecoder() {
    uint8_t *_sps;
    NSUInteger _spsSize;
    uint8_t *_pps;
    NSUInteger _ppsSize;
    CMVideoFormatDescriptionRef _videoFormatDescription;
    VTDecompressionSessionRef _decompressionSession;
}

/**
 settings
 */
@property (nonatomic, strong) RongRTCVideoEncoderSettings *settings;

/**
 callback queue
 */
@property (nonatomic, strong) dispatch_queue_t callbackQueue;

@end

void DecoderOutputCallback(void * CM_NULLABLE decompressionOutputRefCon,
                           void * CM_NULLABLE sourceFrameRefCon,
                           OSStatus status,
                           VTDecodeInfoFlags infoFlags,
                           CM_NULLABLE CVImageBufferRef imageBuffer,
                           CMTime presentationTimeStamp,
                           CMTime presentationDuration ) {
    if (status != noErr) {
        NSLog(@" decoder callback error :%@", @(status));
        return;
    }

    CVPixelBufferRef *outputPixelBuffer = (CVPixelBufferRef *)sourceFrameRefCon;
    *outputPixelBuffer = CVPixelBufferRetain(imageBuffer);
    RongRTCVideoDecoder *decoder = (__bridge RongRTCVideoDecoder *)(decompressionOutputRefCon);
    dispatch_async(decoder.callbackQueue, ^{
        [decoder.delegate didGetDecodeBuffer:imageBuffer];
        CVPixelBufferRelease(imageBuffer);
    });
}

@implementation RongRTCVideoDecoder

@synthesize settings = _settings;
@synthesize callbackQueue = _callbackQueue;

- (BOOL)configWithSettings:(RongRTCVideoEncoderSettings *)settings onQueue:(dispatch_queue_t)queue {
    self.settings = settings;
    if (queue) {
        _callbackQueue = queue;
    } else {
        _callbackQueue = dispatch_get_main_queue();
    }
    return YES;
}

- (BOOL)createVT {
    if (_decompressionSession) {
        return YES;
    }

    const uint8_t * const parameterSetPointers[2] = {_sps, _pps};
    const size_t parameterSetSizes[2] = {_spsSize, _ppsSize};
    int naluHeaderLen = 4;
    OSStatus status = CMVideoFormatDescriptionCreateFromH264ParameterSets(kCFAllocatorDefault, 2, parameterSetPointers, parameterSetSizes, naluHeaderLen, &_videoFormatDescription );
    if (status != noErr) {
        NSLog(@"CMVideoFormatDescriptionCreateFromH264ParameterSets error:%@", @(status));
        return false;
    }

    NSDictionary *destinationImageBufferAttributes =
                                        @{
                                            (id)kCVPixelBufferPixelFormatTypeKey: [NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange],
                                            (id)kCVPixelBufferWidthKey: [NSNumber numberWithInteger:self.settings.width],
                                            (id)kCVPixelBufferHeightKey: [NSNumber numberWithInteger:self.settings.height],
                                            (id)kCVPixelBufferOpenGLCompatibilityKey: [NSNumber numberWithBool:true]
                                        };

    VTDecompressionOutputCallbackRecord CallBack;
    CallBack.decompressionOutputCallback = DecoderOutputCallback;
    CallBack.decompressionOutputRefCon = (__bridge void * _Nullable)(self);
    status = VTDecompressionSessionCreate(kCFAllocatorDefault, _videoFormatDescription, NULL, (__bridge CFDictionaryRef _Nullable)(destinationImageBufferAttributes), &CallBack, &_decompressionSession);

    if (status != noErr) {
        NSLog(@"VTDecompressionSessionCreate error:%@", @(status));
        return false;
    }

    status = VTSessionSetProperty(_decompressionSession, kVTDecompressionPropertyKey_RealTime,kCFBooleanTrue);
    return YES;
}

- (CVPixelBufferRef)decode:(uint8_t *)frame withSize:(uint32_t)frameSize {
    CVPixelBufferRef outputPixelBuffer = NULL;
    CMBlockBufferRef blockBuffer = NULL;
    CMBlockBufferFlags flag0 = 0;

    OSStatus status = CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault, frame, frameSize, kCFAllocatorNull, NULL, 0, frameSize, flag0, &blockBuffer);

    if (status != kCMBlockBufferNoErr) {
        NSLog(@"VCMBlockBufferCreateWithMemoryBlock code=%d", (int)status);
        CFRelease(blockBuffer);
        return outputPixelBuffer;
    }

    CMSampleBufferRef sampleBuffer = NULL;
    const size_t sampleSizeArray[] = {frameSize};

    status = CMSampleBufferCreateReady(kCFAllocatorDefault, blockBuffer, _videoFormatDescription, 1, 0, NULL, 1, sampleSizeArray, &sampleBuffer);

    if (status != noErr || !sampleBuffer) {
        NSLog(@"CMSampleBufferCreateReady failed status=%d", (int)status);
        CFRelease(blockBuffer);
        return outputPixelBuffer;
    }

    VTDecodeFrameFlags flag1 = kVTDecodeFrame_1xRealTimePlayback;
    VTDecodeInfoFlags infoFlag = kVTDecodeInfo_Asynchronous;

    status = VTDecompressionSessionDecodeFrame(_decompressionSession, sampleBuffer, flag1, &outputPixelBuffer, &infoFlag);

    if (status == kVTInvalidSessionErr) {
        NSLog(@"decode frame error with session err status =%d", (int)status);
        [self resetVT];
    } else  {
        if (status != noErr) {
            NSLog(@"decode frame error with  status =%d", (int)status);
        }
    }

    CFRelease(sampleBuffer);
    CFRelease(blockBuffer);

    return outputPixelBuffer;
}

- (void)resetVT {
    [self destorySession];
    [self createVT];
}

- (void)decode:(NSData *)data {
    uint8_t *frame = (uint8_t*)[data bytes];
    uint32_t length = data.length;
    uint32_t nalSize = (uint32_t)(length - 4);
    uint32_t *pNalSize = (uint32_t *)frame;
    *pNalSize = CFSwapInt32HostToBig(nalSize);

    int type = (frame[4] & 0x1F);
    CVPixelBufferRef pixelBuffer = NULL;
    switch (type) {
        case 0x05:
            if ([self createVT]) {
                pixelBuffer= [self decode:frame withSize:length];
            }
            break;
        case 0x07:
            self->_spsSize = length - 4;
            self->_sps = (uint8_t *)malloc(self->_spsSize);
            memcpy(self->_sps, &frame[4], self->_spsSize);
            break;
        case 0x08:
            self->_ppsSize = length - 4;
            self->_pps = (uint8_t *)malloc(self->_ppsSize);
            memcpy(self->_pps, &frame[4], self->_ppsSize);
            break;
        default:
            if ([self createVT]) {
                pixelBuffer = [self decode:frame withSize:length];
            }
            break;
    }
}

- (void)dealloc {
    [self destorySession];
}

- (void)destorySession {
    if (_decompressionSession) {
        VTDecompressionSessionInvalidate(_decompressionSession);
        CFRelease(_decompressionSession);
        _decompressionSession = NULL;
    }
}

@end

1.7 工具類

//
//  RongRTCBufferUtil.m
//  SealRTC
//
//  Created by Sun on 2020/5/8.
//  Copyright © 2020 RongCloud. All rights reserved.
//

#import "RongRTCBufferUtil.h"

// 下面的這些方法，必定要記得release，有的沒有在方法裏面release，可是在外面release了，要否則會內存泄漏

@implementation RongRTCBufferUtil

+ (UIImage *)imageFromBuffer:(CMSampleBufferRef)buffer {    
    CVPixelBufferRef pixelBuffer = (CVPixelBufferRef)CMSampleBufferGetImageBuffer(buffer);
    CIImage *ciImage = [CIImage imageWithCVPixelBuffer:pixelBuffer];

    CIContext *temporaryContext = [CIContext contextWithOptions:nil];
    CGImageRef videoImage = [temporaryContext createCGImage:ciImage fromRect:CGRectMake(0, 0, CVPixelBufferGetWidth(pixelBuffer), CVPixelBufferGetHeight(pixelBuffer))];

    UIImage *image = [UIImage imageWithCGImage:videoImage];
    CGImageRelease(videoImage);

    return image;
}

+ (UIImage *)compressImage:(UIImage *)image newWidth:(CGFloat)newImageWidth {
    if (!image) return nil;

    float imageWidth = image.size.width;
    float imageHeight = image.size.height;
    float width = newImageWidth;
    float height = image.size.height/(image.size.width/width);
    float widthScale = imageWidth /width;
    float heightScale = imageHeight /height;
    UIGraphicsBeginImageContext(CGSizeMake(width, height));

    if (widthScale > heightScale) {
        [image drawInRect:CGRectMake(0, 0, imageWidth /heightScale , height)];
    }
    else {
        [image drawInRect:CGRectMake(0, 0, width , imageHeight /widthScale)];
    }

    UIImage *newImage = UIGraphicsGetImageFromCurrentImageContext();
    UIGraphicsEndImageContext();
    return newImage;
}

+ (CVPixelBufferRef)CVPixelBufferRefFromUiImage:(UIImage *)img {
    CGSize size = img.size;
    CGImageRef image = [img CGImage];

    NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
                             [NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey,
                             [NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey, nil];
    CVPixelBufferRef pxbuffer = NULL;
    CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, size.width, size.height, kCVPixelFormatType_32ARGB, (__bridge CFDictionaryRef) options, &pxbuffer);

    NSParameterAssert(status == kCVReturnSuccess && pxbuffer != NULL);

    CVPixelBufferLockBaseAddress(pxbuffer, 0);
    void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
    NSParameterAssert(pxdata != NULL);

    CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();
    CGContextRef context = CGBitmapContextCreate(pxdata, size.width, size.height, 8, 4*size.width, rgbColorSpace, kCGImageAlphaPremultipliedFirst);
    NSParameterAssert(context);

    CGContextDrawImage(context, CGRectMake(0, 0, CGImageGetWidth(image), CGImageGetHeight(image)), image);

    CGColorSpaceRelease(rgbColorSpace);
    CGContextRelease(context);

    CVPixelBufferUnlockBaseAddress(pxbuffer, 0);

    return pxbuffer;
}

+ (CMSampleBufferRef)sampleBufferFromPixbuffer:(CVPixelBufferRef)pixbuffer time:(CMTime)time {
    CMSampleBufferRef sampleBuffer = NULL;

    //獲取視頻信息
    CMVideoFormatDescriptionRef videoInfo = NULL;
    OSStatus result = CMVideoFormatDescriptionCreateForImageBuffer(NULL, pixbuffer, &videoInfo);
    CMTime currentTime = time;

    //    CMSampleTimingInfo timing = {currentTime, currentTime, kCMTimeInvalid};
    CMSampleTimingInfo timing = {currentTime, currentTime, kCMTimeInvalid};
    result = CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault,pixbuffer, true, NULL, NULL, videoInfo, &timing, &sampleBuffer);
    CFRelease(videoInfo);
    return sampleBuffer;
}

+ (size_t)getCMTimeSize {
    size_t size = sizeof(CMTime);
    return size;
}

@end

此工具類中實現是由 CPU 處理，當進行 CMSampleBufferRef 轉 UIImage、UIImage 轉 CVPixelBufferRef、 CVPixelBufferRef 轉 CMSampleBufferRef 以及裁剪圖片時，這裏須要注意將使用後的對象及時釋放，不然會出現內存大量泄漏。

2. 視頻發送

2.1 準備階段

使用融雲的 RongRTCLib 的前提須要一個 AppKey，請在官網（https://www.rongcloud.cn/)獲取，經過 AppKey 取得 token 以後進行 IM 鏈接，在鏈接成功後加入 RTC 房間，這是屏幕共享發送的準備階段。

- (void)broadcastStartedWithSetupInfo:(NSDictionary<NSString *,NSObject *> *)setupInfo {
    // User has requested to start the broadcast. Setup info from the UI extension can be supplied but optional.

    // 請填寫您的 AppKey
    self.appKey = @"";
    // 請填寫用戶的 Token
    self.token = @"";
    // 請指定房間號
    self.roomId = @"123456";

    [[RCIMClient sharedRCIMClient] initWithAppKey:self.appKey];
    [[RCIMClient sharedRCIMClient] setLogLevel:RC_Log_Level_Verbose];

    // 鏈接 IM
    [[RCIMClient sharedRCIMClient] connectWithToken:self.token
                                           dbOpened:^(RCDBErrorCode code) {
        NSLog(@"dbOpened: %zd", code);
    } success:^(NSString *userId) {
        NSLog(@"connectWithToken success userId: %@", userId);
        // 加入房間
        [[RCRTCEngine sharedInstance] joinRoom:self.roomId
                                    completion:^(RCRTCRoom * _Nullable room, RCRTCCode code) {
            self.room = room;
            self.room.delegate = self;
            [self publishScreenStream];
        }];
    } error:^(RCConnectErrorCode errorCode) {
        NSLog(@"ERROR status: %zd", errorCode);
    }];
}

如上是鏈接 IM 和加入 RTC 房間的全過程，其中還包含調用發佈自定義視頻 [self publishScreenStream]; 此方法在加入房間成功後才能夠進行。

- (void)publishScreenStream {
    RongRTCStreamParams *param = [[RongRTCStreamParams alloc] init];
    param.videoSizePreset = RongRTCVideoSizePreset1280x720;
    self.videoOutputStream = [[RongRTCAVOutputStream alloc] initWithParameters:param tag:@"RongRTCScreenVideo"];
    [self.room publishAVStream:self.videoOutputStream extra:@"" completion:^(BOOL isSuccess, RongRTCCode desc) {
        if (isSuccess) {
            NSLog(@"發佈自定義流成功");
        }
    }];
}

自定義一個 RongRTCAVOutputStream 流便可，使用此流發送屏幕共享數據。

2.2 開始發送屏幕共享數據

上面咱們已經鏈接了融雲的 IM 和加入了 RTC 房間，而且自定義了一個發送屏幕共享的自定義流，接下來，如何將此流發佈出去呢？

- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType {
    switch (sampleBufferType) {
        case RPSampleBufferTypeVideo:
            // Handle video sample buffer
            [self.videoOutputStream write:sampleBuffer error:nil];
            break;
        case RPSampleBufferTypeAudioApp:
            // Handle audio sample buffer for app audio
            break;
        case RPSampleBufferTypeAudioMic:
            // Handle audio sample buffer for mic audio
            break;
        default:
            break;
    }
}

但咱們接收到了蘋果上報的數據以後，調用 RongRTCAVOutputStream 中的 write:error: 方法，將 sampleBuffer 發送給遠端，至此，屏幕共享數據就發送出去啦。

[self.videoOutputStream write:sampleBuffer error:nil];

融雲的核心代碼就是經過上面的鏈接 IM，加入房間，發佈自定義流，而後經過自定義流的 write:error: 方法將 sampleBuffer 發送出去。

無論是經過 ReplayKit 取得屏幕視頻，仍是使用 Socket 在進程間傳輸，都是爲最終的 write:error: 服務。

總結

1. Extension 內存是有限制的，最大 50M，因此在 Extension 裏面處理數據須要格外注意內存釋放；
2. 若是 VideotoolBox 在後臺解碼一直失敗，只需把 VideotoolBox 重啓一下便可，此步驟在上面的代碼中有體現；
3. 若是不須要將 Extension 的數據傳到主 App，只需在 Extension 裏直接將流經過 RongRTCLib 發佈出去便可，缺點是 Extension 中發佈自定義流的用戶與主 App 中的用戶不是同一個，這也是上面經過 Socket 將數據傳遞給主 App 要解決的問題；
4. 若是主 App 須要拿到屏幕共享的數據處理，使用 Socket 將流先發給主 App，而後在主 App 裏面經過 RongRTCLib 將流發出去。

最後附上 Demo