人工智能時代的到來,相信你們已耳濡目染,虹軟免費、離線開放的人臉識別 SDK,正推進着全行業進入刷臉時代,爲了方便開發者接入,虹軟提供了多種語言,多種平臺的人臉識別SDK的支持,使用場景普遍。產品主要功能有:人臉檢測、追蹤、特徵提取、特徵比對、屬性檢測,活體檢測,圖像質量檢測等,此外,虹軟提供的是基於本地算法特徵的離線識別SDK,提供全平臺的離線支持。html
現現在,人臉查找及跟蹤這例Demo很是火,以前個人大學室友也曾用python調opencv庫函數來實現過相似的功能,包括不少比賽,也會在此基礎上構造賽題,而虹軟也正是提供了這方面的技術支持。所以做爲初學者的我,也想嘗試基於虹軟的SDK來寫我的臉查找及跟蹤的樣例,並寫此文章進行記錄,向廣大初學開發者做分享。java
此Demo採用Maven做爲項目管理工具,並基於Windows x64,Java 8,SDK是基於虹軟人臉識別SDK3.0python
SDK依賴Jar包 可從虹軟官網獲取 點擊「免費獲取」 」登陸「後 選擇 「具體平臺/版本/語言」進行獲取git
pom.xml 依賴包括github
<dependencies> <dependency> <groupId>org.bytedeco</groupId> <artifactId>javacv-platform</artifactId> <version>1.5.2</version> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>1.18.8</version> </dependency> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-core</artifactId> <version>1.2.3</version> </dependency> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-access</artifactId> <version>1.2.3</version> </dependency> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>1.2.3</version> </dependency> <dependency> <groupId>com.arcsoft.face</groupId> <artifactId>arcsoft-sdk-face</artifactId> <version>1.0.0</version> <scope>system</scope> <systemPath>${basedir}/libs/arcsoft-sdk-face-3.0.0.0.jar</systemPath> </dependency> <dependency> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> <version>26.0-jre</version> </dependency> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-lang3</artifactId> <version>3.8.1</version> </dependency> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-pool2</artifactId> <version>2.6.0</version> </dependency> <dependency> <groupId>commons-io</groupId> <artifactId>commons-io</artifactId> <version>2.6</version> </dependency> </dependencies>
此類繼承BasePooledObjectFactory
抽象類,爲FaceEngine
對象池算法
1)成員變量說明apache
//人臉識別引擎庫路徑 private String libPath; //SDK的APP_ID private String appId; //SDK的SDK_KEY private String sdkKey; //引擎配置類 private EngineConfiguration engineConfiguration;
其中 人臉識別引擎庫,APP_ID,SDK_KEY 可經過虹軟官網」開發者中心「 進行 「登陸」後 在「個人應用「中進行獲取canvas
2)create()方法數組
public FaceEngine create() throws Exception { FaceEngine faceEngine = new FaceEngine(libPath); int activeCode = faceEngine.activeOnline(appId, sdkKey); log.info("faceEngineActiveCode:" + activeCode); int initCode = faceEngine.init(engineConfiguration); log.info("faceEngineInitCode:" + initCode); return faceEngine; }
FaceEngine
人臉識別引擎此方法,經過libPath(SDK引擎庫的路徑)實例化FaceEngine
對象,再根據APP_ID,SDK_KEY調用activeOnline()
方法激活引擎(聯網狀態下)安全
成功激活引擎後,根據EngineConfiguration
引擎配置類 調用init()
方法初始化引擎
3)wrap()方法
public PooledObject<FaceEngine> wrap(FaceEngine faceEngine) { return new DefaultPooledObject<>(faceEngine); }
FaceEngine
人臉識別引擎PooledObject
包裝類此方法,經過PooledObject
包裝器對象 將faceEngine
進行包裝,便於維護引擎的狀態
4)destroyObject()方法
public void destroyObject(PooledObject<FaceEngine> p) throws Exception { FaceEngine faceEngine = p.getObject(); int result = faceEngine.unInit(); super.destroyObject(p); }
PooledObject
包裝類此方法,從PooledObject
包裝器對象中獲取faceEngine
引擎,隨後卸載引擎
1)成員變量說明
//VIDEO模式人臉檢測引擎,用於預覽幀人臉追蹤 private FaceEngine ftEngine; //人臉註冊引擎 private FaceEngine regEngine; //用於人臉識別的引擎池 private GenericObjectPool<FaceEngine> frEnginePool; //存放視頻中識別到的人臉信息(faceId爲key,FaceResult爲Value) (FaceId用來標記一張人臉,從進入畫面到離開畫面這個值不變,可使用FaceId判斷用戶) //ConcurrentHashMap:在多線程運行狀況下,增/刪faceResultRegistry中的鍵值對時,保證其線程安全 //volatile關鍵字:在多線程運行狀況下,增/刪faceResultRegistry中的鍵值對後,保證其線程之間的可見性 private volatile ConcurrentHashMap<Integer, FaceResult> faceResultRegistry = new ConcurrentHashMap<>(); //線程池 private ExecutorService frService = Executors.newFixedThreadPool(20); //存放 註冊照與註冊照人臉特徵值 的映射 public ConcurrentHashMap<String, byte[]> faceFeatureRegistry = new ConcurrentHashMap<>(); //記錄上次清理過期人臉時間 private long lastClearTime = System.currentTimeMillis(); //封裝 視頻中檢測到的人臉與註冊照人臉 比對結果 @Data public class FaceResult { private boolean flag = false; private String name; private float score; } //封裝 視頻中檢測到的人臉信息 @Data public class FacePreviewInfo { private FaceInfo faceInfo; private int age; private boolean liveness; }
2)init()方法
public void initEngine(String libPath,String appId,String sdkKey) { //引擎配置 ftEngine = new FaceEngine(libPath); int activeCode = ftEngine.activeOnline(appId, sdkKey); EngineConfiguration ftEngineCfg = new EngineConfiguration(); ftEngineCfg.setDetectMode(DetectMode.ASF_DETECT_MODE_VIDEO); ftEngineCfg.setFunctionConfiguration(FunctionConfiguration.builder().supportFaceDetect(true).build()); int ftInitCode = ftEngine.init(ftEngineCfg); //引擎配置 regEngine = new FaceEngine(libPath); EngineConfiguration regEngineCfg = new EngineConfiguration(); regEngineCfg.setDetectMode(DetectMode.ASF_DETECT_MODE_IMAGE); regEngineCfg.setFunctionConfiguration(FunctionConfiguration.builder().supportFaceDetect(true).supportFaceRecognition(true).build()); int regInitCode = regEngine.init(regEngineCfg); GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig(); poolConfig.setMaxIdle(5); poolConfig.setMaxTotal(5); poolConfig.setMinIdle(5); poolConfig.setLifo(false); EngineConfiguration frEngineCfg = new EngineConfiguration(); frEngineCfg.setFunctionConfiguration(FunctionConfiguration.builder().supportFaceRecognition(true).build()); frEnginePool = new GenericObjectPool(new FaceEngineFactory(libPath, appId, sdkKey, frEngineCfg), poolConfig);//底層庫算法對象池 if (!(activeCode == ErrorInfo.MOK.getValue() || activeCode == ErrorInfo.MERR_ASF_ALREADY_ACTIVATED.getValue())) { log.error("activeCode: " + activeCode); throw new RuntimeException("activeCode: " + activeCode); } if (ftInitCode != ErrorInfo.MOK.getValue()) { log.error("ftInitEngine: " + ftInitCode); throw new RuntimeException("ftInitEngine: " + ftInitCode); } if (regInitCode != ErrorInfo.MOK.getValue()) { log.error("regInitEngine: " + regInitCode); throw new RuntimeException("regInitEngine: " + regInitCode); } }
此方法,根據傳入的libPath,APP_ID,SDK_KEY去初始化用於人臉檢測跟蹤引擎(VIDEO模式,開啓人臉檢測功能) 以及用於人臉註冊的引擎(IMAGE模式,開啓人臉識別功能),而後再去實例化人臉識別引擎池,設置引擎池對應屬性後,實例化EngineConfiguration
對象(開啓人臉識別功能),最後經過FaceEngineFactory
的構造方法去初始化引擎並獲取對象池。
3)registerFace()方法 註冊人臉
public void registerFace(String imagePath) { log.info("正在註冊人臉"); int count = 0; if (regEngine != null) { File file = new File(imagePath); File[] files = file.listFiles(); for (File file1 : files) { ImageInfo imageInfo = ImageFactory.getRGBData(file1); if (imageInfo != null) { List<FaceInfo> faceInfoList = new ArrayList<>(); int code = regEngine.detectFaces(imageInfo.getImageData(), imageInfo.getWidth(), imageInfo.getHeight(), imageInfo.getImageFormat(), faceInfoList); if (code == 0 && faceInfoList.size() > 0) { FaceFeature faceFeature = new FaceFeature(); int resCode = regEngine.extractFaceFeature(imageInfo.getImageData(), imageInfo.getWidth(), imageInfo.getHeight(),imageInfo.getImageFormat(), faceInfoList.get(0), faceFeature); if (resCode == 0) { int lastIndexOf = file1.getName().lastIndexOf("."); String name = file1.getName().substring(0, lastIndexOf); faceFeatureRegistry.put(name, faceFeature.getFeatureData()); log.info("成功註冊人臉:" + name); count++; } } } } log.info("人臉註冊完成,共註冊:" + count + "張人臉"); } else { throw new RuntimeException("註冊失敗,引擎未初始化或初始化失敗"); } }
此方法,將參數目錄下的每一個文件解析爲ImageInfo
類型的RGB圖像信息數據,再調用FaceEngineService
對象的detectFaces()
方法檢測並獲取人臉信息數據(其所需參數有 圖像數據 ,圖像寬度(4的倍數),圖片高度,圖像的顏色格式,存放檢測到的人臉信息List)。成功檢測到人臉後,再經過extractFaceFeature()
方法提取人臉特徵值(其所需參數有圖像數據,圖像寬度(4的倍數),圖像高度,圖像的顏色格式,人臉信息,存放提取到的人臉特徵信息)。成功獲取到人臉特徵值後,將圖片文件名和人臉特徵值以key-value的形式存放於ConcurrentHashMap
中。
4)detectFaces()方法 人臉檢測
public List<FacePreviewInfo> detectFaces(ImageInfo imageInfo) { if (ftEngine != null) { List<FaceInfo> faceInfoList = new ArrayList<>(); int code = ftEngine.detectFaces(imageInfo.getImageData(), imageInfo.getWidth(), imageInfo.getHeight(), imageInfo.getImageFormat(), faceInfoList); List<FacePreviewInfo> previewInfoList = new LinkedList<>(); for (FaceInfo faceInfo : faceInfoList) { FacePreviewInfo facePreviewInfo = new FacePreviewInfo(); facePreviewInfo.setFaceInfo(faceInfo); previewInfoList.add(facePreviewInfo); } clearFaceResultRegistry(faceInfoList); return previewInfoList; } return null; }
ImageInfo
圖像信息FacePreviewInfo
列表信息此方法,根據傳入的ImageInfo
類型的RGB圖像信息數據,調用ftEngine引擎的detectFaces()
方法 獲取人臉信息,遍歷獲取到的人臉信息列表設置於FacePreviewInfo
類型對象中,隨後將faceInfoList
列表 傳入clearFaceResultRegistry()
方法,清理過期的人臉,並返回FacePreviewInfo
列表。
5)clearFaceResultRegistry()方法 清理過期人臉
private void clearFaceResultRegistry(List<FaceInfo> faceInfoList) { if (System.currentTimeMillis() - lastClearTime > 5000) { Iterator<Integer> iterator = faceResultRegistry.keySet().iterator(); for (; iterator.hasNext(); ) { Integer next = iterator.next(); boolean flag = false; for (FaceInfo faceInfo : faceInfoList) { if (next.equals(faceInfo.getFaceId())) { flag = true; } } if (!flag) { iterator.remove(); } } } }
FacePreviewInfo
列表信息此方法,若當前時間距離上次清理過期人臉已有5s(用戶可根據須要自行設置),則遍歷faceResultRegistry
的key,判斷faceResultRegistry
與faceInfoList
(即以前識別到的與新識別到的人臉)是否存在相同FaceId
的人臉,是,則刪除faceResultRegistry
中此過期的人臉信息。
6)getFaceResult() 方法
public FaceResult getFaceResult(FaceInfo faceInfo, ImageInfo imageInfo) { FaceResult faceResult = faceResultRegistry.get(faceInfo.getFaceId()); if (faceResult == null) { faceResult = new FaceResult(); faceResultRegistry.put(faceInfo.getFaceId(), faceResult); frService.submit(new FaceInfoRunnable(faceInfo, imageInfo, faceResult)); } else if (faceResult.isFlag()) { return faceResult; } return null; }
ImageInfo
圖像信息FaceResult
結果此方法,先嚐試根據faceId
從faceResultRegistry
中獲取 FaceResult
(即以前是否比對過相同人臉),若不存在則實例化一個faceResult
並將其以faceId
爲Key,faceResult
爲value 存放到faceResultRegistry
中,同時新建一個FaceInfoRunnable
線程並將faceInfo
, imageInfo
, faceResult
三者傳入線程中 運行;若存在,則判斷 faceResult
的flag是否爲true(便是否可從註冊照找到類似人臉), 若爲true 直接返回便可 。
7)FaceInfoRunnable
此類爲一個實現 Runnable 接口的線程實現類
成員變量說明
//傳入 視頻中識別到的人臉信息 private FaceInfo faceInfo; //傳入 ImageInfo圖像信息 private ImageInfo imageInfo; //人臉比對結果封裝 private FaceResult faceResult;
run()方法
@Override public void run() { FaceEngine frEngine = null; try { frEngine = frEnginePool.borrowObject(); if (frEngine != null) { FaceFeature faceFeature = new FaceFeature(); int resCode = frEngine.extractFaceFeature(imageInfo.getImageData(), imageInfo.getWidth(), imageInfo.getHeight(), imageInfo.getImageFormat(), faceInfo, faceFeature); if (resCode == 0) { float score = 0.0F; Iterator<Map.Entry<String, byte[]>> iterator = faceFeatureRegistry.entrySet().iterator(); for (; iterator.hasNext(); ) { Map.Entry<String, byte[]> next = iterator.next(); FaceFeature faceFeatureTarget = new FaceFeature(); faceFeatureTarget.setFeatureData(next.getValue()); FaceSimilar faceSimilar = new FaceSimilar(); frEngine.compareFaceFeature(faceFeatureTarget, faceFeature, faceSimilar); if (faceSimilar.getScore() > score) { score = faceSimilar.getScore(); faceResult.setName(next.getKey()); } } log.info("類似度:" + score); if (score >= 0.8f) { faceResult.setScore(score); faceResult.setFlag(true); faceResultRegistry.put(faceInfo.getFaceId(), faceResult); } else { faceResultRegistry.remove(faceInfo.getFaceId()); } } } } catch (Exception e) { } finally { if (frEngine != null) { frEnginePool.returnObject(frEngine); } } }
run()
方法,根據成員變量FaceInfo
,ImageInfo
調用frEngine的extractFaceFeature()
方法獲取人臉特徵值。成功獲取特徵值後,遍歷faceFeatureRegistry
(註冊照人臉)中的特徵值,結合剛獲取到的特徵值經過compareFaceFeature()
方法比對 倆人臉類似度(其所需參數有人臉特徵值1,人臉特徵值2,比對模型,存放比對類似值結果),並以類似度最高的註冊照命名faceResult的Name
,最終,若類似度大於等於0.8(用戶可根據須要自行設置) 則將類似度值設置於faceResult
對象並將其flag設爲true(即註冊照中找到類似人臉),並以faceId
爲key 再次put到faceResultRegistry
中,不然remove此faceId
的faceResul
t,最後釋放引擎。
此類是視頻播放類
1)成員變量說明
//視頻幀抓取器 private FFmpegFrameGrabber fFmpegFrameGrabber; //視頻播放監聽器 private VideoListener videoListener; //管理定時任務(true表示其關聯的線程設爲守護線程) private Timer timer = new Timer(true);
2)start()方法
public void start() { try { fFmpegFrameGrabber.setPixelFormat(AV_PIX_FMT_BGR24 ); fFmpegFrameGrabber.start(); videoListener.onStart(); } catch (FrameGrabber.Exception e) { videoListener.onError(e); } final int[] lengthInVideoFrames = {fFmpegFrameGrabber.getLengthInVideoFrames()}; OpenCVFrameConverter.ToIplImage converter = new OpenCVFrameConverter.ToIplImage();//轉換器 TimerTask task = new TimerTask() { @Override public void run() { try { Frame grab = fFmpegFrameGrabber.grabImage(); lengthInVideoFrames[0]--; if (grab != null) { IplImage iplImage = converter.convert(grab); if (iplImage != null) { videoListener.onPreview(iplImage); } } if (lengthInVideoFrames[0] <= 0) { stop(); } } catch (Exception e) { videoListener.onError(e); } } }; timer.schedule(task, 0, 40); }
此方法,用於開始播放視頻,首先爲幀抓取器設置 要轉換成的圖像數據格式,隨後啓動幀抓取器和視頻播放監聽器。lengthInVideoFrames
數組中存放視頻的幀數,而converter
變量爲幀與圖片之間的轉換器。經過TimerTask
生成一個線程,在線程run()
方法中去抓取視頻中的每一幀並將其轉換爲圖像,獲取到的圖像交給videoListener
的onPreview()
回調方法進行處理,若幀數已處理完則中止運行。另外,TimerTask
線程將由timer
進行管理,每40毫秒執行一次。
此類爲包含main()
方法的主類,右擊 執行 Run ‘MainApplication.main()’ 便可運行此Demo
main() 主方法
//虹軟引擎庫存放路徑 String libPath = "D:\\arcsoft_lib"; //sdk的APP_ID String appId = "9iSfMeAhjA52N**************iW1aKes2TpSrd"; //sdk的SDK_KEY String sdkKey = "BuRTH3hGs91m**************dxEgyP9xu6fiFG7G"; //視頻文件路徑(從視頻中查找並跟蹤人臉) String videoPath="D:\icon-man.mp4"; //須要識別人的註冊照目錄路徑 String imagePath="D:\\photo";
Loader.load(opencv_imgproc.class); Loader.load(CvPoint.class); Loader.load(CvFont.class); CanvasFrame canvas = new CanvasFrame("預覽"); canvas.setDefaultCloseOperation(EXIT_ON_CLOSE); VideoPlayer videoPlayer = new VideoPlayer(videoPath);
首先,加載opencv_imgproc
,CvPoint
,CvFont
等程序所需類,並實例化預覽窗口(設置程序退出即窗口關閉)
同時向VideoPlayer
的構造方法傳入視頻路徑(爲FFmpegFrameGrabber
成員變量 指定具體視頻) 實例化VideoPlayer
FaceRecognize faceRecognize = new FaceRecognize(); faceRecognize.initEngine(libPath,appId,sdkKey); faceRecognize.registerFace(imagePath); OpenCVFrameConverter.ToIplImage converter = new OpenCVFrameConverter.ToIplImage();
以後,實例化FaceRecognize
並根據LibPath,APP_ID,SDK_KEY調用initEngine()
方法初始化FaceRecognize
各引擎,同時實例化轉換器
videoPlayer.setListener(new VideoListener() { CvScalar color = cvScalar(0, 0, 255, 0); // blue [green] [red] CvFont cvFont = cvFont(opencv_imgproc.FONT_HERSHEY_DUPLEX); @Override public void onStart() { } @Override public void onPreview(IplImage iplImage) { ImageInfo imageInfo = new ImageInfo(); imageInfo.setWidth(iplImage.width()); imageInfo.setHeight(iplImage.height()); imageInfo.setImageFormat(ImageFormat.CP_PAF_BGR24); byte[] imageData = new byte[iplImage.imageSize()]; iplImage.imageData().get(imageData); imageInfo.setImageData(imageData); List<FaceRecognize.FacePreviewInfo> previewInfoList = faceRecognize.detectFaces(imageInfo); for (FaceRecognize.FacePreviewInfo facePreviewInfo : previewInfoList) { int x = facePreviewInfo.getFaceInfo().getRect().getLeft(); int y = facePreviewInfo.getFaceInfo().getRect().getTop(); int xMax = facePreviewInfo.getFaceInfo().getRect().getRight(); int yMax = facePreviewInfo.getFaceInfo().getRect().getBottom(); CvPoint pt1 = cvPoint(x, y); CvPoint pt2 = cvPoint(xMax, yMax); opencv_imgproc.cvRectangle(iplImage, pt1, pt2, color, 1, 4, 0); FaceRecognize.FaceResult faceResult = faceRecognize.getFaceResult(facePreviewInfo.getFaceInfo(), imageInfo); if (faceResult != null) { try { CvPoint pt3 = cvPoint(x, y - 2); opencv_imgproc.cvPutText(iplImage, faceResult.getName(), pt3, cvFont, color); } catch (Exception e) { e.printStackTrace(); } } } Frame frame = converter.convert(iplImage); canvas.showImage(frame); } @Override public void onCancel() { } @Override public void onError(Exception e) { } }); videoPlayer.start();
上述代碼 ,首先爲VideoPlayer
設置監聽器,再啓動VideoPlayer
。而監聽器VideoListener
的onPreview()
方法中,先將傳入的IplImage
類型圖像信息數據(即由幀抓取器獲取到的圖片)設置到ImageInfo類型對象中,以後調用faceRecognize
的detectFaces()
方法獲取人臉信息。成功獲取到人臉後,根據人臉信息肯定人臉方位座標,調用opencv_imgproc繪製方形矩陣。再根據識別到的人臉信息和圖像信息數據 調用faceRecognize
的getFaceResult()
方法獲取FaceResult
(即註冊照中是否擁有與視頻中類似的人臉)。若FaceResult
不爲空,則調用 opencv_imgproc
將對應圖片文件名寫於人臉框上方,最後將圖片轉化爲視頻幀做用於canvas上進行展現。
如有想一塊兒學習虹軟SDK,感覺人臉識別奧祕的同窗,可經過點擊此連接獲取Demo源碼
瞭解更多人臉識別產品相關內容請到虹軟視覺開放平臺哦