Facial landmark detection using Dlib (left) and CLM-framework (right).python
Who sees the human face correctly: the photographer, the mirror, or the painter? — Pablo Picassolinux
If Picasso was alive today, he would have definitely added one more professional to that list — a computer vision engineer!git
As computer vision engineers and researchers we have been trying to understand the human face since the very early days. The most obvious application of facial analysis is Face Recognition. But to be able to identify a person in an image we first need to find where in the image a face is located. Therefore, face detection — locating a face in an image and returning a bounding rectangle / square that contains the face — was a hot research area. In 2001, Paul Viola and Michael Jones pretty much nailed the problem with their seminal paper titled 「Rapid Object Detection using a Boosted Cascade of Simple Features.」 In the early days of OpenCV and to some extent even now, the killer application of OpenCV was a good implementation of the Viola and Jones face detector.github
Once you have a bounding box around the face, the obvious research problem is to see if you can find the location of different facial features ( e.g. corners of the eyes, eyebrows, and the mouth, the tip of the nose etc ) accurately. Facial feature detection is also referred to as 「facial landmark detection」, 「facial keypoint detection」 and 「face alignment」 in the literature, and you can use those keywords in Google for finding additional material on the topic.web
Applications of Facial Keypoint Detection
There are several interesting applications of keypoint detection in human faces. A few of them are listed below.算法
Facial feature detection improves face recognition
Facial landmarks can be used to align facial images to a mean face shape, so that after alignment the location of facial landmarks in all images is approximately the same. Intuitively it makes sense that facial recognition algorithms trained with aligned images would perform much better, and this intuition has been confirmed by many research papers.macos
Head pose estimation
Once you know a few landmark points, you can also estimate the pose of the head. In other words you can figure out how the head is oriented in space, or where the person is looking. E.g. CLM-Framework described in this post also returns the head pose.windows
Face Morphing
Facial landmarks can be used to align faces that can then be morphed to produce in-between images. An example is shown in Figure 1.
Figure 1. A morph between President Bush and the The Governator!
Virtual Makeover
At my company ( TAAZ.com ) we had written our own facial landmark detector. The detected landmarks were used to the calculate contours of the mouth, eyes etc. to render makeup virtually. An example is shown in Figure 2.
Figure 2. Landmark detection for virtual makeover.
Face Replacement
If you have facial feature points estimated on two faces, you can align one face to the other, and then seamlessly clone one face onto the other. You can also do something goofy like this
https://auduno.github.io/clmtrackr/examples/facesubstitution.html
In a previous post, we showed how to use facial features to predict facial attractiveness.
Clearly, the ability to detect facial features in images and videos open up possibilities of a ton of interesting applications. Let us now get our hands dirty with some tools that will allow us to do this.
Facial Feature Detection & Tracking Libraries
There has been a flurry of activity in this area in the last 5 years. Part of the reason for this activity is the availability of large annotated datasets like LFPW and Helen. I have listed a bunch of papers in the next section. However, I do not recommend implementing these papers from scratch because now we have access to high quality open source implementations of some of these papers.
In the video below, you can see two of the libraries, Dlib and CLM-framework in action.
Dlib ( C++ / Python )
Dlib is a collection of miscellaneous algorithms in Machine Learning, Computer Vision, Image Processing, and Linear Algebra. Most of the library is just header files that you can include in your C++ application. Oh you prefer python ? No problem, it has a python API as well.
I personally like Dlib more than any other facial feature detection & tracking library because the code is very clean, well documented, the license permits use in commercial applications, the algorithm they have chosen to implement is very fast and accurate, and you can easily integrate the library in your C++ project by just including the header files.
How to compile Dlib ?
- Download a copy from github
1
git clone https://github.com/davisking/dlib.git - Build examples ( OSX / Linux )
12345
cd dlib/examplesmkdir buildcd buildcmake ..cmake --build . --config ReleaseThese examples are a great way to start using dlib. Make a copy of an example cpp file, modify it, modify examples/CMakeLists.txt and compile again using the instructions above. Easy!
- Compile dlib python module
12
cd dlib/python_examples./compile_dlib_python_module.bat - Set PYTHONPATH environment variable
12
# Put the following line in .bashrc or .profileexport PYTHONPATH=/path/to/dlib/python_examples:$PYTHONPATH - Test python module
1
python -c "import dlib"
If the above line does not give an error, you are all set.
In case you run into any compilation issues, there are additional instructions at Dlib.net
How to run Dlib’s facial landmark detector ?
After you have built the examples, to run the facial landmark detector using a webcam, do the following.
|
1
2
3
4
5
|
cd examples/build/
#Download the face landmark model
tar xvjf shape_predictor_68_face_landmarks.dat.bz2
./webcam_face_pose_ex
|
If you want to run it on a single image, you can try
|
1
|
./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat faces/*.jpg
|
CLM-Framework (C++)
CLM-framework, also known as the Cambridge Face Tracker, is a C++ library for facial keypoint detection and head pose estimation. You can see how well it works in the included video. Compiling this library on OSX was bit of a challenge but it was not too bad. The library depends on OpenCV 3 and requires X11.
There are two important ways in which Dlib beats CLM-Framework. First, DLib is much faster than CLM-Framework. Second, Dlib’s license allows you to use it in commercial applications. If I had to pick, I would use Dlib. Interestingly, CLM-Framework depends on Dlib!
How to compile CLM-Framework ?
Compiling CLM-Framework was a bit involved for OSX. For windows and linux there are detailed instructions here. For compiling version 1.3.0 on OSX, I used the instructions for linux but made the following changes.
Most of the dependencies were installed using brew.
In file CMakeLists.txt ( red text was replaced with green text ).
find_package( OpenCV 3.0 REQUIRED )
INCLUDE_DIRECTORIES(${OpenCV_INCLUDE_DIRS})
INCLUDE_DIRECTORIES(/opt/X11/include)
In file exe/SimpleCLM/SimpleCLM.cpp
How to run CLM-Framework’s Facial Feature Detector ?
After compiling CLM-Framework, the executables are in the bin directory. For the webcam demo shown in this post, you can use
|
1
|
bin/SimpleCLM
|
Face++ ( FacePlusPlus ) : Web API
One of the best implementations of facial landmark detection is by FacePlusPlus. They won the 300 Faces In-the-Wild Landmark Detection Challenge, 2013. They provide an easy to use API. The problem is that you need to upload an image to their servers and that raises a lot of privacy concerns. But if privacy is not an issue, Face++ is very good option. You can see a demo at
http://www.faceplusplus.com/demo-landmark/
Facial Feature Detection Research
Many different approaches have been used to solve this problem and it is difficult to summarize them in a blog post. I am simply linking to some important papers ( with major bias toward recent work ) for people who want to investigate more.
- Active Appearance Model (AAM) by T. Cootes, G. Edwards and C. J. Taylor. [1998]
- Face Alignment through Subspace Constrained Mean-Shifts by Jason M. Saragih, Simon Lucey and Jeffrey F. Cohn. [2009]
- Localizing Parts of Faces Using a Consensus of Exemplars by Peter N. Belhumeur, David W. Jacobs, David J. Kriegman, Neeraj Kumar [ 2011 ]
- Face Alignment by Explicit Shape Regression by Xudong Cao Yichen Wei Fang Wen Jian Sun [2012]
- Supervised Descent Method and Its Applications to Face Alignment by Xuehan Xiong and Fernando De la Torre [2013]
- Constrained Local Neural Fields for robust facial landmark detection in the wild by Tadas Baltrusaitis, Peter Robinson, and Louis-Philippe Morency. [2013]
- Extensive Facial Landmark Localization with Coarse-to-fine Convolutional Network Cascade by Erjin Zhou, Haoqiang Fan, Zhimin Cao, Yuning Jiang and Qi Yin. [2013]
- Face alignment at 3000 fps via regressing local binary features by S Ren, X Cao, Y Wei, J Sun. [2014]
- Facial Landmark Detection by Deep Multi-task Learning by Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. [2014]
- One Millisecond Face Alignment with an Ensemble of Regression Trees by Vahid Kazemi and Josephine Sullivan. [2014]
Subscribe
If you liked this article, please subscribe to our newsletter and receive a free
Computer Vision Resource guide. In our newsletter we share OpenCV tutorials and examples written in C++/Python, and Computer Vision and Machine Learning algorithms and news.
中文翻譯
源地址：http://blog.csdn.net/xiamentingtao/article/details/50908190
原文地址：http://www.learnopencv.com/facial-landmark-detection/#comment-2471797375
做爲計算機視覺研究員，咱們很早就開始研究人臉。人臉分析領域最廣爲人知的就是人臉識別（face recognition）.可是爲了識別一幅圖像中的人臉，咱們首先必需要找到圖像中人臉的位置。所以人臉檢測（face detection)-定位一幅圖像中的人臉而且返回一個包圍人臉的矩形或者正方形（bounding rectangle/square）是一個熱門的研究領域。2001年，Paul Viola 和Michael Jones 發表了史詩級論文<< 「Rapid Object Detection using a Boosted Cascade of Simple Features.>>.在OpenCV早期甚至某種程度下在如今，OpenCV的致命武器就是對
Viola and Jones face detector的一個比較好的實現。
一旦你找到了人臉附近的包圍盒，最顯然的研究固然是準確識別人臉不一樣特徵的位置（好比，眼角、瞳孔、嘴巴、鼻子等）。人臉特徵檢測（face feature detection）也稱爲 「facial landmark detection」, 「facial keypoint detection」 and 「face alignment」，你能夠在Google找到相似的文獻。
Facial Keypoint Detection
人臉關鍵點檢測有不少應用。以下作了一些列舉：
Facial feature detection improves face recognition
人臉特徵點能夠被用來將人臉對齊到平均人臉（mean face shape）,這樣在對齊以後全部圖像中的人臉特徵點的位置幾乎是相同的。直觀上來看，用對齊後的圖像訓練的人臉識別算法更加有效，這個直覺已經被不少論文驗證。
Head pose estimation
一旦你知道了一些特徵點的位置，你也能夠估計頭部的姿式。換句話說，你能夠解決頭部在空間中的定向問題，或者通俗的講就是人朝那裏看的問題。
Face Morphing （人臉變形）
人臉特徵點能夠對齊人臉，這樣能夠生成兩張人臉的中間圖像。以下圖：
Virtual Makeover（虛擬化妝）
在個人公司
咱們已經寫了本身的人臉特徵點檢測器。檢測出的特徵點被用來計算嘴的輪廓，眼睛等用來渲染虛擬化妝。Figure2z展現了這一效果:
Face Replacement
如何兩張人臉的特徵點已經估計出來了，你能夠將一張人臉對齊到另外一張人臉，而且能夠無縫換臉。你也能夠作像下面同樣傻瓜的事。
https://auduno.github.io/clmtrackr/examples/facesubstitution.html
先前的報告中，咱們展現瞭如何使用人臉特徵點去預測人臉的吸引力。
很明顯，在圖片和視頻上進行人臉特徵點檢測爲許多有趣的應用提供了不少的可能性。下面咱們就將介紹一些有用的特徵點檢測工具。
Facial Feature Detection & Tracking Libraries
過去五年來，這個領域很火，部分緣由是大量能夠用來訓練的數據如LFPW、Helen被提供。我在下一節列了不少論文。可是我不建議胡亂實現這些論文，由於已經有開源的實現。
下面的視頻中，你能夠看到兩個庫Dlib和CLM-framework.
http://7xrqgw.com1.z0.glb.clouddn.com/dlib_clm.mp4
Dlib(C++/Python)
Dlib是機器學習，計算機視覺，圖像處理，線性代數中衆多算法的集合。庫中大多數是頭文件，你能夠直接直接包含在C++應用中。或者你更喜歡Python?沒問題，他也有一個Python接口.
我我的更喜歡Dlib由於代碼是簡潔的，有大量的註釋，也能夠被用來商用。他們選擇實現的算法是很是快的，而且是準確的，你能夠很容易集成這個庫到你的C++工程中，而你須要作的僅僅是包含頭文件.
如何編譯Dlib?
- 從Github上下載：
git clone https://github.com/davisking/dlib.git
- 1
- 創建Examples（OSX\Linux）
cd dlib/examples mkdir build cd build cmake .. cmake --build . --config Release
- 1
- 2
- 3
- 4
- 5
這些例子是一個開始使用Dlib的很是好的方法。拷貝一個例子的cpp文件，修改它，修改examples/CMakeLists.txt 而且像上面同樣再一次編譯它。很容易吧！
3. 編譯dlib python 模塊
cd dlib/python_examples ./compile_dlib_python_module.bat
- 1
- 2
- 設置 PYTHONPATH 環境變量
# Put the following line in .bashrc or .profile export PYTHONPATH=/path/to/dlib/python_examples:$PYTHONPATH
- 1
- 2
- 3
- 測試python模塊
python -c "import dlib"
- 1
若是以上都沒有問題的話，你就設置好了。
How to run Dlib’s facial landmark detector ?
當你編譯好examples後，爲了在網絡攝像頭上運行人臉特徵點檢測器，能夠這樣作:
cd examples/build/
#Download the face landmark model wget http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2 tar xvjf shape_predictor_68_face_landmarks.dat.bz2 ./webcam_face_pose_ex
- 1
- 2
- 3
- 4
- 5
若是你想要在單個圖像上運行，你能夠這樣試試:
./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat faces/*.jpg
- 1
- 2
CLM-Framework (C++)
CLM-framework,也被稱爲劍橋人臉跟蹤器，是一個用來進行人臉特徵點檢測和頭部姿式估計的C++庫。你能夠看看他在包含的video文件裏工做的多麼好啊！在OSX上編譯這個庫有點兒挑戰可是也不太難。庫依賴於OpenCV3和X11.
有兩個重要的事說明Dlib能夠挑戰CLM-Framework。首先，Dlib比CLM-Framework更快。其次，Dlib的license容許你商用。若是要挑一個的，我會使用Dlib.有趣的是，CLM-Framework依賴於Dlib.
如何編譯CLM-Framework？
編譯CLM-Framework在OSX上有點兒複雜。對於Windows和linux，這裏有一份詳細的說明.爲了在OSX上編譯version 1.3.0,我使用了linux的指示，可是發生了不少改變。
許多依賴項可使用brew安裝.
在文件CMakeLists.txt(以下劃掉的被後面的取代)
find_package( OpenCV 3.0 REQUIRED )
find_package( OpenCV 3.0 REQUIRED HINTS /path/to/opencv )
INCLUDE_DIRECTORIES(${OpenCV_INCLUDE_DIRS})
INCLUDE_DIRECTORIES(/opt/X11/include)
在文件exe/SimpleCLM/SimpleCLM.cpp中
writerFace = VideoWriter(tracked_videos_output[f_n], CV_FOURCC(‘D’,’I’,’V’,’X’), 30, captured_image.size(), true);
writerFace = VideoWriter(tracked_videos_output[f_n], CV_FOURCC(‘M’,’P’,’4′,’V’), 15, captured_image.size(), true);
如何運行CLM-Framework人臉檢測器？
編譯後，可執行文件在bin路徑中.對於視頻中展示的網絡攝像頭Demo，你能夠這樣使用：
bin/SimpleCLM
- 1
Face++ ( FacePlusPlus ) : Web API
人臉特徵點檢測最好的實現之一就是Face++.他們在300 Faces in-the-Wild
Landmark Detection Challenge,2013取得了冠軍。他們提供了一個易用的API。問題是你須要上傳一張圖片到他們的服務器，這個將帶來不少隱私上的擔心。可是若是隱私不是問題的話，Face++是一個好的選擇。你能夠在
http://www.faceplusplus.com/demo-landmark/
看到一個Demo.
Facial Feature Detection Research
許多不一樣的方法均可以用來解決這個問題。很難再博客中對其歸類。我簡單地列出了一些重要論文。 1. Active Appearance Model (AAM) by T. Cootes, G. Edwards and C. J. Taylor. [1998] 2. Face Alignment through Subspace Constrained Mean-Shifts by Jason M. Saragih, Simon Lucey and Jeffrey F. Cohn. [2009] 3. Localizing Parts of Faces Using a Consensus of Exemplars by Peter N. Belhumeur, David W. Jacobs, David J. Kriegman, Neeraj Kumar [ 2011 ] 4. Face Alignment by Explicit Shape Regression by Xudong Cao Yichen Wei Fang Wen Jian Sun [2012] 5. Supervised Descent Method and Its Applications to Face Alignment by Xuehan Xiong and Fernando De la Torre [2013] 6. Constrained Local Neural Fields for robust facial landmark detection in the wild by Tadas Baltrusaitis, Peter Robinson, and Louis-Philippe Morency. [2013] 7. Extensive Facial Landmark Localization with Coarse-to-fine Convolutional Network Cascade by Erjin Zhou, Haoqiang Fan, Zhimin Cao, Yuning Jiang and Qi Yin. [2013] 8. Face alignment at 3000 fps via regressing local binary features by S Ren, X Cao, Y Wei, J Sun. [2014] 9. Facial Landmark Detection by Deep Multi-task Learning by Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. [2014] 10.One Millisecond Face Alignment with an Ensemble of Regression Trees by Vahid Kazemi and Josephine Sullivan. [2014]