Face Tracking by Skin Color Segmentation

Face tracking from image frames or videos may be done by finding a way to segment the skin from the image frames. In this activity, the segmentation of the skin was done using non parametric color image segmentation. Parametric and non parametric color image segmentation were already done in the previous activity. You can find it here for more detailed implementation of the color image segmentation.

Initially, we captured a video of ourselves as we move from one lighting condition to the other. It is observed that the color of the skin changes for different lighting conditions. Some frames in the video was chosen to represent each of the lighting condition. In the method, skin patches were cropped under the different lighting condition and the histogram of the skin patch was plotted in rg normalized chromaticity space. I first obtain the histogram of the skin under sunny lighting condition, as shown in the figure below. 

Histogram of skin under sunny lighting condition.  

This histogram was used to segment the skin on the background. Histogram backprojection was used to replace the value of every pixel location of the image frame by the corresponding value of its location on the histogram. After backprojecting, the resulting image was thresholded with just a small value of 0.005 to remove those pixels that have very small possibility of belonging to the skin segment, Using blob analysis, bounding box was placed around the blob that has the largest area. The result of the backprojection on an image frame, the thresholded image, and the image that has a bounding box representing the largest blob area are shown in the figure below. 

The bounding box representing the largest blob will also serve as the detected face during the face tracking (since, of course, our face consists of mostly skin). The histogram of the skin under the sunny lighting condition was used to image frames that are under other lighting conditions. The figure below illustrate the resulting face track and as predicted, the face was not able to recognize on the cloudy and/or dark lighting condition. Also, we note that the face that was tracked under the fluorescent lighting condition seems smaller. 

We now consider the histogram of the skin under different lighting conditions including sunny, cloudy, fluorescent and dark. The histogram is presented below and it was observed that the skin locus now covers a big area on the 2D histogram plot. 

Histogram of skin under differen lighting conditions.

The resulting face tracks were now also to recognize the face under different lighting condition and the bounding boxes were observed to be able to enclose the entire face (or the entire skin patch) on the image frames under different light conditions. 

The final output of the face tracking of the image frames in the whole video is now presented below. 

Final Output

--
Reference
[1] Soriano, Maricor, et al. "Adaptive skin color modeling using the skin locus for selecting training pixels." Pattern Recognition 36.3 (2003): 681-690.