Face Tracking by Skin Color Segmentation

Face tracking from image frames or videos may be done by finding a way to segment the skin from the image frames. In this activity, the segmentation of the skin was done using non parametric color image segmentation. Parametric and non parametric color image segmentation were already done in the previous activity. You can find it here for more detailed implementation of the color image segmentation.

Initially, we captured a video of ourselves as we move from one lighting condition to the other. It is observed that the color of the skin changes for different lighting conditions. Some frames in the video was chosen to represent each of the lighting condition. In the method, skin patches were cropped under the different lighting condition and the histogram of the skin patch was plotted in rg normalized chromaticity space. I first obtain the histogram of the skin under sunny lighting condition, as shown in the figure below. 

Histogram of skin under sunny lighting condition.  

This histogram was used to segment the skin on the background. Histogram backprojection was used to replace the value of every pixel location of the image frame by the corresponding value of its location on the histogram. After backprojecting, the resulting image was thresholded with just a small value of 0.005 to remove those pixels that have very small possibility of belonging to the skin segment, Using blob analysis, bounding box was placed around the blob that has the largest area. The result of the backprojection on an image frame, the thresholded image, and the image that has a bounding box representing the largest blob area are shown in the figure below. 

The bounding box representing the largest blob will also serve as the detected face during the face tracking (since, of course, our face consists of mostly skin). The histogram of the skin under the sunny lighting condition was used to image frames that are under other lighting conditions. The figure below illustrate the resulting face track and as predicted, the face was not able to recognize on the cloudy and/or dark lighting condition. Also, we note that the face that was tracked under the fluorescent lighting condition seems smaller. 

We now consider the histogram of the skin under different lighting conditions including sunny, cloudy, fluorescent and dark. The histogram is presented below and it was observed that the skin locus now covers a big area on the 2D histogram plot. 

Histogram of skin under differen lighting conditions.

The resulting face tracks were now also to recognize the face under different lighting condition and the bounding boxes were observed to be able to enclose the entire face (or the entire skin patch) on the image frames under different light conditions. 

The final output of the face tracking of the image frames in the whole video is now presented below. 

Final Output

--
Reference
[1] Soriano, Maricor, et al. "Adaptive skin color modeling using the skin locus for selecting training pixels." Pattern Recognition 36.3 (2003): 681-690.

Texture Recognition

One way to analyze the texture pattern in an image is through Local Binary Pattern (LBP). The computation of the LBP is done as follows:

(1) for every pixel location, get the difference of the center pixels from each of its eight neighboring pixel.
(2) set a threshold in which all the negative values are zeroes, otherwise ones.
(3) set a fixed weight for each of the 8 position of the neighbor, that is, $2^{n-1}$ for the nth neighbor where the movement is in clockwise direction.  Multiply this weight to the corresponding thresholded values.
(4) compute the sum of these products. The sum is the LBP value for that neighborhood.

The summary of the computation of LBP is illustrated in the figure below

Three texture classes that were considered for this activity are sand, sponge and wood. At least five samples for each class were considered. The first set of texture samples is shown below. 

Sand texture samples in grayscale. 

Information about the texture are presented here through histogram. The LBP histogram for each sample of the sand texture is shown below. Aside from the peak at the very dark and the very light values or the two ends of the histogram, the other bins don't have drastic difference in values. This histogram pattern was consistent for all five sand samples. 

LBP histogram of five sand texture samples.

Sponge texture samples

Sponge texture samples in grayscale.

The LBP histogram for each sample of the sponge texture. Ideally, different texture have different histogram. The difference of the sand LBP histogram between the sponge LBP histogram are easily noticeable. The histogram of the sponge texture as shown below have sudden drop and rise in their values but this is relatively the same result for all five sponge textures. 

Wood texture samples

Wood texture samples in grayscale.

Lastly, the LBP histogram for each sample of the wood textures was acquired. It was observed from the figure below that the three fork shape at the middle was prominent for all the histograms. But for other part, we can reasonably say that they are similar to each other. 

But for strong argument sake, I presented below their corresponding histograms using smaller bins for this set of texture. Now, we can observed the similarity between the five histograms better. The middle values have relatively negligible density as compared to the rest where the the right side seems to have heavier weight than the left side. This is different from the other textures because the LBP histograms of sand and sponge have pretty much symmetric right and left sides.

References

[1] Matti Pietikäinen. Local Binary Patterns. 2010. http://www.scholarpedia.org/article/Local_Binary_Patterns

Invariant moments

Computing the moments of invariant is a way to recognize the visual patterns and characters independent on its position, size and orientation[1-2].

The moments of invariant of images were computed in this activity. We have a set of images which consists of the original image, rotated image and scaled image. Consider first a set of grayscale images as shown below.

Set of grayscale images. Original image, rotated image, and scaled image (from left to right) . 

The 2D moment of order (p+q) of an image was given by
$$m_{pq} = \sum_{x=0}^{N-1} \sum_{y=0}^{M-1} x^p y^q f(x,y)$$
The central moment is then given by
$$\mu_{pq} = \sum_{x=0}^{N-1} \sum_{y=0}^{M-1} (x-\bar{x})^p (y-\bar{y})^q f(x,y)$$
where $\bar{x}$ and $\bar{y}$ are calculated using the following equations
$$\bar{x} = \frac{m_{10}}{m_{00}}; \bar{y} = \frac{m_{01}}{m_{00}}$$
The normalized central moments $\eta_{pq}$ is now computed as
$$\eta_{pq} = \frac{\mu_{pq}}{\mu_{00}^\gamma}; \gamma = \frac{p+q}{2} +1$$

From the second and third order of moments, a set of seven invariant moments was derived.

In matlab, the computation is done using for loop to access each pixel position and the value of the image in that position. The calculated invariant moments for the grayscale image set are summarized in the table below. For each invariant moment, the values are nearly the same for the three images. The closely similar values of the invariant moments just justifies that the images are recognized to have the same pattern regardless of the rotation and the size.

The same procedure was done for other set of images. Unlike the grayscale images which have an image function $f$ that ranges from 0 to 255, the succeeding sets of images are binary images which have values that are only either 0 or 1. Edge images are like Dirac delta function which has peak values (equal to 1) located at the edge otherwise, the values were zero. 

I applied the same procedure first for my chromosome edge image in the last activity but the resulting invariant moment values were not that similar to each other. When discussing with Anjali, she thought that maybe there were too little details that this approach wasn't able to recognize the patterns. So I tried other edge image that have a many details as shown in the figure below. 

The values of the moments invariant for the set of edge image are tabulated below. Now, the values for each moment invariant were observed to be closely similar. And since the edge images are mostly zeros in values, the invariant moment values are also smaller as compared to that of the grayscale set of images. The eight invariant moments for this set is just in the range of [0.2, 7.4].

Lastly, the moments invariants for the set of synthetic symmetric images were also computed. They were like step function which have zero values for the background, then they have values of ones when they pass through the boundary of their shape. You may refer to the figure below.

Again, I just first tried the method for simple images but they resulted to not so similar vaules. Also, this computation of invariant moments by Hu tends to be sensitive to symmetry. That's why another way of computing moment invariant that considers symmetry was studied by Flusser and Suk[2]. Table below shows the invariant moments for the set of edge images. There are only nice results for the first three moments but the rest have relativity not similar values.

So again,  I choose a more detailed set of images and their invariant moments resulted to closely equal values. The set of more detailed symmetric synthetic images and their corresponding moment invariants are shown below.

---

Reference

[1] Hu, Ming-Kuei. "Visual pattern recognition by moment invariants." Information Theory, IRE Transactions on 8.2 (1962): 179-187.
[2] Flusser, Jan, and Tomás Suk. "Rotation moment invariants for recognition of symmetric objects." IEEE Transactions on Image Processing 15.12 (2006): 3784-3790.
[3] Gonzalez, Rafael C. Digital image processing. Chapter 11, Pearson Education India, 2009.