Skin detection in digital images

Face detection has been quite "hot" topic lately. Perhaps one reason is that
modern computers are now powerful enough to process sequential video frames
in real time.

There are many techniques for face detection, one approach is to first classify the pixels to be "skin" or "not skin" pixels. When the "skin" pixels have been found, the noise is removed and pixels are combined to create larger groups. If we are lucky one or more of this kind of pixel groups are part of a face. The pixel group(s) can then be inputted to a face detection algorithm, which makes the final detection for eyes, nose and mouth etc.

Pixel based operations needed for skin color detection are simple, and that is what makes the skin color techniques so attractive.

The skin color seems to be quite easy to spot, "just choose pixels which loosely matches skin color". But actually the problem is more difficult than it first seems. The skin color varies from dark to light and is heavily influenced by lightning.

The lightning conditions can make person's skin color completely different. The lightning color can be yellow, blue or white, and the image can be taken with or without a flash. There can be distracting highlights on skin of the face or the background can contain similar colors as skin.

As many other algorithms, also my algorithm expects images to be shot in normal daylight, without excessive higlights or shadows.

To actually detect a face, there must be some assumptions about the image. The face must be in the image and there should be as little as possible other exposed skin in the image. Suitable images are, for example, images that are coming from the laptop webcams.

To maximize the detection of skin colored pixels, many have proposed to use several detection mechanisms and then combining the results. I am here doing the same thing. In the following, I am presenting three methods for detecting pixels with skin color; using RGB-space, YCrCb-space and HSI-space. 

Note: I am not attaching the full source code here. If there is a huge demand for that, I can do it later on... 

Example images

Here you can see some of the test images downloaded from The USC-SIPI Image Database (http://sipi.usc.edu/database/) and from Kodak (http://r0k.us/graphics/kodak/).

I have ran the skin detection algorithm described below, and the pixels that the algorithm "guesses" to be skin are highlighted with red color.

Skin color cluster in RGB color space

Skin color cluster in RGB color space (R,G,B) is classified as skin if [1]:

R > 95 and G > 40 and B > 20 and
max{R,G,B}-min{R,G,B} > 15 and 
|R-G| > 15 and R > G and R > B

This can be written to a Java function:

// We are assuming here that most pixels in the random image
// are non-skin pixels. So let's try to get out of the function
// as fast as possible to avoid slowest operations, min and max.
public boolean isSkinRGB(int r, int g, int b) {
    // first easiest comparisons
    if ( (r<95) | (g<40) | (b<20) | (r<g) | (r<b) ) {
        return false; // no match, stop function here
    }
    int d = r-g;
    if ( -15<d && d<15) {
        return false; // no match, stop function here
    }
    // we have left most time consuming operation last
    // hopefully most of the time we are not reaching this point
    int max = max(r,g,b);
    int min = min(r,g,b);
    if ((max-min)<15) {
        // this is the worst case
        return false; // no match, stop function 
    }
    // all comparisons passed
    return true;
}

Skin color cluster in YCrCb color space 

First we need to convert RGB color values to YCrCb color values.

To convert RGB color values to YCrCb color values we can use following formula [5]:

Y  = (0.257*R) + (0.504*G) + (0.098*B) + 16 
Cr = (0.439*R) - (0.368*G) - (0.071*B) + 128
Cb = -(0.148*R)- (0.291*G) + (0.439*B) + 128

Using these values we can classify (Y,Cr,Cb) as skin if [3]:

Cr <= 1.5862 × Cb + 20 AND
Cr <= 0.3448 × Cb + 76.2069 AND
Cr <= -4.5652 × Cb + 234.5652 AND
Cr >= -1.15 × Cb + 301.75 AND
Cr >= -2.2857 × Cb + 432.85 AND

This can be written to a Java function:

class YCrCb {

    double Y = 0.0;
    double Cr = 0.0;
    double Cb = 0.0;
    
    public static YCrCb RGB2YCbCr(double R, double G, double B) {
        YCrCb result = new YCrCb();
        result.Y  = (0.257*R) + (0.504*G) + (0.098*B) + 16; 
        result.Cr = (0.439*R) - (0.368*G) - (0.071*B) + 128;
        result.Cb = -(0.148*R)- (0.291*G) + (0.439*B) + 128;
        return result;
    }
}

// Like in RGB case, let's get out of the function as
// fast as possible to avoid unnecessary multiplications. 
public static boolean isSkinYCbCr(double r, double g, double b) {
    YCrCb result = YCrCb.RGB2YCbCr(r, g, b);
    double cr = result.Cr;
    double cb = result.Cb;
    if (cr >= ((1.5862*cb) + 20)) {
        return false;
    }
    if (cr <= ((0.3448*cb) + 76.2069)) {
        return false;
    }
    if (cr <= ((-4.5652*cb) + 234.5652)) {
        return false;
    }
    if (cr >= ((-1.15*cb) + 301.75)) {
        return false;
    }
    if (cr >= ((-2.2857*cb) + 432.85)) {
        return false;
    }
    // all comparisons passed
    return true;
}

Skin color cluster in HSI color space 

For calculating HSI color values, we can use following method [4]:

First we need to normalize (R,G,B):

r = R / R + G + B
g = G / R + G + B
b = B / R + G + B

Then we can use the values:

h = cos^-1((R-G)+(R-B)/2*sqrt((R-G)^2+(R-B)(G-B)))
s = 1-3*min(r,g,b) : 0 <= s <= 1
i = (R + G + B) / (3*255) : 0 <= i <= 1
h = 360 - h, if (B>G) 

This can be written in Java like this:

class HSI {
    double h = 0.0;
    double s = 0.0;
    double i = 0.0;

    public HSI RGB2HSI(int R, int G, int B) {
        HSI result = new HSI();
        result.i = (R+G+B)/3.0;    // we have calculated I!
        if (R==G&&G==B) {
            return result;    // return result with h=0.0 and s=0.0
        }
        double r = R/i;            // normalize R
        double g = G/i;            // normalize G
        double b = B/i;            // normalize B
        double w = 0.5*(R-G+R-B) / Math.sqrt((R-G)*(R-G)+(R-B)*(G-B));
        if (w>1) W = 1.0;       // clip input for acos to -1 <= w <= 1
        if (w<-1) W = -1.0;     // clip input for acos to -1 <= w <= 1
        result.h = Math.acos(w);   // the value is 0 <= h <= Math.PI
        if (B>G) {
            result.h = 2*Math.PI - result.h;
        }
        // finally the last component s
        result.s = 1-3*min(r,g,b);

    }

}

The skin color cluster in HSI space is simply:

H < 25 
H > 230

Thus, we can write skin color detection in HSI-space in Java like this:

public boolean isSkinHSI(double r, double g, double b) {
    HSI hsi = RGB2HSI(r,g,b);
    if (hsi.h<25||hsi.h>230) {
        return true;
    )
    return false;
}

All in one

We can now use all these methods combined to make as good as possible guess about pixels being part of skin or not. It is enough to load an image and make the test for each pixel.

The Java-function to be used to mark skin pixels with color red (0x00FF0000) with any sized BufferedImage could look like this:

public static BufferedImage detect(BufferedImage in) {
    BufferedImage out = new BufferedImage(in.getWidth(),in.getHeight(),BufferedImage.TYPE_INT_RGB);
    int red = 0x00FF0000;
    int[] pixels = new int[in.getWidth()*in.getHeight()];
    in.getRGB(0,0, in.getWidth(), in.getHeight(), pixels, 0, in.getWidth());
    for (int i=0;i<pixels.length;i++) {
        int c = pixels[i];
        int r = (c&0x00FF0000)>>16;
        int g = (c&0x0000FF00)>>8;
        int b = (c&0x000000FF)>>0;
        int a = 0;
        if (isSkinYCbCr(r,g,b)) {
            a+=1;
        }
        if (isSkinRGB(r, g, b)) {
            a+=1;
        }
        if (isSkinHSI(r, g, b)) {
            a+=1;
        }
        if (a==3) { // if all the methods respond "true", mark the pixel
            pixels[i]= red;
        }
    }
    out.setRGB(0,0, in.getWidth(), in.getHeight(), pixels, 0, in.getWidth());
    return out;
}    

Face detection

This algorithm can be used to detect face, or at least give quite good guess where the face could be, when attached to the image capturing software like in my post: Capturing webcam image with Java Media Framework.

References and further reading:

[1]
Jure Kovaˇc, Peter Peer, and Franc Solina
Human Skin Colour Clustering for Face Detection
http://www.lrv.fri.uni-lj.si/~peterp/publications/eurocon03.pdf

[2]
Vladimir Vezhnevets, Vasili Sazonov, Alla Andreeva 
A Survey on Pixel-Based Skin Color Detection Techniques
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.5.521&rep=rep1&type=pdf

[3]
Nusirwan Anwar bin Abdul Rahman, Kit Chong Wei and John See
RGB-H-CbCr Skin Colour Model for Human Face Detection
http://pesona.mmu.edu.my/~johnsee/research/papers/files/rgbhcbcr_m2usic06.pdf

[4]
Conversion from RGB to HSI
Ruye Wang
http://fourier.eng.hmc.edu/e161/lectures/color_processing/node3.html

[5] 
RGB to YCbCr 
http://www.fourcc.org/fccyvrgb.php