Hand Detection Using Cascade of Softmax Classifiers

3.1. Multiclass Hand Posture Classification by Softmax Regression

3.2. Softmax-Based Cascade Architecture for Human Hand Detection

In the experimental part, k₀ is set at 2. For ease of understanding, the flowchart for the window image classification is provided in Figure 1.

3.3. Multiresolution HOG Feature for Different Stage-Classifiers

For sliding-window-based hand detection, there are tens of thousands window images to be classified in single frame, which makes the detection system lack efficiency. To improve the efficiency, here in this work, the multiresolution HOG features are adopted for posture representations [24]. The cascade is designed so that the HOG features with low resolutions are utilized by classifiers of lower stage-levels, and HOG features with high resolutions are utilized by classifiers of higher stage-levels. The varied feature resolutions can be achieved by adjusting the density of cell splits in window images as discussed in [24]. With such multiresolution scheme, a large number of background windows can be excluded by the classifiers using low-resolution HOG features. And only few difficult background windows need to be further classified by the HOG features of high resolutions which are more discriminative and more computationally costly. In this way, the detection speed can be greatly improved without sacrificing the detection accuracy. Concretely, let J_k denote the time consumption for single window classification with B_k(·), and denote the percentage of windows through the kth stage as follows: ρ_k = number of windows through the first k stage-classifiers/number of all windows generated from the full-sized image. Then, based upon the proposed multiresolution and cascade scheme, the average time expense for classifying one window image is $$. However, if the detection system adopts a single softmax with HOG features of the highest resolution, the time expense would be J_K+1, which is usually several times as much as E₁.

To promote the understanding, details of the training process for the proposed method are described in Algorithm 1. In Step (1), the training data is prepared and some hyperparameters are defined to control the training process. In Step (2), the first stage-classifier is trained, while the rest of stage-classifiers are trained one by one in Step (3). During training of the first stage, the initial N_t negatives are randomly cropped, and all the rest are acquired using hard example mining techniques (Step (2.4)). Such strategy could enhance the discriminative ability of the first stage-classifier. For stage larger than 1, all the N_T negatives are directly mined based upon the previous stage-classifier (Step (3.2)). In the kth stage, the multiclass softmax model is firstly learned, and then based upon the predefined pass-rate hyperparameters β_k, the modified SftB classifiers B_k(·) and C_k(·) can be generated. Once the stage reaches the predefined K + 1, the procedure could stop and return the set of cascade components $$.

4.1. Datasets and Experimental Settings

4.1.1. Datasets

The experimental dataset comprises four predefined posture categories. For each category, there are around 2000 positive examples with normalized size of 80 × 80 pixels. The samples are obtained by cropping hand regions from the full images that are collected from ten subjects under various backgrounds and lighting conditions. The negative samples are generated during training process by randomly cropping image regions from 500 extra complicated pictures of full size. These full-sized images comprise various undefined hand postures but contain no hand posture of predefined categories. Except for the training samples, we also prepare 4000 full-sized images to evaluate the performance of the proposed method, and each image contains at least one predefined posture instance. Examples for the defined posture categories are presented in Figure 2.

4.1.2. Experimental Settings

In the experiment, training samples are normalized into the resolution of 80 × 80 pixels. HOG features of various resolutions are utilized for classification, where different resolutions are achieved by adopting different cell splits. Cell splits for the adopted 3 resolutions are illustrated in Figure 3. Parameters for HOG features of all resolutions are fixed as unsigned gradient orientation, 9 equally distributed angle bins, 2 × 2 cells per block, and block steps equaling to cell size. Totally four stages-classifiers are incorporated into the softmax structure. The first three are SftB classifiers, and the last one is SftM classifier. Feature configuration for each stage-classifier is presented in Table 1. In addition, to improve the detection efficiency, changing window size is employed for multiscale search rather than resizing the image itself (e.g., we could take window size of 64 × 64, 80 × 80, 96 × 96, and 120 × 120 to detect hands of different scales in the frame. For window size of s × s, the region of cell c(i, j) will be taken as [x + x1, x + x2, y + y1, y + y2], where (x, y) are the top left coordinates of this window image, x1 = floor((i − 1)∗s/rc) + 1, x2 = ceil(i∗s/rc), y1 = floor((j − 1)∗s/rc) + 1, y2 = ceil(j∗s/rc), and rc is the cell number at horizontal or vertical direction (totally rc∗rc cells as shown in Figure 3). To sum up, the cell size changes with the window size. Although such calculation for cell location is not so accurate when s is not divisible by rc, the feature is still effective. In video-based detection, if the application scenario requires the users to be near the camera, the window sizes should be larger, while if the users are required to stay far away from the camera, the window sizes should be smaller) and the window step is set as 0.05 times of the window size. For live hand detection, the web-camera is set so that image with 320 × 240 resolution could be captured. All experiments are conducted on a PC equipped with Intel(R) Pentium(R) G3220 @3.00GHz CPU, 4.00GB RAM, and under the visual studio 2013 platform.

Table 1. The feature configuration for each cascade stage.

Cascade stage	Cell split	Block layout	Feature dimension	Output domain
Stage 1	5×5	4×4	576	{0,1}
Stage 2	8×8	7×7	1764	{0,1}
Stage 3	10×10	9×9	2916	{0,1}
Stage 4	10×10	9×9	2916	{0, ⋯, P}

4.2. Effectiveness of the Proposed SftB Classifiers

To evaluate the proposed SftB classifier, we, respectively, use the softmax and logistic regression (LR) techniques to train the first three binary stage-classifiers to produce the final four-stage cascade. During the SftB cascade training period, all samples prepared for the pth stage-classifiers C_(p,SftB)(·) are divided into the training set $$ and the testing set $$. C_(p,SftB)(·) is learned from dataset $$, and ROC curve for C_(p,SftB)(·) is calculated based on the testing set $$ (the ROC describes the variation relation between false positive rates (FPR) and true positive rates (TPR). Different TPR of $$ are achieved by adjusting the value of threshold ζ_k. And varying ζ_k can in return produce varying FPR on $$. In this way, the ROC curve for SftB can be produced). Similarly, we can train C_(p,LR)(1 ≤ p ≤ 3). In this way, totally six ROC curves are produced based upon $$. In addition, an extra ROC curve is also generated for a SftB classifier based upon $$ and using HOG features of the first resolution. All the seven ROC curves are displayed in Figure 4, where the notations “stage2&Reso2&LR” and “stage2&Reso2&SftB,” respectively, represent the LR and SftB classifiers trained with HOG features of the second resolution. Other notations can be explained in a similar way.

From Figure 4, we can see that, with the same HOG resolution and for fixed TPR (Table 4), the FPR (Table 4) under SftB classifier is much smaller than that calculated with LR classifier. This is because that the SftB is modified from a multiclass classifier, which essentially provides the decision boundaries among different posture categories and therefore can decompose the complex space formed by multiclass posture examples. Moreover, we find that the classifier “Stage2&Reso1&SftB” seriously underperforms the others, which indicates that increasing the resolution of HOG features is crucial to guarantee the classification accuracy.

In addition, the histograms for outputs from $$ (see (8)) are calculated and presented in Figure 5, so that more knowledge can be gained about the proposed softmax-based binary classification. In the illustration, the upper histogram is calculated based upon the background examples and the bottom one is calculated based upon the predefined hand posture examples.

4.3. Effectiveness of the Proposed Softmax-Based Cascade Detector

The four confusion matrices corresponding to the four detectors are presented in Table 2, where the Softmax+Resolution1, Softmax+Resolution2, and Softmax+Resolution3, respectively, represent the confusion matrix computed from the three noncascade detectors. Note that the confusion matrix here is different from that for classification problem. In fact, for sliding-window-based detection, one target instance may be covered by many windows, and the postprocessing is only applied to windows that are classified into the same category. As a result, one region can be finally predicted into more than one posture category. For this reason, the sum of elements in each row does not necessarily equal to one.

From Table 2, we can see that the hand detection with noncascade softmax detectors may cause high false detection rates at the background areas and high confusion rates among different posture categories. By contrast, the proposed softmax-based cascade could significantly suppress all kinds of false detections without sacrificing the recall rates. This is because the complexity of background space can be effectively decomposed by the usage of multiple stage-classifiers, and therefore it becomes much easier for the final multiclass softmax model to discriminate among the predefined postures and the minorities of remaining backgrounds.

To make more direct and intuitional comparisons, multiple performance values based on summary measures are also computed and provided in Table 3. The measures mean recall rate and mean correct rate, respectively, represent the averaged recall rates and the averaged confusion rates among the four predefined posture categories. For the definition of FPPI and mean correct rate, please refer to Table 4.

From Table 3, we can see that the detection accuracy with Softmax+Resolution3 is the highest among the three noncascade classifiers. However, by comparison, the proposed multiclass cascade detector further improves the mean recall rate from 0.9225 to 0.9448 and boosts the mean correct rate from 0.5155 to 0.8475. Meanwhile, the mean confusion rate is reduced from 0.2182 to 0.0515, and the FPPI is reduced from 0.2248 to 0.0169. In addition, the proposed detector is faster than Softmax+Resolution3 by almost 4 times.

Figure 6 shows some hand posture detection result based on a normal web-camera. From the results, we can see that the proposed method can detect the defined hand postures under various environments. And the system can reach a real-time running speed of 27 FPS under our experimental setup.

4.4. The Influences of the Settings for Posture Example Pass-Rates

Performance of the proposed cascade is directly affected by the thresholds ζ_k of its stage-classifiers as shown in (8). The thresholds affect not only the detection results but also the training process, since the background samples for the pth stage are acquired by the previous (p-1) stage-classifiers. These thresholds are determined based upon the settings for pass-rates of posture samples (for H_k(·), $$ could be computed based upon the posture examples set $$ which is used for learning Θ_k. Sort $$ in ascending order to produce vector $$, and take the value ξ_k = χ(floor((1 − β_k)N_ps)) as threshold, where β_k are the preset posture examples pass-rates for the kth stage SftB during training period) which are set at the training stage to control the training process. To acquire better cascade detector, we prepare multiple groups of settings for the pass-rates and then train the four-stage cascade classifier with each group of settings. After that, the FPPW and detection rate (Table 4) are computed based upon each of these cascade detectors, and the best group of settings is selected by comparing the values of all FPPW and detection rates. Note that the detection rate does not necessarily equal to the mean correct rate, since confusion detections may exist among different posture categories.

The six groups of pass-rates being compared are, respectively, [97%#98%#99%], [99%#98%#97%], [94%#96%#98%], [98%#96%#94%], [95%#97%#99%], and [99%#97%#95%]. The notation [97%#98%#99%] means that, for the first three stage-classifiers, the pass-rate of posture examples is successively set to 97%, 98%, and 99%. Each group contains exact three pass-rate settings because there are exact four stage-classifiers in each cascade detector, while the fourth stage is a multiclass softmax model that will not be modified. The curves for variation relations of FPPW with the stage-level are presented in Figure 7, and the detection rates are illustrated in Figure 8. Except that FPPW and detection rate are both increasing with the product of three pass-rates, we have another important observation. That is, when the product of the three pass-rates is fixed, the detection rate in Case1 (Table 4) is significantly higher than that in Case2 (Table 4), while the FPPW in both cases are very close. This indicates that the detectors trained in Case1 are more discriminative than those trained in Case2. This observation suggests that, to achieve good performance, it is better to set low pass-rates for classifiers at low stages and set higher pass-rates for classifiers at higher stages.