The ChaLearn gesture dataset (CGD 2011)

Abstract

This paper describes the data used in the ChaLearn gesture challenges that took place in 2011/2012, whose results were discussed at the CVPR 2012 and ICPR 2012 conferences. The task can be described as: user-dependent, small vocabulary, fixed camera, one-shot-learning. The data include 54,000 hand and arm gestures recorded with an RGB-D $\hbox {Kinect}^\mathrm{TM}$camera. The data are organized into batches of 100 gestures pertaining to a small gesture vocabulary of 8–12 gestures, recorded by the same user. Short continuous sequences of 1–5 randomly selected gestures are recorded. We provide man-made annotations (temporal segmentation into individual gestures, alignment of RGB and depth images, and body part location) and a library of function to preprocess and automatically annotate data. We also provide a subset of batches in which the user’s horizontal position is randomly shifted or scaled. We report on the results of the challenge and distribute sample code to facilitate developing new solutions. The data, datacollection software and the gesture vocabularies are downloadable from http://gesture.chalearn.org. We set up a forum for researchers working on these data http://groups.google.com/group/gesturechallenge.

Appendix

1.1 Results by challenge participant

We used the code provided by 15 top ranking participants in both challenge rounds to compute performances on the validation and final evaluation sets (Table 5). We also provide results on 20 other batches selected for our translation experiments. Untranslated data are referred to as “utran” and translated data as “tran”. Details on the methods employed by the participants are found in reference [24] and on the website of the challenge.

1.2 Development data lexicons

The development data were recorded using a subset of thirty lexicons (Table 6). They were recorded at least 11 times each by different users. We list in Table 7 the lexicons used for validation and final evaluation data. Note that some validation lexicons are also present in development data but that the final evaluation data include only new lexicons found in no other sets.

Table 6 Lexicons recorded in development data

Table 7 Results by batch. This table lists for validation and final evaluation data batches their lexicon, identity of the subject (user) that recorded the data, and performance of recognition of 15 top ranking participants in round 1 and 2. The performance score is the average generalized Levenshtein distance, which is analogous to an error rate. Best is the lowest score, Mean is the average score and Std is the standard deviation

1.3 Results by data batch

We show in Table 7 the performances by batch. We computed the best and average performance over 15 top ranking participants in round 1 and 2: Alfnie1, Alfnie2, BalazsGodeny, HITCS, Immortals, Joewan, Manavender, OneMillionMonkeys, Pennect, SkyNet, TurtleTamers, Vigilant, WayneZhang, XiaoZhuWudi, and Zonga.

1.4 Depth parameters

We also provide the parameters necessary to reconstruct the original depth data from normalized values (Table 8).

Table 8 Depth parameters. This table lists for validation and final evaluation data batches the parameters necessary to approximately reconstruct the original depth values from the normalized values. This is achieved by averaging the R,G, and B values to get a value v, then perform v/255*(MaxDepth$-$MinDepth) +MinDepth. The depth parameters for the development data are found on the website of the challenge. “Date” is the data at which the data batch was recorded in YY MM DD HH MM format. “DephRes” is the number of unique levels of depth