Motion Capture of Hands in Action using
Discriminative Salient Points

European Conference on Computer Vision (ECCV) 2012

Abstract

Capturing the motion of two hands interacting with an object is a very challenging task due to the large number of degrees of freedom, self-occlusions, and similarity between the fingers, even in the case of multiple views observing the scene. In this paper we propose to use discriminatively learned salient points on the fingers and to estimate the finger-salient point associations and the hand poses at the same time. We introduce a differentiable objective function that also takes edges, optical flow and collisions into account.

Our qualitative and quantitative results show that the proposed approach achieves very accurate results for several challenging sequences containing hands in action.

Paper

Original sequences

Original datasets used in the paper (compressed using LJPG):

		Finger Tips Touching and Praying		[01, 02, 03, 04, 05, 06, 07, 08]    [Calibration]    [Results]
		Fingers Crossing and Twisting		[01, 02, 03, 04, 05, 06, 07, 08]    [Calibration]    [Results]
		Fingers Folding		[01, 02, 03, 04, 05, 06, 07, 08]    [Calibration]    [Results]
		Fingers Walking		[01, 02, 03, 04, 05, 06, 07, 08]    [Calibration]    [Results]
		Holding and Passing a Ball		[01, 02, 03, 04, 05, 06, 07, 08]    [Calibration]    [Results]
		Taking off a Ring		[01, 02, 03, 04, 05, 06, 07, 08]    [Calibration]    [Results]
		Paper Folding		[01, 02, 03, 04, 05, 06, 07, 08]    [Calibration]    [Results]
		Rope Folding		[01, 02, 03, 04, 05, 06, 07, 08]    [Calibration]    [Results]
		Models		[Hands]    [Ball]    [Ring]    [Paper]    [Rope]    [Hand 3DMax]
		File format		[FileFormat.txt]    [c++ bone struct code]
		GroundTruth		[Holding and Passing a Ball]

Binaries

To view the results in full 3D download the viewer binaries below:

		Software to visualize the results in 3D		[ZIP]
		3DS Max Exporter		[ZIP]

BibTex reference

@INPROCEEDINGS{InteractingHands12,
   author = "Luca Ballan and Aparna Taneja and Jürgen Gall and Luc Van Gool and Marc Pollefeys",
   title = "Motion Capture of Hands in Action using Discriminative Salient Points",
   booktitle = "European Conference on Computer Vision (ECCV)",
   month = "October",
   year = "2012",
   address = "Firenze"
   pages = "640--653"
}


Related works:

Acquiring Shape and Motion of Interacting People from Videos L. Ballan [PDF] [web] [video] [bibtex] PhD's thesis, Department of Information Engineering, University of Padova, 2009

Marker-less Motion Capture of Skinned Models in a Four Camera Set-up using Optical Flow and Silhouettes L. Ballan and G. M. Cortelazzo [PDF] [web] [video] [bibtex] 3DPVT 2008, Atlanta, GA, USA

Acknowledgments

The research leading to these results has received funding from the ERC under the EC’s Seventh Framework Programme (FP7/2007-2013) / ERC grant #210806, from the Swiss National Science Foundation, and from Microsoft's ICES.