Semantic 3D Modeling
In semantic 3D modeling the goal is to find a dense geometric model from images and at the same time also infer the semantic classes of the individual parts of the reconstructed model. Having a semantically annotated dense 3D model gives a much richer representation of the scene than just the geometry. For example questions such as what is the volume of a building can directly be answered. This is difficult with just a geometric model where the knowledge about which parts of the geometry belong to the building is not present. Also by solving the problem of dense 3D reconstruction and class segmentation jointly, prior knowledge such as the ground is usually a surface which is close to horizontal can be included.
Traditionally, volumetric 3D reconstruction is done as a two label problem where each voxel gets label into either being in the free space or in the occupied space. For our semantic formulation we extend the representation to a multi-label problem. A voxel either belongs to the free space or to one out of multiple semantic classes describing the occupied space. While such a formulation is quite memory intensive it allows for a very rich description of the scene. For example also not directly observed surfaces between semantic labels can be represented, for example the ground can extend underneath a building. Priors on the surface orientation, that are learnt from training data, are used to faithfully fill in such surfaces between different semantic labels.
Tutorial: CVPR 2016
Christian Häne, Ľubor Ladický and Marc Pollefeys
Slides
- Part 1 + 3 (Christian): [PDF]
- Part 2 + 4 (Lubor): [PDF]
Tutorial: 3DV 2015
Speakers: Christian Häne and Marc Pollefeys
Slides
Datasets
- Car and bottle datasets from: C. Häne, N. Savinov, M. Pollefeys, Class Specific 3D Object Shape Priors Using Surface Normals, CVPR 2014
[ZIP]
Publications
- N. Savinov, C. Häne, L. Ladicky, M. Pollefeys, Semantic 3D Reconstruction with Continuous Regularization and Ray Potentials Using a Visibility Consistency Constraint, Proc. IEEE Int. Conf. on Computer Vision and Pattern Recognition (CVPR) 2016
[PDF]
- C. Häne, L. Ladicky, M. Pollefeys, Direction Matters: Depth Estimation with a Surface Normal Classifier, Proc. IEEE Int. Conf. on Computer Vision and Pattern Recognition (CVPR) 2015
- R. Karimi, C. Häne, M. Pollefeys, Segment Based 3D Object Shape Priors, Proc. IEEE Int. Conf. on Computer Vision and Pattern Recognition (CVPR) 2015
- N.Savinov, L. Ladicky, C. Häne, M. Pollefeys, Discrete Optimization of Ray Potentials for Semantic 3D Reconstruction, Proc. IEEE Int. Conf. on Computer Vision and Pattern Recognition (CVPR) 2015
- C. Häne, N. Savinov, M. Pollefeys, Class Specific 3D Object Shape Priors Using Surface Normals, Proc. IEEE Int. Conf. on Computer Vision and Pattern Recognition (CVPR) 2014
- C. Häne, C. Zach, A. Cohen, R. Angst, M. Pollefeys, Joint 3D Scene Reconstruction and Class Segmentation, Proc. IEEE Int. Conf. on Computer Vision and Pattern Recognition (CVPR) 2013
|
