ETH Zurich - D-INFK - IVC - CVG - Research - Learning Priors for Semantic 3D Reconstruction

Learning Priors for Semantic 3D Reconstruction

Ian Cherabier, Johannes L. Schönberger, Martin R. Oswald,
Marc Pollefeys, and Andreas Geiger

Department of Computer Science, ETH Zürich, Switzerland,
MPI-IS and University of Tübingen
ECCV 2018


We present a novel semantic 3D reconstruction framework which embeds variational regularization into a neural network. Our net- work performs a fixed number of unrolled multi-scale optimization iterations with shared interaction weights. In contrast to existing variational methods for semantic 3D reconstruction, our model is end-to-end trainable and captures more complex dependencies between the semantic labels and the 3D geometry. Compared to previous learning-based approaches to 3D reconstruction, we integrate powerful long-range dependencies using variational coarse-to-fine optimization. As a result, our network architecture requires only a moderate number of parameters while keeping a high level of expressiveness which enables learning from very little data. Experiments on real and synthetic datasets demonstrate that our network achieves higher accuracy compared to a purely variational approach while at the same time requiring two orders of magnitude less iterations to converge. Moreover, our approach handles ten times more semantic class labels using the same computational resources.


  • Learning Priors for Semantic 3D Reconstruction.
    Ian Cherabier*, Johannes L. Schönberger*, Martin R. Oswald, Marc Pollefeys, and Andreas Geiger.
    European Conference on Computer Vision (ECCV), 2018. (*Equal contribution)


© CVG, ETH Zürich webmaster: