Yifei Shi's Homepage

PlaneMatch: Patch Coplanarity Prediction for Robust

RGB-D Reconstruction

Yifei Shi1,2, Kai Xu1,2, Matthias Niessner3, Szymon Rusinkiewicz1, Thomas Funkhouser1,4

1Princeton University, 2National University of Defense Technology,

3Technical University of Munich, 4Google

European Conference on Computer Vision (ECCV) 2018, Oral Presentation

Fig. 1: Scene reconstruction based on coplanarity matching of patches across different views, for both overlapping (left two pairs) and nonoverlapping (right two pairs) patch pairs. The two pairs to the right are long-range, without overlapping. The bottom shows a zoomed-in comparison between our method (left) and key-point matching based method (right).

Abstract We introduce a novel RGB-D patch descriptor designed for detecting coplanar surfaces in SLAM reconstruction. The core of our method is a deep convolutional neural net that takes in RGB, depth, and normal information of a planar patch in an image and outputs a descriptor that can be used to find coplanar patches from other images.We train the network on 10 million triplets of coplanar and non-coplanar patches, and evaluate on a new coplanarity benchmark created from commodity RGB-D scans. Experiments show that our learned descriptor outperforms alternatives extended for this new task by a significant margin. In addition, we demonstrate the benefits of coplanarity matching in a robust RGBD reconstruction formulation.We find that coplanarity constraints detected with our method are sufficient to get reconstruction results comparable to state-of-the-art frameworks on most scenes, but outperform other methods on standard benchmarks when combined with a simple keypoint method.

Fig. 2: An overview of our method. We train an embedding network (c-d) to predict coplanarity for a pair of planar patches across different views, based on the co-planar patches (b) sampled from training sequences with ground-truth camera poses (a). Given a test sequence, our robust optimization performs reconstruction (f) based on predicted co-planar patches (e).
Fig. 3: Network architecture of the local and global towers. Layers shaded in the same color share weights.

 Bibtex       @inproceedings{shi2018planematch, 

                                author = {Yifei Shi and Kai Xu and Matthias Nie{\ss}ner and Szymon Rusinkiewicz and Thomas Funkhouser}, 

                                booktitle = {Proceedings of the European Conference on Computer Vision ({ECCV})}, 

                                title = {PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction}, 

                                year = {2018}