Semantic Labeling and Instance Segmentation of 3D Point Clouds using Patch Context Analysis and Multiscale Processing

Shi-Min Hu

Jun-Xiong Cai

Yu-Kun Lai


A example of semantic labeling and instance segmentation.

Abstract

We present a novel algorithm for semantic segmentation and labeling of 3D point clouds of indoor scenes, where objects in point clouds can have significant variations and complex configurations. Effective segmentation methods decomposing point clouds into semantically meaningful pieces are highly desirable for object recognition, scene understanding, scene modeling, etc. However, existing segmentation methods based on low-level geometry tend to either under-segment or over-segment point clouds. Our method takes a fundamentally different approach, where semantic segmentation is achieved along with labeling. To cope with substantial shape variation for objects in the same category, we first segment point clouds into surface patches and use unsupervised clustering to group patches in the training set into clusters, providing an intermediate representation for effectively learning patch relationships. During testing, we propose a novel patch segmentation and classification framework with multiscale processing, where the local segmentation level is automatically determined by exploiting the learned cluster based contextual information. Our method thus produces robust patch segmentation and semantic labeling results, avoiding parameter sensitivity. We further learn object-cluster relationships from the training set, and produce semantically meaningful object level segmentation. Our method outperforms state-of-the-art methods on several representative point cloud datasets, including S3DIS, SceneNN, Cornell RGB-D and ETH.

Paper

TVCG 2018

Overview

ETH RGBD Dataset

Raw Ply data with xyz and normal

PCD data with xyz, normal, label and obeject.

For the format and input/output of pcd file, you can refer to pcl library.

Other datasets

Cornell RGBD dataset

SceneNN dataset

S3DIS dataset

Citation

@article{hu2018semantic,
		author = {Hu, Shi-Min and Cai, Jun-Xiong and Lai, Yu-Kun},
		title = {Semantic Labeling and Instance Segmentation of 3D Point Clouds using Patch Context Analysis and Multiscale Processing},
  		journal={IEEE transactions on visualization and computer graphics},
		year = {2018}
		publisher={IEEE}
		}

Acknowledgements

This work was supported by the National Key Technology R&D Program (Project Number 2017YFB1002604), the Natural Science Foundation of China (Project Number 61521002, 61761136018) and Tsinghua-Tencent Joint Laboratory for Internet Innovation Technology. We would like to thank the authors and contributors of the ETH dataset, the Cornell RGB-D dataset, the SceneNN dataset and the S3DIS dataset for making the datasets available.