Efficient Hierarchical Graph-Based Segmentation of RGBD Videos

Abstract

We present an efficient and scalable algorithm for seg- menting 3D RGBD point clouds by combining depth, color, and temporal information using a multistage, hierarchical graph-based approach. Our algorithm processes a moving window over several point clouds to group similar regions over a graph, resulting in an initial over-segmentation. These regions are then merged to yield a dendrogram us- ing agglomerative clustering via a minimum spanning tree algorithm. Bipartite graph matching at a given level of the hierarchical tree yields the final segmentation of the point clouds by maintaining region identities over arbitrarily long periods of time. We show that a multistage segmentation with depth then color yields better results than a linear com- bination of depth and color. Due to its incremental process- ing, our algorithm can process videos of any length and in a streaming pipeline. The algorithm’s ability to produce robust, efficient segmentation is demonstrated with numer- ous experimental results on challenging sequences from our own as well as public RGBD data sets.

Authors

Steven Hickson
College of Computing,
Georgia Tech
shickson [at] gatech.edu
Stan Birchfield
Microsoft
stanleyb [at] microsoft.com
Irfan Essa
College of Computing,
Georgia Tech
irfan [at] cc.gatech.edu
Henrik Christensen
College of Computing,
Georgia Tech
hic [at] cc.gatech.edu

Paper

Download (PDF, 2.18MB)

Video

Poster

Download (PDF, 9.01MB)

Code/Data

The code requires OpenCV and PCL and should be compatible in Windows and Linux with the most time-of-flight and projected infrared devices. It can be downloaded at https://github.com/StevenHickson/4D_Segmentation

Acknowledgement

The authors would like to thank Prof. Mubarak Shah and his PhD Student Gonzalo Vaca-Castano for their mentorship and guidance to the primary author of this paper, when he participated in the National Science Foundation funded ” REU Site: Research Experience for Undergraduates in Computer Vision” (#1156990) in 2012 at University of Central Florida’s Center for Research in Computer Vision . In addition, we would also like to thank TUM and NYU for providing datasets. Details and links forthcoming.

Citation

  • [DOI] S. Hickson, S. Birchfield, I. Essa, and H. Christensen, “Efficient hierarchical graph-based segmentation of rgbd videos,” in Computer vision and pattern recognition (cvpr), 2014 ieee conference on, 2014, pp. 344-351.
    [Bibtex]
    @INPROCEEDINGS{2014-Hickson-EHGSRV,
    author={Hickson, Steven and Birchfield, Stan and Essa, Irfan and Christensen, Henrik},
    booktitle={Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on},
    title={Efficient Hierarchical Graph-Based Segmentation of RGBD Videos},
    year={2014},
    month={June},
    pages={344-351},
    abstract={We present an efficient and scalable algorithm for segmenting 3D RGBD point clouds by combining depth, color, and temporal information using a multistage, hierarchical graph-based approach. Our algorithm processes a moving window over several point clouds to group similar regions over a graph, resulting in an initial over-segmentation. These regions are then merged to yield a dendrogram using agglomerative clustering via a minimum spanning tree algorithm. Bipartite graph matching at a given level of the hierarchical tree yields the final segmentation of the point clouds by maintaining region identities over arbitrarily long periods of time. We show that a multistage segmentation with depth then color yields better results than a linear combination of depth and color. Due to its incremental processing, our algorithm can process videos of any length and in a streaming pipeline. The algorithm's ability to produce robust, efficient segmentation is demonstrated with numerous experimental results on challenging sequences from our own as well as public RGBD data sets.},
    keywords={Histograms;Image color analysis;Image segmentation;Spatiotemporal phenomena;Three-dimensional displays;Tin;Videos;4D Segmentation;Point Cloud Segmentation;Segmentation;grouping and shape representation},
    doi={10.1109/CVPR.2014.51},
    url_link={http://dx.doi.org/10.1109/CVPR.2014.51}
    }

Copyright

The documents contained in these directories are included by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author’s copyright. These works may not be reposted without explicit permission of the copyright holder.

Posted in CVPR.