RepFinder: Finding Approximately Repeated Scene Elements for Image Editing

Ming-Ming Cheng¹ Fang-Lue Zhang¹ Niloy J. Mitra² Xiaolei Huang³ Shi-Min Hu¹

¹TNList, Department of Computer Science and Technology, Tsinghua University, Beijing
²KAUST/IIT Delhi ³Lehigh University

Abstract

Repeated elements are ubiquitous and abundant in both manmade and natural scenes. Editing such images while preserving the repetitions and their relations is nontrivial due to overlap, missing parts, deformation between instances, illumination variation, etc. Manually enforcing such relations is laborious and error prone. We propose a novel framework where simple user input in the form of scribbles are used to guide detection and extraction of such repeated elements. Our detection process is based on a novel boundary band method, and robustly extracts the repetitions along with their mutual depth relations. We then use topological sorting to establish a partial depth ordering of overlapping repeated instances. Missing parts on occluded instances are completed using information from other instances. The extracted repeated instances can then be seamlessly edited and manipulated for a variety of high level tasks that are otherwise difficult to perform. We demonstrate the versatility of our framework on a large set of inputs of varying complexity, showing applications to image rearrangement, edit transfer, deformation propagation, and instance replacement.

Paper

RepFinder: Finding Approximately Repeated Scene Elements for Image Editing [Bib] [21.3M pdf] [3.64M pdf] [PPT Zip File]
ACM Transactions on Graphics, 29, 4, 83:1-8, 2010.
Ming-Ming Cheng, Fang-Lue Zhang, Niloy J. Mitra, Xiaolei Huang, Shi-Min Hu

System Pipeline

Results

Video

Video Download from tsinghua

Supplementary Materials

1. Source code

Our system is a huge project with many components. To help readers implementing our system, we supply some pointers to important source code components of our system. These components are:

Grab-cut (Including color GMM and graph cut)
Active contours: We use opencv version
Shape context (with thin plate splines)
Matting
Hierarchical segmentation

2. FAQs:

Edge-based vs. point based matching: Our first attempt to solve this problem was based on employing a state-of-the-art point-based matching method to establish correspondences. However, in images with repeated elements, SIFT matching leads to much ambiguity due to the presence of a large number of points having similar texture. Even spectral correspondence methods fail to resolve the ambiguities. Thus, we moved to an edge-based approach that uses shape information, to resolve ambiguity, and to enforce local coherence of correspondence. We have found it to be simple, effective and robust. [An earlier experiment]