Sketch2Photo: Internet Image Montage

Tao Chen1  Ming-Ming Cheng1  Ping Tan2  Ariel Shamir3  Shi-Min Hu1

1TNList, Department of Computer Science and Technology, Tsinghua University
2National University of Singapore    3The Interdisciplinary Center

Abstract

We present a system that composes a realistic picture from a simple freehand sketch annotated with text labels. The composed picture is generated by seamlessly stitching several photographs in agreement with the sketch and text labels; these are found by searching the Internet. Although online image search generates many inappropriate results, our system is able to automatically select suitable photographs to generate a high quality composition, using a filtering scheme to exclude undesirable images. We also provide a novel image blending algorithm to allow seamless image composition. Each blending result is given a numeric score, allowing us to find an optimal combination of discovered images. Experimental results show the method is very successful; we also evaluate our system using the results from two user studies.

Paper

Sketch2Photo: Internet Image Montage
ACM SIGGRAPH ASIA 2009, ACM Transactions on Graphics, to appear
Tao Chen, Ming-Ming Cheng, Ping Tan, Ariel Shamir, Shi-Min Hu

System Pipeline

Retrieval Results

Composition Results

Video

or

Supplementary Materials

1. Intermediate results

General supplementary materials, including intermediate results and comparisons. [pdf]

2. User studies

High resolution compositions and detailed statistics of the user studies. [pdf]

3. Software

The original Sketch2Photo downloaded and processed images from internet after user drawing sketches. Although it was a brilliant concept, we found this impracticable for users -- they lost patience while waiting for the hours of downloading and processing. Now, we bring the Sketch2Photo to a new phase: User can watch the result almost instantly after him drawing the sketch. We achieve this by a server-client scheme in which the server pre-processes all the images offline, then the client will only need to query desired images by sketching. We also modified the query algorithm so that it now takes almost no time (but you still have to wait for downloading those queried images). We also add more interactive functions to make the use more flexible.

Note this software is still very primitive, we release it on our cooperative partner, M.I.S.Technology Co.’s website (download guide) for a public test. There are some unfinished functions and differences from the previous Sketch2Photo. We list them below:

  1. It only supports limited keywords for backgrounds and scene items, which are pre-processed on our server. We will add more available keywords in further version, and add the function to let the server download new images according to user inputted keywords if they are not exist in the server.
  2. The querying by sketch method is now very fast, but suffers the quality loss. We are stilling looking for the balance between quality and speed. There is also no horizontal line alignment, although the horizontal information contains in backgrounds. It will be added back in next version.
  3. Since in this version we allow user adjusting object size and position, the composition items combination cannot be optimized as before, this will be added together with horizontal alignment.
  4. The advanced composition is absent. We are working on a more efficient implementation, and it’s not bug free for release. So the current composition result looks trivial.

So if you hope to use this software to generate gorgeous photographs, you should probably skip this version, but we welcome you to test and give us feed back. We will continue improving it according to above four points, and any suggestions are welcome!

Sktech2Photo Team

Tao Chen, Kun Xu, Fang-Lve Zhang, Meng Ding and Ming-Ming Cheng

Update: A web-based Sketch2Photo application: click here (Chinese), collaborated with Tencent.

Acknowledgments

This work was supported by the National Basic Research Project of China (Project Number 2006CB303106), the National High Technology Research and Development Program of China (Project Number 2009AA01Z330) and FDCT, Macau (Project Number 008/2008/A1). Tan Ping is supported by Singapore FRC Grant R-263-000-477-112.

Note

Original Name: PhotoSketch. We change the name since it's conflict with existing software: http://www.brainstormllc.com/.

Locations of visitors to this page Locations of visitors to this page