Deep Learning For Visual Search and Style Similarity

Learning visual similarity for product design with convolutional neural networks

Sean Bell, and Kavita Bala

ACM Transactions on Graphics (SIGGRAPH 2015), 34(4), August 2015

Download: [ Paper PDF (13.2 MB)]

Abstract: Popular sites like Houzz, Pinterest, and LikeThatDecor, have communities of users helping each other answer questions about products in images. In this paper we learn an embedding for visual search in interior design. Our embedding contains two different domains of product images: products cropped from internet scenes, and prod- ucts in their iconic form. With such a multi-domain embedding, we demonstrate several applications of visual search including identify- ing products in scenes and finding stylistically similar products. To obtain the embedding, we train a convolutional neural network on pairs of images. We explore several training architectures including re-purposing object classifiers, using siamese networks, and using multitask learning. We evaluate our search quantitatively and qualita- tively and demonstrate high quality results for search across multiple visual domains, enabling new applications in interior design.


Page maintained by Kavita Bala
Last update: Sep 2016
Valid HTML 4.01 Transitional