CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering

Zhengqi Li    Noah Snavely

Cornell University/Cornell Tech

In ECCV, 2018

Abstract

Intrinsic image decomposition is a challenging, long-standing computer vision problem for which ground truth data is very difficult to acquire. We explore the use of synthetic data for training CNN-based intrinsic image decomposition models, then applying these learned models to real-world images. To that end, we present CGIntrinsics , a new, large-scale dataset of physically-based rendered images of scenes with full ground truth decompositions. The rendering process we use is carefully designed to yield high-quality, realistic images, which we find to be crucial for this problem domain. We also propose a new end-to-end training method that learns better decompositions by leveraging CGIntrinsics, and optionally IIW and SAW, two recent datasets of sparse annotations on real world images. Surprisingly, we find that a decomposition network trained solely on our synthetic data outperforms the state-of-the-art on both IIW and SAW, and performance improves even further when IIW and SAW data is added during training. Our work demonstrates the suprising effectiveness of carefully-rendered synthetic data for the intrinsic images task.

paper thumbnail

[Paper] [Supplemental]

Zhengqi Li and Noah Snavely. "CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering".

@inProceedings{li2018cgintrinsics,
  title={CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering},
  author={Zhengqi Li and Noah Snavely},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2018}
}



Dataset

The CGIntrinsics dataset includes > 20,000 high quality rendered images from the Mitsuba Renderer.

CGIntrinsics v1 Dataset

We also provide extra training data (such as our precomputed dense pairwise IIW judgements) that we use for training with IIW and SAW data jointly with CGIntrinsics. Note that we do not provide original images and labels; those could be downloaded from the original IIW and SAW project websites.
IIW Dataset
SAW Dataset

Pretrained Models

Pretrained Model ("Ours (CGI+IIW+SAW)")

This model corresponds to the full model ("Ours (CGI+IIW+SAW)") trained with CGIntrisics, IIW, and SAW data, as described in the paper.

Pretrained Model ("Ours (CGI)")

This model corresponds to the model trained on CGIntrisics dataset ("Ours (CGI)"), as described in the paper.

Precomputed predictions on IIW test set using full model("Ours (CGI+IIW+SAW)")
Precomputed predictions on SAW test set using full model("Ours (CGI+IIW+SAW)")
Precomputed predictions on IIW test set using model trained on CGI dataset ("Ours (CGI)")
Precomputed predictions on SAW test set using model trained on CGI dataset ("Ours (CGI)")
Precomputed predictions on IIW training set using model trained on CGI dataset ("Ours (CGI)")

Since it seems that some people have difficulty running evaluation, we provide precomputed predictions on IIW test set and SAW test set.

* To get evalution results on IIW test set, download IIW dataset and run 'compute_iiw_whdr.py', you need to change judgement_path in this script to fit to your IIW data path, see readme for more details.

* To get evalution results on SAW test set, download SAW dataset and run 'compute_saw_ap.py'. You need modify 'full_root' in this script and to point to the SAW directory you download, see readme for more details

* Note: our released model was trained on public released CGI dataset, and if you run on SAW test set, you might get slightly better performance than what was described in the original ECCV camera ready paper. We have updated paper in ArXiv to reflect this change.

Training/Evaluation Code

Github

Acknowledgements

We thank Jingguang Zhou for his help with data generation. We also thank the anonymous reviewers for their valuable comments. This work was funded by the National Science Foundation through grant IIS-1149393, and by a grant from Schmidt Sciences.