CS 4670 ‘10 Fall

Professor Noah Snavely

Final Project: HDR Imaging on the Nokia N900

Team: Nicolas Savva (nss45), Yoon Kyung Shim (ys367)

1. Introduction

Cameras are limited in their dynamic range--they can only capture a certain range of intensities in a single photo. This creates the problem of details being lost if the light contrast in a photo is too large. HDR (High Dynamic Ranging) solves this problem by computing the response curve from a set of images taken with different exposures and reconstructing the radiance map. For our final project, we have tried to build an HDR program which runs on the Nokia N900 phone.

2. Related Work

A lot of work has been done in this area. For recovering the radiance map, we used the algorithm from Debevec ’s SIGGRAPH 1997 paper. We implemented Reinhard’s tone mapping operator, most of the information for which was from the paper “Photographic Tone Reproduction for Digital Images”. We also referenced Professor Li Zhang’s project and lecture slides, as well as our own class slides, for an overall layout of what to do. The FCam project has an app for HDRCapture, which was also mentioned in the related SIGGRAPH 2010 paper. A full list of references is at the end.

3. Technical Description

(1) Setup

Our idea was to have something which runs on the phone by itself, at the moment of taking a picture, like the FCam’s HDRCapture app. When the shutter button is pressed, 5 consecutive pictures with different exposures are taken. Autoexposure is used to determine the amount of exposure; this number is multiplied by 100 for the first picture, then at every successive shot the exposure time is divided by 10. Thus the exposure time for the first picture is 100*(autoexposure time), for the second picture is 10*(autoexposure time), for the third one is (autoexposure time), for the fourth and fifth are 1/10*(autoexposure time) and 1/100*(autoexposure time), respectively. Right after the pictures are taken, the HDR methods are run on them, and the 5 original pictures plus the resulting image are saved.

(2) Debevec’s algorithm for recovering radiance maps

We implemented this ourselves followingDebevec’s paper. It consists of four methods, debevec, weight, iterPixel, and trueRadiance, which are in CameraThread.cpp.

debevec takes four matrices (Z, B, g, lE) and a float (lambda) as parameters. Z stores a sampling of pixels from all images in a way such that all pixels of the same image are in the same column; the pixel i of image j can be accessed at Z(i,j). In our current implementation, we sample 130 pixels from each image, with a sample in the same location in each image. B is a one-dimensional matrix storing the (natural) log of the exposure time for each picture. Empty one-dimensional matrices are passed in for g and lE, which store the results of the algorithm. After running the method, g contains the log exposure corresponding to each pixel value z at location g(z, 0), and lE contains the log irradiance for a pixel i at lE(i, 0). Lambda is the smoothness constant (a smaller lambda corresponds to greater smoothness). The problem of recovering the radiance map is basically an overdetermined linear least-squares system, which we solve using cvSolve().

weight is a helper function used in debevec and trueRadiance which, given a pixel value, calculates the amount of weight that should be assigned to that pixel. The closer a pixel value is to the min or the max values, the less weight is given to that pixel.

iterPixel and trueRadiance compute the relative radiance values for each pixel in the final HDR image. This is done by using the second part of Debevec’s algorithm, where for a given pixel the value of that pixel across all images is combined, weighted, and normalized. This procedure reduces noise in the recovered radiance values and imaging artifacts such as granularity. Debevec’s algorithm is for a single-channel image. To run the procedure on a 3-channel RGB image, we first split up the image, run debevec on each channel separately, and then combine the three channels in iterPixel.

(3) Tone mapping

We have implemented Reinhard’s global tone mapping operator. This is done in method toneMap. The image key value is the log-average of the “world” luminance values of all the pixels in the image, which we obtain for each pixel by (Luminance = 0.2125*red + 0.7154*green + 0.0721*blue) where red, green, and blue are the values of the corresponding channel of the image. The luminance is then scaled to a midpoint region by (Scaled luminance = pixel luminance * (0.18 / image key value)), and a final adjustment is made to ensure that all scaled pixel values are within displayable range: (Final luminance = scaled luminance / (1+scaled luminance)).

4. Experimental Results

From our initial implementation of Debevec’s algorithm we obtained a radiance map. True HDR images cannot be stored in conventional image file formats, so the visible images we obtain suffer some loss of precision (i.e. they are not the actual radiance mappings).

<Set 1>

Radiance map

<Set 2>

*this set was taken in very low-light conditions, and two of the five pictures came out totally black

Radiance map

<Set 3>

Radiance map

<Set 4>

Radiance map

<Set 5>

Radiance maps

Tone-mapped image

5. Discussion of Results

As can be seen from the experimental results, most of the radiance maps turn out green. Spots of very bright light -- or in the case of set 2, extremely dark areas -- in the picture often exhibit patterns of red, blue, and yellow, however. We think this might be a compound problem of pointer issues and possibly some mistake in extracting the radiance mappings. The output of the radiance map also varies depending on the file format it is saved in—this may be normal, since an HDR image loses precision in a normal image format. The results of the tone-mapped images, however, are almost identical to .tif radiance maps. This may result from an underlying issue in the radiance map stage, or from a bug in the tone-mapping code. The radiance maps also show noise when the original images were taken in conditions of low light, especially for indoors pictures.

Computation on the phone is rather slow: it takes each image about 3~5 minutes to process.

6. Future Work

For our particular project, if we had more time we would like to fix the bugs, make a better interface, and optimize the speed. Letting the user choose the exposures at which the pictures are taken, to allow for greater flexibility and perhaps artistic scope would also be a possible goal. For the project in general, a possible extension would be to do HDR on video.

7. Team Logistics

Since we are a two-person team, we did most of the work side-by-side, or taking turns to code and debug. This report was also written with the input of both team members on all parts.

8. Note on the code

We piggy-backed on the code for Project 1b. The only classes relevant for the final project, however, is CameraThread.cpp.

9. References

Adams et al., Stanford University, The Frankencamera: an experimental platform for computational photography

Paul E. Debevec, Jitendra Malik, “Recovering High Dynamic Range Radiance Maps from Photographs”, SIGGRAPH 1997.

Erik Reinhard, Michael Stark, Peter Shirley, Jim Ferwerda,” Photographic Tone Reproduction for Digital Images”, SIGGRAPH 2002.

Noah Snavely, Cornell University, class slides (http://www.cs.cornell.edu/courses/cs4670/2010fa/lectures/lectures.html)

Li Zhang, University of Wisconsin, project (http://pages.cs.wisc.edu/~lizhang/courses/cs766-2010f/projects/hdr/index.htm)

FCam API (http://fcam.garage.maemo.org)

GameDev.net (http://www.gamedev.net/reference/articles/article2208.asp)

9. Motion from Stereo

Is in the file named FundamentalMatrix.cpp. We have an integrated version on Yoon’s Windows machine; however, it was infected a few days ago and we have been working only on Nicolas’s computer since then. The basic computation method is in the submitted file; however, it lacks some of the displaying and helper code we were trying to debug.