In this project, you will implement a system to combine a series of horizontally overlapping photographs into a single panoramic image. Using the ORB feature detector and descriptors (which you are familiar with through the Features project), you will first detect discriminating features in the images and find the best matching features in the other images. Then, using RANSAC, you will automatically align the photographs (determine their overlap and relative positions) and then blend the resulting images into a single seamless panorama. We have provided you with a graphical interface that lets you view the results of the various intermediate steps of the process. We have also provided you with some test images and skeleton code to get you started with the project.
The project will consist of a pipeline of tabs visualized through
autostichUI that will operate on images or intermediate
results to produce the final panorama output.
The steps required to create a panorama are listed below. You will be creating two ways to stitch a panorama: using translations (where you'll need to presphericallywarp the input images) and homographies, where you align the input images directly. The steps in square brackets are only used with the spherical warping route:

Step 

1. 
Take pictures on a tripod (or handheld) 

2. 
[Warp to spherical coordinates] 

3. 
Extract features 

4. 
Match features 

5. 
Align neighboring pairs using RANSAC 

6. 
Write out list of neighboring translations 

7. 
Correct for drift 

8. 
Read in [warped] images and blend them 

9. 
Crop the result and import into a viewer 

Finally, to make programming easier, this assignment will be in Python, with the help of NumPy and SciPy. Python+NumPy+SciPy is a very powerful scientific computing environment, and makes computer vision tasks much easier. A crashcourse on Python and NumPy can be found here.
For those that are already using git to work in groups, you can still share code with your partner by having multiple masters to your local repository (one being this original repository and the other some remote service like github where you host the code you are working on); here's a reference with more information.
You can run the skeleton program by running,
You will use the feature detection and matching component to combine a series of photographs into a 360 degree panorama. Your software will automatically align the photographs (determine their overlap and relative positions) and then blend the resulting photos into a single seamless panorama. You will then be able to view the resulting panorama inside an interactive Web viewer. To start this component, you will be supplied with some test images and skeleton code.
[TODO 1] Compute the inverse map to warp the image by filling in the skeleton code in the computeSphericalWarpMappings routine to:
(Note: You will have to use the focal length f estimates for the halfresolution images provided above (you can either take pictures and save them in small files or save them in large files and reduce them afterwards) . If you use a different image size, do remember to scale f according to the image size.)
(Note 2: This step is not used when estimating homographies between images, only translations.)
[TODO 2, 3] computeHomography takes two feature sets from image 1 and image 2, f1 and f2 and a list of feature matches matches and estimates a homography from image 1 to image 2.
(Note 3: In computeHomography, you will compute the bestfit homography using the Singular Value Decomposition. From lecture 11: "the solution h is the eigenvector of A'A with smallest eigenvalue. " Recall that the SVD decomposes a matrix by A=USV' where U and V are the left and right singular vectors, and S is a diagonal matrix of singular values, conventionally ordered from largest to smallest. Furthermore, there is a very strong connection between singular vectors and eigenvectors. Consider: A'A = (VSU')(USV') = V(S^2)V'. That is, right singular vectors of A are eigenvectors of A'A, and eigenvalues of A'A are the squares of singular vectors of A. Returning to the problem, this means that the solution h is the right singular vector corresponding to the smallest singular value. For more details, the wikipedia article on the svd is very good.)
[TODO 4] AlignPair takes two feature sets, f1 and f2, the list of feature matches obtained from the feature detecting and matching component (described in the first part of the project), a motion model, m (described below), and estimates and interimage transform matrix M. For this project, the enum MotionModel takes two possible values: eTranslate and eHomography. AlignPair uses RANSAC (RAndom SAmpling Consensus) to pull out a minimal set of feature matches (one match for the case of translations, four for homographies), estimates the corresponding motion (alignment) and then invokes getInliers to get the indices of feature matches (indexing into matches) that agree with the current motion estimate. After repeated trials, the motion estimate with the largest number of inliers is used to compute a least squares estimate for the motion, which is then returned in the motion estimate M.
[TODO 5]
getInliers computes the indices of the matches that
have a Euclidean distance below RANSACthresh given
features f1 and f2 from
image 1 and image 2 and an interimage tranformation matrix from image 1 to image 2.
[TODO 6, 7] LeastSquaresFit computes a least squares estimate for the translation or homograpy using all of the matches previously estimated as inliers. It returns the resulting translation or homography output transform M.
[TODO 8] Given an image and a homography, figure out the box bounding the image after applying the homography.(imageBoundingBox.)
[TODO 9] Given the warped images and their relative displacements, figure out how large the final stitched image will be and their absolute displacements in the panorama.(blendImages.)
[TODO 10] Then, resample each image to its final location (you will need to use inverse mapping here) and blend it with its neighbors. Try a simple feathering function as your weighting function (see mosaics lecture slide on "feathering") (this is a simple 1D version of the distance map described in [Szeliski & Shum]). For extra credit, you can try other blending functions or figure out some way to compensate for exposure differences. (accumulateBlend.) Remember to set the alpha channel of the resulting panorama to opaque! (normalizeBlend.)
[TODO 11] Crop the resulting image to make the left and right edges seam perfectly. The horizontal extent can be computed in the previous blending routine since the first image occurs at both the left and right end of the stitched sequence (draw the "cut" line halfway through this image). Use a linear warp to the mosaic to remove any vertical "drift" between the first and last image. This warp, of the form y' = y + ax, should transform the y coordinates of the mosaic such that the first image has the same ycoordinate on both the left and right end. Calculate the value of 'a' needed to perform this transformation. (blendImages)
Summary of potentially useful functions (you do not have to use any of these):The skeleton code that we provide comes with a graphical interface, with the module autostichUI.py, which makes it easy for you to do the following:
You can use the GUI visualizations to check whether your program is running correctly.
Here is a list of suggestions for extending the program for extra credit. You are encouraged to come up with your own extensions. We're always interested in seeing new, unanticipated ways to use this program! Please use the extracredit flag in autostitchUI.py. You will need to use args in line 511 and modify the code as necessary. If we run your program without the flag, it must perform the basic implementation.
First, your source code should be zipped up into an archive called 'code.zip', and uploaded to CMS. . In addition, turn in a panorama as JPG as your artifact. In particular, turn in a panorama from a handheld sequence. This panorama can be either translationaligned (360 or not), or aligned with homographies (your choice).
Last modified on March 10th, 2015