Assignment Day Wednesday, July 06, 2016 (PM)
Due Date Tuesday, July 12, 2016 5 PM.

 


[H4] Blending     

Overview:

In this homework (from Jay Summet's class), you will be putting together a pyramid blending pipeline that takes three images and generates a blended images. You will require you to create a blend using your own images). This homework is to be done individually. You may ask others for help on Piazza, but you may NOT use other people's (current or present) code outside of the code we have provided to you.

Description:

Before we get started, download the homework files under (zip file here). The zip file contains the files you will be working with (use the ones not with "small" in their file title).

Once you have extracted the file above, you are ready to get started.

Above you can see the sample images which are provided for you in the images/source/sample directory. Every time you run the code, it will look inside each folder under images/, and attempt to find a folder that has images with filenames that contain 'white', 'black' and 'mask'. Once it finds a folder that contains three images with those respective names, it will apply a blending procedure to them, and save the output to images/output/. Along with the output image, it will create visualizations of the gaussian and laplacian pyramids used in the process.

The blending procedure takes two images and the mask and splits them into their red, green, and blue channels. It then blends each channel separately. You do not have to worry about dealing with three channels, you can assume your images take in grayscale images (as we have done in earlier assignments).

The code will construct a Laplacian pyramid for the two images. It will then scale the mask image to the range [0,1] and construct a Gaussian pyramid for it. Finally, it will blend the two pyramids and collapse them to the output image.

Pixel values of 255 in the mask image are scaled to the value 1 in the mask pyramid, and assigned the strongest weight to the image labeled 'white' during blending. Pixel values of 0 in the mask image are scaled to the value 0 in the mask pyramid, and assigned the strongest weight to the image labeled 'black' during blending.

In order to facilitate this process, you will be providing six (6) key functions throughout this programming homework that are the building blocks of blending two images.

Grading and Testing

We will test your homework with the python program file homework4_test.py, provided in the zipfile, directly with your code. In addition it provides helpful feedback, and you may use it to debug your functions.

Part 0: Reduce and Expand functions:

reduce

This function takes an image, convolves it, and then subsamples it down to a quarter of the size (dividing the height and width by two). Note that we say subsample here. We recommend you look into numpy indexing techniques to accomplish the subsampling (essentially you want to index every other row and every other column).

Within the code, you are provided with a generating_kernel( a ) function. This function takes a floating point number a, and returns a 5x5 generating kernel. For the reduce and expand functions, you should use

a = 0.4.

We have already covered how convolve works in class, for this assignment we allow you to use the scipy library implementation of convolve.

scipy.signal.convolve2d(image, kernel, 'same')

This call will convolve the image and kernel, and return an array of the 'same' size as image.

expand

This function takes an image and supersamples it to four times the size (multiplying the height and width by two (2)). After increasing the size, we have to interpolate the missing values by using the same convolution we used in reduce, and lastly we scale our output by 4.

Some tips in terms of how to super sample an image.

  1. -----1. Create an image that is twice the size of the input.
  2. -----2. Review your reduce code. Look at what you did to get your output.
  3. -----3. Instead of choosing every other row / col in reduce for your output, can you assign every other row / col in           expand for your output?
  4. -----4. As stressed above, look into numpy indexing. A key thing to note is the basic slice syntax, think about how            you can use that to your advantage.


For this part of the assignment, please use the generating kernel with a = 0.4 and the convolve2d function from the reduce function discussion above.

Part 1: Gaussian and Laplacian Pyramids

In this part of the assignment, you will be implementing functions that create Gaussian and Laplacian pyramids. As mentioned above you will use a homework4_test.py to test your code. In addition, assignme4_test.py defines the functions viz_gauss_pyramid and viz_lapl_pyramid, which take a pyramid as input and return an image visualizaiton. You might want to use these functions to visualize your pyramids while debugging -- we use these at the end of the testing to output your results.

gaussPyramid

This function takes an image and builds a pyramid out of it. The first layer of this pyramid is the original image, and each subsequent layer of the pyramid is the reduced form of the previous layer. Put simply, you are iteratively calling the reduce function on the output of the previous call, with the first call simply being the input image.

Please use the reduce function that you implemented in the previous part in order to write this function.

laplPyramid

This function takes a Gaussian pyramid constructed by the previous function, and turns it into a Laplacian pyramid. The doc string contains further information about the operations you should perform for each layer.

Like with Gaussian pyramids, Laplacian pyramids are represented as lists of numpy arrays in the code.

Please use the expand function that you implemented in the previous part of the code in order to write this function.

Part 2: Writing the blend and collapse functions.ssian and Laplacian Pyramids

In this part, you will be completing the pipeline by writing the actual blend function, and creating a collapse function that will allow us to convert our Laplacian pyramid into an output image.

As mentioned above, here as well you will be using homework4_test.py to test your code. There are a few funcion that you will be modifying, and they are mentioned below.

blend

This function takes three pyramids:

This function should perform an alpha-blend of the two Laplacian pyramids according to the mask pyramid. So you will be blending each pair of layers together using the mask of that layer as the weight.

As described in the doc string, pixels where the mask is 1 should be taken from the white image, pixels where the mask is 0 should be taken from the black image. Pixels with value 0.5 in the mask should be an equal blend of the white and black images. This is mathematically described in the assignment comments.

You may assume that all of the provided pyramids are of the same dimensons, and have dtype float. You may further assume that the mask pyramid has values in the range [0,1].

Your output pyramid should be of the same dimensions as all the inputs, and dtype float.

collapse

This function is given a laplacian pyramid and is expected to 'flatten' it to an image.

We need to take the bottom (smallest) layer, expand it, and then add it to the next layer. We continue this process until we reach the top of the pyramid. Once you add the second layer to the top layer, the top layer is your output (a common mistake is to re-add the top layer to the top layer so if your values look like they are twice as big as what the expected output is, that is what you are doing).

This function should return a numpy array of the same shape as the top (largest) layer of the pyramid, and dtype float.

Write-Up

PDF specifciations:

  1. -----1. Choose a black image and a white image that you would like to blend.
  2. -----2. Choose a unique mask (don't use the one we provide) that you created and explain how you created it.
  3. -----3. Demonstrate these three images in the PDF.
  4. -----4. Explain what you did to tackle each function. Note: For the expand function, we also want you to explicitly           state why you have to scale your output by 4.
  5. -----5. Anything else you wish to note, feel free to include the pyramid outputs if you found them interesting,           (e.g., to solicit Peer Feedback).

 

What to turn in:

 homework4.py - Your code.
 homework4.pdf - Your code, see above for the writeup, don't forget to include your images!

How we will grade your submission:

part 0:
---- 10 pts reduce
---- 10 pts expand
part 1:
---- 20 pts gaussPyramid --> laplPyramid
part 2:
---- 20 pts blend
---- 20 pts collape
          blend and collapse: finish image correctly blended.
---- -40 pts (deduction) if no actual blend is occuring
writeup:
----- 20 pts
          (1) How and why did you chose the images you are submitting.
          (2) How did you make your mask?
          (3) For each function explain what you did.

Resources:

** zip file here.
** Wikipedia Pyramid Description: (here)
** Paper 1: Burt and Adelson's Original Pyramid Paper (here)
** Paper 2: "Pyramid methods in image processing" (here)
** Paper 3: "A multiresolution spline with application to image mosaics"
          applications of blending, and masks ( here ) in detail. recommended.

http://persci.mit.edu/pub_pdfs/spline83.pdf
** Gaussian-Laplacian Pyramid Image Coding: (here)

Acknowledgement(s):

Project description courtesy Jay Summet and Irfan Essa.