Quantitative Results Using Error Metric

Error Metric

I selected to use the mean squared error as the error metric to evaluate the effectiveness of the system. The benefit of using this metric is that it provides a method for calculating the error that gives a numerical value that can be used to compare errors from different reconstructions regardless of the size of the images in question.

Where

X – Original High resolution image

X’ – Reconstructed High resolution image

L – Number of pixels along each axis (assumes square image)

Quantitative Results

The image below shows the original high resolution image that was used in order to assess the performance of the super-resolution algorithm when applied to various low resolution sets.

20 Pixels Super-Resolved to 30

Original High Resolution Image

Low Resolution Input frames (Each one is 30 x 30)

Output Image

Rank of Model Matrix = 900 (Full Rank)

Mean Squared Error = 1.6491e-024

15 Pixels Super-Resolved to 30

I wont include images for this case as they look pretty similar to the images for the 20 pixel low resolution case.

Rank of Model Matrix = 900 (Full Rank)

Mean Squared Error = 9.0744e-023

10 Pixels Super-Resolved to 30

Rank of Model Matrix = 900 (Full Rank)

Mean Squared Error = 1.2279e-025

6 Pixels Super-Resolved to 30

6 Pixels was the lowest that could achieved whilst still having the model matrix have full rank.

Low Resolution Input frames (Each one is 6 x 6)

Reconstructed High Resolution Image

Rank of Model Matrix = 900 (Full Rank)

Mean Squared Error = 1.1463e-024

4 Pixels Super-Resolved to 30

Reconstructed High Resolution Image

Rank of Model Matrix = 400

Mean Squared Error = 0.0182

Down Sampling & Up Sampling Test Images

Since Friday I have been working towards applying the forward model mentioned in my previous post to a high resolution image, and then attempting to obtain the original high resolution image using the produced low resolution images and knowledge about the transformations that were applied to it.

Forward Model

In order to simplify my first attempt at applying the forward model and then reversing its affects I choose to implement the warping operation applied to the high resolution images to being a shifting affect only. The blurring operation was implemented using a Gaussian kernel with a σ = 0.5. Finally I chose to decimate the image using bilinear interpolation.

Applying the forward Model

The only parameter that was varied in order to produce the low resolution images was the amount that the original high resolution image was displaced before the blurring and decimation operations were performed.

The image below is the original high resolution image that I applied the forward model to.

The images below show some examples of the result of applying the forward model to the high resolution image.

Calculating the High Resolution Image

First Attempt at Reconstructing the original high resolution image

Second Attempt at Reconstructing the high resolution image

As suggested by Serge i have now inverted the original high resolution image before creating the low resolution frames, and then applying the super-resolution process to them, this has been done purely for cosmetic reasons so that the black lines that creep in at the borders as the image is shifted dont look out of place.

The image below shows the new high resolution image.

The image below shows 9 out of the 25 low resolution frames that were generated.

Since inverting the images i have noticed a slight bug in the matrix that is being used to decimate the high resolution images. This bug didn't show up when performing the super-resolution with an image with a white background. However along the right, and lower side of the reconstructed high resolution image a greyish strip can be seen. To solve this problem i believe i need to adjust the weights of the decimation matrix. But not wanting to rush this and break the whole program i will wait until i have more time before adjusting them.

Next Step(s)

• Fix bug in the decimation matrix
• Implement a function to calculate the error between the reconstructed image and the original image - probably in the form of mean square error
• Attempt to automatically determine the relative motions between the low resolution images, and then compare the new reconstruction error with the previous reconstruction error. Serge recommended implementing this using the optical flow algorithm.
  
• Look into implementing maximum likelihood estimation in order to reconstruct the high resolution image, rather than using the pseudo-inverse.
  
• Effects of noise on the system. Currently i have been using low resolution images that contain no noise, once maximum likelihood estimation is implemented a comparison as to which is more robust against the effects of noise will be performed.
  

Super-Resolution Papers & Models

Papers

The last week or so has been spent pouring over the various papers on multi-frame super resolution and trying to understand the approaches that have been taken in order to solve the super-resolution problem.

Having found an abundance of super-resolution papers online I have selected four papers which were written over the period of 1997 – 2004, which adopt the same approach towards solving the super-resolution problem, each of which builds on the ideas that were presented in the previous paper. The links to these papers are below, and are in chronological order.

“Restoration of a Single Superresolution Image from Several Blurred, Noisy, and Undersampled Measured Images”, Michael Elad and Arie Feuer

“A computationally efficient superresolution image reconstruction algorithm” Nhat Nguyen_, Peyman Milanfar and Gene Golub

“A Fast Super-Resolution Reconstruction Algorithm for Pure Translation Motion and Common Space-Invariant Blur” Michael Elad, Yacov Hel-Or

“Advances and Challenges in Super-Resolution”, Sina Farsiu, Dirk Robinson, Michael Elad, Peyman Milanfar

In the papers mentioned above super-resolution is defined as an inverse problem in which the formation of the low resolution images is modelled as series of successive transformations that are performed on a high resolution image. They then propose that super-resolution can be achieved by obtaining and then applying to the low resolution images the inverse of the forward transformation in order to reconstruct the original high resolution data. The forward model will be discussed below.

Forward Model

The forward model is the model that is used to describe the transformation from a high resolution image to a low resolution version of the same image. The low resolution image is treated as a high resolution image that has been subjected to motion (or warp), camera blur (the cameras point spread function), and down sampling (decimation) operations, the low resolution image is also treated as contained a noise component.

The model connecting the k^th low resolution image to the high resolution image is shown below in a mathematical form:

Where

Y_k is the k^th low resolution image

D_k is the down sampling matrix

C_k is the Blurring matrix

F_k is the motion or warp matrix

X is the high resolution image

E_k is the k^th noise vector

A more general model grouping the equations for all the low resolution frames into one equation is shown below:

...

Multi-frame Vs Single-frame enhancement

Having spent some time reading over the two approaches to performing super-resolution i have decided that i am going to focus on multi-frame rather than single-frame enhancement. I selected the multi frame approach because i believe it will produce more robust results over a wider range of input images. One of the main limitations of the single-frame approach is that the process will only be effective if the database contains a high resolution image which is similar to the image which is trying to be enhanced.

The multi-frame approach on the other and uses the additional data which is contained in the extra input frames to interpolate the pixel values in the high resolution image, this allows the multi-frame approach to be applied to any input image, even if a similar image was not used during the design of the algorithm.

Super Resolution Introduction

Super-Resolution is the process of constructing a high resolution image from a set of one or more low resolution input images. The two approaches to performing super-resolution enhancement are single frame and multi-frame enhancement. Since low resolution images inherently contain less information than higher resolution images, the process of constructing high resolution images form an input of one or more low resolution images requires that the missing high resolution data be calculated.