This article is part of the series Super-Resolution Imaging: Analysis, Algorithms, and Applications.

Open Access Research Article

Video-to-Video Dynamic Super-Resolution for Grayscale and Color Sequences

Sina Farsiu1*, Michael Elad2 and Peyman Milanfar1

  • * Corresponding author: Sina Farsiu

Author Affiliations

1 Electrical Engineering Department, University of California Santa Cruz, Santa Cruz, CA 95064, USA

2 Computer Science Department, Technion – Israel Institute of Technology, Haifa 32000, Israel

For all author emails, please log on.

EURASIP Journal on Advances in Signal Processing 2006, 2006:061859  doi:10.1155/ASP/2006/61859


The electronic version of this article is the complete one and can be found online at: http://asp.eurasipjournals.com/content/2006/1/061859


Received: 17 December 2004
Revisions received: 10 March 2005
Accepted: 15 March 2005
Published: 8 February 2006

© 2006 Farsiu et al.

We address the dynamic super-resolution (SR) problem of reconstructing a high-quality set of monochromatic or color super-resolved images from low-quality monochromatic, color, or mosaiced frames. Our approach includes a joint method for simultaneous SR, deblurring, and demosaicing, this way taking into account practical color measurements encountered in video sequences. For the case of translational motion and common space-invariant blur, the proposed method is based on a very fast and memory efficient approximation of the Kalman filter (KF). Experimental results on both simulated and real data are supplied, demonstrating the presented algorithms, and their strength.

References

  1. TS Huang, RY Tsai, Multi-frame image restoration and registration. Advances in Computer Vision and Image Processing (JAI Press, Greenwich, Conn, USA, 1984) 1, pp. 317–339 chapter 7 OpenURL

  2. N Nguyen, P Milanfar, GH Golub, A computationally efficient super-resolution image reconstruction algorithm. Transactions on Image Processing 10(4), 573–583 (2001). PubMed Abstract | Publisher Full Text OpenURL

  3. M Irani, S Peleg, Improving resolution by image registration. CVGIP: Graphical Models and Image Processing 53(3), 231–239 (1991). Publisher Full Text OpenURL

  4. M Elad, A Feuer, Restoration of a single super-resolution image from several blurred, noisy, and undersampled measured images. Transactions on Image Processing 6(12), 1646–1658 (1997). PubMed Abstract | Publisher Full Text OpenURL

  5. A Zomet, S Peleg, Efficient super-resolution and applications to mosaics. Proceedings of IEEE 15th International Conference on Pattern Recognition (ICPR '00), September 2000, Barcelona, Spain 1, 579–583

  6. MK Ng, NK Bose, Mathematical analysis of super-resolution methodology. IEEE Signal Processing Magazine 20(3), 62–74 (2003). Publisher Full Text OpenURL

  7. Y Altunbasak, AJ Patti, RM Mersereau, Super-resolution still and video reconstruction from MPEG-coded video. IEEE Transactions on Circuits and Systems for Video Technology 12(4), 217–226 (2002). Publisher Full Text OpenURL

  8. S Farsiu, MD Robinson, M Elad, P Milanfar, Fast and robust multi-frame super-resolution. IEEE Transactions on Image Processing 13(10), 1327–1344 (2004). PubMed Abstract | Publisher Full Text OpenURL

  9. CA Segall, AK Katsaggelos, R Molina, J Mateos, Bayesian resolution enhancement of compressed video. IEEE Transactions on Image Processing 13(7), 898–911 (2004). PubMed Abstract | Publisher Full Text OpenURL

  10. S Borman, RL Stevenson, Super-resolution from image sequences—a review. Proceedings of Midwest Symposium on Circuits and Systems (MWSCAS '98), August 1998, Notre Dame, Ind, USA, 374–378

  11. SC Park, MK Park, MG Kang, Super-resolution image reconstruction: a technical overview. IEEE Signal Processing Magazine 20(3), 21–36 (2003). Publisher Full Text OpenURL

  12. S Farsiu, MD Robinson, M Elad, P Milanfar, Advances and challenges in super-resolution. International Journal of Imaging Systems and Technology 14(2), 47–57 (2004). Publisher Full Text OpenURL

  13. M Elad, A Feuer, Super-resolution restoration of an image sequence: adaptive filtering approach. IEEE Transactions on Image Processing 8(3), 387–395 (1999). PubMed Abstract | Publisher Full Text OpenURL

  14. M Elad, A Feuer, Super-resolution reconstruction of image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence 21(9), 817–834 (1999). Publisher Full Text OpenURL

  15. M Elad, Y Hel-Or, A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur. IEEE Transactions on Image Processing 10(8), 1187–1193 (2001). PubMed Abstract | Publisher Full Text OpenURL

  16. S Farsiu, M Elad, P Milanfar, Multiframe demosaicing and super-resolution from undersampled color images. Computational Imaging II, January 2004, San Jose, Calif, USA, Proceedings of SPIE 5299, 222–233

  17. S Farsiu, M Elad, P Milanfar, Multiframe demosaicing and super-resolution of color images. IEEE Transactions on Image Processing 15(1), 141–159 (2006). PubMed Abstract OpenURL

  18. SM Kay, Fundamentals of Statistical Signal Processing, Volume I: Estimation Theory (Prentice-Hall, Englewood Cliffs, NJ, USA, 1993)

  19. AH Jazwinski, Stochastic Processes and Filtering Theory (Academic Press, New York, NY, USA, 1970)

  20. M Elad, in Super-resolution reconstruction of continuous image sequence, Ph, ed. by . D. dissertation (Technion-Israel Institute of Technology, Haifa, Israel, 1997)

  21. HE Rauch, CT Striebel, F Tung, Maximum likelihood estimates of dynamic linear systems. American Institute of Aeronautics and Astronautics 3(8), 1445–1450 (1965)

  22. AC Harvey, Forecasting, Structural Time Series Models and the Kalman Filter (Cambridge University Press, Cambridge, UK, 1990)

  23. A Bovik, Handbook of Image and Video Processing (Academic Press, New York, NY, USA, 2000)

  24. LI Rudin, S Osher, E Fatemi, Nonlinear total variation based noise removal algorithms. Physica D 60(1–4), 259–268 (1992)

  25. Y Li, F Santosa, A computational algorithm for minimizing total variation in image restoration. IEEE Transactions on Image Processing 5(6), 987–995 (1996). PubMed Abstract | Publisher Full Text OpenURL

  26. TF Chan, S Osher, J Shen, The digital TV filter and nonlinear denoising. IEEE Transactions on Image Processing 10(2), 231–241 (2001). PubMed Abstract | Publisher Full Text OpenURL

  27. C Tomasi, R Manduchi, Bilateral filtering for gray and color images. Proceedings of IEEE 6th International Conference on Computer Vision (ICCV '98), January 1998, Bombay, India, 839–846

  28. M Elad, On the origin of the bilateral filter and ways to improve it. IEEE Transactions on Image Processing 11(10), 1141–1151 (2002). PubMed Abstract | Publisher Full Text OpenURL

  29. C Laroche, M Prescott, Apparatus and method for adaptive for adaptively interpolating a full color image utilizing chrominance gradients United States Patent 5,373,322, 1994

  30. R Kimmel, Demosaicing: image reconstruction from color CCD samples. IEEE Transactions on Image Processing 8(9), 1221–1228 (1999). PubMed Abstract | Publisher Full Text OpenURL

  31. D Keren, M Osadchy, Restoring subsampled color images. Machine Vision and Applications 11(4), 197–202 (1999). Publisher Full Text OpenURL

  32. Y Hel-Or, D Keren, Demosaicing of color images using steerable wavelets. in Tech, ed. by . Rep. HPL-2002-206R1 20020830 (HP Laboratories Israel, Haifa, Israel, 2002) (Online, available: http://citeseer, 2002), . nj.nec.com/ webcite OpenURL

  33. DD Muresan, TW Parks, Optimal recovery demosaicing. Proceedings of IASTED International Conference on Signal and Image Processing (SIP '02), August 2002, Kauai, Hawaii, USA

  34. BK Gunturk, Y Altunbasak, RM Mersereau, Color plane interpolation using alternating projections. IEEE Transactions on Image Processing 11(9), 997–1013 (2002). PubMed Abstract | Publisher Full Text OpenURL

  35. D Alleysson, S Süsstrunk, J Hérault, Color demosaicing by estimating luminance and opponent chromatic signals in the Fourier domain. Proceedings of IS&T/SID 10th Color Imaging Conference, November 2002, Scottsdale, Ariz, USA, 331–336 PubMed Abstract | Publisher Full Text OpenURL

  36. R Ramanath, WE Snyder, GL Bilbro, WA Sander, Demosaicking methods for the Bayer color arrays. Journal of Electronic Imaging 11(3), 306–315 (2002). Publisher Full Text OpenURL

  37. S-C Pei, I-K Tam, Effective color interpolation in CCD color filter arrays using signal correlation. IEEE Transactions on Circuits and Systems for Video Technology 13(6), 503–513 (2003). Publisher Full Text OpenURL

  38. A Zomet, S Peleg, Multi-sensor super-resolution. Proceedings of IEEE 6th Workshop on Applications of Computer Vision (WACV '02), December 2002, Orlando, Fla, USA, 27–31

  39. T Gotoh, M Okutomi, Direct super-resolution and registration using raw CFA images. Proceedings of IEEE Computer Society Conference on Computer Vision and Patern Recognition (CVPR '04), June–July 2004, Washington, DC, USA 2, 600–607

  40. NR Shah, A Zakhor, Resolution enhancement of color video sequences. IEEE Transactions on Image Processing 8(6), 879–885 (1999). PubMed Abstract | Publisher Full Text OpenURL

  41. BC Tom, AK Katsaggelos, Resolution enhancement of monochrome and color video using motion compensation. IEEE Transactions on Image Processing 10(2), 278–287 (2001). PubMed Abstract | Publisher Full Text OpenURL

  42. WK Pratt, Digital Image Processing, 3rd edn. (John Wiley & Sons, New York, NY, USA, 2001)

  43. P Golland, AM Bruckstein, Motion from color. Computer Vision and Image Understanding 68(3), 346–362 (1997). Publisher Full Text OpenURL

  44. JR Bergen, P Anandan, KJ Hanna, R Hingorani, Hierarchical model-based motion estimation. Proceedings of European Conference on Computer Vision (ECCV '92), May 1992, Santa Margherita Ligure, Italy, 237–252