Multi-Spectral Fusion and Denoising of Color and Near-Infrared Images Using Multi-Scale Wavelet Analysis †
Abstract
:1. Introduction
1.1. Related Work
1.1.1. Gradient Processing
1.1.2. Image Fusion
1.2. Contributions
- We propose the MFD framework for RGB and NIR image fusion in the wavelet domain to achieve both texture transfer and noise removal.
- We provide the discrepancy model based on the wavelet scale map (correlation between RGB and NIR data) to deal with the discrepancy between RGB and NIR images.
- We combine three probability terms of contrast preservation, gradient denoising, and fusion denoising into the MFD framework to resolve the discrepancy while reducing the noise in the fusion.
- We enhance the color based on the luminance variation after fusion. The enhanced colors are more vivid with less color distortion.
2. Proposed Method
2.1. Problem Formulation
2.2. Multi-Spectral Fusion and Denoising Framework
2.3. Parameter Description
2.3.1. Parameters and
2.3.2. Parameters and
2.3.3. Parameters and
2.3.4. Parameters
2.4. Numerical Solution
2.5. Application to Low Pass Fusion
2.6. Unified MFD Framework for RGB and NIR Image Fusion
Algorithm 1 Multi-scale fusion and denoising of NIR and RGB images. |
Input: Noisy gray image from RGB image, NIR image
Initialize: ,,, , , , , , , . 1. Perform DT-CWT on noisy gray and NIR images. 2. Detail layer: For m=1:M (M: Maximum decomposition) For n=1:N (N: Maximum iteration number) a. Calculate of and [24]; b. Calculate from by (13); c. Calculate by (14)–(17); d. Optimize by (21); e. Optimize by (23); if and ; break; end For end For 3. Base layer:For n=1:N (N: Maximum iteration number) a. Calculate of and [24]; b. Calculate from g by (26); c. Calculate by (14); d. Optimize and by (24)–(25); if and ; break; end For 4. Perform inverse DT-CWT.Output: Fused gray image. |
2.7. Chroma Denoising and Color Enhancement
3. Experimental Results
3.1. Multi-Spectral Fusion of NIR and RGB Images
3.2. Parameter Analysis
3.3. Comparison between Different Wavelet Transforms
3.4. Comparison with Different Color Spaces
3.5. Computational Complexity
3.6. Application to RGB-NIR Images under Normal Illumination
3.7. Fusion of RGB Luminance Channel and NIR Image in a Local Manner
3.8. Application to RGB-NIR Images with JPEG Compression
3.9. Limitation and Future Work
4. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Deng, G. Guided Wavelet Shrinkage for Edge-Aware Smoothing. IEEE Trans. Image Process. 2017, 26, 900–914. [Google Scholar] [CrossRef] [PubMed]
- Shen, Y.; Liu, Q.; Lou, S.; Hou, Y.L. Wavelet-Based Total Variation and Nonlocal Similarity Model for Image Denoising. IEEE Signal Process. Lett. 2017, 24, 877–881. [Google Scholar] [CrossRef]
- Ding, Y.; Selesnick, I.W. Artifact-free wavelet denoising: Non-convex sparse regularization, convex optimization. IEEE Signal Process. Lett. 2015, 22, 1364–1368. [Google Scholar] [CrossRef] [Green Version]
- Yamashita, H.; Sugimura, D.; Hamamoto, T. Enhancing low-light color images using an RGB-NIR single sensor. In Proceedings of the IEEE International Conference on Visual Communications and Image Processing (VCIP), Singapore, 13–16 December 2015; pp. 1–4. [Google Scholar]
- Yamashita, H.; Sugimura, D.; Hamamoto, T. RGB-NIR imaging with exposure bracketing for joint denoising and deblurring of low-light color images. In Proceedings of the IEEE International Conference onAcoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA, 5–9 March 2017; pp. 6055–6059. [Google Scholar]
- Son, C.H.; Zhang, X.P. Near-Infrared Fusion via Color Regularization for Haze and Color Distortion Removals. IEEE Trans. Circuits Syst. Video Technol. 2017, 28, 3111–3126. [Google Scholar] [CrossRef]
- Son, C.H.; Zhang, X.P. Near-Infrared Coloring via a Contrast-Preserving Mapping Model. IEEE Trans. Image Process. 2017, 26, 5381–5394. [Google Scholar] [CrossRef] [PubMed]
- Shen, X.; Yan, Q.; Xu, L.; Ma, L.; Jia, J. Multispectral joint image restoration via optimizing a scale map. IEEE Trans. Pattern Anal. Mach. Intell. 2015, 37, 2518–2530. [Google Scholar] [CrossRef]
- Krishnan, D.; Fergus, R. Dark Flash Photography. ACM Trans. Graph. 2009, 28, 96:1–96:11. [Google Scholar] [CrossRef]
- Fredembach, C.; Süsstrunk, S. Colouring the near-infrared. In Proceedings of the Color and Imaging Conference. Society for Imaging Science and Technology, Portland, OR, USA, 10–14 November 2008; Volume 2008, pp. 176–182. [Google Scholar]
- Zhuo, S.; Zhang, X.; Miao, X.; Sim, T. Enhancing low light images using near infrared flash images. In Proceedings of the IEEE International Conference on Image Processing (ICIP), Hong Kong, China, 26–29 September 2010; pp. 2537–2540. [Google Scholar]
- Li, W.; Zhang, J.; Dai, Q.H. Robust blind motion deblurring using near-infrared flash image. J. Vis. Commun. Image Represent. 2013, 24, 1394–1413. [Google Scholar] [CrossRef]
- Yan, Q.; Shen, X.; Xu, L.; Zhuo, S.; Zhang, X.; Shen, L.; Jia, J. Cross-field joint image restoration via scale map. In Proceedings of the IEEE International Conference on Computer Vision, Sydney, NSW, Australia, 1–8 December 2013; pp. 1537–1544. [Google Scholar]
- Sugimura, D.; Mikami, T.; Yamashita, H.; Hamamoto, T. Enhancing color images of extremely low light scenes based on RGB/NIR images acquisition with different exposure times. IEEE Trans. Image Process. 2015, 24, 3586–3597. [Google Scholar] [CrossRef]
- Son, C.H.; Zhang, X.P. Layer-based approach for image pair fusion. IEEE Trans. Image Process. 2016, 25, 2866–2881. [Google Scholar] [CrossRef]
- Shibata, T.; Tanaka, M.; Okutomi, M. Versatile visible and near-infrared image fusion based on high visibility area selection. J. Electron. Imaging 2016, 25, 013016. [Google Scholar] [CrossRef]
- Li, H.; Wu, X. DenseFuse: A Fusion Approach to Infrared and Visible Images. IEEE Trans. Image Process. 2019, 28, 2614–2623. [Google Scholar] [CrossRef] [Green Version]
- Zhang, L.; Zeng, G.; Wei, J.; Xuan, Z. Multi-Modality Image Fusion in Adaptive-Parameters SPCNN Based on Inherent Characteristics of Image. IEEE Sens. J. 2020, 20, 11820–11827. [Google Scholar] [CrossRef]
- Zheng, M.; Qi, G.; Zhu, Z.; Li, Y.; Wei, H.; Liu, Y. Image Dehazing by an Artificial Image Fusion Method Based on Adaptive Structure Decomposition. IEEE Sens. J. 2020, 20, 8062–8072. [Google Scholar] [CrossRef]
- Jung, H.; Kim, Y.; Jang, H.; Ha, N.; Sohn, K. Unsupervised Deep Image Fusion With Structure Tensor Representations. IEEE Trans. Image Process. 2020, 29, 3845–3858. [Google Scholar] [CrossRef]
- Su, H.; Jung, C. Multi-spectral Fusion and Denoising of RGB and NIR Images Using Multi-scale Wavelet Analysis. In Proceedings of the 2018 24th IAPR International Conference on Pattern Reconition (ICPR), Beijing, China, 20–24 August 2018. [Google Scholar]
- Bioucas-Dias, J.M. Bayesian wavelet-based image deconvolution: A GEM algorithm exploiting a class of heavy-tailed priors. IEEE Trans. Image Process. 2006, 15, 937–951. [Google Scholar] [CrossRef]
- Bhatnagar, G.; Wu, Q.J.; Liu, Z. Directive contrast based multimodal medical image fusion in NSCT domain. IEEE Trans. Multimed. 2013, 15, 1014–1024. [Google Scholar] [CrossRef]
- Hill, P.; Al-Mualla, M.E.; Bull, D. Perceptual Image Fusion Using Wavelets. IEEE Trans. Image Process. 2017, 26, 1076–1088. [Google Scholar] [CrossRef] [Green Version]
- Xu, L.; Yan, Q.; Xia, Y.; Jia, J. Structure Extraction from Texture via Natural Variation Measure. ACM Trans. Graph. (Siggraph Asia) 2012, 31. [Google Scholar] [CrossRef]
- Farbman, Z.; Fattal, R.; Lischinski, D.; Szeliski, R. Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Trans. Graph. (Tog). 2008, 27, 67. [Google Scholar] [CrossRef]
- Wang, W.; Shui, P.; Feng, X. Variational Models for Fusion and Denoising of Multifocus Images. IEEE Signal Process. Lett. 2008, 15, 65–68. [Google Scholar] [CrossRef]
- He, K.; Sun, J.; Tang, X. Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 2013, 35, 1397–1409. [Google Scholar] [CrossRef] [PubMed]
- Dinh, K.Q.; Canh, T.N.; Jeon, B. Color image denoising via cross-channel texture transferring. IEEE Signal Process. Lett. 2016, 23, 1071–1075. [Google Scholar] [CrossRef]
- Mantiuk, R.; Mantiuk, R.; Tomaszewska, A.; Heidrich, W. Color Correction for Tone Mapping. Comput. Graph. Forum (Proc. Eurographics 2009) 2009, 28, 193–202. [Google Scholar] [CrossRef]
- Dabov, K.; Foi, A.; Katkovnik, V.; Egiazarian, K. Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering. IEEE Trans. Image Process. 2007, 16, 2080–2095. [Google Scholar] [CrossRef]
- Shannon, C.E. A mathematical theory of communication. ACM SIGMOBILE Mob. Comput. Commun. Rev. 2001, 5, 3–55. [Google Scholar] [CrossRef]
- Zhang, L.; Zhang, L.; Bovik, A.C. A feature-enriched completely blind image quality evaluator. IEEE Trans. Image Process. 2015, 24, 2579–2591. [Google Scholar] [CrossRef] [Green Version]
- Panetta, K.; Gao, C.; Agaian, S. No reference color image contrast and quality measures. IEEE Trans. Consum. Electron. 2013, 59, 643–651. [Google Scholar] [CrossRef]
- Liu, Z.; Karam, L.J.; Watson, A.B. JPEG2000 encoding with perceptual distortion control. IEEE Trans. Image Process. 2006, 15, 1763–1778. [Google Scholar] [CrossRef] [Green Version]
- Ma, J.; Chen, C.; Li, C.; Huang, J. Infrared and visible image fusion via gradient transfer and total variation minimization. Inf. Fusion 2016, 31, 100–109. [Google Scholar] [CrossRef]
- Ham, B.; Cho, M.; Ponce, J. Robust Guided Image Filtering Using Nonconvex Potentials. IEEE Trans. Pattern Anal. Mach. Intell. 2018, 40, 192–207. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Metrics | DWLS | SM | GWS | Dense. | UDIF | Pro. |
---|---|---|---|---|---|---|
DE | 7.082 | 7.046 | 7.030 | 6.841 | 6.865 | 7.128 |
FBIQE | 29.639 | 27.269 | 27.909 | 29.600 | 30.469 | 26.836 |
CIQ | 0.912 | 0.914 | 0.904 | 0.788 | 0.841 | 0.961 |
Metrics | HSV | CIE LAB | YCbCr |
---|---|---|---|
DE | 7.015 | 7.032 | 7.128 |
FBIQE | 28.057 | 29.747 | 26.836 |
CIQ | 0.764 | 0.896 | 0.961 |
Methods | DWLS | SM | GWS | Dense. | UDIF | Pro. |
---|---|---|---|---|---|---|
Time(s/image) | 3.42 | 7.99 | 0.60 | 0.40 | 0.29 | 14.94 |
Metrics | 0.25 | 0.5 | 0.75 | png |
---|---|---|---|---|
DE | 7.137 | 7.134 | 7.132 | 7.128 |
FBIQE | 29.283 | 27.486 | 26.895 | 26.836 |
CIQ | 0.803 | 0.826 | 0.934 | 0.961 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Su, H.; Jung, C.; Yu, L. Multi-Spectral Fusion and Denoising of Color and Near-Infrared Images Using Multi-Scale Wavelet Analysis. Sensors 2021, 21, 3610. https://doi.org/10.3390/s21113610
Su H, Jung C, Yu L. Multi-Spectral Fusion and Denoising of Color and Near-Infrared Images Using Multi-Scale Wavelet Analysis. Sensors. 2021; 21(11):3610. https://doi.org/10.3390/s21113610
Chicago/Turabian StyleSu, Haonan, Cheolkon Jung, and Long Yu. 2021. "Multi-Spectral Fusion and Denoising of Color and Near-Infrared Images Using Multi-Scale Wavelet Analysis" Sensors 21, no. 11: 3610. https://doi.org/10.3390/s21113610
APA StyleSu, H., Jung, C., & Yu, L. (2021). Multi-Spectral Fusion and Denoising of Color and Near-Infrared Images Using Multi-Scale Wavelet Analysis. Sensors, 21(11), 3610. https://doi.org/10.3390/s21113610