Medical image fusion based on local Laplacian decomposition and iterative joint filter
Corresponding Author
Weisheng Li
Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China
Correspondence
Weisheng Li, Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, 400065, China.
Email: [email protected]
Search for more papers by this authorFeifei Chao
Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China
Search for more papers by this authorGuofen Wang
Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China
Search for more papers by this authorJun Fu
Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China
Search for more papers by this authorXiuxiu Peng
Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China
Search for more papers by this authorCorresponding Author
Weisheng Li
Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China
Correspondence
Weisheng Li, Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, 400065, China.
Email: [email protected]
Search for more papers by this authorFeifei Chao
Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China
Search for more papers by this authorGuofen Wang
Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China
Search for more papers by this authorJun Fu
Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China
Search for more papers by this authorXiuxiu Peng
Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China
Search for more papers by this authorFunding information: National Key Research and Development Program of China, Grant/Award Numbers: 2019YFE0110800, 2016YFC1000307-3; National Natural Science Foundation of China, Grant/Award Numbers: 61972060, U1713213, 62027827; Natural Science Foundation of Chongqing, Grant/Award Numbers: cstc2020jcyj-zdxmX0025, cstc2019cxcyljrc-td0270, cstc2019jcyj-cxttX0002
Abstract
Previous multi-modal medical image fusion methods have suffered from color distortion, blurring, and noise. To address these problems, we propose a method for integrating the information contained in functional and anatomical medical images. In the proposed method, multi-scale image representation of input images is produced by local Laplacian filtering. The rgb2ycbcr algorithm and iterative joint filters are then used to produce fused approximate images. The residual images are divided into regions of interest and noninterest regions, and then a local energy maximization scheme and local energy average scheme are used to combine these regions. Fused interest areas and fused noninterest areas are combined to produce fused residual images. Finally, an inverse local Laplacian filter is used as a reconstruction tool to produce a fused image. Experimental results indicated that our method has a distinct advantage over existing state-of-the-art algorithms in terms of vision quality and objective metrics.
Open Research
DATA AVAILABILITY STATEMENT
Data sharing is not applicable to this article as no new data were created or analyzed in this study.
REFERENCES
- 1Liu Y, Liu S, Wang Z. A general framework for image fusion based on multi-scale transform and sparse representation. Inf Fusion. 2015; 24: 147-164. doi:10.1016/j.inffus.2014.09.004
- 2Sahu A, Bhateja V, Krishn A, Himanshi. Medical image fusion with Laplacian pyramids. International Conference on Medical Imaging, m-Health and Emerging Communication Systems (MedCom). 2014; 448-453. doi:10.1109/MedCom.2014.7006050
10.1109/MedCom.2014.7006050 Google Scholar
- 3Du J, Li W, Xiao B, Nawaz Q. Union Laplacian pyramid with multiple features for medical image fusion. Neurocomputing. 2016; 194: 326-339. doi:10.1016/j.neucom.2016.02.047
- 4Li H, Manjunath BS, Mitra SK. Multisensor image fusion using the wavelet transform. Graph Models Image Process. 1995; 57(3): 235-245. doi:10.1006/gmip.1995.1022
- 5Li S, Yang B. Multifocus image fusion by combining curvelet and wavelet transform. Pattern Recogn Lett. 2008; 29(9): 1295-1301. doi:10.1016/j.patrec.2008.02.002
- 6Hill P, al-Mualla ME, Bull D. Perceptual image fusion using wavelets. IEEE Trans Image Process. 2017; 26(3): 1076-1088. doi:10.1109/TIP.2016.2633863
- 7Cands E, Demanet L, Donoho D, Ying L. Fast discrete curvelet transforms. Multiscale Model Simul. 2006; 5(3): 861-899. doi:10.1137/05064182X
- 8Cunha A, Zhou J, Do MN. The nonsubsampled contourlet transform: theory, design, and applications. IEEE Trans Image Process. 2006; 15(10): 3089-3101. doi:10.1109/TIP.2006.877507
- 9Liu S, Shi M, Zhu Z, Zhao J. Image fusion based on complex-shearlet domain with guided filtering. Multidim Syst Sign Process. 2017; 28: 207-224. doi:10.1007/s11045-015-0343-6
- 10Liu X, Mei W, Du H. Structure tensor and nonsubsampled shearlet transform based algorithm for CT and MRI image fusion. Neurocomputing. 2017; 235: 131-139. doi:10.1016/j.neucom.2017.01.006
- 11Wang Z, Cui Z, Zhu Y. Multi-modal medical image fusion by Laplacian pyramid and adaptive sparse representation, Comput. Biol Med. 2020; 123:103823. doi:10.1016/j.compbiomed.2020.103823
10.1016/j.compbiomed.2020.103823 Google Scholar
- 12Bhataria KC, Shah BK. A review of image fusion techniques. Paper presented at: 2018 Second International Conference on Computing Methodologies and Communication (ICCMC), IEEE. 2018; 114-123. doi:10.1109/ICCMC.2018.8487686
- 13Talbar SN, Chavan SS, Pawar A. Non-subsampled complex wavelet transform based medical image fusion. Paper presented at: Proceedings of the Future Technologies Conference. 2018; 880: 548-556. doi:10.1007/978-3-030-02686-8_41
- 14Ali FE, el-Dokany IM, Saad AA, Abd el-Samie FE. A curvelet transform approach for the fusion of MR and CT images. J Mod Opt. 2010; 57(4): 273-286. doi:10.1080/09500340903541056
- 15Li T, Wang Y. Biological image fusion using a NSCT based variable-weight method. Inf Fusion. 2011; 12(2): 85-92. doi:10.1016/j.inffus.2010.03.007
- 16Adu J, Gan J, Wang Y, Huang J. Image fusion based on nonsubsampled contourlet transform for infrared and visible light image. Infrared Phys Technol. 2013; 61: 94-100. doi:10.1016/j.infrared.2013.07.010
- 17Bhatnagar G, Wu QMJ, Liu Z. Directive contrast based multimodal medical image fusion in NSCT domain. IEEE Multimedia. 2013; 15(5): 1014-1024. doi:10.1109/TMM.2013.2244870
- 18Li B, Peng H, Wang J. A novel fusion method based on dynamic threshold neural P systems and nonsubsampled contourlet transform for multi-modality medical images. Signal Process. 2021; 178:107793. doi:10.1016/j.sigpro.2020.107793
- 19Bhateja V, Srivastava A, Moin A, Lay-Ekuakille A. Multispectral medical image fusion scheme based on hybrid Contourlet and Shearlet transform domains. Rev Sci Instrum. 2018; 89:084301–084307. doi:10.1063/1.5016947
- 20Singh S, Gupta D, Anand RS, Kumar V. Nonsubsampled shearlet based CT and MR medical image fusion using biologically inspired spiking neural network. Biomed Signal Process. 2015; 18: 91-101. doi:10.1016/j.bspc.2014.11.009
- 21Yin M, Liu X, Liu Y, Chen X. Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain. IEEE Trans Instrum Meas. 2018; 68(1): 49-64. doi:10.1109/TIM.2018.2838778
- 22Li B, Peng H, Luo X, et al. Medical image fusion method based on coupled neural P systems in nonsubsampled shearlet transform domain. Int J Neural Syst. 2020; 31:2050050. doi:10.1142/S0129065720500501
- 23Shen R, Cheng I, Basu A. Cross-scale coefficient selection for volumetric medical image fusion. IEEE Trans Biomed Eng. 2013; 60(4): 1069-1079. doi:10.1109/TBME.2012.2211017
- 24Du J, Li W, Lu K, Xiao B. An overview of multi-modal medical image fusion. Neurocomputing. 2016; 215: 3-20. doi:10.1016/j.neucom.2015.07.160
- 25Shahdoosti HR, Ghassemian H. Combining the spectral PCA and spatial PCA fusion methods by an optimal filter. Inf Fusion. 2016; 27: 150-160. doi:10.1016/j.inffus.2015.06.006
- 26Qu XB, Yan J-W, Xiao H-Z, Zhu Z-Q. Image fusion algorithm based on spatial frequency-motivated pulse coupled neural networks in nonsubsampled contourlet transform domain. Acta Automat Sin. 2008; 34(12): 1508-1514. doi:10.1016/S1874-1029(08)60174-3
10.1016/S1874?1029(08)60174?3 Google Scholar
- 27Tan W, Tiwari P, Pandey HM, Moreira C, Jaiswal AK. Multimodal medical image fusion algorithm in the era of big data. Neural Comput Applic. 2020. doi:10.1007/s00521-020-05173-2
- 28Yin H, Li S, Fang L. Simultaneous image fusion and super-resolution using sparse representation. Inf Fusion. 2013; 14(3): 229-240. doi:10.1016/j.inffus.2012.01.008
- 29Liu Y, Chen X, Liu A, Ward RK, Wang ZJ. Recent advances in sparse representation based medical image fusion. IEEE Instrum Meas Mag. 2021; 24(2): 45-53. doi:10.1109/MIM.2021.9400960
- 30Liu Y, Chen X, Peng H, Wang Z. Multi-focus image fusion with a deep convolutional neural network. Inf Fusion. 2017; 36: 191-207. doi:10.1016/j.inffus.2016.12.001
- 31Liu SQ, Yin L, Miao S, Ma J, Cong S, Hu S. Multimodal medical image fusion using rolling guidance filter with CNN and nuclear norm minimization. Curr Med Imag. 2020; 16(10): 1243-1258. doi:10.2174/1573405616999200817103920
- 32Song X, Wu X-J, Li H. MSDNet for medical image fusion. Paper presented at: International Conference on Image and Graphics, 11902. 2019; 278-288. doi:10.1007/978-3-030-34110-7_24
- 33Paris S, Hasinoff SW, Kautz J. Local Laplacian filters: edge-aware image processing with a Laplacian pyramid. ACM Trans Graph. 2011; 30(4): 1244-1259.
- 34Du J, Li WS, Xiao B. Anatomical-functional image fusion by information of interest in local laplacian filtering domain. IEEE Trans Image Process. 2017; 26(12): 5855-5866.
- 35Xu Z. Medical image fusion using multi-level local Extrema. Inf Fusion. 2014; 19: 38-48. doi:10.1016/j.inffus.2013.01.001
- 36Zheng Y, Essock EA, Hansen BC, Haun AM. A new metric based on extended spatial frequency and its application to DWT based fusion algorithms. Inf Fusion. 2007; 8(2): 177-192. doi:10.1016/j.inffus.2005.04.003
- 37Tan W, Thitøn W, Xiang P, Zhou H. Multi-modal brain image fusion based on multi-level edge-preserving filtering. Biomed Signal Process. 2021; 64:102280. doi:10.1016/j.bspc.2020.102280
- 38Tomasi C, Manduchi R. Bilateral filtering for gray and color images. Paper presented at: IEEE Proc of ICCV. 1998; 6: 839-846. doi:10.1109/ICCV.1998.710815
- 39He K, Sun J, Tang X. Guided image filtering. IEEE Trans Pattern Anal. 2013; 35(6): 1397-1409. doi:10.1109/TPAMI.2012.213
- 40Petschnigg G, Szeliski R, Agrawala M, Cohen M, Hoppe H, Toyama K. Digital photography with flash and no-flash image pairs. ACM Trans Graph. 2004; 23(3): 664-672. doi:10.1145/1186562.1015777
- 41Gastal ES, Oliveira MM. Domain transform for edge-aware image and video processing. ACM Trans Graph. 2011; 30(4): 69. doi:10.1145/2010324.1964964
- 42Li W, Xie Y, Zhou H, Han Y, Zhan K. Structure-aware image fusion. Int J Light Electron Optics. 2018; 172: 1-11. doi:10.1016/j.ijleo.2018.06.123
- 43Johnson KA, Alex Becker J. The Whole Brain ALTAS. http://www.med.harvard.edu/aanlib
- 44Wang L, Li B, Tian LF. EGGDD: an explicit dependency model for multi-modal medical image fusion in shift-invariant shearlet transform domain. Inf Fusion. 2014; 19(11): 29-37. doi:10.1016/j.inffus.2013.04.005
- 45Daneshvar S, Ghassemian H. MRI and PET image fusion by combining IHS and retina-inspired models. Inf Fusion. 2010; 11(2): 114-123. doi:10.1016/j.inffus.2009.05.003
- 46Li S, Kang X, Hu J. Image fusion with guided filtering. IEEE Trans Image Process. 2013; 22(7): 2864-2875. doi:10.1109/TIP.2013.2244222
- 47Li X, Guo X, Han P, Wang X, Li H, Luo T. Laplacian re-decomposition for multimodal medical image fusion. IEEE Trans Instrum Meas. 2020; 69: 6880-6890. doi:10.1109/TIM.2020.2975405
- 48Du J, Fang M, Yu Y, Lu G. An adaptive two-scale biomedical image fusion method with statistical comparisons. Comput Methods Prog Biomed. 2020; 196:105603. doi:10.1016/j.cmpb.2020.105603
- 49Das S, Kundu MK. A neuro-fuzzy approach for medical image fusion. IEEE Trans Biomed Eng. 2013; 60(12): 3347-3353. doi:10.1109/TBME.2013.2282461
- 50Zhang Y, Liu Y, Sun P, Yan H, Zhao X, Zhang L. IFCNN: a general image fusion framework based on convolutional neural network. Inf Fusion. 2020; 54: 99-118. doi:10.1016/j.inffus.2019.07.011
- 51Zhang K, Huang Y, Zhao C. Remote sensing image fusion via RPCA and adaptive PCNN in NSST domain. Int J Wavelets, Multi. 2018; 16:1850037. doi:10.1142/S0219691318500376
- 52Balasubramaniam P, Ananthi VP. Image fusion using intuitionistic fuzzy sets. Inf Fusion. 2014; 20: 21-30. doi:10.1016/j.inffus.2013.10.011
- 53Wang Y, Du H, Xu J, Liu Y. A no-reference perceptual blur metric based on complex edge analysis. Paper presented at: 3rd IEEE International Conference on Network Infrastructure and Digital Content, Beijing. 2012; 487-491. doi:10.1109/ICNIDC.2012.6418801
- 54Singh R, Khare A. Multiscale medical image fusion in wavelet domain. Sci World J. 2013; 2013: 1-10. doi:10.1155/2013/521034
- 55Chabi N, Yazdi M, Entezarmahdi M. An efficient image fusion method based on dual tree complex wavelet transform. Paper presented at: 8th Iranian Conference on Machine Vision and Image Processing (MVIP), Zanjan. 2013; 403-407. doi:10.1109/IranianMVIP.2013.6780019
- 56Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process. 2004; 13(4): 600-612. doi:10.1109/TIP.2003.819861