Abstract
In laparoscopic surgery, image quality is often degraded by surgical smoke or by side effects of the illumination system, such as reflections, specularities, and non-uniform illumination. The degraded images complicate the work of the surgeons and may lead to errors in image-guided surgery. Existing enhancement algorithms mainly focus on enhancing global image contrast, overlooking local contrast. Here, we propose a new Patch Adaptive Structure Decomposition utilizing the Multi-Exposure Fusion technique to enhance the local contrast of laparoscopic images for better visualization. The set of under-exposure level images is obtained from a single input blurred image by using gamma correction. Spatial linear saturation is applied to enhance image contrast and to adjust the image saturation. The Multi-Exposure Fusion (MEF) is used on a series of multi-exposure images to obtain a single clear and smoke-free fused image. MEF is applied by using adaptive structure decomposition on all image patches. Image entropy based on the texture energy is used to calculate image energy strength. The texture entropy energy determined the patch size that is useful in the decomposition of image structure. The proposed method effectively eliminate smoke and enhance the degraded laparoscopic images. The qualitative results showed that the visual quality of the resultant images is improved and smoke-free. Furthermore, the quantitative scores computed of the metrics: FADE, Blur, JNBM, and Edge Intensity are significantly improved as compared to other existing methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
Enquiries about data availability should be directed to the authors.
References
Ancuti CO, Ancuti C (2013) Single image dehazing by multi-scale fusion. IEEE Trans Image Process 22(8):3271–3282. https://doi.org/10.1109/TIP.2013.2262284
Azam MA, Khan KB, Ahmad M, Mazzara M (2021) Multimodal medical image registration and fusion for quality enhancement. Comput Mater Contin 68(1):821–840. https://doi.org/10.32604/cmc.2021.016131
Azam MA et al (2021) Deep learning applied to white light and narrow band imaging videolaryngoscopy: toward real-time laryngeal cancer detection. Laryngoscope. https://doi.org/10.1002/lary.29960
Baid A, Kotwal A, Bhalodia R, Merchant SN, Awate SP (2017) Joint desmoking, specularity removal, and denoising of laparoscopy images via graphical models and Bayesian inference. Proc Int Sympos Biomed Imaging. https://doi.org/10.1109/ISBI.2017.7950623
Bansal B, Singh Sidhu J, Jyoti K (2017) A review of image restoration based image Defogging algorithms. Int J Image Graph Signal Process 9(11):62–74. https://doi.org/10.5815/ijigsp.2017.11.07
Choi LK, You J, Bovik AC (2015) Referenceless prediction of perceptual fog density and perceptual image defogging. IEEE Trans Image Process 24(11):3888–3901. https://doi.org/10.1109/TIP.2015.2456502
Crete F, Dolmiere T, Ladret P, Nicolas M (2007) The blur effect: perception and estimation with a new no-reference perceptual blur metric. Hum vis Electron Imaging XII 6492:64920I. https://doi.org/10.1117/12.702790
Fan Y, Chen R, Li Y, Zhang T (2021) Deep neural de-raining model based on dynamic fusion of multiple vision tasks. Soft Comput 25(3):2221–2235. https://doi.org/10.1007/s00500-020-05291-y
Fattal R (2008) Single image dehazing. ACM Trans Graph. https://doi.org/10.1145/1360612.1360671
Ferzli R, Karam LJ (2006) A no-reference objective image sharpness metric based on just-noticeable blur and probability summation. Proc Int Conf Image Process ICIP 3:445–448. https://doi.org/10.1109/ICIP.2007.4379342
Ferzli R, Karam LJ (2009) A no-reference objective image sharpness metric based on the notion of Just Noticeable Blur (JNB). IEEE Trans Image Process 18(4):717–728. https://doi.org/10.1109/TIP.2008.2011760
Galdran A (2018) Image dehazing by artificial multiple-exposure image fusion. Signal Process 149:135–147. https://doi.org/10.1016/j.sigpro.2018.03.008
Hahn KY, Kang DW, Azman ZAM, Kim SY, Kim SH (2017) Removal of hazardous surgical smoke using a built-in-filter trocar: a study in laparoscopic rectal resection. Surg Laparosc Endosc Percutaneous Tech 27(5):341–345. https://doi.org/10.1097/SLE.0000000000000459
Hautière N, Tarel JP, Aubert D, Dumont É (2008) Blind contrast enhancement assessment by gradient ratioing at visible edges. Image Anal Stereol 27(2):87–95. https://doi.org/10.5566/ias.v27.p87-95
He K, Sun J, Tang X (2010) ECCV2010—guided image filtering. Eccv 2010:1–14
He K, Sun J, Tang X (2011) Single image haze removal using dark channel prior. IEEE Trans Pattern Anal Mach Intell 33(12):2341–2353. https://doi.org/10.1109/TPAMI.2010.168
Jobson DJ (2004) Retinex processing for automatic image enhancement. J Electron Imaging 13(1):100. https://doi.org/10.1117/1.1636183
Khan ZA et al (2020) Towards a video quality assessment based framework for enhancement of laparoscopic videos. Electr Eng Syst Sci. https://doi.org/10.1117/12.2549266
Kotwal A (2016) Joint desmoking and denoising of laparoscopy images Department of Electrical Engineering Indian Institute of Technology (IIT) Bombay Department of Computer Science and Engineering Indian Institute of Technology (IIT) Bombay, pp. 1050–1054
Li H, Qiu H, Yu Z, Zhang Y (2016) Infrared and visible image fusion scheme based on NSCT and low-level visual features. Infrared Phys Technol 76:174–184. https://doi.org/10.1016/j.infrared.2016.02.005
Li Y, Miao Q, Liu R, Song J, Quan Y, Huang Y (2018a) A multi-scale fusion scheme based on haze-relevant features for single image dehazing. Neurocomputing 283:73–86. https://doi.org/10.1016/j.neucom.2017.12.046
Li H, He X, Tao D, Tang Y, Wang R (2018b) Joint medical image fusion, denoising and enhancement via discriminative low-rank sparse dictionaries learning. Pattern Recognit 79:130–146. https://doi.org/10.1016/j.patcog.2018.02.005
Li Y et al (2018c) A novel multi-exposure image fusion method based on adaptive patch structure. Entropy 20(12):1–17. https://doi.org/10.3390/e20120935
Li H, Wang Y, Yang Z, Wang R, Li X, Tao D (2020) Discriminative dictionary learning-based multiple component decomposition for detail-preserving noisy image fusion. IEEE Trans Instrum Meas 69(4):1082–1102. https://doi.org/10.1109/TIM.2019.2912239
Ma K, Li H, Yong H, Wang Z, Meng D, Zhang L (2017) Robust multi-exposure image fusion: a structural patch decomposition approach. IEEE Trans Image Process 26(5):2519–2532. https://doi.org/10.1109/TIP.2017.2671921
Nair D, Sankaran P (2022) Benchmarking single image dehazing methods. SN Comput Sci. https://doi.org/10.1007/s42979-021-00925-w
Nan D, Bi DY, He LY, Ma SP, Fan ZL (2016) A variational framework for single image dehazing based on restoration. KSII Trans Internet Inf Syst 10(3):1182–1194. https://doi.org/10.3837/tiis.2016.03.013
Qi G, Chang L, Luo Y, Chen Y, Zhu Z, Wang S (2020) A precise multi-exposure image fusion method based on low-level features. Sensors (switzerland) 20(6):1–18. https://doi.org/10.3390/s20061597
Rong Z, Jun WL (2014) Improved wavelet transform algorithm for single image dehazing. Optik (stuttg) 125(13):3064–3066. https://doi.org/10.1016/j.ijleo.2013.12.077
Salazar-Colores S, Cruz-Aceves I (2018) Single image dehazing using a multilayer perceptron. J Electron Imaging 27(4):043022
Salazar-Colores S, Alberto-Moreno H, Ortiz-Echeverri CJ, Flores G (2020) Desmoking laparoscopy surgery images using an image-to-image translation guided by an embedded dark channel. pp. 1–9. http://arxiv.org/abs/2004.08947.
Sdiri B, Beghdadi A, Cheikh FA, Pedersen M, Elle OJ (2016) “An adaptive contrast enhancement method for stereo endoscopic images combining binocular just noticeable difference model and depth information. IST Int Sympos Electron Imaging Sci Technol. https://doi.org/10.2352/ISSN.2470-1173.2016.13.IQSP-212
Stoyanov D (2012) Surgical vision. Ann Biomed Eng 40(2):332–345. https://doi.org/10.1007/s10439-011-0441-z
Tan RT (2008a) Visibility in bad weather. Comput vis Pattern Recogn CVPR 2008:1–8
Tan RT (2008b) Visibility in bad weather from a single image. 26th IEEE Conf Comput vis Pattern Recognit CVPR. https://doi.org/10.1109/CVPR.2008.4587643
Tarel JP, Hautière N (2009) Fast visibility restoration from a single color or gray level image. Proc IEEE Int Conf Comput vis 2009:2201–2208. https://doi.org/10.1109/ICCV.2009.5459251
Thomas G, Flores-Tapia D, Pistorius S (2011) Histogram specification: a fast and flexible method to process digital images. IEEE Trans Instrum Meas 60(5):1565–1578. https://doi.org/10.1109/TIM.2010.2089110
Twinanda AP, Shehata S, Mutter D, Marescaux J, De Mathelin M, Padoy N (2017) EndoNet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging 36(1):86–97. https://doi.org/10.1109/TMI.2016.2593957
Vese LA, Osher SJ (2003) Modeling textures with total variation minimization and oscillating patterns in image processing. J Sci Comput 19(1–3):553–572. https://doi.org/10.1023/A:1025384832106
Yin L, Zheng M, Qi G, Zhu Z, Jin F, Sim J (2019) A novel image fusion framework based on sparse representation and pulse coupled neural network. IEEE Access 7:98290–98305. https://doi.org/10.1109/ACCESS.2019.2929303
Yu Z, Bajaj C (2004) A fast and adaptive method for image contrast enhancement. Proc Int Conf Image Process ICIP 5:1001–1004. https://doi.org/10.1109/icip.2004.1419470
Zhu Q, Mai J, Shao L (2015) A fast single image haze removal algorithm using color attenuation prior. IEEE Trans Image Process 24(11):3522–3533. https://doi.org/10.1109/TIP.2015.2446191
Zhu Z, Chai Y, Yin H, Li Y, Liu Z (2016) A novel dictionary learning approach for multi-modality medical image fusion. Neurocomputing 214:471–482. https://doi.org/10.1016/j.neucom.2016.06.036
Funding
This research received no external funding.
Author information
Authors and Affiliations
Contributions
Conceptualization: MAA, KBK; Investigation: ER,SUK, Methodology: MAA, Supervision: KBK
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no conflict of interest.
Additional information
Communicated by Jia-Bao Liu.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Azam, M.A., Khan, K.B., Rehman, E. et al. Smoke removal and image enhancement of laparoscopic images by an artificial multi-exposure image fusion method. Soft Comput 26, 8003–8015 (2022). https://doi.org/10.1007/s00500-022-06990-4
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-022-06990-4