When Multi-Focus Image Fusion Networks Meet Traditional Edge-Preservation Technology

Abstract

Generating the decision map with accurate boundaries is the key to fusing multi-focus images. In this paper, we introduce edge-preservation (EP) techniques into neural networks to improve the quality of decision maps, supported by an interesting phenomenon we found: the maps generated by traditional EP techniques are similar to the feature maps in the trained network with excellent performance. Based on the manifold theory in the field of edge-preservation, we propose a novel edge-aware layer derived from isometric domain transformation and a recursive filter, which effectively eliminates burrs and pseudo-edges in the decision map by highlighting the edge discrepancy between the focused and defocused regions. This edge-aware layer is incorporated to a Siamese-style encoder and a decoder to form a complete segmentation architecture, termed Y-Net, which can contrastively learn and capture the feature differences of the sourced images with a relatively small number of training data (i.e., 10,000 image pairs). In addition, a new strategy based on randomization is devised to generate masks and simulate multi-focus images with natural images, which alleviates the absence of ground-truth and the lack of training sets in multi-focus image fusion (MFIF) task. The experimental results on four publicly available datasets demonstrate that Y-Net with the edge-aware layers is superior to other state-of-the-art fusion networks in terms of qualitative and quantitative comparison.

Appendix

To simplify the nearest neighbor strategy, we only consider the distance between $x_i$ and $x_{i+1}$. $x_{i+1}$ and $x_{i+2}$ are represented by $x_i+h$ and $x_i+2h$, respectively, since sapling interval is equal to h. Since only one interval is calculated, the coefficients of the slope change term and the distance change term are reset to 1. The desired 1D domain is expressed as ${\mathscr {D}}(x_i)$, where ${\mathscr {D}}(x_i)={\mathscr {T}}(x_i,I(x_i))$. Considering Eq. (3), the distance between the nearest $x_i$ and $x_{i+1}$ in 1D domain can be computed by:

$$\begin{aligned} \begin{array}{r} \left| {\mathscr {D}}\left( x_{i}\!\!+\!\!h\right) \!\!-\!\!{\mathscr {D}}\left( x_{i}\right) \right| \!=\!L_2(x_i, x_{i}\!\!+\!\!h)\!+\!SC(x_i)\!+\!DC(x_i) \end{array} \nonumber \\ \end{aligned}$$

(12)

To eliminate the absolute value sign on the left side of the equation, appropriate constraints need to be imposed on ${\mathscr {D}}(x_{i}+h)\ge {\mathscr {D}}(x_i)$. Divide both sides of the equation by h, and then takes the limit $h \rightarrow 0$, as follows:

$$\begin{aligned} \begin{array}{l} {\mathop {\lim }\limits _{h \rightarrow 0} \left| {\frac{{\mathcal{D}\left( {{x_i} + h} \right) - \mathcal{D}\left( {{x_i}} \right) }}{h}} \right| } = \mathop {\lim }\limits _{h \rightarrow 0}\!\! \sqrt{\frac{{{h^2} + {{\left( {\xi \left( {{x_i} + h} \right) - \xi \left( {{x_i}} \right) } \right) }^2}}}{{{h^2}}}}\\ \qquad \qquad \qquad \qquad \qquad \qquad + \left| {\frac{{\xi \left( {{x_i} + h} \right) - \xi \left( {{x_i}} \right) }}{{{h^2}}} \!\!- } \right. {\mathop {\frac{{\xi \left( {{x_i} + 2h} \right) - \xi \left( {{x_i}} \right) }}{{2{h^2}}}\vert }}\\ \qquad \qquad \qquad \qquad \qquad \qquad { + \left| {\frac{{\xi \left( {{x_i} + 2h} \right) + \xi \left( {{x_i}} \right) }}{{2h}} - \frac{{\xi \left( {{x_i} + h} \right) }}{h}} \right| }\\ \qquad \qquad \qquad \qquad \qquad \quad =\mathop {\lim }\limits _{h \rightarrow 0}\!\! \sqrt{\!1 \!\!+\!\! {{\!\left( {\frac{{\!\!\xi \left( {{x_i} + h} \right) - \xi \left( {{x_i}} \right) }}{h}} \!\right) }^2}}\\ \qquad \qquad \qquad \qquad \qquad \qquad + \left| \! {\frac{{\xi ({x_i} + h) - \xi ({x_i})}}{{{h^2}}} \!\!-\!\! \frac{{\xi ({x_i} + 2h) \!-\!\! \xi ({x_i})}}{{2{h^2}}}} \!\right| \\ \qquad \qquad \qquad \qquad \qquad \qquad + \left| {\frac{{\xi \left( {{x_i} + 2h} \right) - \xi \left( {{x_i} + h} \right) - \left( {\xi \left( {{x_i} + h} \right) - \xi \left( {{x_i}} \right) } \right) }}{{2h}}} \right| \\ \qquad \qquad \qquad \qquad \qquad \quad = \mathop {\lim }\limits _{h \rightarrow 0}\!\! \sqrt{1 \!\!+\!\! {{\left( {{\xi ^\prime }({x_i})} \right) }^2}} \\ \qquad \qquad \qquad \qquad \qquad \qquad + \frac{1}{2}\left| {\frac{{\frac{{\xi ({x_i} + 2h) - \xi ({x_i+h})}}{h} - \frac{{\xi ({x_i} + h) - \xi ({x_i})}}{h}}}{h}} \right| \\ \qquad \qquad \qquad \qquad \qquad \qquad + \frac{1}{2}\left| {\frac{{\xi \left( {{x_i} + 2h} \right) - \xi \left( {{x_i} + h} \right) }}{h} {-} \frac{{\xi \left( {{x_i} + h} \right) - \xi \left( {{x_i}} \right) }}{h}} \right| \\ \quad \qquad \qquad \qquad \qquad \qquad =\mathop {\lim }\limits _{h \rightarrow 0} \!\!\sqrt{\!1 \!\!+\!\! {{\left( {{\xi ^\prime }({x_i})} \right) }^2}} \!\!+\!\! \left| {\frac{{{\xi ^\prime }({x_i} + h) - {\xi ^\prime }({x_i})}}{h}} \right| \\ \quad \qquad \qquad \qquad \qquad \qquad \quad + \frac{1}{2}\!\!\left| {{\xi ^\prime }\!\left( {{x_i} \!\!+\!\! h} \right) \!\!-\!\! {\xi ^\prime }\!\left( {{x_i}} \right) } \right| \\ \quad \quad \qquad \qquad \qquad \qquad \quad =\sqrt{1 \!+\! {{\left( {{\xi ^\prime }({x_i})} \right) }^2}} \!\!+\!\! \frac{1}{2}\left| {{\xi ^{\prime \prime }}({x_i})} \right| \\ \quad \qquad \qquad \qquad \qquad \qquad \qquad \!\!+\!\! \mathop {\lim }\limits _{h \rightarrow 0} \frac{1}{2}\left| {{\xi ^\prime }\left( {{x_i} \!+\! h} \right) \!-\! {\xi ^\prime }\left( {{x_i}} \right) } \right| \mathrm{{}} \end{array} \nonumber \\ \end{aligned}$$

(13)

Considering that the minimum sampling interval h is 1 for images, Eq. (13) is expressed by Eq. (14):

$$\begin{aligned} \begin{array}{l} {{\mathscr {D}}}^{\prime }\!\left( x_{i}\right) =\sqrt{1+\left( \xi ^{\prime }\left( x_{i}\right) \right) ^{2}}+\left. \frac{1}{2}\right. \vert \xi ^{\prime \prime }(x_i)\vert \\ \qquad \quad \qquad \quad +\frac{1}{2}\left| \xi ^{\prime }\left( x_{i+1}\right) -\xi ^{\prime }\left( x_{i}\right) \right| \\ \quad \qquad \quad \quad =\sqrt{1+\left( \xi ^{\prime }\!\left( x_{i}\right) \!\right) ^{2}}+\left. \frac{1}{2}\right. \vert \xi ^{\prime \prime }(x_i)\vert +\frac{1}{2}\vert \xi ^{\prime \prime }(x_i)\vert \\ \quad \qquad \quad \quad =\sqrt{1+\left( \xi ^{\prime }\left( x_{i}\right) \right) ^{2}}+\left. \right. \vert \xi ^{\prime \prime }(x_i)\vert , \end{array} \end{aligned}$$

(14)

where ${{\mathscr {D}}}^{\prime }(x_i)$ denotes the derivative of ${{\mathscr {D}}}(x_i)$ with respect to $x_i$. By integrating ${{\mathscr {D}}}^{\prime }(x_i)$ and granting ${\mathscr {D}}(0)=0$, ${\mathscr {D}}(x_i)$ is expressed as:

$$\begin{aligned} {{\mathscr {D}}}\left( x_{i}\right) =\int _{0}^{x_{i}}\sqrt{1+\left( \xi ^{\prime }\left( x\right) \right) ^{2}}\!+\!\left| \xi ^{\prime \prime }\left( x\right) \right| dx \end{aligned}$$

(15)

${\mathscr {D}}$ shows how ${\mathscr {T}}$ transforms curve C from ${\mathbb {R}}^2$ to ${\mathbb {R}}$ while preserving the edges. If the signal $\xi$ has c channels, we obtain the ${\mathscr {D}}$ shown in Eq. (16) to transform the curve in ${\mathbb {R}}^{c+1}$ to ${\mathbb {R}}$.

$$\begin{aligned} \begin{array}{l} {{\mathscr {D}}}\left( x_{i}\right) \!=\!\int _{0}^{x_{i}} \sqrt{1\!+\!\sum \limits _{m=1}^{c}\left( \xi _{m}^{\prime }\left( x\right) \right) ^{2}}\!+\!\sum \limits _{m=1}^{c} \left| \xi _{m}^{\prime \prime }\left( x\right) \right| dx \end{array} \end{aligned}$$

(16)

For a 1D signal $\xi$ with c channels, a warping ${\mathscr {D}}$ of 1D spatial domain is introduced and denoted by ${\mathscr {T}}$: ${\mathbb {R}}^{c+1} \rightarrow {\mathbb {R}}$, where ${\mathscr {D}}(x)=(x, \xi _1(x), \xi _2(x),...,\xi _c(x))$. ${\mathscr {D}}$ is named domain transformation in this study. Benefiting from the design of simultaneously processing multiple channels, ${\mathscr {D}}$ protects the edges from artifacts.

Besides, two parameters are constructed, $\sigma _r$ and $\sigma _s$, which can be determined by the input feature maps to realize the self-adaption of ${\mathscr {D}}$ so as not hinder network training. These two parameters can be used to control the space and range of the signal from the perspective of the signal. Then, the complete form of ${\mathscr {D}}$ is represented as:

$$\begin{aligned} \begin{array}{l} {{\mathscr {D}}}\left( x_{i}\right) =\int _{0}^{x_{i}} \sqrt{1+\left( \frac{\sigma _{s}}{\sigma _{r}}\right) \sum \limits _{m=1}^{c}\left( \xi _{m}^{\prime }\left( x\right) \right) ^{2}}+\frac{\sigma _{s}}{\sigma _{r}} \sum \limits _{m=1}^{c} \\ \quad \qquad \qquad \frac{1}{2}\left| \xi _{m}^{\prime \prime }\left( x\right) \right| +\sigma _{r}\sum \limits _{m=1}^{c}\frac{1}{2}\left| \xi _{m}^{\prime \prime }\left( x\right) \right| dx, \quad x_i \in \Psi \end{array} \nonumber \\ \end{aligned}$$

(17)