MPIN: a macro-pixel integration network for light field super-resolution

Abstract

Most existing light field (LF) super-resolution (SR) methods either fail to fully use angular information or have an unbalanced performance distribution because they use parts of views. To address these issues, we propose a novel integration network based on macro-pixel representation for the LF SR task, named MPIN. Restoring the entire LF image simultaneously, we couple the spatial and angular information by rearranging the four-dimensional LF image into a two-dimensional macro-pixel image. Then, two special convolutions are deployed to extract spatial and angular information, separately. To fully exploit spatial-angular correlations, the integration resblock is designed to merge the two kinds of information for mutual guidance, allowing our method to be angular-coherent. Under the macro-pixel representation, an angular shuffle layer is tailored to improve the spatial resolution of the macro-pixel image, which can effectively avoid aliasing. Extensive experiments on both synthetic and real-world LF datasets demonstrate that our method can achieve better performance than the state-of-the-art methods qualitatively and quantitatively. Moreover, the proposed method has an advantage in preserving the inherent epipolar structures of LF images with a balanced distribution of performance.

摘要

现有的大多数光场超分辨率方法不能充分利用角度信息, 或者由于利用部分视图而产生不均衡的性能。为解决这些问题, 本文提出一种基于宏像素表示的光场图像超分辨率聚合网络模型(称为MPIN)。该网络通过将四维光场图像重新排列成二维宏像素图像, 将空间和角度信息进行耦合, 从而同时恢复整张光场图像。网络利用两种特殊的卷积分别提取空间和角度信息。为充分利用空间—角度相关性, 所设计的聚合残差模块融合两种信息使其相互引导, 以实现角度相干性。在宏像素表示下, 该网络通过扩展角度混洗层来提高宏像素图像的空间分辨率, 有效避免了混叠。在合成和真实光场数据集上的大量实验表明, 本文提出的方法在定性和定量上均实现了比现有方法更好的性能。此外, 该方法在保持光场图像固有极线结构的同时, 具有均衡性能分布的优点。