Abstract
The precise reconstruction of accelerated magnetic resonance imaging (MRI) brings about notable advantages, such as enhanced diagnostic precision and decreased examination costs. In contrast, traditional cardiac MRI necessitates repetitive acquisitions across multiple heartbeats, resulting in prolonged acquisition times. Significant strides have been made in accelerating MRI through deep learning-based reconstruction methods. However, these existing methods encounter certain limitations: (1) The intricate nature of heart reconstruction involving multiple complex time-series data poses a challenge in exploring nonlinear dependencies between temporal contexts. (2) Existing research often overlooks weight sharing in iterative frameworks, impeding the effective capturing of non-local information and, consequently, limiting improvements in model performance. In order to improve cardiac MRI reconstruction, we propose a novel temporal-spatial transformer with a strategy in this study. Based on the multi-level encoder and decoder transformer architecture, we conduct multi-level spatiotemporal information feature aggregation over several adjacent views, that create nonlinear dependencies among features and efficiently learn important information among adjacent cardiac temporal frames. Additionally, in order to improve contextual awareness between neighboring views, we add cross-view attention for temporal information fusion. Furthermore, we introduce an iterative strategy for training weights during the reconstruction process, which improves feature fusion in critical locations and reduces the number of computations required to calculate global feature dependencies. Extensive experiments have demonstrated the substantial superiority of this procedure over the most advanced techniques, suggesting that it has broad potential for clinical use.
- [1] . 2018. MoDL: Model-based deep learning architecture for inverse problems. IEEE Transactions on Medical Imaging 38, 2 (2018), 394–405.Google Scholar
Cross Ref
- [2] . 2020. Free-breathing and ungated dynamic mri using navigator-less spiral storm. IEEE Transactions on Medical Imaging 39, 12 (2020), 3933–3943.Google ScholarCross Ref
- [3] . 2010. AR-REHAB: An augmented reality framework for poststroke-patient rehabilitation. IEEE Transactions on Instrumentation and Measurement 59, 10 (2010), 2554–2563.Google ScholarCross Ref
- [4] . 2021. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 6836–6846.Google ScholarCross Ref
- [5] . 2017. Quo vadis, action recognition? A new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6299–6308.Google ScholarCross Ref
- [6] . 2021. Spatial-temporal transformer for dynamic scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 16372–16382.Google ScholarCross Ref
- [7] . 2020. Super-resolution reconstruction of single anisotropic 3D MR images using residual convolutional neural network. Neurocomputing 392 (2020), 209–220.Google ScholarCross Ref
- [8] . 2007. The potential of haptics technologies. IEEE Instrumentation and Measurement Magazine 10, 1 (2007), 10–17.Google ScholarCross Ref
- [9] . 2018. Digital twins: The convergence of multimedia technologies. IEEE Multimedia 25, 2 (2018), 87–92.Google ScholarDigital Library
- [10] . 2021. Task transformer network for joint MRI reconstruction and super-resolution. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VI 24. Springer, 307–317.Google ScholarDigital Library
- [11] Rui Guo, Hossam El-Rewaidy, Salah Assana, Xiaoying Cai, Amine Amyar, Kelvin Chow, Xiaoming Bi, Tuyen Yankama, Julia Cirillo, Patrick Pierce, Beth Goddu, Long Ngo, and Reza Nezafat. 2022. Accelerated cardiac T1 mapping in four heartbeats with inline MyoMapNet: a deep learning-based T1 estimation approach. Journal of Cardiovascular Magnetic Resonance 24, 1 (2022), 1–15.Google Scholar
- [12] . 2021. Ssan: Separable self-attention network for video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12618–12627.Google ScholarCross Ref
- [13] . 2017. Learning spatio-temporal features with 3d residual networks for action recognition. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 3154–3160.Google ScholarCross Ref
- [14] . 2019. Axial attention in multidimensional transformers. arXiv:1912.12180. Retrieved from https://arxiv.org/abs/1912.12180Google Scholar
- [15] . 2019. Smart healthcare monitoring: A voice pathology detection paradigm for smart cities. Multimedia Systems 25 (2019), 565–575.Google ScholarCross Ref
- [16] . 2019. MRI reconstruction via cascaded channel-wise attention network. In Proceedings of the 2019 IEEE 16th International Symposium on Biomedical Imaging. IEEE, 1622–1626.Google ScholarCross Ref
- [17] . 2007. Improved k–t BLAST and k–t SENSE using FOCUSS. Physics in Medicine and Biology 52, 11 (2007), 3201.Google ScholarCross Ref
- [18] . 2022. Transformer-empowered multi-scale contextual matching and aggregation for multi-contrast MRI super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 20636–20645.Google ScholarCross Ref
- [19] . 2022. Vrt: A video restoration transformer. arXiv:2201.12288. Retrieved from https://arxiv.org/abs/2201.12288Google Scholar
- [20] . 2021. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1833–1844.Google ScholarCross Ref
- [21] Jing Lin, Yuanhao Cai, Xiaowan Hu, Haoqian Wang, Youliang Yan, Xueyi Zou, Henghui Ding, Yulun Zhang, Radu Timofte, and Luc Van Gool. 2022. Flow-guided sparse transformer for video deblurring. In International Conference on Machine Learning. PMLR, 13334–13343.Google Scholar
- [22] . 2011. Accelerated dynamic MRI exploiting sparsity and low-rank structure: kt SLR. IEEE Transactions on Medical Imaging 30, 5 (2011), 1042–1054.Google ScholarCross Ref
- [23] Guangming Wang, Jun Lyu, Fanwen Wang, Chengyan Wang, and Jing Qin. 2024. Multi-level temporal information sharing transformer-based feature reuse network for cardiac MRI reconstruction. In Statistical Atlases and Computational Models of the Heart. Regular and CMRxRecon Challenge Papers (STACOM’23), Oscar Camara, et al. (Eds.)., Lecture Notes in Computer Science, vol 14507. Springer, Cham. Google ScholarDigital Library
- [24] . 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 10012–10022.Google ScholarCross Ref
- [25] . 2018. Performance of U-net based pyramidal lucas-kanade registration on free-breathing multi-b-value diffusion MRI of the kidney. The British Journal of Radiology 91, 1086 (2018), 20170813.Google Scholar
- [26] . 2021. Transfer learning enhanced generative adversarial networks for multi-channel MRI reconstruction. Computers in Biology and Medicine 134 (2021), 104504.Google ScholarDigital Library
- [27] . 2021. PIC-GAN: A parallel imaging coupled generative adversarial network for accelerated multi-channel MRI reconstruction. Diagnostics 11, 1 (2021), 61.Google ScholarCross Ref
- [28] . 2018. Respiratory motion correction for free-breathing 3D abdominal MRI using CNN-based image registration: A feasibility study. The British Journal of Radiology 91, xxxx (2018), 20170788.Google Scholar
- [29] . 2023. Region-focused multi-view transformer-based generative adversarial network for cardiac cine MRI reconstruction. Medical Image Analysis 85 (2023), 102760.Google ScholarCross Ref
- [30] . 2022. DuDoCAF: Dual-domain cross-attention fusion with recurrent transformer for fast multi-contrast MR imaging. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 474–484.Google ScholarDigital Library
- [31] . 2019. Recon-glgan: A global-local context based generative adversarial network for mri reconstruction. In Machine Learning for Medical Image Reconstruction: 2nd International Workshop, MLMIR 2019, Held in Conjunction with MICCAI 2019, Shenzhen, China, October 17, 2019, Proceedings 2. Springer, 3–15.Google ScholarDigital Library
- [32] . 2015. Low-rank plus sparse matrix decomposition for accelerated dynamic MRI with separation of background and dynamic components. Magnetic Resonance in Medicine 73, 3 (2015), 1125–1136.Google ScholarCross Ref
- [33] . 2023. Rethinking video vits: Sparse video tubes for joint image and video learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2214–2224.Google ScholarCross Ref
- [34] . 2018. Convolutional recurrent neural networks for dynamic MR image reconstruction. IEEE Transactions on Medical Imaging 38, 1 (2018), 280–290.Google ScholarCross Ref
- [35] . 2020. DC-WCNN: A deep cascade of wavelet based convolutional neural networks for MR image reconstruction. In Proceedings of the 2020 IEEE 17th International Symposium on Biomedical Imaging. IEEE, 1069–1073.Google ScholarCross Ref
- [36] . 2016. State of the art: Clinical applications of cardiac T1 mapping. Radiology 278, 3 (2016), 658–676.Google ScholarCross Ref
- [37] . 2017. A deep cascade of convolutional neural networks for MR image reconstruction. In Information Processing in Medical Imaging: 25th International Conference, IPMI 2017, Boone, NC, USA, June 25-30, 2017, Proceedings 25. Springer, 647–658.Google ScholarCross Ref
- [38] . 2014. Two-stream convolutional networks for action recognition in videos. Advances in Neural Information Processing Systems 27 (2014).Google Scholar
- [39] . 2016. T1 mapping: Basic techniques and clinical applications. JACC: Cardiovascular Imaging 9, 1 (2016), 67–81.Google ScholarCross Ref
- [40] Alina L. Machidon and Veljko Pejovic. 2021. Deep learning techniques for compressive sensing-based reconstruction and inference–A ubiquitous systems perspective. arXiv preprint arXiv:2105.13191Google Scholar
- [41] . 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017).Google Scholar
- [42] Chengyan Wang, et al., 2023. CMRxRecon: An open cardiac MRI dataset for the competition of accelerated image reconstruction. arXiv preprint arXiv:2309.10836 (2023)Google Scholar
- [43] . 2023. Free-breathing myocardial T1 mapping using inversion-recovery radial FLASH and motion-resolved model-based reconstruction. Magnetic Resonance in Medicine 89, 4 (2023), 1368–1384.Google ScholarCross Ref
- [44] . 2021. End-to-end video instance segmentation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8741–8750.Google ScholarCross Ref
- [45] Syed Umar Amin, Mansour Alsulaiman, Ghulam Muhammad, Mohamed Amine Mekhtiche, and M. Shamim Hossain. 2019. Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion, Future Generation Computer Systems, 101, (2019), 542–554.Google Scholar
- [46] . 2022. NestedFormer: Nested modality-aware transformer for brain tumor segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 140–150.Google ScholarDigital Library
- [47] . 2022. Multiview transformers for video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3333–3343.Google ScholarCross Ref
- [48] . 2022. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10819–10829.Google ScholarCross Ref
Index Terms
- Iterative Temporal-spatial Transformer-based Cardiac T1 Mapping MRI Reconstruction
Recommendations
Multi-level Temporal Information Sharing Transformer-Based Feature Reuse Network for Cardiac MRI Reconstruction
Statistical Atlases and Computational Models of the Heart. Regular and CMRxRecon Challenge PapersAbstractThe accurate reconstruction of accelerated Magnetic Resonance Imaging (MRI) brings significant clinical benefits, including improved diagnostic accuracy and reduced examination costs. Traditional cardiac MRI requires repetitive acquisitions over ...
Model-based reconstruction for T1 mapping using single-shot inversion-recovery radial FLASH
Quantitative parameter mapping in MRI is typically performed as a two-step procedure where serial imaging is followed by pixelwise model fitting. In contrast, model-based reconstructions directly reconstruct parameter maps from raw data without explicit ...
Cine Cardiac MRI Reconstruction Using a Convolutional Recurrent Network with Refinement
Statistical Atlases and Computational Models of the Heart. Regular and CMRxRecon Challenge PapersAbstractCine Magnetic Resonance Imaging (MRI) allows for understanding of the heart’s function and condition in a non-invasive manner. Undersampling of the k-space is employed to reduce the scan duration, thus increasing patient comfort and reducing the ...
Comments