CONFERENCE PROCEEDINGS
Advanced Search
Home > Journals > J. Med. Imag. > Volume 10 > Issue 3 > Article
28 June 2023 Fusion of imaging and non-imaging data for disease trajectory prediction for coronavirus disease 2019 patients
Amara Tariq, Siyi Tang, Hifza Sakhi, Leo Anthony G. Celi, Janice M. Newsome, Daniel L. Rubin, Hari Trivedi, Judy W. Gichoya, Imon Banerjee
Author Affiliations +
Journal of Medical Imaging, Vol. 10, Issue 3, 034004 (June 2023). https://doi.org/10.1117/1.JMI.10.3.034004
Abstract

Purpose

Our study investigates whether graph-based fusion of imaging data with non-imaging electronic health records (EHR) data can improve the prediction of the disease trajectories for patients with coronavirus disease 2019 (COVID-19) beyond the prediction performance of only imaging or non-imaging EHR data.

Approach

We present a fusion framework for fine-grained clinical outcome prediction [discharge, intensive care unit (ICU) admission, or death] that fuses imaging and non-imaging information using a similarity-based graph structure. Node features are represented by image embedding, and edges are encoded with clinical or demographic similarity.

Results

Experiments on data collected from the Emory Healthcare Network indicate that our fusion modeling scheme performs consistently better than predictive models developed using only imaging or non-imaging features, with area under the receiver operating characteristics curve of 0.76, 0.90, and 0.75 for discharge from hospital, mortality, and ICU admission, respectively. External validation was performed on data collected from the Mayo Clinic. Our scheme highlights known biases in the model prediction, such as bias against patients with alcohol abuse history and bias based on insurance status.

Conclusions

Our study signifies the importance of the fusion of multiple data modalities for the accurate prediction of clinical trajectories. The proposed graph structure can model relationships between patients based on non-imaging EHR data, and graph convolutional networks can fuse this relationship information with imaging data to effectively predict future disease trajectory more effectively than models employing only imaging or non-imaging data. Our graph-based fusion modeling frameworks can be easily extended to other prediction tasks to efficiently combine imaging data with non-imaging clinical data.
© 2023 Society of Photo-Optical Instrumentation Engineers (SPIE)
Citation Download Citation
Amara Tariq, Siyi Tang, Hifza Sakhi, Leo Anthony G. Celi, Janice M. Newsome, Daniel L. Rubin, Hari Trivedi, Judy W. Gichoya, and Imon Banerjee "Fusion of imaging and non-imaging data for disease trajectory prediction for coronavirus disease 2019 patients," Journal of Medical Imaging 10(3), 034004 (28 June 2023). https://doi.org/10.1117/1.JMI.10.3.034004
Received: 3 November 2022; Accepted: 13 June 2023; Published: 28 June 2023
ACCESS THE FULL ARTICLE
ORGANIZATIONAL
Sign in with credentials provided by your organization.
INSTITUTIONAL
Select your institution to access the SPIE Digital Library.
SELECT YOUR INSTITUTION
PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.
PERSONAL SIGN IN
No SPIE Account? Create one
PURCHASE THIS CONTENT
SUBSCRIBE TO DIGITAL LIBRARY
50 downloads per 1-year subscription
Members: $195
Non-members: $335 ADD TO CART
25 downloads per 1 - year subscription
Members: $145
Non-members: $250 ADD TO CART
PURCHASE SINGLE ARTICLE
Includes PDF, HTML & Video, when available
Members: $24.00
Non-members: $28.00 ADD TO CART
JOURNAL ARTICLE
 16 PAGES
DOWNLOAD PAPER SAVE TO MY LIBRARY
GET CITATION
Advertisement
Advertisement
RIGHTS & PERMISSIONS
Get copyright permission  Get copyright permission on Copyright Marketplace
KEYWORDS
Data modeling
Data fusion
Diseases and disorders
Image fusion
Chest imaging
Performance modeling
Modeling
RELATED CONTENT
Subscribe to Digital Library
Receive Erratum Email Alert
Back to Top