Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Brief Communication
  • Published:

Inferring super-resolution tissue architecture by integrating spatial transcriptomics with histology

Abstract

Spatial transcriptomics (ST) has demonstrated enormous potential for generating intricate molecular maps of cells within tissues. Here we present iStar, a method based on hierarchical image feature extraction that integrates ST data and high-resolution histology images to predict spatial gene expression with super-resolution. Our method enhances gene expression resolution to near-single-cell levels in ST and enables gene expression prediction in tissue sections where only histology images are available.

This is a preview of subscription content, access via your institution

Access options

Buy this article

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Workflow and super-resolution gene expression prediction accuracy of iStar.
Fig. 2: Tissue annotation using iStar.

Similar content being viewed by others

Data availability

We analyzed the following publicly available ST datasets: (1) 10x Xenium human breast cancer data (https://www.10xgenomics.com/products/xenium-in-situ/preview-dataset-human-breast); (2) 10x Xenium mouse brain data (https://www.10xgenomics.com/resources/datasets/fresh-frozen-mouse-brain-replicates-1-standard); (3) human HER2-positive breast cancer ST data reported in Anderson et al. (https://github.com/almaan/her2st); (4) 10x Visium human breast cancer data (https://www.10xgenomics.com/resources/datasets/human-breast-cancer-visium-fresh-frozen-whole-transcriptome-1-standard); (5) 10x Visium human colorectal cancer data (https://www.10xgenomics.com/resources/datasets/human-colorectal-cancer-whole-transcriptome-analysis-1-standard-1-2-0); (6) 10x Visium human prostate cancer data (https://www.10xgenomics.com/resources/datasets/human-prostate-cancer-adenocarcinoma-with-invasive-carcinoma-ffpe-1-standard-1-3-0); (7) human prostate cancer data reported in Erickson et al. (https://doi.org/10.17632/svw96g68dv.1); (8) human clear cell renal cell carcinoma primary tumors reported in Meylan et al. (GSE175540); (9) 10x Visium mouse kidney data (https://www.10xgenomics.com/resources/datasets/adult-mouse-kidney-ffpe-1-standard-1-3-0); (10) 10x Visium mouse brain coronal cut data (https://www.10xgenomics.com/resources/datasets/mouse-brain-coronal-section-2-ffpe-2-standard); (11) 10x Visium mouse brain sagittal cut posterior data (https://www.10xgenomics.com/resources/datasets/mouse-brain-serial-section-2-sagittal-posterior-1-standard); (12) 10x Visium mouse brain olfactory bulb data (https://www.10xgenomics.com/resources/datasets/adult-mouse-olfactory-bulb-1-standard-1). Details of the datasets analyzed in this paper are described in Supplementary Table 3. Gene expression visualizations for other spatial resolutions in the 10x Xenium breast cancer and mouse brain data are available at https://zenodo.org/doi/10.5281/zenodo.10071636. Source data are provided with this paper.

Code availability

The iStar algorithm was implemented in Python and is available on GitHub at https://github.com/daviddaiweizhang/istar.

References

  1. Burgess, D. J. Spatial transcriptomics coming of age. Nat. Rev. Genet. 20, 317 (2019).

    Article  CAS  PubMed  Google Scholar 

  2. Asp, M., Bergenstrahle, J. & Lundeberg, J. Spatially resolved transcriptomes-next generation tools for tissue exploration. Bioessays 42, e1900221 (2020).

    Article  PubMed  Google Scholar 

  3. Crosetto, N., Bienko, M. & van Oudenaarden, A. Spatially resolved transcriptomics and beyond. Nat. Rev. Genet. 16, 57–66 (2015).

    Article  CAS  PubMed  Google Scholar 

  4. Moor, A. E. & Itzkovitz, S. Spatial transcriptomics: paving the way for tissue-level systems biology. Curr. Opin. Biotechnol. 46, 126–133 (2017).

    Article  CAS  PubMed  Google Scholar 

  5. Hu, J. et al. SpaGCN: integrating gene expression, spatial location and histology to identify spatial domains and spatially variable genes by graph convolutional network. Nat. Methods 18, 1342–1351 (2021).

    Article  PubMed  Google Scholar 

  6. Sun, S., Zhu, J. & Zhou, X. Statistical analysis of spatial expression patterns for spatially resolved transcriptomic studies. Nat. Methods 17, 193–200 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  7. Svensson, V., Teichmann, S. A. & Stegle, O. SpatialDE: identification of spatially variable genes. Nat. Methods 15, 343–346 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Dries, R. et al. Giotto: a toolbox for integrative analysis and visualization of spatial expression data. Genome Biol. 22, 78 (2021).

    Article  CAS  PubMed  Google Scholar 

  9. Pham, D. et al. stLearn: integrating spatial location, tissue morphology and gene expression to find cell types, cell–cell interactions and spatial trajectories within undissociated tissues. Preprint at bioRxiv https://doi.org/10.1101/2020.05.31.125658 (2020).

  10. Asp, M. et al. A spatiotemporal organ-wide gene expression and cell atlas of the developing human heart. Cell 179, 1647–1660 e1619 (2019).

    Article  CAS  PubMed  Google Scholar 

  11. Wang, X. et al. Three-dimensional intact-tissue sequencing of single-cell transcriptional states. Science 361, eaat5691 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  12. Lubeck, E., Coskun, A. F., Zhiyentayev, T., Ahmad, M. & Cai, L. Single-cell in situ RNA profiling by sequential hybridization. Nat. Methods 11, 360–361 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  13. Shah, S., Lubeck, E., Zhou, W. & Cai, L. In situ transcription profiling of single cells reveals spatial organization of cells in the mouse hippocampus. Neuron 92, 342–357 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Eng, C. L. et al. Transcriptome-scale super-resolved imaging in tissues by RNA seqFISH. Nature 568, 235–239 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Moffitt, J. R. et al. Molecular, spatial, and functional single-cell profiling of the hypothalamic preoptic region. Science 362, eaau5324 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  16. Chen, K. H., Boettiger, A. N., Moffitt, J. R., Wang, S. & Zhuang, X. RNA imaging. Spatially resolved, highly multiplexed RNA profiling in single cells. Science 348, aaa6090 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  17. Stickels, R. R. et al. Highly sensitive spatial transcriptomics at near-cellular resolution with Slide-seqV2. Nat. Biotechnol. 39, 313–319 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  18. Chen, A. et al. Spatiotemporal transcriptomic atlas of mouse organogenesis using DNA nanoball-patterned arrays. Cell 185, 1777–1792 e1721 (2022).

    Article  CAS  PubMed  Google Scholar 

  19. Badea, L. & Stanescu, E. Identifying transcriptomic correlates of histology using deep learning. PLoS ONE 15, e0242858 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. Ash, J. T., Darnell, G., Munro, D. & Engelhardt, B. E. Joint analysis of expression levels and histological images identifies genes associated with tissue morphology. Nat. Commun. 12, 1609 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  21. Schmauch, B. et al. A deep learning model to predict RNA-Seq expression of tumours from whole slide images. Nat. Commun. 11, 3877 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  22. Chen, R. J. et al. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 16144–16155 (IEEE, 2022).

  23. Liu, Z. et al. Swin transformer: hierarchical vision transformer using shifted windows. In Proc. IEEE/CVF International Conference on Computer Vision, 10012–10022 (2021).

  24. Han, K. et al. Transformer in transformer. Adv. Neural Inf. Process. Syst. 34, 15908–15919 (2021).

    Google Scholar 

  25. Caron, M. et al. Emerging properties in self-supervised vision transformers. In Proc. IEEE/CVF International Conference On Computer Vision 9650–9660 (2021).

  26. Bao, H., Dong, L., Piao, S. & Wei, F. Beit: Bert pre-training of image transformers. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. (ICLR, 2022).

  27. Janesick, A. et al. High resolution mapping of the breast cancer tumor microenvironment using integrated single cell, spatial and in situ analysis of FFPE tissue. Preprint at bioRxiv https://doi.org/10.1101/2022.10.06.510405 (2022).

  28. Bergenstråhle, L. et al. Super-resolved spatial transcriptomics by deep data fusion. Nat. Biotechnol. 40, 476–479 (2022).

    Article  PubMed  Google Scholar 

  29. Wang, Z., Bovik, A. C., Sheikh, H. R. & Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004).

    Article  PubMed  Google Scholar 

  30. Hamerly, G. & Elkan, C. Learning the k in k-means. Adv. Neural Inf. Process. Syst. 16, 281–288 (2003).

    Google Scholar 

  31. Andersson, A. et al. Spatial deconvolution of HER2-positive breast cancer delineates tumor-associated cell type interactions. Nat. Commun. 12, 6012 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  32. Stahl, P. L. et al. Visualization and analysis of gene expression in tissue sections by spatial transcriptomics. Science 353, 78–82 (2016).

    Article  CAS  Google Scholar 

  33. Wu, S. Z. et al. A single-cell and spatially resolved atlas of human breast cancers. Nat. Genet. 53, 1334–1347 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  34. Sautes-Fridman, C., Petitprez, F., Calderaro, J. & Fridman, W. H. Tertiary lymphoid structures in the era of cancer immunotherapy. Nat. Rev. Cancer 19, 307–325 (2019).

    Article  CAS  PubMed  Google Scholar 

  35. Fridman, W. H. et al. B cells and tertiary lymphoid structures as determinants of tumour immune contexture and clinical outcome. Nat. Rev. Clin. Oncol. 19, 441–457 (2022).

    Article  CAS  PubMed  Google Scholar 

  36. Helmink, B. A. et al. B cells and tertiary lymphoid structures promote immunotherapy response. Nature 577, 549–555 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  37. Petitprez, F. et al. B cells are associated with survival and immunotherapy response in sarcoma. Nature 577, 556–560 (2020).

    Article  CAS  PubMed  Google Scholar 

  38. Cabrita, R. et al. Tertiary lymphoid structures improve immunotherapy and survival in melanoma. Nature 577, 561–565 (2020).

    Article  CAS  PubMed  Google Scholar 

  39. Lin, J. R. et al. Multiplexed 3D atlas of state transitions and immune interaction in colorectal cancer. Cell 186, 363–381 e319 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  40. benchmark. University of Pennsylvania https://upenn.box.com/v/istar-results-benchmark

  41. Steiner, A. et al. How to train your ViT? Data, augmentation, and regularization in vision transformers. Trans. Mach. Learn Res. https://openreview.net/pdf?id=4nPswr1KcP (2022).

  42. Hölscher, D. L. et al. Next-generation morphometry for pathomics-data mining in histopathology. Nat. Commun. 14, 470 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  43. Rappez, L. et al. SpaceM reveals metabolic states of single cells. Nat. Methods 18, 799–805 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  44. Xu, B., Wang, N., Chen, T. & Li, M. Empirical evaluation of rectified activations in convolutional network. Preprint at arXiv https://doi.org/10.48550/arXiv.1505.00853 (2015).

  45. Clevert, D.-A., Unterthiner, T. & Hochreiter, S. Fast and accurate deep network learning by exponential linear units (ELUs). In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. (ICLR, 2016).

  46. Ringnér, M. What is principal component analysis? Nat Biotechnol. 26, 303–304 (2008).

    Article  PubMed  Google Scholar 

  47. McInnes, L., Healy, J. & Melville, J. UMAP: uniform manifold approximation and projection for dimension reduction. Preprint at arXiv https://doi.org/10.48550/arXiv.1802.03426 (2018).

  48. Schurch, C. M. et al. Coordinated cellular neighborhoods orchestrate antitumoral immunity at the colorectal cancer invasive front. Cell 182, 1341–1359 e1319 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  49. Aran, D. et al. Reference-based analysis of lung single-cell sequencing reveals a transitional profibrotic macrophage. Nat. Immunol. 20, 163–172 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  50. Hu, J. et al. Iterative transfer learning with neural network for clustering and cell type classification in single-cell RNA-seq analysis. Nat. Mach. Intell. 2, 607–618 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  51. Lu, Y. The level weighted structural similarity loss: a step away from MSE. In Proc. AAAI Conference on Artificial Intelligence 33, 9989–9990 (2019).

    Google Scholar 

  52. Lai, W.-S., Huang, J.-B., Ahuja, N. & Yang, M.-H. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 624–632 (2017).

  53. Dahl, R., Norouzi, M. & Shlens, J. Pixel recursive super resolution. Proc. IEEE International Conference on Computer Vision 5439–5448 (2017).

  54. Masutani, E. M., Bahrami, N. & Hsiao, A. Deep learning single-frame and multiframe super-resolution for cardiac MRI. Radiology 295, 552–561 (2020).

    Article  PubMed  Google Scholar 

  55. Anwar, S., Khan, S. & Barnes, N. A deep journey into super-resolution: a survey. ACM Comput. Surv. 53, 1–34 (2020).

    Article  Google Scholar 

  56. Wang, Z., Chen, J. & Hoi, S. C. Deep learning for image super-resolution: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 43, 3365–3387 (2020).

    Article  Google Scholar 

  57. Ma, Y. & Zhou, X. Spatially informed cell-type deconvolution for spatial transcriptomics. Nat. Biotechnol. 40, 1349–1359 (2022).

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

M.L. was supported by the following NIH grants: R01GM125301, R01EY030192, R01HL150359, R01HG013185 and P01AG066597. E.B.L. was supported by NIH grant P01AG066597. L.W. was supported in part by NIH grant R01CA266280, the Cancer Prevention and Research Institute of Texas (CPRIT) award RP200385, the University Cancer Foundation via the Institutional Research Grant Program at the University of Texas MD Anderson Cancer Center, the Andrew Sabin Family Foundation, and the Break Through Cancer Foundation. We thank M. Meylan and W. H. Fridman for sharing the kidney cancer histology image data. We also thank Erickson, Lamb and Lundberg for sharing the prostate cancer histology image and clone annotation data.

Author information

Authors and Affiliations

Authors

Contributions

This study was conceived of and led by M.L. D.Z. designed the model and algorithm, implemented the iStar software and led data analyses with input from M.L., E.E.F., L.W., K.S., G.X.X., M.D.F. and E.B.L. E.E.F. examined histology images. A.S., H.Y., M.Y.Y.L., K.S.C. and J.H. helped with data analyses. L.W. provided marker genes for TLS detection and interpreted results related to cancer. D.Z. and M.L. wrote the paper with feedback from the other co-authors.

Corresponding authors

Correspondence to Daiwei Zhang or Mingyao Li.

Ethics declarations

Competing interests

M.L. receives research funding from Biogen Inc. unrelated to the current manuscript. The other authors declare no competing financial interests.

Peer review

Peer review information

Nature Biotechnology thanks the anonymous reviewers for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Correlation between super-resolution gene expression and histology.

Comparison of iStar’s super-resolution gene expression patterns with the paired histology image in the Xenium-derived pseudo-Visium data. Spot boundaries are highlighted.

Extended Data Fig. 2 Resolution enhancement at various scales.

Visualization of predicted super-resolution gene expressions by iStar at various scales of resolution enhancement for three breast cancer-related genes (ESR1, ERBB2, and PGR) in the Xenium-derived pseudo-Visium data.

Extended Data Fig. 3 Numerical evaluation of prediction accuracy.

Prediction accuracy of iStar and XFuse as measured by root mean squared error (RMSE) and structural similarity index measure (SSIM) for all the 313 genes in the Xenium-derived pseudo-Visium data obtained from a breast cancer patient. In each scatter plot, a dot represents a gene. In this analysis, Section 1 was first treated as the training sample in which in-sample prediction was perfromed for Section 1 and then out-of-sample prediction was perfromed for Section 2. We then repeated this analysis by treating Section 2 as the ‘in-sample’ and Section 1 as the ‘out-of-sample’. The evaluation metrics in’Average’ is the average of those in ‘Sections 1’ and ‘Section 2’.

Source data

Extended Data Fig. 4 Visualization of single-cell level gene expression prediction.

Visualization of single-cell level gene expression predicted by iStar for the pseudo-Visium breast cancer data derived from Xenium data. Shown on the left is the ground truth single-cell level gene expression directly measured by Xenium, and shown on the right is the single-cell level gene expression predicted by iStar. For each gene, the root mean squared error (RMSE) and Pearson’s correlation coefficient (PCC) between the prediction and the ground truth across the whole tissue and within the shown region are displayed.

Extended Data Fig. 5 Prediction accuracy evaluation of single-cell level gene expression prediction.

Prediction accuracy of iStar and XFuse for single-cell level gene expression prediction as measured by root mean squared error (RMSE) for all the 313 genes in the Xenium-derived pseudo-Visium data obtained from a breast cancer patient. In each scatter plot, a dot represents a gene. In this analysis, Section 1 was first treated as the training sample in which in-sample prediction was performed for Section 1 and then out-of-sample prediction was performed for Section 2. We then repeated this analysis by treating Section 2 as the ‘in-sample’ and Section 1 as the ‘out-of-sample’. The evaluation metrics in’Average’ is the average of those in ‘Sections 1’ and ‘Section 2’. We stratified cells by the quantiles of their cell size.

Source data

Extended Data Fig. 6 Super-resolution vs spot-level signature scoring of tertiary lymphoid structures (TLSs).

iStar detected tertiary lymphoid structures (TLSs) in the Anderson et al. (2021) HER2+ breast cancer dataset. Displayed are the predicted TLS scores by iStar and the original publication, along with the pathologist’s manual annotation reported in the original publication. Super-resolution was performed with 128x resolution enhancement.

Extended Data Fig. 7 Super-resolution gene expression prediction in the Xenium-derived pseudo-Visium mouse brain data.

Visualization of the spot-level training data, ground truth gene expression, and predicted super-resolution gene expressions by iStar and XFuse for 24 highly variable genes, whose variances are in the 80%-100% quantiles among all the 248 genes in the Xenium-derived pseudo-Visium data obtained from mouse brain. The variance quantiles of the genes in the order of top-left, top-right, bottom-left, bottom-right are equally spaced from 100% to 80% (in descending order). Super-resolution gene expressions are visualized at the scale of 8x resolution enhancement.

Extended Data Fig. 8 Prediction accuracy evaluation and gene-based segmentation comparison in the Xenium-derived pseudo-Visium mouse brain data.

a. Prediction accuracy of by iStar and XFuse as measured by root mean squared error (RMSE) and structural similarity index measure (SSIM) for all the 248 genes in the Xenium-derived pseudo-Visium data obtained from mouse brain. In each scatter plot, a dot represents one gene. b. Segmentation of the Xenium-derived pseudo-Visium data obtained from mouse brain by iStar and XFuse using all 248 genes available in this dataset. Super-resolution was performed with 128x resolution enhancement.

Extended Data Fig. 9 Gene-based segmentation of mouse brain datasets.

Analyses of a. mouse brain (coronal cut), b. mouse brain posterior (sagittal cut), and c. mouse brain ofactory bulb Visium datasets generated by 10x Genomics. For each dataset, iStar was applied to enhance the resolution of the top 1000 most highly variable genes. Super-resolution was performed with 128x resolution enhancement. Segmentations by iStar identified fine-grained tissue structures in the mouse brain and agreed with the Allen Brain Atlas annotations.

Extended Data Fig. 10 Gene-based tissue segmentation and tertiary lymphoid structure (TLS) signature scoring of cancer datasets and a mouse kidney dataset.

Comparison of gene-based segmentation by iStar with manual tissue annotation in a. mouse kidney and b. prostate cancer Visium datasets generated by 10x Genomics. c. Gene-based segmentation by iStar in a colorectal cancer Visium dataset by 10x Genomics. d. Comparison of tertiary lymphoid structure (TLS) signature score by iStar with manual TLS annotation in a kidney cancer Visium dataset generated by Meylan et al. (2022).

Supplementary information

Supplementary Information

Supplementary Tables 1–3 and Figs. 1–24.

Reporting Summary

Source data

Source Data Extended Data Fig. 3

Statistical source data.

Source Data Extended Data Fig. 5

Statistical source data.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, D., Schroeder, A., Yan, H. et al. Inferring super-resolution tissue architecture by integrating spatial transcriptomics with histology. Nat Biotechnol (2024). https://doi.org/10.1038/s41587-023-02019-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1038/s41587-023-02019-9

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing