DeepSpot

DeepSpot: Leveraging Spatial Context for Enhanced Spatial Transcriptomics Prediction from H&E Images

Authors: Kalin Nonchev, Sebastian Dawo, Karina Silina, Holger Moch, Sonali Andani, Tumor Profiler Consortium, Viktor Hendrik Koelzer, and Gunnar Rätsch

The preprint is available here.

News

• [09.2026] Introducing Aurora - a no-code platform for virtual spatial transcriptomics from H&E images. Aurora makes virtual spatial transcriptomics accessible without requiring users to have their own GPUs, with a fully automated workflow designed to ensure that spatial transcriptomics predictions are performed correctly. Try the interactive Aurora demo.
• [06.2026] Meet the next generation of DeepSpot: DeepSpot-M: a multimodal foundation model for transcriptome-wide virtual spatial transcriptomics from histology. Code.
• [05.2026] The AESTETIK model we build on, presented in Representation learning for multi-modal spatially resolved transcriptomics data, has been published in Bioinformatics (Oxford)
• [03.2026] Cross-sample AESTETIK, the spatial transcriptomics integration model powering DeepSpot, will be presented at the ICLR 2026 Learning Meaningful Representations of Life.
• [01.2026] Invited talk at 10x Genomics Single Cell & Spatial Discovery Symposium, 2026, Bern, Switzerland
• [12.2025] Contributed talk at NeurIPS 2025 Workshop on Multi-modal Foundation Models and Large Language Models for Life Sciences, 2025, San Diego, USA
• [10.2025] DeepSpot2Cell: Predicting Virtual Single-Cell Spatial Transcriptomics from H&E images using Spot-Level Supervision at NeurIPS 2025 Imageomics. Code.
• [9.2025] DeepSpot was featured in the Eric and Wendy Schmidt Center article “Machine Learning Teams Push the Boundaries of Virtual Spatial Biology in Global Autoimmune Disease Challenge.”
• [5.2025] Updated manuscript with new in-depth experiments and benchmarks, single-cell spatial transcriptomics prediction from H&E images, new datasets link.
• [04.2025] DeepSpot secured 1st place at the Autoimmune ML Challenge organized by the Broad Institute of MIT and Harvard and CrunchDao for predicting single-cell spatial transcriptomics from H&E images link.

Changelog

NEW version (May 2025)

• DeepCell for spatial transcriptomics at single-cell resolution
• new tutorials
• change in the module structure
  • from deepspot.spot import DeepSpot
  • from deepspot.cell import DeepCell
• 8 TB of predicted TCGA spatial transcriptomics data - LUSC, LUAD, SKCM, KIRC
  • 3780 samples
  • 56 317 393 spots
  • 4 cancer types
• more DeepSpot pretrained models on Visium and Xenium data

Do you want to generate spatial transcriptomics data using your H&E images?

We introduce DeepSpot, a novel deep-learning model that predicts spatial transcriptomics from H&E images. DeepSpot employs a deep-set neural network to model spots as bags of sub-spots and integrates multi-level tissue details and spatial context. This integration, supported by the robust pre-trained H&E models, significantly enhances the accuracy and granularity of gene predictions from H&E images.

Fig.: DeepSpot leverages pathology foundation models and spatial tissue context. Workflow of DeepSpot: H&E slides are first divided into tiles, each corresponding to a spot. For each spot, we create a bag of sub-spots by dividing it into sub-tiles that capture the local morphology, and a bag of neighboring spots to represent the global context. A pretrained pathology model extracts tile features, which are input to the model. The concatenated representations are then fed into the gene head predictor, ρgene, to predict spatial gene expression.

Setup

git clone https://github.com/ratschlab/DeepSpot
cd DeepSpot
conda env create --file=environment.yaml
conda activate deepspot
python setup.py install

NB: Please ensure you have installed pyvips depending on your machine’s requirements. We suggest installing pyvips through conda:

conda install conda-forge::pyvips

Install jupyter kernel

python -m ipykernel install --user --name deepspot --display-name "deepspot"

Getting Started

Please take a look at our notebook collection to get started with DeepSpot for Visium or the adapted version of DeepSpot - DeepCell, for Xenium. We provide a small toy examples.

DeepSpot for spatial transcriptomics at spot resolution:

• Spatial transcriptomics data preprocessing spot level resolution
• DeepSpot training
• DeepSpot inference
• DeepSpot inference with pretrained model

DeepCell for spatial transcriptomics at single-cell resolution:

• Spatial transcriptomics data preprocessing single-cell level resolution
• DeepCell training
• DeepCell inference

Pretrained DeepSpot weights

Moreover, we provide pretrained weights for DeepSpot, which were generated during the training of the model in our publication and were used, for example, to generate spatial transcriptomics data for TCGA skin melanoma and kidney cancer slides. Download DeepSpot weights here.

Pathology foundation models

Please ensure that you download the weights for the pathology foundation models and update their file path deepspot/utils/utils_image.py. You may need to agree to specific terms and conditions before downloading.

• UNI weights https://huggingface.co/MahmoodLab/UNI
• Hoptimus0 weights https://huggingface.co/bioptimus/H-optimus-0
• Phikon weights https://huggingface.co/owkin/phikon

TCGA spatial transcriptomics data

We provide publicly the predicted spatial transcriptomics data with over 56 million spots from 3 780 TCGA patients with melanoma or kidney cancer. You can find the data here. Please navigate to the Hugging Face dataset card for more information.

How to start?

pip install datasets

Logging

from huggingface_hub import login, hf_hub_download, snapshot_download
import squidpy as sq
import pandas as pd
import scanpy as sc
import datasets

login(token="YOUR HUGGINGFACE TOKEN")

Load metadata information

# Define dataset details
repo_id = "nonchev/TCGA_digital_spatial_transcriptomics"
filename = "metadata_2025-01-11.csv" # please check which is the latest one

# Create path
file_path = hf_hub_download(repo_id=repo_id, filename=filename, repo_type="dataset")
# Load metata
metadata = pd.read_csv(file_path)
metadata.head()

        dataset slide_type                                          sample_id  n_spots                                          file_path
0     TCGA_SKCM       FFPE  TCGA-BF-AAP6-01Z-00-DX1.EFF1D6E1-CDBC-4401-A10...     5860  TCGA_SKCM/FFPE/TCGA-BF-AAP6-01Z-00-DX1.EFF1D6E...
1     TCGA_SKCM       FFPE  TCGA-FS-A1ZU-06Z-00-DX3.0C477EE6-C085-42BE-8BA...     2856  TCGA_SKCM/FFPE/TCGA-FS-A1ZU-06Z-00-DX3.0C477EE...
2     TCGA_SKCM       FFPE  TCGA-D9-A1X3-06Z-00-DX1.17AC16CC-5B22-46B3-B9C...     6236  TCGA_SKCM/FFPE/TCGA-D9-A1X3-06Z-00-DX1.17AC16C...

Download a single TCGA spatial transcriptomics sample

local_dir = 'TCGA_data'  # Change the folder path as needed

snapshot_download("nonchev/TCGA_digital_spatial_transcriptomics", 
                  local_dir=local_dir,
                  allow_patterns="TCGA_SKCM/FFPE/TCGA-D9-A3Z3-06Z-00-DX1.C4820632-C64D-4661-94DD-9F27F75519C3.h5ad.gz",
                  repo_type="dataset")

adata = sc.read_h5ad("path/to/h5ad.gz")
sq.pl.spatial_scatter(adata, 
                      color=["SOX10", "CD37", "COL1A1", "predicted_label"],
                      size=20, img_alpha=0.8, ncols=2)

Download the entire TCGA digital spatial transcriptomics dataset

local_dir = 'TCGA_data'  # Change the folder path as needed

# Note that the full dataset is around 7TB

snapshot_download("nonchev/TCGA_digital_spatial_transcriptomics", 
                  local_dir=local_dir,
                  repo_type="dataset")

NB: To distinguish in-tissue spots from the background, tiles with a mean RGB value above 200 (near white) were discarded. Additional preprocessing can remove potential image artifacts.

Papers Citing DeepSpot

1. Moritz Schaefer, Zoe Piran, Nils Philipp Walter, Animesh Awasthi, Christoph Bock, J. Leskovec, and Zinaida Good “Transitive Representation Learning Enhances Histopathology Annotation.”. DOI
2. Alice Driessen, Emil Malmsten, Steffen Wolf, Bernd Illing, Isinsu Katircioglu, Adriano Martinelli, R. Stahel, S. Peters, E. Felip, Sanjay Popat, K. Homicsko, Raphael Gottardo, Marianna Rapsomaniki, and Fei Tang “HistoTx: Early fusion of H&E images and spatial transcriptomics at varying spatial transcriptomics resolution with self-supervised learning.” (2026). DOI
3. Zi-Jun Gao, Yi-Jiang Chen, Xiang-De Luo, and Pheng-Ann Heng “CellPlex: Benchmarking Single-Cell Spatial Proteomics Prediction from H&E.” (2026). DOI
4. F. Giraud-Sauveur, Quentin Blampey, Pierre Marza, H. Benkirane, Maria Vakalopoulou, P. Cournède, and S. Christodoulidis “ALHENA, an H&E encoder distilled from transcriptomic foundation models for multi-scale slide analysis.” bioRxiv (2026). DOI
5. Ulysse McConnell, K. Nonchev, V. Koelzer, and G. Rätsch “Slide-level batch structure limits histology-guided supervision of transcriptomic foundation models.” bioRxiv (2026). DOI
6. Qing Wang, Si-Pei Gu, Qi-Zhen Lan, Xiao-Qian Jiang, Yi-Jiang Chen, and Qian-Qian Song “Octave: Scale-Resolved Evaluation of Spatial Gene Expression Prediction from Histology.” bioRxiv (2026). DOI
7. Zi-Jun Gao, Chun-Bin Gu, Jin-Xi Xiang, Xiang-De Luo, and Pheng-Ann Heng “Towards Scalable Context-Aware Single-Cell Spatial Transcriptomics Prediction from Histology Images.” arXiv.org (2026). DOI
8. Shi-Ting Ruan, Xi-Tong Ling, Qi-Ming He, Zi-You Yan, Huai-Tian Yuan, Tian Guan, Ying Xiao, Xu Guan, and Yong-Hong He “SpaFactor: Lightweight Spatial Context-Aware Gene Program Modeling for Histology-to-Transcriptomics Inference.” arXiv.org (2026). DOI
9. Hyun-jong Jang, Kwangil Yim, and Sung-Hak Lee “Artificial Intelligence for Molecular Biomarker Identification in Gastrointestinal and Hepatobiliary Cancers.” International Journal of Molecular Sciences (2026). DOI
10. Zi-Jun Wang, Chong-Yi Yang, Xiao-Ya Tang, En-Zhi Yin, Yu-Xin Yao, Yue-Jun Luo, Jie He, and N. Sun “Trimodal, uncertainty-guided whole-slide framework for genome-scale spatial expression and image-only virtual perturbation in cancer cohorts.” Genome Medicine (2026). DOI
11. Sagnik Bhadury, Ashley P. Tsang, Dipankar Ray, K. Lagisetty, and Arvind Rao “Functional Depth Biomarkers Distinguish Lung Squamous Cell Carcinoma from Lung Adenocarcinoma.” Research Square (2026). DOI
12. Marija Pizurica, Eric Zimmermann, Neil A. Tenenholtz, J. Hall, O. Gevaert, A. Amini, L. Crawford, and Kristen A. Severson “JASPR: Joint Spatial Representation learning of histology and spatial genomics for improved virtual genomic screening and clinical prognostication.” arXiv.org (2026). DOI
13. Pantelis R. Vlachas, K. Nonchev, V. Koelzer, and G. Rätsch “Transcriptomics-Conditioned Virtual Tissue Synthesis via Diffusion Transformers.” bioRxiv (2026). DOI
14. Manuel Tran, R. Gindra, Philipp Putze, Sen-Bai Kang, G. Palla, Tina Kos, C. Falcomatà, Chen Wang, Ruifeng Guo, M. Boxberg, L. Berclaz, L. Lindner, L. Bergmayr, T. Knösel, P. Jurmeister, Frederick Klauschen, K. Homicsko, Raphael Gottardo, Markus Eckstein, C. Matek, Andreas Mock, F. Theis, Dieter Saur, and Ting-Ying Peng “Pan-cancer virtual spatial transcriptomics from routine histology with Phoenix.” bioRxiv (2026). DOI
15. Ju Dai, Kalin Nonchev, V. Koelzer, and G. Rätsch “Towards Cross-Sample Alignment for Multi-Modal Representation Learning in Spatial Transcriptomics.” bioRxiv (2026). DOI
16. Benjamin S. Strope, D. Varghese, William Z. Bowie, Stacy Wang, and Qian Zhu “CancerSTFormer enables multi-scale analysis of spot-resolution spatial transcriptomes and dissects gene and immune regulatory responses to targeted therapies.” bioRxiv (2026). DOI
17. Ashley P. Tsang, Santhoshi N. Krishnan, Reva Kulkarni, Sagnik Bhadury, M. P. di Magliano, T. Frankel, and Arvind Rao “Encoding functional edges in graphs to model spatially varying relationships in the tumor microenvironment.” npj Artificial Intelligence (2026). DOI
18. Till Richter, Eric Zimmermann, J. Hall, Fabian J. Theis, Srivatsan Raghavan, Peter S. Winter, A. Amini, and L. Crawford “Beyond alignment: synergistic integration is required for multimodal cell foundation models.” bioRxiv (2026). DOI
19. Alex Beachum, Xue Xiao, Yuansheng Zhou, Qi-Wei Li, Guang-Hua Xiao, and Lin Xu “Advances in predicting omics profiles from imaging data.” Briefings Bioinform. (2026). DOI
20. Kalin Nonchev, Glib Manaiev, V. Koelzer, and G. Rätsch “DeepSpot2Cell: Predicting Virtual Single-Cell Spatial Transcriptomics from H&E images using Spot-Level Supervision.” bioRxiv (2025). DOI
21. C. Delrue, and M. Speeckaert “Transcriptomic Signatures in IgA Nephropathy: From Renal Tissue to Precision Risk Stratification.” International Journal of Molecular Sciences (2025). DOI
22. Jonathan Xu, Michelle Jiang, Shunsuke Koga, Nancy R. Zhang, and Zhi Huang “SpatialFinder: a human-in-the-loop vision-language framework for prioritizing high-value regions in spatial transcriptomics.” bioRxiv (2025). DOI
23. Moritz Schaefer, Kalin Nonchev, Animesh Awasthi, Jake Burton, V. Koelzer, G. Rätsch, and Christoph Bock “Molecularly informed analysis of histopathology images using natural language.” bioRxiv (2025). DOI
24. Sushant Patkar, T. Rosean, Palak Patel, Stephanie A. Harmon, Peter L. Choyke, Tamara Jamaspishvili, and B. Turkbey “Towards interpretable molecular and spatial analysis of the tumor microenvironment from digital histopathology images with HistoTME-v2.” bioRxiv (2025). DOI

This list is automatically updated weekly via GitHub Actions using the Semantic Scholar and OpenCitations APIs.

Related Projects

• AESTETIK — AutoEncoder for learning multi-modal spatial transcriptomics representations. Powers cross-sample integration in DeepSpot.
• DeepSpot2Cell — Predicts virtual single-cell spatial transcriptomics from H&E images using spot-level supervision. Presented at NeurIPS 2025 Imageomics.

Citation

In case you found our work useful, please consider citing us:

@article{nonchev2025deepspot,
  title={DeepSpot: Leveraging Spatial Context for Enhanced Spatial Transcriptomics Prediction from H\&E Images},
  author={Nonchev, Kalin and Dawo, Sebastian and Silina, Karina and Moch, Holger and Andani, Sonali and Tumor Profiler Consortium and Koelzer, Viktor H and Raetsch, Gunnar},
  journal={medRxiv},
  pages={2025--02},
  year={2025},
  publisher={Cold Spring Harbor Laboratory Press}
}

The code for reproducing the paper results can be found here.

Contact

In case, you have questions, please get in touch with Kalin Nonchev.

NB: Computational data analysis was performed at Leonhard Med (https://sis.id.ethz.ch/services/sensitiveresearchdata/) secure trusted research environment at ETH Zurich. Our pipeline aligns with the specific cluster requirements and resources.