DEEP BIOLOGICAL PATHWAY INFORMED PATHOLOGY-GENOMIC MULTIMODAL SURVIVAL PREDICTION

Abstract

The integration of multi-modal data, such as pathological images and genomic data, is essential for understanding cancer heterogeneity and complexity for personalized treatments, as well as for enhancing survival predictions. Despite the progress made in integrating pathology and genomic data, most existing methods cannot mine the complex inter-modality relations thoroughly. Additionally, identifying explainable features from these models that govern preclinical discovery and clinical prediction is crucial for cancer diagnosis, prognosis, and therapeutic response studies. We propose PONET-a novel biological pathway informed pathology-genomic deep model that integrates pathological images and genomic data not only to improve survival prediction but also to identify genes and pathways that cause different survival rates in patients. Empirical results on six of The Cancer Genome Atlas (TCGA) datasets show that our proposed method achieves superior predictive performance and reveals meaningful biological interpretations. The proposed method establishes insight on how to train biological informed deep networks on multimodal biomedical data which will have general applicability for understanding diseases and predicting response and resistance to treatment.

1. INTRODUCTION

Manual examination of haematoxylin and eosin (H&E)-stained slides of tumour tissue by pathologists is currently the state-of-the-art for cancer diagnosis (Chan, 2014) . The recent advancements in deep learning for digital pathology have enabled the use of whole-slide images (WSI) for computational image analysis tasks, such as cellular segmentation (Pan et al., 2017; Hou et al., 2020) , tissue classification and characterisation (Hou et al., 2016; Hekler et al., 2019; Iizuka et al., 2020) . While H&E slides are important and sufficient to establish a profound diagnosis, genomics data can provide a deep characterisation of the tumour on the molecular level potentially offering the chance for prognostic and predictive biomarker discovery. Cancer prognosis via survival outcome prediction is a standard method used for biomarker discovery, stratification of patients into distinct treatment groups, and therapeutic response prediction (Cheng et al., 2017; Ning et al., 2020) . WSIs exhibit enormous heterogeneity and can be as large as 150,000 × 150,000 pixels. Most approaches adopt a two-stage multiple instance learning-based (MIL) approach for representation learning of WSIs, in which: 1) instance-level feature representations are extracted from image patches in the WSI, and then 2) global aggregation schemes are applied to the bag of instances to obtain a WSI-level representation for subsequent supervision (Hou et al., 2016; Courtiol et al., 2019; Wulczyn et al., 2020; Lu et al., 2021) . Therefore, multimodal survival prediction faces an additional challenge due to the large data heterogeneity gap between WSIs and genomics, and many existing approaches use simple multimodal fusion mechanisms for feature integration, which prevents mining important multimodal interactions (Mobadersany et al., 2018; Chen et al., 2022b; a) . The incorporation of biological pathway databases in a model takes advantage of leveraging prior biological knowledge so that potential prognostic factors of well-known biological functionality can be identified (Hao et al., 2018) . Moreover, encoding biological pathway information into the neural networks achieved superior predictive performance compared with established models (Elmarakeby et al., 2021) . Based on the current challenges in multimodal fusion of pathology and genomics and the potential prognostic interpretation to link pathways and clinical outcomes in pathway-based analysis, we

