Research

Our group is at the forefront of pioneering the development of Generalist Medical AI systems, which can closely resemble doctors in their ability to reason through a wide range of medical tasks, incorporate multiple data modalities, and communicate in natural language.

Foundation Models

We develop self-supervision and pre-training techniques for adaptable medical (‘foundation’) models, reducing the need for extensive data annotation. For instance, we developed an innovative AI model capable of detecting diseases on chest X-rays without relying on explicit labels (Tiu, Nature BME 22). We successfully built and leveraged foundation models for medical domains including for chest X-rays (Sowrirajan, MIDL 21; Vu, PMLR 21; Iyer, preprint 23), electrocardiograms (Gopal, PMLR 21; Jin, JAMIA 22), lung and heart sounds (Soni, Patterns 22), and CTs (Ke, MIDL 23). Our recent review (Krishnan, Nature BME 22) further explores how self-supervised learning can free medical AI from its reliance on labeled data.

Multimodal Learning

We develop methods to combine diverse data sources, like images, sensors, and language, to improve decision-making and generalization in medicine. We developed models from physiological data to monitor high-risk patients in the emergency department (Sundrani, NPJ Digital Med 23). We also developed models capable of learning to interpret images by learning from paired medical images and radiology reports (Tiu, Nature BME 22). We established a benchmark for modality agnostic learning across medical image and sensor data (Wantlin, preprint 23). We discuss other opportunities for learning across modalities (Acosta, Nature Med, 2022) and across time (Acosta, Radiology, 22) in our recent reviews.

Generative AI

We build generative AI models where models can interpret medical images in natural language and enable interactive communication with clinicians (Endo et al., ML4H, 21). We recently introduced datasets to address some of the biggest challenges with automated report generation (Ramesh et al, ML4H, 22; Yu et al., preprint 23). Furthermore, we are pioneering a copilot approach to radiology report generation, where AI models provide initial drafts of reports that clinicians can edit (Jeong et al. MIDL 23). We investigate the implications of providing AI assistance to clinicians in terms of their behavior (Moehring, preprint 23), and trustworthiness of explainability methods (Saporta, Nature MI, 22).