Modality attention and sampling enables deep learning with heterogeneous marker combinations in fluorescence microscopy
METADATA ONLY
Loading...
Author / Producer
Date
2021-09
Publication Type
Journal Article
ETH Bibliography
yes
Citations
Altmetric
METADATA ONLY
Data
Rights / License
Abstract
Fluorescence microscopy allows for a detailed inspection of cells, cellular networks and anatomical landmarks by staining with a variety of carefully selected markers visualized as colour channels. Quantitative characterization of structures in acquired images often relies on automatic image analysis methods. Despite the success of deep learning methods in other vision applications, their potential for fluorescence image analysis remains underexploited. One reason lies in the considerable workload required to train accurate models, which are normally specific for a given combination of markers and therefore applicable to a very restricted number of experimental settings. We herein propose ‘marker sampling and excite’—a neural network approach with a modality sampling strategy and a novel attention module that together enable (1) flexible training with heterogeneous datasets with combinations of markers and (2) successful utility of learned models on arbitrary subsets of markers prospectively. We show that our single neural network solution performs comparably to an upper bound scenario in which an ensemble of many networks is naively trained for each possible marker combination separately. We also demonstrate the feasibility of this framework in high-throughput biological analysis by revising a recent quantitative characterization of bone-marrow vasculature in three-dimensional confocal microscopy datasets and further confirm the validity of our approach on another substantially different dataset of microvessels in foetal liver tissues. Not only can our work substantially ameliorate the use of deep learning in fluorescence microscopy analysis, but it can also be utilized in other fields with incomplete data acquisitions and missing modalities.
Permanent link
Publication status
published
External links
Editor
Book title
Journal / series
Volume
3 (9)
Pages / Article No.
799 - 811
Publisher
Nature
Event
Edition / version
Methods
Software
Geographic location
Date collected
Date created
Subject
Biomedical engineering; Computer science; Image processing; Machine learning
Organisational unit
09528 - Göksel, Orçun (ehemalig) / Göksel, Orçun (former)
Notes
Funding
179116 - Imaging Soft Tissue Elasticity (SNF)
Related publications and datasets
Is new version of: