Generation of a 3D cardiac model based on 2D echocardiography video data

Abstract

Echocardiography is a non-invasive medical imaging tool that serves to assess the cardiac functionality by producing images of the heart’s tissues and chambers using non-ionizing ultrasound waves. We propose a generative model that learns to produce personalized 4D (3D + time) heart models from 3D (2D + time) echocardiograms. We argue that such heart model may facilitate downstream tasks such as disease diagnosis and prediction. Our model is based on 2 Autoencoders and a Cycle GAN. The 2 Autoencoders process sequences of images (echocardiograms) and sequences of meshes (3D heart) to extract low dimensional representations of these sequences. We develop the Autoencoder model for processing sequences of meshes based on [1]. We also develop the Cycle GAN model which translates between the 2 data modalities (image sequences and mesh sequences) by translating between their latent space representations in a cycle-consistent manner. We evaluate the performance of the models we develop qualitatively and quantitatively and show that our full 3D to 4D model achieves promising results.