Deep Learning Based Cardiac MRI Segmentation: Do We Need Experts?

Abstract

Deep learning methods are the de facto solutions to a multitude of medical image analysis tasks. Cardiac MRI segmentation is one such application, which, like many others, requires a large number of annotated data so that a trained network can generalize well. Unfortunately, the process of having a large number of manually curated images by medical experts is both slow and utterly expensive. In this paper, we set out to explore whether expert knowledge is a strict requirement for the creation of annotated data sets on which machine learning can successfully be trained. To do so, we gauged the performance of three segmentation models, namely U-Net, Attention U-Net, and ENet, trained with different loss functions on expert and non-expert ground truth for cardiac cine–MRI segmentation. Evaluation was done with classic segmentation metrics (Dice index and Hausdorff distance) as well as clinical measurements, such as the ventricular ejection fractions and the myocardial mass. The results reveal that generalization performances of a segmentation neural network trained on non-expert ground truth data is, to all practical purposes, as good as that trained on expert ground truth data, particularly when the non-expert receives a decent level of training, highlighting an opportunity for the efficient and cost-effective creation of annotations for cardiac data sets.

1. Introduction

Deep neural networks (more specifically, convolutional neural networks) have deeply percolated through healthcare R&D, addressing various problems, such as survival prediction, disease diagnostics, image registration, anomaly detection, and the segmentation of images, be it magnetic resonance images (MRI), computed tomography (CT) or ultrasound (US), to name a few [1]. The roaring success of deep learning methods is rightly attributed to the unprecedented amount of annotated data across domains. However, ironically, while solutions to decade-long medical problems are at hand [2], the use of neural networks in day-to-day practice is still pending. This can be explained in part by the following two observations. First, while being accurate on average, neural networks can nonetheless be sometimes wrong [3], as they provide no strict clinical guarantees. In other words, any neural network within the intra-expert variability is excellent on average but not immune to sparse erroneous (yet degenerated) results, which is problematic in clinical practice [2]. Second, machine learning methods are known to suffer from domain adaptation problems, one of the most glaring medical imaging issue of our times [4]. As such, clinically accurate machine learning methods trained on a specific set of data almost always see their performance drop when tested on a data set acquired by following a different protocol. These problems are derived in good part from the fact that current data sets are still relatively small. According to Maier-Hein et al. [5] most medical imaging challenges organized so far contain less than 100 training and testing cases. This shows that medical applications cannot rely on a very large medical data set encompassing tens of thousands of annotated data acquired in various conditions, with machines of various vendors showing clinical conditions and anatomical configurations of all kinds.

This is unlike non-medical computer vision problems, which have had access for a long time to large and varied data sets, such as ImageNet, Coco, PascalVOC, ADE20k, and Youtube-8M, to name a few [6]. The annotation of these data sets relies on non-experts, often through online services, such as Mechanical Turk [7]. Unfortunately, obtaining similarly large annotated data sets in medical imaging is difficult. The challenge stems from the nature of the data, which is sensitive and requires navigating a complicated regulatory framework and privacy safeguards. Furthermore, labeling medical data sets is quite resource intensive and prohibitively costly, as it requires a domain expertise.

For these reasons, the medical imaging literature has had an increasing number of publications, whose goal is to compensate for the lack of expert annotations [8]. While some methods leverage partly-annotated data sets [9], others use domain adaptation strategies to compensate for small training data sets [10]. Some other approaches artificially increase the number of annotated data with generative adversarial networks (GANs) [11,12], while others use third-party neural networks to help experts annotate images more rapidly [13].

While these methods have been shown to be effective for their specific test cases, it is widely accepted that large manually-annotated data sets bring indisputable benefits [14]. In this work, we depart from trying to improve the segmentation methods and focus on the data sets as we challenge the idea that medical data, cardiac cine MRI specifically, needs to be labeled by experts only, and explore the consequences of using non-expert annotations on the generalization capabilities of a neural network. Non-expert here refers to a non-physician who cannot be regarded as a reference in the field. While non-expert annotations are easier and cheaper to obtain, they could be used to build larger data sets faster and at a reduced cost.

This idea was tested on cardiac cine-MRI segmentation. To this end, we had two non-experts labeling cardiac cine-MRI images and compared the performance of neural networks trained on non-expert and expert data. The evaluation between both approaches was conducted with geometric metrics (Dice index and Hausdorff distance) as well as clinical parameters, namely, the ejection fraction for the left and right ventricles and the myocardial mass.

2. Methods and Data

As mentioned before, medical data annotation requires rightful expertise so that the labeling can be used with full confidence. Expert annotators are typically medical doctors or medical specialists whose training and experience are reliable sources of truth for the problem at hand. These experts often have close collaborators working daily with medical data, typically computer scientists, technicians, biophysicist, etc. While their understanding of the data is real, these non-experts are typically not considered a reliable source of truth. Non-experts are thus considered as people who can manually label data but whose annotations are biased and/or noisy and thus unreliable.

In this study, two non-experts were asked to label 1902 cardiac images. We defined a non-expert as someone with no professional expertise on cardiac anatomy nor on cine-MR images. Non-Expert 1 is a technician in biotechnology who received 30 min of training by a medical expert on how to recognize and outline cardiac structures. The training was done on a few examples where the expert showed what the regions of interest in the image look like and where their boundaries lie. Training also came with an introduction to the cardiac anatomy and its temporal dynamics. Non-Expert 2 is a computer scientist with 4 years of active research in cardiac cine-MRI with several months of training. In the case of Non-Expert 2, the training spanned several months, where directions about the imaging modality as well as the anatomy and pathologies were thoroughly explained. In addition, fine delineation guidelines were provided to disambiguate good from poor annotations. In this study, we both gauge the effect of training a neural network on non-expert data and also verify how the level of training of the non-experts impact the overall results.

The non-experts were asked to delineate three structures of the heart, namely, the left ventricular cavity (endocardial border), the left ventricle myocardium (epicardial border) and the endocardial border of the right ventricle. No further quality control was done to validate the non-expert annotations. Segmentations were used as is for the subsequent tasks.

We used the gold standard for medical image segmentation U-Net [15] as the baseline network. In addition, the well-known Attention U-Net [16] and ENet [17] networks were trained in order to ensure that the results are affected by the differences in annotations and not the network architecture. We first trained the the segmentation models (U-Net, Attention U-Net and ENet) on the original ACDC data set (Automated Cardiac Diagnosis Challenge) [2] with its associated ground truth, using a classical supervised training scheme, with a combined cross-entropy and Dice loss:

$$L_{ce}=-\frac{1}{N}\sum _{i=1}^N\sum _{k=1}^3{y}_{ki}log{\widehat{y}}_{ki}$$

(1)

$$L_{dice}=1-\frac{1}{N}\left[\sum _{i=1}^N\frac{2\times {\sum }_k{\widehat{y}}_{ki}{y}_{ki}}{{\sum }_k{\widehat{y}}_{ki}+{\sum }_k{y}_{ki}}\right]$$

(2)

where ${\widehat{y}}_{ki}$ is the probabilistic output for image $i\in N$ (N is the number of images in the batch) and class $k\in \{1,2,3\}$ (3 is the number of classes). $\widehat{y}$ is the predicted output of the network, y is a one-hot encoding of the ground truth segmentation map.

We then re-trained the neural networks with the non-expert labels, using the same training configuration. Furthermore, considering that the non-expert annotations can be seen as noisy versions of the true annotation (i.e., $y^{\prime }=y+ϵ$ where $y^{\prime }$ is the non-expert annotation, y is the ground truth and $ϵ$ a random variable), we also trained the networks with a mean absolute error loss which, as shown by Ghosh et al. [18], has the solve property of compensating for labeling inaccuracies.

3. Experimental Setup

To test whether non-expert annotated data sets hold any value for cardiac MRI segmentation, the following two cardiac cine MRI data sets were used:

• Automated Cardiac Diagnosis Challenge (ACDC) data set [2]: This data set comprises 150 exams acquired at the University Hospital of Dijon (all from different patients). It is divided into 5 evenly distributed subgroups (4 pathological plus 1 healthy subject groups) and split into 100 exams for training, and 50 are held out set for testing. The exams were acquired using two MRI scanners with different magnetic strengths (1.5 T and 3 T). The pixel spacing varies from 0.7 mm to 1.9 mm with a slice spacing varying between 5 mm and 10 mm. An example of images with the different expert and non-expert annotations is shown in Figure 1.
• Multi-Centre, Multi-Vendor and Multi-Disease Cardiac Image Segmentation (M&M) data set [19]: This data set consists of 375 cases from 3 different countries (Spain, Germany and Canada) totaling 6 different centers with 4 different MRI manufacturers (Siemens, General Electric, Philips and Canon). The cohort is composed of patients with hypertrophic and dilated cardiomyopathies as well as healthy subjects. The cine MR images were annotated by experienced clinicians from the respective centers.

We trained the segmentation models on the 100 ACDC training subjects on either the expert or non-expert ground truth data. Training was done with a fixed set of hyper-parameters, chosen through a cross-validated hyper-parameters search to best fit the 3 annotators, without tuning it further. The networks were trained three times in order to reduce the effect of the stochastic nature of the training process on the results.

As mentioned before, we first trained the neural networks on non-expert data with exactly the same setup as for the expert annotations. Then, we retrained from scratch the neural networks (U-Net, Attention U-Net and ENet) with a L1 loss, which was shown to be robust to noisy labels [18].

We then tested in turn on the 50 ACDC test subjects and the 150 M&Ms training data. The M&Ms data set constitutes data with ground truth that are not biased toward either of the annotators of the training set, be it the expert or the non-expert. Moreover, testing on different data sets provided an inter-expert variability range as well as a domain generalization setup.

4. Results and Discussion

The first set of results are laid out in

Table 1. Dice score (DSC), Hausdorff distance (HD), average ejection fraction (EF) error, and end diastolic volume (EDV) of left ventricular cavity for U-Net, Attention U-Net and E-Net with expert and non-expert annotations with data augmentation.

U-Net
Test Set	ACDC Trainset	Loss	Avg. DSC	Avg. HD	Avg. EF (%)	Avg EF Err.	Avg. EDV (mL)
	Expert	CE+Dice	$0.92\pm 0.08$	$11.84\pm 11.43$	$47.57\pm 19.62$	$3.59\pm 3.17$	$175.70\pm 69.03$
ACDC	Non-Expert 1	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.92\pm 0.08\\ 0.91\pm 0.08\end{array}$	$\begin{array}{c}12.15\pm 10.63\\ 12.80\pm 10.48\end{array}$	$\begin{array}{c}50.27\pm 20.00\\ 50.37\pm 20.14\end{array}$	$\begin{array}{c}4.37\pm 3.69\\ 4.52 *\pm 4.32\end{array}$	$\begin{array}{c}173.05\pm 67.06\\ 173.57\pm 70.29\end{array}$
	Non-Expert 2	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}\mathbf{0.93}\pm \mathbf{0.09}\\ 0.92\pm 0.09\end{array}$	$\begin{array}{c}\mathbf{11.68}\pm \mathbf{12.36}\\ 11.76\pm 12.34\end{array}$	$\begin{array}{c}49.73\pm 20.70\\ 49.45\pm 20.68\end{array}$	$\begin{array}{c}\mathbf{3.35}\pm \mathbf{3.21}\\ 3.69\pm 3.59\end{array}$	$\begin{array}{c}182.98\pm 72.51\\ 181.19\pm 70.93\end{array}$
	Expert	CE+Dice	$0.86\pm 0.11$	$15.03\pm 8.07$	$54.22\pm 15.11$	$7.47\pm 5.79$	$162.57\pm 60.22$
M&Ms	Non-Expert 1	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.86\pm 0.10\\ 0.86\pm 0.10\end{array}$	$\begin{array}{c}11.76\pm 6.04\\ \mathbf{11.52}\pm \mathbf{5.55}\end{array}$	$\begin{array}{c}56.81 *\pm 15.37\\ 57.14 *\pm 15.23\end{array}$	$\begin{array}{c}5.53 *\pm 5.22\\ \mathbf{5.39} *\pm \mathbf{5.38}\end{array}$	$\begin{array}{c}149.75 *\pm 57.33\\ 151.08 *\pm 58.08\end{array}$
	Non-Expert 2	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}\mathbf{0.88}\pm \mathbf{0.09}\\ 0.88\pm 0.10\end{array}$	$\begin{array}{c}11.58\pm 6.93\\ 11.84\pm 7.93\end{array}$	$\begin{array}{c}56.70 *\pm 15.33\\ 56.74 *\pm 15.45\end{array}$	$\begin{array}{c}5.54 *\pm 4.57\\ 6.17 *\pm 5.11\end{array}$	$\begin{array}{c}164.95\pm 62.00\\ 165.78\pm 62.00\end{array}$
Attention U-Net
Test Set	ACDC Trainset	Loss	Avg. DSC	Avg. HD	Avg. EF (%)	Avg EF Err.	Avg. EDV
	Expert	CE+Dice	$0.92\pm 0.08$	$11.78\pm 11.14$	$46.83\pm 19.50$	$3.81\pm 3.63$	$177.77\pm 71.08$
ACDC	Non-Expert 1	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.92\pm 0.08\\ 0.92\pm 0.09\end{array}$	$\begin{array}{c}12.07\pm 11.20\\ 13.42\pm 12.62\end{array}$	$\begin{array}{c}50.27\pm 20.07\\ 50.35\pm 20.28\end{array}$	$\begin{array}{c}4.28\pm 3.94\\ 4.74 *\pm 4.15\end{array}$	$\begin{array}{c}170.86\pm 67.86\\ 173.43\pm 67.61\end{array}$
	Non-Expert 2	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}\mathbf{0.93}\pm \mathbf{0.07}\\ 0.93\pm 0.08\end{array}$	$\begin{array}{c}\mathbf{11.05}\pm \mathbf{11.45}\\ 12.52\pm 11.62\end{array}$	$\begin{array}{c}50.56\pm 21.24\\ 49.78\pm 21.10\end{array}$	$\begin{array}{c}\mathbf{3.50}\pm \mathbf{3.86}\\ 3.93\pm 3.96\end{array}$	$\begin{array}{c}180.28\pm 70.50\\ 178.82\pm 68.16\end{array}$
	Expert	CE+Dice	$\mathbf{0.88}\pm \mathbf{0.09}$	$11.90\pm 7.38$	$55.77\pm 15.23$	$\mathbf{5.59}\pm \mathbf{4.73}$	$164.93\pm 62.67$
M&Ms	Non-Expert 1	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.86\pm 0.12\\ 0.86\pm 0.10\end{array}$	$\begin{array}{c}11.90\pm 7.73\\ 12.40\pm 6.79\end{array}$	$\begin{array}{c}57.67\pm 17.34\\ 57.40\pm 17.44\end{array}$	$\begin{array}{c}6.64 *\pm 6.45\\ 6.23\pm 6.18\end{array}$	$\begin{array}{c}148.25 *\pm 57.84\\ 148.22 *\pm 57.37\end{array}$
	Non-Expert 2	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.87\pm 0.10\\ 0.87\pm 0.09\end{array}$	$\begin{array}{c}\mathbf{11.10}\pm \mathbf{6.59}\\ 11.14\pm 6.43\end{array}$	$\begin{array}{c}58.51 *\pm 16.33\\ 58.20 *\pm 16.34\end{array}$	$\begin{array}{c}5.99\pm 5.01\\ 6.29 *\pm 5.36\end{array}$	$\begin{array}{c}161.20\pm 61.40\\ 161.29\pm 62.34\end{array}$
E-Net
Test Set	ACDC Trainset	Loss	Avg. DSC	Avg. HD	Avg. EF (%)	Avg EF Err.	Avg. EDV
	Expert	CE+Dice	$0.92\pm 0.09$	$12.28\pm 11.38$	$46.68\pm 19.06$	$3.94\pm 3.86$	$178.48\pm 70.68$
ACDC	Non-Expert 1	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.91\pm 0.09\\ 0.91\pm 0.10\end{array}$	$\begin{array}{c}12.72\pm 12.28\\ 13.07\pm 15.00\end{array}$	$\begin{array}{c}48.64\pm 20.33\\ 50.10\pm 19.99\end{array}$	$\begin{array}{c}4.31\pm 4.99\\ 4.15\pm 3.65\end{array}$	$\begin{array}{c}172.60\pm 67.69\\ 171.53\pm 67.29\end{array}$
	Non-Expert 2	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.92\pm 0.09\\ \mathbf{0.92}\pm \mathbf{0.08}\end{array}$	$\begin{array}{c}12.18\pm 14.01\\ \mathbf{11.69}\pm \mathbf{11.16}\end{array}$	$\begin{array}{c}49.05\pm 20.52\\ 48.83\pm 20.57\end{array}$	$\begin{array}{c}\mathbf{3.25}\pm \mathbf{3.09}\\ 3.45\pm 3.53\end{array}$	$\begin{array}{c}180.77\pm 71.20\\ 181.07\pm 70.45\end{array}$
	Expert	CE+Dice	$0.86\pm 0.10$	$14.46\pm 7.54$	$53.91\pm 16.04$	$7.19\pm 5.98$	$165.77\pm 61.59$
M&Ms	Non-Expert 1	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.86\pm 0.12\\ 0.86\pm 0.12\end{array}$	$\begin{array}{c}13.64\pm 10.60\\ \mathbf{11.90}\pm \mathbf{8.59}\end{array}$	$\begin{array}{c}53.10\pm 21.96\\ 55.90{}^{†}\pm 18.26\end{array}$	$\begin{array}{c}7.41\pm 12.67\\ \mathbf{5.81}{*}^{,†}\pm \mathbf{7.64}\end{array}$	$\begin{array}{c}152.59*\pm 59.87\\ 150.79*\pm 57.79\end{array}$
	Non-Expert 2	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}\mathbf{0.87}\pm \mathbf{0.10}\\ \mathbf{0.87}\pm \mathbf{0.10}\end{array}$	$\begin{array}{c}12.39\pm 7.63\\ 12.72\pm 8.41\end{array}$	$\begin{array}{c}53.53\pm 17.49\\ 54.99\pm 16.29\end{array}$	$\begin{array}{c}6.39\pm 7.25\\ 6.76\pm 5.56\end{array}$	$\begin{array}{c}165.05\pm 61.73\\ 165.36\pm 60.71\end{array}$

* p-value $<0.05$ non-expert vs. expert; ${}^{†}$ p-value $<0.05$ MAE vs. CE.

. It corresponds to standard geometrical metrics, i.e., the Dice score and the Hausdorff distance (HD) for the left ventricular (LV) cavity (Table 1), the myocardium (MYO) (considering the endocardial and epicardial of the left ventricle) (

Table 2. Dice score, Hausdorff distance and mass error for the myocardium for U-Net, Attention U-Net and E-Net with expert and non-expert annotations with data augmentation.

U-Net
Test Set	ACDC Trainset	Loss	Avg. DSC	Avg. HD	Avg. Mass Err.
	Expert	CE+Dice	$0.88\pm 0.03$	$11.02\pm 6.25$	$16.88\pm 9.66$
ACDC	Non-Expert 1	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.82\pm 0.03\\ 0.86\pm 0.03\end{array}$	$\begin{array}{c}11.45\pm 4.88\\ 11.69\pm 5.18\end{array}$	$\begin{array}{c}36.40*\pm 13.37\\ 15.93{}^{†}\pm 8.95\end{array}$
	Non-Expert 2	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.87\pm 0.03\\ \mathbf{0.89}\pm \mathbf{0.03}\end{array}$	$\begin{array}{c}\mathbf{10.13}\pm \mathbf{5.02}\\ 10.43\pm 5.00\end{array}$	$\begin{array}{c}11.68*\pm 7.74\\ \mathbf{6.58}{*}^{,†}\pm \mathbf{5.93}\end{array}$
	Expert	CE+Dice	$\mathbf{0.80}\pm \mathbf{0.06}$	$17.57\pm 7.65$	$17.36\pm 12.87$
M&Ms	Non-Expert 1	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.76\pm 0.08\\ 0.78\pm 0.09\end{array}$	$\begin{array}{c}14.91\pm 8.60\\ 14.32\pm 7.95\end{array}$	$\begin{array}{c}22.14*\pm 19.71\\ 18.78{}^{†}\pm 17.80\end{array}$
	Non-Expert 2	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.80\pm 0.07\\ 0.80\pm 0.07\end{array}$	$\begin{array}{c}14.29\pm 7.53\\ \mathbf{13.76}\pm \mathbf{6.79}\end{array}$	$\begin{array}{c}\mathbf{16.36}\pm \mathbf{13.69}\\ 19.44{*}^{,†}\pm 16.79\end{array}$
Attention U-Net
Test Set	ACDC Trainset	Loss	Avg. DSC	Avg. HD	Avg. Mass Err.
	Expert	CE+Dice	$\mathbf{0.88}\pm \mathbf{0.03}$	$11.57\pm 7.13$	$17.95\pm 10.28$
ACDC	Non-Expert 1	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.84\pm 0.03\\ 0.86\pm 0.03\end{array}$	$\begin{array}{c}\mathbf{10.27}\pm \mathbf{5.25}\\ 13.11\pm 8.65\end{array}$	$\begin{array}{c}33.99*\pm 13.17\\ 20.12{}^{†}\pm 8.82\end{array}$
	Non-Expert 2	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.87\pm 0.03\\ \mathbf{0.88}\pm \mathbf{0.03}\end{array}$	$\begin{array}{c}10.86\pm 6.17\\ 11.53\pm 6.62\end{array}$	$\begin{array}{c}13.76*\pm 8.99\\ \mathbf{9.68}{*}^{,†}\pm \mathbf{7.32}\end{array}$
	Expert	CE+Dice	$\mathbf{0.82}\pm \mathbf{0.05}$	$15.85\pm 9.84$	$16.33\pm 12.75$
M&Ms	Non-Expert 1	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.78\pm 0.08\\ 0.77\pm 0.09\end{array}$	$\begin{array}{c}13.95\pm 8.04\\ 16.19\pm 9.78\end{array}$	$\begin{array}{c}19.44*\pm 17.61\\ 20.80*\pm 20.08\end{array}$
	Non-Expert 2	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.80\pm 0.07\\ 0.78\pm 0.08\end{array}$	$\begin{array}{c}\mathbf{13.43}\pm \mathbf{7.85}\\ 13.75\pm 7.56\end{array}$	$\begin{array}{c}\mathbf{16.17}\pm \mathbf{13.58}\\ 26.89{*}^{,†}\pm 21.43\end{array}$
E-Net
Test Set	ACDC Trainset	Loss	Avg. DSC	Avg. HD	Avg. Mass Err.
	Expert	CE+Dice	$\mathbf{0.87}\pm \mathbf{0.03}$	$10.99\pm 5.61$	$17.14\pm 9.05$
ACDC	Non-Expert 1	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.82\pm 0.04\\ 0.86\pm 0.04\end{array}$	$\begin{array}{c}11.91\pm 7.80\\ 11.51\pm 6.03\end{array}$	$\begin{array}{c}39.81*\pm 15.58\\ 21.26{*}^{,†}\pm 9.49\end{array}$
	Non-Expert 2	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.85\pm 0.04\\ \mathbf{0.87}\pm \mathbf{0.03}\end{array}$	$\begin{array}{c}10.96\pm 6.11\\ \mathbf{10.49}\pm \mathbf{5.16}\end{array}$	$\begin{array}{c}15.64\pm 9.59\\ \mathbf{7.83}{*}^{,†}\pm \mathbf{6.24}\end{array}$
	Expert	CE+Dice	$0.80\pm 0.07$	$15.81\pm 7.55$	$17.86\pm 15.63$
M&Ms	Non-Expert 1	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.77\pm 0.10\\ 0.78\pm 0.09\end{array}$	$\begin{array}{c}16.12\pm 12.23\\ \mathbf{14.51}\pm \mathbf{9.26}\end{array}$	$\begin{array}{c}28.40*\pm 23.91\\ 18.35{}^{†}\pm 17.10\end{array}$
	Non-Expert 2	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}\mathbf{0.80}\pm \mathbf{0.06}\\ 0.79\pm 0.08\end{array}$	$\begin{array}{c}15.18\pm 8.41\\ 15.69\pm 8.39\end{array}$	$\begin{array}{c}\mathbf{16.08}\pm \mathbf{14.98}\\ 18.14{}^{†}\pm 15.82\end{array}$

* p-value $<0.05$ non-expert vs. expert; ${}^{†}$ p-value $<0.05$ MAE vs. CE.

) and the cavity of the right ventricle (RV) (

Table 3. Dice score, Hausdorff distance, average ejection fraction (EF) error, and end diastolic volume (EDV) for the right ventricular cavity for U-Net, Attention U-Net and E-Net with expert and non-expert annotations with data augmentation.

U-Net
Test Set	ACDC Train set	Loss	Avg. DSC	Avg. HD	Avg. EF (%)	Avg EF Err.	Avg. EDV (mL)
	Expert	CE+Dice	$\mathbf{0.90}\pm \mathbf{0.06}$	$14.72\pm 6.12$	$42.77\pm 15.25$	$\mathbf{6.69}\pm \mathbf{6.21}$	$183.73\pm 70.68$
ACDC	Non-Expert 1	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.78\pm 0.11\\ 0.84\pm 0.09\end{array}$	$\begin{array}{c}19.59\pm 8.07\\ 16.78\pm 7.23\end{array}$	$\begin{array}{c}32.22*\pm 13.49\\ 35.51{*}^{,†}\pm 14.64\end{array}$	$\begin{array}{c}14.77*\pm 9.88\\ 13.52*\pm 8.88\end{array}$	$\begin{array}{c}216.16*\pm 69.96\\ 181.61{}^{†}\pm 61.85\end{array}$
	Non-Expert 2	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.86\pm 0.07\\ 0.89\pm 0.07\end{array}$	$\begin{array}{c}15.25\pm 6.00\\ \mathbf{14.65}\pm \mathbf{6.34}\end{array}$	$\begin{array}{c}40.10\pm 14.42\\ 41.61\pm 14.85\end{array}$	$\begin{array}{c}7.98\pm 7.34\\ 7.38\pm 7.27\end{array}$	$\begin{array}{c}200.67*\pm 73.35\\ 186.59\pm 71.56\end{array}$
	Expert	CE+Dice	$\mathbf{0.82}\pm \mathbf{0.15}$	$\mathbf{16.91}\pm \mathbf{12.65}$	$52.90\pm 19.79$	$\mathbf{9.12}\pm \mathbf{12.73}$	$145.92\pm 61.94$
M&Ms	Non-Expert 1	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.74\pm 0.15\\ 0.78\pm 0.15\end{array}$	$\begin{array}{c}22.51\pm 15.82\\ 16.93\pm 10.95\end{array}$	$\begin{array}{c}37.68*\pm 21.06\\ 38.75*\pm 36.59\end{array}$	$\begin{array}{c}17.88*\pm 18.47\\ 18.12*\pm 33.08\end{array}$	$\begin{array}{c}173.34*\pm 70.88\\ 144.36{}^{†}\pm 60.57\end{array}$
	Non-Expert 2	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.79\pm 0.16\\ 0.81\pm 0.17\end{array}$	$\begin{array}{c}20.31\pm 17.56\\ 18.05\pm 15.60\end{array}$	$\begin{array}{c}44.11*\pm 29.89\\ 39.17*\pm 91.83\end{array}$	$\begin{array}{c}12.86*\pm 26.88\\ 20.72*\pm 88.32\end{array}$	$\begin{array}{c}168.76*\pm 66.89\\ 149.17{}^{†}\pm 66.41\end{array}$
Attention U-Net
Test Set	ACDC Train set	Loss	Avg. DSC	Avg. HD	Avg. EF (%)	Avg EF Err.	Avg. EDV (mL)
	Expert	CE+Dice	$\mathbf{0.89}\pm \mathbf{0.07}$	$16.62\pm 7.71$	$38.73\pm 15.58$	$8.71\pm 7.46$	$194.05\pm 71.01$
ACDC	Non-Expert 1	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.79\pm 0.10\\ 0.83\pm 0.09\end{array}$	$\begin{array}{c}20.60\pm 9.19\\ 20.08\pm 11.27\end{array}$	$\begin{array}{c}30.12*\pm 13.11\\ 33.00*\pm 14.62\end{array}$	$\begin{array}{c}15.96*\pm 9.27\\ 13.57{*}^{,†}\pm 9.68\end{array}$	$\begin{array}{c}218.32*\pm 69.08\\ 198.36{}^{†}\pm 68.13\end{array}$
	Non-Expert 2	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.86\pm 0.07\\ 0.88\pm 0.09\end{array}$	$\begin{array}{c}16.33\pm 7.92\\ \mathbf{16.30}\pm \mathbf{9.18}\end{array}$	$\begin{array}{c}37.96\pm 15.19\\ 39.65\pm 15.58\end{array}$	$\begin{array}{c}9.20\pm 7.15\\ \mathbf{8.31}\pm \mathbf{8.28}\end{array}$	$\begin{array}{c}200.98\pm 71.70\\ 185.30\pm 70.51\end{array}$
	Expert	CE+Dice	$\mathbf{0.82}\pm \mathbf{0.15}$	$\mathbf{17.94}\pm \mathbf{12.54}$	$50.23\pm 15.46$	$\mathbf{9.39}\pm \mathbf{8.21}$	$156.45\pm 62.88$
M&Ms	Non-Expert 1	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.76\pm 0.13\\ 0.77\pm 0.14\end{array}$	$\begin{array}{c}20.96\pm 11.07\\ 24.24\pm 14.61\end{array}$	$\begin{array}{c}37.69*\pm 19.13\\ 37.94*\pm 18.12\end{array}$	$\begin{array}{c}17.86*\pm 15.87\\ 17.67*\pm 15.47\end{array}$	$\begin{array}{c}179.85*\pm 67.54\\ 167.74{*}^{,†}\pm 60.49\end{array}$
	Non-Expert 2	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.79\pm 0.15\\ 0.80\pm 0.17\end{array}$	$\begin{array}{c}18.09\pm 13.02\\ 20.72\pm 15.26\end{array}$	$\begin{array}{c}47.28*\pm 19.16\\ 44.72*\pm 27.57\end{array}$	$\begin{array}{c}12.20*\pm 12.84\\ 13.22*\pm 23.62\end{array}$	$\begin{array}{c}159.95\pm 64.95\\ 151.40{}^{†}\pm 63.41\end{array}$
E-Net
Test Set	ACDC Train set	Loss	Avg. DSC	Avg. HD	Avg. EF (%)	Avg EF Err.	Avg. EDV (mL)
	Expert	CE+Dice	$\mathbf{0.88}\pm \mathbf{0.07}$	$\mathbf{15.76}\pm \mathbf{6.38}$	$38.65\pm 15.27$	$\mathbf{9.11}\pm \mathbf{7.17}$	$192.65\pm 70.02$
ACDC	Non-Expert 1	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.77\pm 0.11\\ 0.83\pm 0.10\end{array}$	$\begin{array}{c}22.14\pm 9.01\\ 17.47\pm 8.29\end{array}$	$\begin{array}{c}30.12*\pm 13.12\\ 34.48{*}^{,†}\pm 13.68\end{array}$	$\begin{array}{c}16.29*\pm 10.02\\ 13.00{*}^{,†}\pm 9.11\end{array}$	$\begin{array}{c}223.07*\pm 71.29\\ 189.00{}^{†}\pm 65.27\end{array}$
	Non-Expert 2	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.84\pm 0.09\\ 0.87\pm 0.08\end{array}$	$\begin{array}{c}16.19\pm 6.29\\ 16.46\pm 7.34\end{array}$	$\begin{array}{c}36.25\pm 14.13\\ 36.87\pm 13.84\end{array}$	$\begin{array}{c}10.46\pm 7.91\\ 9.90\pm 7.91\end{array}$	$\begin{array}{c}203.49\pm 71.15\\ 194.54\pm 69.42\end{array}$
	Expert	CE+Dice	$0.79\pm 0.18$	$19.57\pm 16.32$	$48.86\pm 20.66$	$\mathbf{12.06}\pm \mathbf{13.44}$	$151.92\pm 66.12$
M&Ms	Non-Expert 1	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.72\pm 0.16\\ 0.78\pm 0.14\end{array}$	$\begin{array}{c}28.38\pm 17.94\\ 19.30\pm 12.58\end{array}$	$\begin{array}{c}34.20*\pm 30.51\\ 41.63{*}^{,†}\pm 19.67\end{array}$	$\begin{array}{c}22.27*\pm 26.54\\ 14.83{*}^{,†}\pm 15.06\end{array}$	$\begin{array}{c}178.67*\pm 72.98\\ 156.65{}^{†}\pm 61.99\end{array}$
	Non-Expert 2	$\left\{\begin{array}{c}\mathrm{C}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\\ \mathrm{M}\mathrm{A}\mathrm{E}+\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}\end{array}\right.$	$\begin{array}{c}0.79\pm 0.14\\ \mathbf{0.80}\pm \mathbf{0.14}\end{array}$	$\begin{array}{c}\mathbf{19.13}\pm \mathbf{12.09}\\ 20.73\pm 13.26\end{array}$	$\begin{array}{c}41.88*\pm 18.65\\ 42.94*\pm 16.39\end{array}$	$\begin{array}{c}14.17*\pm 13.61\\ 13.35\pm 11.36\end{array}$	$\begin{array}{c}166.83*\pm 66.72\\ 169.43*\pm 63.54\end{array}$

* p-value $<0.05$ non-expert vs. expert; ${}^{†}$ p-value $<0.05$ MAE vs. CE.

). It also contains the end-diastolic volume (EDV) as well as the ejection fraction (EF) for the LV and the RV and the myocardial mass error. For all three tables, the networks (U-Net, Attention U-Net and ENet) were trained on the ACDC training set and tested on the ACDC testing set and the M&Ms training set.

Results for the ACDC testing set reveal that for the LV (Table 1) the networks trained on the non-expert annotations (Non-Expert 1 as well as Non-Expert 2) manage to achieve performance that is statistically indistinguishable from that of the expert. This is true regardless of the training loss (CE+Dice vs MAE+Dice) and the metric (Dice, HD, and EF). The only exception is for the EF error for Non-Expert 1 with loss function MAE+Dice.

The situation, however, is more fuzzy for the MYO and the RV. In both cases, we can see that results for the Non-Expert 1 are almost always worse than that of the expert, especially for the CE+Dice loss. For example, there is a Dice score drop of $12\%$ on the myocardium. In addition, the clinical results on the RV (Table 3) show a clear gap between the Non-Expert 1 and the other annotators. However, we can appreciate how the MAE+Dice loss improves results for both non-experts. Overall, for the MYO and the RV, results for the Non-Expert 2 are very close (if not better) than that of the expert. This is obvious when considering the average myocardial mass error in Table 2. Although, one recurrent result from our experiments is the hit-and-miss performance of all the evaluated networks on the M&Ms data set, where in a number of cases, the output segmentation is completely degenerated as shown in Figure 2. Moreover, the difference in segmentation performance between all the annotators is similar regardless of the segmentation network (U-Net, Attention U-Net or ENet) used, although Attention U-Net shows the best performance overall, which is to be expected, given its larger capacity.

Further analysis of the segmentation performance on the different sections of the heart (Figure 3), namely the base, the middle and the apex, show that the differences between the non-experts and the expert annotations lie heavily on the two ends of the heart. The performance gap is more pronounced on the apex for the three anatomical structures. In parallel, when we look at the performance from the disease groups (Figure 4), we can distinguish a relative similarity in the Dice score between the different annotators and disease groups.

Our experiments also reveal some interesting results on the M&Ms data set, a data set with different acquisition settings than ACDC. In that case, we see that the gap in performance between the expert and the non-expert decreases substantially. For example, when comparing the Non-Expert 1 results with MAE+Dice loss and those from the expert annotation, we see that the Dice difference for the RV decreases from a 6% on the ACDC data set to a mere 4% on the M&Ms data set; overall, the results of Non-Expert 2 are similar (and sometimes better) than those of the expert.

Throughout our experiments, the performance of the three neural networks (U-Net, Attention U-Net and ENet) trained on the Non-Expert 2 annotations with MAE+Dice loss has been roughly on par, if not better, with those trained on the expert annotations. This is especially true for the LV. For Non-Expert 1, most likely due to a lack of proper training, the results on both test sets and most MYO and RV metrics are worse than those of the expert. In fact, a statistical test reveals that the results from Non-Expert 1 are almost always statistically different than those of the expert. We also evaluated the statistical difference between the CE+Dice and the MAE+Dice losses and observed that the MAE+Dice loss provides overall better results for both non experts.

Overall on M&Ms, while the expert obtained a better MYO mass error and a better RV EF error, the MYO HD was lower for Non-Expert 2 and the Dice score and the RV HD of Non-Expert 2 are statistically similar. These results underline the idea that well-trained non-expert and expert annotations could be used interchangeably to build reliable annotated cardiac data sets. In contrast, the number of non-experts we evaluated might be considered a limitation of our study; however, this still provides encouraging results for settings where experts are not readily available to annotate whole data sets, but can provide training to a non-expert to effectively annotate in their stead. We leave the investigation on more data sets to future works that could transpose the setup to more difficult problems and a larger number of non-experts. Our work supplements previous endeavors that rely on non-experts to annotate medical data sets; Heim et al. [20] showcased the ability of crowd-sourced expertise to reliably annotate the liver data set, although their approach proposes initial segmentations to the non-expert, which might have biased their decision. Likewise, Ganz et al. [21] proposed to make use of non-experts as a crowd-sourced error detection framework. In contrast, our approach evaluated the effectiveness of non-expert knowledge without any prior input. This further reinforces the idea that crowd-sourced medical annotations are a viable solution for the lack of data.

5. Conclusions

In this work, we studied the usefulness of training deep learning models with non-expert annotations for the segmentation of cardiac MR images. The need for medical experts was probed in a comparative study with non-physician sourced labels. Through framing the problem of relying on non-expert annotations as noisy data, we managed to obtain good performance on two public data sets, one of which was used to emulate an out-of-distribution data set. We found that training a deep neural network, regardless of its capacity (U-Net, Attention U-Net or ENet), with data labeled by a well-trained non-expert achieved comparable performance than on expert data. Moreover, the performance gap between the networks with non-expert and expert annotations on the out-of-distribution data set was less pronounced than the gap on the training data set. Future endeavors could focus on crowd sourcing large-scale medical data sets and tailoring approaches that take their noisiness into account.