Original Research

Deep-Learning-Based Neural Tissue Segmentation of MRI in Multiple Sclerosis: Effect of Training Set Size

Corresponding Author

Ponnada A. Narayana PhD

[email protected]

orcid.org/0000-0002-5703-4346

Department of Diagnostic and Interventional Imaging, McGovern Medical School, University of Texas Health Science Center, Houston, Texas, USA

Address reprint requests to: P.A.N., Department of Diagnostic and Interventional Imaging, McGovern Medical School, University of Texas Health Science Center, Houston, TX 77030. E-mail: [email protected]Search for more papers by this author

Ivan Coronado MS,

Ivan Coronado MS

Department of Diagnostic and Interventional Imaging, McGovern Medical School, University of Texas Health Science Center, Houston, Texas, USA

Search for more papers by this author

Sheeba J. Sujit PhD,

Sheeba J. Sujit PhD

orcid.org/0000-0001-8302-7046

Department of Diagnostic and Interventional Imaging, McGovern Medical School, University of Texas Health Science Center, Houston, Texas, USA

Search for more papers by this author

Jerry S. Wolinsky MD,

Jerry S. Wolinsky MD

Department of Neurology, McGovern Medical School, University of Texas Health Science Center, Houston, Texas, USA

Search for more papers by this author

Fred D. Lublin MD,

Fred D. Lublin MD

Department of Neurology, Icahn School of Medicine at Mount Sinai, New York, New York, USA

Search for more papers by this author

Refaat E. Gabr PhD,

Refaat E. Gabr PhD

Department of Diagnostic and Interventional Imaging, McGovern Medical School, University of Texas Health Science Center, Houston, Texas, USA

Search for more papers by this author

Ponnada A. Narayana PhD,

Corresponding Author

Ponnada A. Narayana PhD

[email protected]

orcid.org/0000-0002-5703-4346

Department of Diagnostic and Interventional Imaging, McGovern Medical School, University of Texas Health Science Center, Houston, Texas, USA

Ivan Coronado MS,

Ivan Coronado MS

Department of Diagnostic and Interventional Imaging, McGovern Medical School, University of Texas Health Science Center, Houston, Texas, USA

Search for more papers by this author

Sheeba J. Sujit PhD,

Sheeba J. Sujit PhD

orcid.org/0000-0001-8302-7046

Department of Diagnostic and Interventional Imaging, McGovern Medical School, University of Texas Health Science Center, Houston, Texas, USA

Search for more papers by this author

Jerry S. Wolinsky MD,

Jerry S. Wolinsky MD

Department of Neurology, McGovern Medical School, University of Texas Health Science Center, Houston, Texas, USA

Search for more papers by this author

Fred D. Lublin MD,

Fred D. Lublin MD

Department of Neurology, Icahn School of Medicine at Mount Sinai, New York, New York, USA

Search for more papers by this author

Refaat E. Gabr PhD,

Refaat E. Gabr PhD

Department of Diagnostic and Interventional Imaging, McGovern Medical School, University of Texas Health Science Center, Houston, Texas, USA

Search for more papers by this author

First published: 18 October 2019

https://doi.org/10.1002/jmri.26959

Citations: 41

Share a link

Email
Wechat
Bluesky

Abstract

Background

The dependence of deep-learning (DL)-based segmentation accuracy of brain MRI on the training size is not known.

Purpose

To determine the required training size for a desired accuracy in brain MRI segmentation in multiple sclerosis (MS) using DL.

Study Type

Retrospective analysis of MRI data acquired as part of a multicenter clinical trial.

Study Population

In all, 1008 patients with clinically definite MS.

Field Strength/Sequence

MRIs were acquired at 1.5T and 3T scanners manufactured by GE, Philips, and Siemens with dual turbo spin echo, FLAIR, and T₁-weighted turbo spin echo sequences.

Assessment

Segmentation results using an automated analysis pipeline and validated by two neuroimaging experts served as the ground truth. A DL model, based on a fully convolutional neural network, was trained separately using 16 different training sizes. The segmentation accuracy as a function of the training size was determined. These data were fitted to the learning curve for estimating the required training size for desired accuracy.

Statistical Tests

The performance of the network was evaluated by calculating the Dice similarity coefficient (DSC), and lesion true-positive and false-positive rates.

Results

The DSC for lesions showed much stronger dependency on the sample size than gray matter (GM), white matter (WM), and cerebrospinal fluid (CSF). When the training size was increased from 10 to 800 the DSC values varied from 0.00 to 0.86 ± 0.016 for T₂ lesions, 0.87 ± 009 to 0.94 ± 0.004 for GM, 0.86 ± 0.08 to 0.94 ± 0.005 for WM, and 0.91 ± 0.009 to 0.96 ± 0.003 for CSF.

Data Conclusion

Excellent segmentation was achieved with a training size as small as 10 image volumes for GM, WM, and CSF. In contrast, a training size of at least 50 image volumes was necessary for adequate lesion segmentation.

Level of Evidence: 1

Technical Efficacy Stage: 1

J. Magn. Reson. Imaging 2020;51:1487–1496.

References

1Wallin MT, Culpepper WJ, Campbell JD, et al. The prevalence of MS in the United States: A population-based estimate using health claims data. Neurology 2019; 92: e1029–e1040.
10.1212/WNL.0000000000007035
PubMed Web of Science® Google Scholar
2Moccia M, de Stefano N, Barkhof F. Imaging outcome measures for progressive multiple sclerosis trials. Mult Scler 2017; 23: 1614–1626.
10.1177/1352458517729456
PubMed Web of Science® Google Scholar
3Sastre-Garriga J, Pareto D, Rovira À. Brain atrophy in multiple sclerosis: Clinical relevance and technical aspects. Neuroimaging Clin 2017; 27: 289–300.
10.1016/j.nic.2017.01.002
PubMed Web of Science® Google Scholar
4Ontaneda D, Fox RJ. Imaging as an outcome measure in multiple sclerosis. Neurotherapeutics 2017; 14: 24–34.
10.1007/s13311-016-0479-6
CAS PubMed Web of Science® Google Scholar
5Hemond CC, Bakshi R. Magnetic resonance imaging in multiple sclerosis. Cold Spring Harb Perspect Med 2018; 8: a028969.
10.1101/cshperspect.a028969
PubMed Web of Science® Google Scholar
6Danelakis A, Theoharis T, Verganelakis DA. Survey of automated multiple sclerosis lesion segmentation techniques on magnetic resonance imaging. Comput Med Imaging Graph 2018; 70: 83–100.
10.1016/j.compmedimag.2018.10.002
PubMed Web of Science® Google Scholar
7Garccia-Lorenzo D, Francis S, Narayanan S, Arnold DL, Collins DL. Review of automatic segmentation methods of multiple sclerosis white matter lesions on conventional magnetic resonance imaging. Med Image Anal 2013; 17: 1–18.
10.1016/j.media.2012.09.004
PubMed Web of Science® Google Scholar
8LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015; 1: 436–444.
10.1038/nature14539
Web of Science® Google Scholar
9Litjens G, Kooi T, Bejnordi BE, et al. A survey on deep learning in medical image analysis. Med Image Anal 2017. 60–88.
10.1016/j.media.2017.07.005
PubMed Web of Science® Google Scholar
10Ravi D, Wong C, Deligianni F, et al. Deep learning for health informatics. IEEE J Biomed Heal Inform 2017; 21: 4–21.
10.1109/JBHI.2016.2636665
PubMed Web of Science® Google Scholar
11Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In: Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics); 2015.
10.1007/978-3-319-24574-4_28
Google Scholar
12Akkus Z, Galimzianova A, Hoogi A, Rubin DL, Erickson BJ. Deep learning for brain MRI segmentation: State of the art and future directions. J Digit Imaging 2017; 449–459.
10.1007/s10278-017-9983-4
PubMed Web of Science® Google Scholar
13Shen D, Wu G, Suk H-I. Deep learning in medical image analysis. Annu Rev Biomed Eng 2017; 19: 221–248.
10.1146/annurev-bioeng-071516-044442
CAS PubMed Web of Science® Google Scholar
14Gabr RE, Coronado I, Robinson M, et al. Brain and lesion segmentation in multiple sclerosis using fully convolutional neural networks: A large-scale study. Mult Scler J 2019;1352458519856843.
10.1177/1352458519856843
PubMed Google Scholar
15Torrey l, Shavlik J. Transfer learning. In: Handbook of research on machine learning applications, vol. 3. 2009; 17–35.
Google Scholar
16Tajbakhsh N, Shin JY, Gurudu SR, et al. Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE Trans Med Imaging 2016; 35: 1299–1312.
10.1109/TMI.2016.2535302
PubMed Web of Science® Google Scholar
17Chartrand G, Cheng PM, Vorontsov E, et al. Deep learning: A primer for radiologists. Radiographics 2017; 2113–2131.
10.1148/rg.2017170077
PubMed Web of Science® Google Scholar
18Wong KCL, Syeda-Mahmood T, Moradi M. Building medical image classifiers with very limited data using segmentation networks. Med Image Anal 2018; 49: 105–116.
10.1016/j.media.2018.07.010
PubMed Web of Science® Google Scholar
19Goodfellow I, Bengio Y, Courville A, Bengio Y. Deep learning, vol. 1. Cambridge, MA: MIT Press; 2016.
Google Scholar
20Cho J, Lee K, Shin E, Choy G, Do S. How much data is needed to train a medical image deep learning system to achieve necessary high accuracy? arXiv Prepr arXiv151106348; 2015.
Google Scholar
21Kalayeh HM, Landgrebe DA. Predicting the required number of training samples. Pattern Anal Mach Intell IEEE Trans 1983; 5: 664–667.
10.1109/TPAMI.1983.4767459
CAS PubMed Web of Science® Google Scholar
22Mukherjee S, Tamayo P, Rogers S, et al. Estimating dataset size requirements for classifying DNA microarray data. J Comput Biol 2003; 10: 119–142.
10.1089/106652703321825928
CAS PubMed Web of Science® Google Scholar
23Tam VH, Kabbara S, Yeh RF, Leary RH. Impact of sample size on the performance of multiple-model pharmacokinetic simulations. Antimicrob Agents Chemother 2006; 50: 3950–3952.
10.1128/AAC.00337-06
CAS PubMed Web of Science® Google Scholar
24Dobbin KK, Zhao Y, Simon RM. How large a training set is needed to develop a classifier for microarray data? Clin Cancer Res 2008; 14: 108–114.
10.1158/1078-0432.CCR-07-0443
CAS PubMed Web of Science® Google Scholar
25Kim S-Y. Effects of sample size on robustness and prediction accuracy of a prognostic gene signature. BMC Inform 2009; 10: 147.
PubMed Web of Science® Google Scholar
26Figueroa RL, Zeng-Treitler Q, Kandula S, Ngo LH. Predicting sample size required for classification performance. BMC Med Inform Decis Mak 2012; 12: 8.
10.1186/1472-6947-12-8
PubMed Web of Science® Google Scholar
27Cortes C, Jackel LD, Solla SA, Vapnik V DJA. Learning curves: Asymptotic values and rate of convergence. In: Adv Neural Inf Process Syst 1994; 327–334.
Google Scholar
28Lindsey JW, Scott TF, Lynch SG, et al. The CombiRx trial of combined therapy with interferon and glatiramer acetate in relapsing remitting MS: Design and baseline characteristics. Mult Scler Relat Disord 2012; 1: 81–86.
10.1016/j.msard.2012.01.006
CAS Web of Science® Google Scholar
29Lublin FD, Cofield SS, Cutter GR, et al. Randomized study combining interferon and glatiramer acetate in multiple sclerosis. Ann Neurol 2013; 73: 327–340.
10.1002/ana.23863
CAS PubMed Web of Science® Google Scholar
30Datta S, Narayana PA. A comprehensive approach to the segmentation of multichannel three-dimensional MR brain images in multiple sclerosis. NeuroImage Clin 2013; 2: 184–196.
10.1016/j.nicl.2012.12.007
PubMed Web of Science® Google Scholar
31Datta S, Sajja BR, He R, Wolinsky JS, Gupta RK, Narayana PA. Segmentation and quantification of black holes in multiple sclerosis. Neuroimage 2006; 29: 467–474.
10.1016/j.neuroimage.2005.07.042
PubMed Web of Science® Google Scholar
32Sajja BR, Datta S, He R, et al. Unified approach for multiple sclerosis lesion segmentation on brain MRI. Ann Biomed Eng 2006; 34: 142–151.
10.1007/s10439-005-9009-0
PubMed Web of Science® Google Scholar
33Bottou L. Large-scale machine learning with stochastic gradient descent. In: Proc COMPSTAT'2010. Berlin: Springer; 2010: 177–186.
10.1007/978-3-7908-2604-3_16
Google Scholar
34Sutskever I, Martens J, Dahl G, Hinton G. On the importance of initialization and momentum in deep learning. In: Int Conf Mach Learn; 2013: 1139–1147.
Google Scholar
35Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In: Proc 13th Int Conf Artif Intell Stat; 2010: 249–256.
Google Scholar
36Sudre CH, Li W, Vercauteren T, Ourselin S, Cardoso MJ. Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In: Deep Learn Med Image Anal Multimodal Learn Clin Decis Support. Berlin: Springer; 2017: 240–248.
10.1007/978-3-319-67558-9_28
Google Scholar
37Chollet F. Keras: Deep learning library for theano and tensorflow. URL https://keras.io/k 2015; 7.
Google Scholar
38Abadi M, Barham P, Chen J, et al. Proceedings of the TensorFlow: A system for large-scale machine learning. In: Proc 12th USENIX Conf Oper Syst Des Implement; 2016.
Google Scholar
39Bengio Y, Louradour J, Collobert R, Weston J. Curriculum learning. In: Proc 26th Annu Int Conf Mach Learn; 2009: 41–48.
10.1145/1553374.1553380
Google Scholar

Citing Literature

Volume51, Issue5

May 2020

Pages 1487-1496

Deep-Learning-Based Neural Tissue Segmentation of MRI in Multiple Sclerosis: Effect of Training Set Size

Abstract

Background

Purpose

Study Type

Study Population

Field Strength/Sequence

Assessment

Statistical Tests

Results

Data Conclusion

References

Citing Literature

References

Information

About Wiley Online Library

Help & Support

Opportunities

Connect with Wiley

Deep-Learning-Based Neural Tissue Segmentation of MRI in Multiple Sclerosis: Effect of Training Set Size

Abstract

Background

Purpose

Study Type

Study Population

Field Strength/Sequence

Assessment

Statistical Tests

Results

Data Conclusion

References

Citing Literature

References

Related

Information