Deep convolutional neural network for segmentation of knee joint anatomy
Zhaoye Zhou
Department of Biomedical Engineering, University of Minnesota, Minneapolis, Minnesota
Search for more papers by this authorGengyan Zhao
Departments of Radiology, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin
Search for more papers by this authorRichard Kijowski
Departments of Radiology, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin
Search for more papers by this authorCorresponding Author
Fang Liu
Departments of Radiology, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin
Correspondence Fang Liu, Department of Radiology, Wisconsin Institutes for Medical Research, 1111 Highland Avenue, Madison, Wisconsin 53705-2275. Email: [email protected]Search for more papers by this authorZhaoye Zhou
Department of Biomedical Engineering, University of Minnesota, Minneapolis, Minnesota
Search for more papers by this authorGengyan Zhao
Departments of Radiology, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin
Search for more papers by this authorRichard Kijowski
Departments of Radiology, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin
Search for more papers by this authorCorresponding Author
Fang Liu
Departments of Radiology, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin
Correspondence Fang Liu, Department of Radiology, Wisconsin Institutes for Medical Research, 1111 Highland Avenue, Madison, Wisconsin 53705-2275. Email: [email protected]Search for more papers by this authorAbstract
Purpose
To describe and evaluate a new segmentation method using deep convolutional neural network (CNN), 3D fully connected conditional random field (CRF), and 3D simplex deformable modeling to improve the efficiency and accuracy of knee joint tissue segmentation.
Methods
A segmentation pipeline was built by combining a semantic segmentation CNN, 3D fully connected CRF, and 3D simplex deformable modeling. A convolutional encoder-decoder network was designed as the core of the segmentation method to perform high resolution pixel-wise multi-class tissue classification for 12 different joint structures. The 3D fully connected CRF was applied to regularize contextual relationship among voxels within the same tissue class and between different classes. The 3D simplex deformable modeling refined the output from 3D CRF to preserve the overall shape and maintain a desirable smooth surface for joint structures. The method was evaluated on 3D fast spin-echo (3D-FSE) MR image data sets. Quantitative morphological metrics were used to evaluate the accuracy and robustness of the method in comparison to the ground truth data.
Results
The proposed segmentation method provided good performance for segmenting all knee joint structures. There were 4 tissue types with high mean Dice coefficient above 0.9 including the femur, tibia, muscle, and other non-specified tissues. There were 7 tissue types with mean Dice coefficient between 0.8 and 0.9 including the femoral cartilage, tibial cartilage, patella, patellar cartilage, meniscus, quadriceps and patellar tendon, and infrapatellar fat pad. There was 1 tissue type with mean Dice coefficient between 0.7 and 0.8 for joint effusion and Baker's cyst. Most musculoskeletal tissues had a mean value of average symmetric surface distance below 1 mm.
Conclusion
The combined CNN, 3D fully connected CRF, and 3D deformable modeling approach was well-suited for performing rapid and accurate comprehensive tissue segmentation of the knee joint. The deep learning-based segmentation method has promising potential applications in musculoskeletal imaging.
Supporting Information
Additional Supporting Information may be found in the online version of this article.
| Filename | Description |
|---|---|
| mrm27229-sup-0001-suppinfoFigs.docx1.2 MB |
FIGURE S1 The training loss curves in the pre-training process using SKI10 images and the training and validation loss curves in 1 training fold using 3D-FSE images FIGURE S2 Examples of tissue segmentation performed on the 3D-FSE images in 2 subjects with knee OA using the CED network only, the CED network combined with 3D fully connected CRF, and the CED network combined with both CRF and 3D deformable modeling. This figure extends Figure 4 to include sagittal, axial, and coronal views |
Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
REFERENCES
- 1Oliveria SA, Felson DT, Reed JI, Cirillo PA, Walker AM. Incidence of symptomatic hand, hip, and knee osteoarthritis among patients in a health maintenance organization. Arthritis Rheum. 1995; 38: 1134-1141.
- 2Felson DT. An update on the pathogenesis and epidemiology of osteoarthritis. Radiol Clin North Am. 2004; 42: 1-9.
- 3Link TM, Steinbach LS, Ghosh S, et al. Osteoarthritis: MR imaging findings in different stages of disease and correlation with clinical findings. Radiology. 2003; 226: 373-381.
- 4Hunter DJ, March L, Sambrook PN. The association of cartilage volume with knee pain. Osteoarthritis Cartilage. 2003; 11: 725-729.
- 5Cicuttini FM, Wluka AE, Wang Y, Stuckey SL. Longitudinal study of changes in tibial and femoral cartilage in knee osteoarthritis. Arthritis Rheum. 2004; 50: 94-97.
- 6Raynauld JP, Martel-Pelletier J, Berthiaume MJ, et al. Long term evaluation of disease progression through the quantitative magnetic resonance imaging of symptomatic knee osteoarthritis patients: correlation with clinical symptoms and radiographic changes. Arthritis Res Ther. 2005; 8: R21.
- 7Felson DT, Chaisson CE, Hill CL, et al. The association of bone marrow lesions with pain in knee osteoarthritis. Ann Intern Med. 2001; 134: 541-549.
- 8Hill CL, Gale DG, Chaisson CE, et al. Knee effusions, popliteal cysts, and synovial thickening: association with knee pain in osteoarthritis. J Rheumatol. 2001; 28: 1330-1337.
- 9Sowers MF, Hayes C, Jamadar D, et al. Magnetic resonance-detected subchondral bone marrow and cartilage defect characteristics associated with pain and X-ray-defined knee osteoarthritis. Osteoarthritis Cartilage. 2003; 11: 387-393.
- 10Peterfy C, Woodworth T, Altman R. Workshop for Consensus on Osteoarthritis Imaging: MRI of the knee. Osteoarthritis Cartilage. 2006; 14: 44-45.
- 11Eckstein F, Kwoh CK, Link TM, OAI investigators. Imaging research results from the Osteoarthritis Initiative (OAI): a review and lessons learned 10 years after start of enrolment. Ann Rheum Dis. 2014; 73: 1289-1300.
- 12Peterfy CG, Guermazi A, Zaim S, et al. Whole-Organ Magnetic Resonance Imaging Score (WORMS) of the knee in osteoarthritis. Osteoarthritis Cartilage. 2004; 12: 177-190.
- 13Hunter DJ, Lo GH, Gale D, Grainger AJ, Guermazi A, Conaghan PG. The reliability of a new scoring system for knee osteoarthritis MRI and the validity of bone marrow lesion assessment: BLOKS (Boston Leeds osteoarthritis knee score). Ann Rheum Dis. 2008; 67: 206-211.
- 14Hunter DJ, Guermazi A, Lo GH, et al. Evolution of semi-quantitative whole joint assessment of knee OA: MOAKS (MRI osteoarthritis knee score). Osteoarthritis Cartilage. 2011; 19: 990-1002.
- 15Li Q, Amano K, Link TM, Ma CB. Advanced imaging in osteoarthritis. Sports Health. 2016; 8: 418-428.
- 16Reichenbach S, Yang M, Eckstein F, et al. Does cartilage volume or thickness distinguish knees with and without mild radiographic osteoarthritis? The Framingham study. Ann Rheum Dis. 2010; 69: 143-149.
- 17Eckstein F, Maschek S, Wirth W, et al. One year change of knee cartilage morphology in the first release of participants from the Osteoarthritis Initiative progression subcohort: association with sex, body mass index, symptoms and radiographic osteoarthritis status. Ann Rheum Dis. 2009; 68: 674-679.
- 18Le Graverand MPH, Buck RJ, Wyman BT, et al. Change in regional cartilage morphology and joint space width in osteoarthritis participants versus healthy controls: a multicentre study using 3.0 Tesla MRI and Lyon-Schuss radiography. Ann Rheum Dis. 2010; 69: 155-162.
- 19Liebl H, Joseph G, Nevitt MC, et al. Early T2 changes predict onset of radiographic knee osteoarthritis: data from the osteoarthritis initiative. Ann Rheum Dis. 2015; 74: 1353-1359.
- 20Prasad AP, Nardo L, Schooler J, Joseph GB, Link TM. T1ρ and T2 relaxation times predict progression of knee osteoarthritis. Osteoarthritis Cartilage. 2013; 21: 69-76.
- 21Liu F, Choi KW, Samsonov A, et al. Articular cartilage of the human knee joint: in vivo multicomponent T2 analysis at 3.0 T. Radiology. 2015; 277: 477-488.
- 22Barr AJ, Dube B, Hensor EMA, et al. The relationship between clinical characteristics, radiographic osteoarthritis and 3D bone area: data from the Osteoarthritis Initiative. Osteoarthritis Cartilage. 2014; 22: 1703-1709.
- 23Hunter D, Nevitt M, Lynch J, et al. Longitudinal validation of periarticular bone area and 3D shape as biomarkers for knee OA progression? Data from the FNIH OA Biomarkers Consortium. Ann Rheum Dis. 2016; 75: 1607-1614.
- 24Bowes MA, Vincent GR, Wolstenholme CB, Conaghan PG. A novel method for bone area measurement provides new insights into osteoarthritis and its progression. Ann Rheum Dis. 2015; 74: 519-525.
- 25Frobell RB, Roos HP, Roos EM, et al. The acutely ACL injured knee assessed by MRI: are large volume traumatic bone marrow lesions a sign of severe compression injury? Osteoarthritis Cartilage. 2008; 16: 829-836.
- 26Driban JB, Price L, Lo GH, et al. Evaluation of bone marrow lesion volume as a knee osteoarthritis biomarker - longitudinal relationships with pain and structural changes: data from the Osteoarthritis Initiative. Arthritis Res Ther. 2013; 15: R112.
- 27Pang J, Driban JB, Destenaves G, et al. Quantification of bone marrow lesion volume and volume change using semi-automated segmentation: data from the osteoarthritis initiative. BMC Musculoskelet Disord. 2013; 14: 3.
- 28Gait AD, Hodgson R, Parkes MJ, et al. Synovial volume vs synovial measurements from dynamic contrast enhanced MRI as measures of response in osteoarthritis. Osteoarthritis Cartilage. 2016; 24: 1392-1398.
- 29Wenger A, Englund M, Wirth W, Hudelmaier M, Kwoh K, Eckstein F. Relationship of 3D meniscal morphology and position with knee pain in subjects with knee osteoarthritis: a pilot study. Eur Radiol. 2012; 22: 211-220.
- 30Roth M, Wirth W, Emmanuel K, Culvenor AG, Eckstein F. The contribution of 3D quantitative meniscal and cartilage measures to variation in normal radiographic joint space width—data from the Osteoarthritis Initiative healthy reference cohort. Eur J Radiol. 2017; 87: 90-98.
- 31Emmanuel K, Quinn E, Niu J, et al. Quantitative measures of meniscus extrusion predict incident radiographic knee osteoarthritis - data from the Osteoarthritis Initiative. Osteoarthritis Cartilage. 2016; 24: 262-269.
- 32Lu M, Chen Z, Han W, et al. A novel method for assessing signal intensity within infrapatellar fat pad on MR images in patients with knee osteoarthritis. Osteoarthritis Cartilage. 2016; 24: 1883-1889.
- 33Steidle-Kloc E, Culvenor AG, Dörrenberg J, et al. Relationship between knee pain and infra-patellar fat pad morphology - a within-and between-person analysis from the Osteoarthritis Initiative. Arthritis Care Res. 2018; 70: 550-557.
- 34Ruhdorfer A, Haniel F, Petersohn T, et al. Between-group differences in infra-patellar fat pad size and signal in symptomatic and radiographic progression of knee osteoarthritis vs non-progressive controls and healthy knees - data from the FNIH Biomarkers Consortium Study and the Osteoarthritis Initiative. Osteoarthritis Cartilage. 2017; 25: 1114-1121.
- 35Conroy MB,
Kwoh CK,
Krishnan E, et al. Muscle strength, mass, and quality in older men and women with knee osteoarthritis. Arthritis Care Res. 2012; 64: 15-21.
10.1002/acr.20588 Google Scholar
- 36Hart HF, Ackland DC, Pandy MG, Crossley KM. Quadriceps volumes are reduced in people with patellofemoral joint osteoarthritis. Osteoarthritis Cartilage. 2012; 20: 863-868.
- 37Pan J, Stehling C, Muller-Hocker C, et al. Vastus lateralis/vastus medialis cross-sectional area ratio impacts presence and degree of knee joint abnormalities and cartilage T2 determined with 3T MRI - an analysis from the incidence cohort of the Osteoarthritis Initiative. Osteoarthritis Cartilage. 2011; 19: 65-73.
- 38McWalter EJ, Wirth W, Siebert M, et al. Use of novel interactive input devices for segmentation of articular cartilage from magnetic resonance images. Osteoarthritis Cartilage. 2005; 13: 48-53.
- 39Fripp J, Crozier S, Warfield SK, Ourselin S. Automatic segmentation and quantitative analysis of the articular cartilages from magnetic resonance images of the knee. IEEE Trans Med Imaging. 2010; 29: 55-64.
- 40Tamez-Pena JG, Farber J, Gonzalez PC, Schreyer E, Schneider E, Totterman S. Unsupervised segmentation and quantification of anatomical knee features: data from the Osteoarthritis Initiative. IEEE Trans Biomed Eng. 2012; 59: 1177-1186.
- 41Prasoon A, Petersen K, Igel C, Lauze F, Dam E, Nielsen M. Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network. Med Image Comput Comput Assist Interv. 2013; 16(Pt 2): 246-253.
- 42Liu F, Zhou Z, Jang H, Samsonov A, Zhao G, Kijowski R. Deep convolutional neural network and 3D deformable approach for tissue segmentation in musculoskeletal magnetic resonance imaging. Magn Reson Med. 2018; 79: 2379-2391.
- 43Liu F, Jang H, Kijowski R, Bradshaw T, McMillan AB. Deep learning MR imaging–based attenuation correction for PET/MR imaging. Radiology. 2018; 286: 676-684.
- 44Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv e-prints arXiv 2014: 1409. 1556.
- 45Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv e-prints arXiv 2015: 1502. 03167.
- 46Nair V, Hinton GE. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning, Haifa, Israel, 2010. p. 807-814.
- 47He K, Zhang X, Ren S, Sun J. Identity mappings in deep residual networks. arXiv e-prints arXiv 2016:1603.05027.
- 48He X, Zemel RSRS, Carreira-Perpinan MAA, et al. Multiscale conditional random fields for image labeling. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, DC, 2004. p. 695-703.
- 49Krähenbühl P, Koltun V. Efficient inference in fully connected CRFs with Gaussian edge potentials. arXiv e-prints arXive 2012:1210.5644.
- 50Lorensen WE,
Cline HE. Marching cubes: a high resolution 3D surface construction algorithm. Comput Graph (ACM). 1987; 21: 163-169.
10.1145/37402.37422 Google Scholar
- 51Delingette H. General object reconstruction based on simplex meshes. Int J Comput Vis. 1999; 32: 111-146.
- 52Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. arXiv e-prints arXiv 2014;1411.4038.
- 53Abadi M, Agarwal A, Barham P, et al. TensorFlow: large-scale machine learning on heterogeneous distributed systems. arXiv e-prints arXiv 2016:1603.04467.
- 54François Chollet. Keras. GitHub 2015. https://github.com/fchollet/keras. Published March 27, 2015. Updated February 13, 2018. Accessed March 27, 2015.
- 55Heimann T, Morrison B, Styner M, Niethammer M, Warfield S. Segmentation of knee images: a grand challenge. Med Image Comput Comput Assist Interv. 2010: 207-214.
- 56He K, Zhang X, Ren S, Sun J. Delving deep into rectifiers: surpassing human-level performance on imagenet classification. arXiv e-prints arXiv 2015;1502.08152.
- 57Kingma DP, Ba J. Adam: a method for stochastic optimization. arXiv e-prints arXiv 2014:1412.6980.
- 58Kamnitsas K, Ledig C, Newcombe VFJ, et al. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med Image Anal. 2017; 36: 61-78.
- 59Milletari F, Navab N, Ahmadi SA. V-Net: fully convolutional neural networks for volumetric medical image segmentation. arXiv e-prints arXiv 2016:1606.04797.
- 60Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O. 3D U-Net: learning dense volumetric segmentation from sparse annotation. arXiv e-prints arXiv 2016:1606.06650.
- 61Christ PF, Ettlinger F, Grün F, et al. Automatic liver and tumor segmentation of CT and MRI volumes using cascaded fully convolutional neural networks. arXiv e-prints arXiv 2017:1702.05970
- 62Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL. Semantic image segmentation with deep convolutional nets and fully connected CRFs. arXiv e-prints arXiv 2014:1412.7062
- 63Liu F, Samsonov A, Wilson JJ, Blankenbaker DG, Block WF, Kijowski R. Rapid in vivo multicomponent T2 mapping of human knee menisci. J Magn Reson Imaging. 2015; 42: 1321-1328.
- 64Kijowski R, Wilson JJ, Liu F. Bicomponent ultrashort echo time T2* analysis for assessment of patients with patellar tendinopathy. J Magn Reson Imaging. 2017; 46: 1441-1447.
- 65van Opbroek A, Ikram MA, Vernooij MW, de Bruijne M. Transfer learning improves supervised image segmentation across imaging protocols. IEEE Trans Med Imaging. 2015; 34: 1018-1030.
