Use of Artificial Intelligence as an Innovative Method for Liver Graft Macrosteatosis Assessment

In liver transplantation (LT), the percentage of hepatic steatosis (HS) in the liver graft is associated with increased risk of graft dysfunction or early nonfunction,^{(1, 2)} and fast and accurate assessment of HS is of paramount importance in the setting of organ procurement for LT.

Although a wide range of methods for the assessment of HS are currently available, pathological examination remains the gold standard for both HS diagnosis and HS grading, and it is used as the reference method for any evaluation of any new HS measurement technique.⁽³⁾ However, routine use of pathological examination is limited by invasiveness or the need of additional time-consuming procedures and instrumentation, which are often unavailable in remote graft procurement hospitals.⁽⁴⁾ Therefore, the decision to accept or discard a liver graft still relies on indirect parameters, such as clinical history, blood tests, liver/spleen attenuation ratio on imaging (when available), and, more importantly, on the harvesting surgeon’s personal evaluation of the liver texture and color. Although this latter parameter achieves an accuracy (Acc) of 86%⁽⁵⁾ for experienced surgeons, it remains a qualitative and subjective evaluation and a source of major biases in case of comparison of repeated measures.

Several recent reports focused on tissue analysis using artificial intelligence (AI), including machine-learning (ML) approaches and fully convolutional neural network (FCNN) concepts, from intraoperative optical images. These types of computerized analyses may help to obtain standardized basic pathological analysis without interobserver variability, classification biases, or technical constraints.^(6-9) On the basis of the recent promising strategy reported by our team on HS graft analysis with AI ML technology, the present study aims at automating liver graft segmentation from smartphone images and validating the robustness of this approach to assess HS on a larger cohort of liver grafts using different cutoff values.

Patients and Methods

Test Methods

The test method was HS tissue classification developed with an AI ML algorithm based on images taken during graft harvesting with a smartphone.

AI Algorithms

For HS classification, ML algorithms were used. ML is an AI method commonly used to build algorithms that learn from data and make predictions based on these algorithms. In this study, we used the support vector machines–single instance learning (SVM-SIL) system. SVM-SIL are semisupervised ML algorithms that use both labeled and unlabeled data to build a predictive algorithm for classification analyses. More specifically, the SVM-SIL system, using the training algorithm, builds a model from a set of training examples (ie, with or without significant HS) that will assign new examples to one category or the other, making it a nonprobabilistic binary linear classifier. This process, performed as many times as the number of included cases, allows assessing the sensitivity, specificity, and Acc of the classification algorithm model. This method is currently the most advanced existing system in ML to avoid selection and classification biases. For liver segmentation from intraoperative photographs, we used FCNN, which is one of the deep-learning strategies that involves some convolutional filters that can learn hierarchical features from data. The role of the filters consists in extracting some characteristics from the input images and collects them in maps that include these features. The number of filters for each layer is chosen according to the time necessary for training the network and the complexity of the problem. In general, a higher number of filters will give better results. The rule is applied only up to a certain threshold because an increase in the number of filters beyond this threshold no longer affects the performance.

Smartphone Images

During the procurement, at least 1 digital image of the liver graft was taken by the personal smartphone of a member of the harvesting surgical team, using a previously published standardized protocol to obtain the best possible image quality and reproducibility.^{(11, 12)} The same smartphone (iPhone 6S; Apple Inc.,^TM Cupertino, CA) was used. The camera was automatically white-balanced and used in the Macro mode. On these smartphones, exposure and focus area selection modes were nonadjustable.

Pathological Analysis

The existence and quantification of HS was estimated on frozen sections from liver graft biopsies, and these sections were considered the reference control test. Surgical biopsies were performed during liver procurement, in the case of suspected severe liver steatosis, or after reperfusion, as routinely performed in both institutions. Presence and proportion of HS was assessed by an experienced pathologist (N.P.), supervised by a referenced pathologist (V.P.), and expressed in percentage of hepatocytes with macrosteatosis steatosis (0%-100%).

Hepatic Steatosis

Test Positivity Cutoffs

An HS rate >60% is associated with a high risk of primary graft failure⁽¹³⁾ and is widely accepted as a cutoff value to discard liver graft discard. However, early allograft dysfunction was reported even in grafts with HS >30% generating different acceptance policies. For this reason and because 30% is the cutoff value used at our centers, the ML method for HS assessment was predetermined for 2 categories of grafts: HS <30% and HS >30%.

AI Cutoff

In our previous study⁽¹⁴⁾ a set of images was created by manually cropping the original photographs so the target organ (liver) would occupy 100% of the frame with elimination of the background. Contrary to this method, in this study, a FCNN was used that was composed of several layers as shown in Fig. 1, inspired by U-net⁽¹⁵⁾ and Resnet.⁽¹⁶⁾ In our case, the model of the network consisted of convolutional (descending) and up-convolutional (ascending) paths. The combination of a convolutional block and 2 identity convolutional blocks is repeated 4 times (in Fig. 1: from stage 1 to stage 4). In each stage, the number of convolutional kernels per layer is doubled. The up-convolutional path is symmetric to the convolutional one. Each stage, repeated 4 times (in Fig. 1: from stage 5 to stage 8), presents an up-convolutional block instead of the convolutional one. The ascending path ends with an up-sampling block of (2, 2) size and a convolutional one, in this case with a (3 × 3) kernel and a sigmoid activation. The network was trained and validated with red green blue (RGB) images and with grayscale images that were obtained by converting the original ones. The results from the automatic segmentation are compared with the ones from the manual segmentation, which is considered the gold standard. To evaluate the segmentation performance of FCNN, we calculated the Acc, recall (Rec), and precision (Prec) on MATLAB software (MathWorks, Inc. Massachusetts 01760 USA).

Once liver graft masks were obtained by FCNN, ML was used to classify features. Because the liver texture is heterogeneous, the texture analysis was performed on liver image patches. Each image was divided into 15 nonoverlapping patches of 100 × 100 pixels. Each patch was classified in transplantable (HS <30%) or nontransplantable (HS >30%) categories, and the features of each patch according to its category were extracted by rotation-invariant local binary patterns, which are resistant to light and camera pose variations and accurately render the liver tissue (Fig. 2). To perform the classification of patches according to donor data, multiple-instance learning on SVM-SIL was used, which has the strong advantage of allowing the fusion of patch-wise information (such as textural features) with image-wise information (such as donor data features). The feature classification was implemented with scikit-learn (http://scikit-learn.org). To perform robust performance evaluation, we performed leave-one-patient-out cross-validation (ie, the images from all patients-1 were used for training and the remaining one for testing). The duration of the test classification analysis was 10⁻³ seconds. Donor feature (photographs and data) classification was analyzed in terms of Acc, Rec, and Prec on MATLAB software.

Pathology

According to the most commonly used scoring system, HS was categorized as follows: normal (grade 0), when the proportion of HS-affected cells ranged from 0% to 5%; mild (grade 1), between 5% and 33%; moderate (grade 2), between 34% and 66%; and severe (grade 3), when the proportion of affected cells was >67%.⁽¹⁷⁾

Blindness

Clinical, biological, and radiological information were available to the surgeons performing the organ procurement, to the pathologist performing the graft biopsy analyses, and to the team in charge of the AI assessment. The results of HS based on the pathology report were available to AI assessors, but pathologists were blinded to AI HS assessment results.

Results

From January to June 2018, 117 consecutive liver grafts from deceased donors were photographed and biopsied with the intention to be transplanted. Of these, 28 had HS >30% and were included in the analysis without being transplanted. A total of 28 liver grafts with HS <30% were procured during the same study period, and they were randomly selected on a 1:1 ratio and included in the control group. This created a balanced data set, which is required for an appropriate SVM-SIL analysis. Thus, the final inclusion cohort for HS classification comprised 56 liver grafts.

A liver biopsy was performed without any complications in all 117 patients. A total of 40 RGB liver graft images (size 3264 × 2448 pixels, 8 megapixels) were taken during each graft procurement and were analyzed. For liver segmentation analysis (liver graft image extraction), all 117 intraoperative pictures were used: 50 for the training data set and 67 for the testing data set. The Prec medians for grayscale and RGB images were equal to 95% and 97%, respectively. The Acc and Rec medians for grayscale and RGB images were 92%, 89% and 98%, 97%, respectively (Fig. 3).

Texture analysis by SVM-SIL was performed on liver image patches, and a balanced data set of 600 patches was obtained. A flow diagram of participants is summarized in Fig. 4, and baseline characteristics are summarized in Table 1. After analysis of the features of each data set, the SVM-SIL showed good results in graft classification according to their acceptance or discard for LT, with a Rec of 93% and a Prec of 82% for transplanted grafts and with a Rec of 97% and Prec of 83% for discarded grafts.

The classification of donor data from the liver picture according to the proportion of steatosis (superior or inferior to 30% of HS) showed an Acc of 89%.

Discussion

This is the first multicenter prospective study to assess the performance of AI technologies to evaluate HS through smartphone pictures and donor data with comparison to liver biopsy. The AI approach reported a sensitivity (Rec) of 97% and 93% for the classification of nonsteatotic and steatotic grafts based on a cutoff value of 30% with an Acc of 89%. These results validate our primary monocentric study where an analysis of 40 liver grafts (2 classes of 20 grafts) showed a sensitivity (Rec) of 80% and 95% for the classification of nonsteatotic, mild, and moderate steatotic versus severe steatotic grafts (cutoff of HS = 60%) with an Acc of 88%.⁽¹⁴⁾ Furthermore, the use of FCNN for liver graft image extraction from the donor picture, first reported in literature, leads to a fully automated HS assessment method. Testing this AI approach with the most frequently used HS cutoff (30% and 60%) for liver graft acceptance policy supports that this method represents a promising step toward a clinically relevant processing system for automatic, noninvasive, and objective HS assessment in the setting of LT.

The main strengths of this study are the noninvasiveness, rapidity, and utilization of a device that is available worldwide (ie, smartphone) for the analysis as compared with the standard reference examination, the liver biopsy. Liver photographs and donor data are processed and classified within seconds with the computer-assisted semisupervised system. No additional procedure or equipment is needed during graft harvesting, and no invasive liver sampling is necessary. On the contrary, a liver biopsy requires a surgical graft sampling, which may cause complications such as bleeding and involves an examination from a pathologist, which is time consuming and frequently unavailable 24/7 in remote places. In contrast, this AI-based steatosis assessment technique is a real-time, noninvasive method to assess HS.

The pictures were acquired with an easy-to-use and widely available smartphone camera. Although smartphones have a significant cost, they are prevalent in the general population (100% of our surgical team). Moreover, the smartphone camera used in this study (Apple iPhone) has a medium performance when compared with other smartphones,⁽¹⁹⁾ so it would be reasonable to assume that newer smartphones could also be used for graft picture acquisition. The second advantage of this method is the standardization of HS evaluation. Even though a liver biopsy is considered as the standard examination for HS assessment, this technique is subject to interobserver variability or sampling size limitation, as underlined by previous publications.⁽³⁾ Furthermore, when pathological examination is not available, the decision to accept or discard a liver graft is usually based on donor data, liver texture, macroscopic evaluation, and subjective clinical experience of the harvesting surgeon, making the graft evaluation highly subjective to bias. By using an AI-based classification, interobserver variability and classification bias are dramatically limited. Indeed, the use of AI allows converting a subjective decision-making process into a standardized, computerized and homogeneous process.

In this experimental study, the highest classification performance was obtained using texture features combined with significant donor data. Indeed, the inclusion of donor features in the algorithm helped increase the Acc of the classification by SVM-SIL, as previously demonstrated.⁽¹⁴⁾ The real innovation of this study, through AI computerization of a human process, is 3-fold: first, to translate the subjective visual assessment of liver texture into an objective and standardized method (smartphone picture); second, to develop a fully automated HS assessment by the liver graft image extraction; and third, to replicate the clinical experience of a transplant surgeon (donor data analysis). A major limitation of this study is the small sample size, which is a common problem within the computer-assisted diagnosis community,⁽²⁰⁾ particularly in the LT setting wherein a larger cohort of discarded grafts (without any intention to be transplanted) are available only for machine perfusion studies.⁽²¹⁾ On the contrary, our sample of discarded grafts was accepted with the intention to be transplanted but then discarded for HS >30% (measured by intraoperative biopsy). A second limit is the impossibility of SVM-SIL analysis to differentiate macrosteatosis and microsteatosis by liver picture and biology when only macrosteatosis is predictive of liver graft dysfunction. Therefore, the present results will require confirmation in larger multicentric studies probably in the machine perfusion study setting. Enlarging the training data set would also allow investigating more advanced machine learning methods. Nevertheless, it has been reported that SVM-SIL achieves competitive results as compared with other more sophisticated semisupervised methods.⁽⁷⁾ The third limit is that the HS assessment method described here, as all methods reported in literature,⁽⁵⁾ could not be a substitute to liver biopsy, as no methods are set to predict other liver conditions (such as balloon degeneration, centrilobular necrosis, or chronic hepatitis), which (more rarely) can also adversely affect graft outcome. Lastly, the use of 30% steatosis as a cutoff value could not reflect the worldwide liver graft acceptance policy and the old smartphone technology (the Iphone 6s was first officially released on September 25, 2015) could limit algorithm performance.

As the LT scientific community reaches for new standards on HS assessment, research to find alternatives to histopathology are highly encouraged. Interesting results are available on magnetic resonance imaging and elastography,^{(22, 23)} but none of them is either practical or performing in the very specific setting of organ procurement.

This research showed that liver texture analysis from pictures and donor data features, analyzed by an AI approach, could represent a promising step toward a helpful processing system to support the surgeon’s decision, particularly the younger surgeons of the future, to accept or discard a liver graft during procurement.