Study design

This was a post hoc analysis of a multicentre retrospective cohort study aimed at describing the treatment strategy for pseudoaneurysms following blunt liver and spleen injuries in paediatric patients (SHIPPS study).¹ The SHIPPS study was a cohort design, with data collected from January 2008 to December 2019 at university, non-university and children's hospitals across Japan. Additional details of the SHIPPS study are described in Table S1–S10.

This study (post hoc analysis) was approved by the Japanese Association for Surgery of Trauma Ethics Committee and the Ethics Committee of the Jichi Medical University Saitama Medical Center (Approval number: S19-012). The study was non-interventional, and the risk to participants was minimal; therefore, informed consent was waived for all patients at all hospitals. However, it is possible to refuse consent under the opt-out policy. This study conformed to the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) guidelines (Table S2).¹⁰

Study setting and population

The inclusion and exclusion criteria for the SHIPPS study are presented in Table S3. This study included paediatric patients with isolated splenic injury who had no confirmed pseudoaneurysm on admission and were managed with NOM. Exclusion criteria in this study were as follows: (1) concomitant liver injury, (2) surgery performed as the initial treatment, (3) pseudoaneurysm formation at presentation and (4) cases with missing data on the presence or absence of aneurysms 24 h or more after hospital admission. The data collected in SHIPPS did not allow for the extraction of whether the pseudoaneurysms were located in the liver or the spleen; thus, our study focused on isolated splenic injury.

Data collection

In the SHIPPS study, basic patient information, physiological signs, trauma severity at presentation, haematology and imaging data, treatment details, blood transfusions, post-treatment course, radiation exposure and treatment facilities were collected.

Details of the data collected in this study are presented in Table S4.

Outcome definition

Outcome was defined as the formation of a pseudoaneurysm first confirmed 24 h or more after admission, following the completion of the initial treatment upon arrival.

The diagnosis of pseudoaneurysm was made by contrast-enhanced CT, contrast-enhanced ultrasound, or angiography. The determination of whether a pseudoaneurysm was present and the imaging schedule were at the discretion of each facility and provider. The data collected by SHIPS did not allow us to understand the reasons for the re-evaluation of images or the clinical reasons.

Predictors

Predictors were determined from data available within 24 h of hospital arrival, which were considered clinically useful and from previous literature.^{2, 7, 11, 12} We considered that the presence or absence of concomitant liver injury could also be a predictor of splenic pseudoaneurysm formation; however, the SHIPPS data collection method made it impossible to distinguish whether the outcome, pseudoaneurysm formation, was hepatic or splenic. Therefore, we could not analyse liver injury complications as a predictor. Predictors included age at the time of injury, sex, origin of injury (fall from height, fall, during sports, bicycle, car, hit by a car while walking, abuse, assault, other), presence of shock vitals at presentation, ISS, AAST-OIS spleen grade 2018, presence of contrast extravascular leakage image at initial contrast-enhanced CT scan, presence of concomitant thoracic, abdominal, or cardiovascular injury, platelet level at presentation, PT-INR, blood products (RBC, platelets, FFP, cryoprecipitate and fibrinogen products) per body weight, tranexamic acid administered or not and the nature of the NOM (angiography only and no therapeutic intervention performed, Transarterial embolisation (TAE) with gelatine sponge, TAE with metal coil, TAE with other substances). The variables of age at the time of injury, platelet level at presentation, PT-INR and blood products per body weight were treated as continuous variables. Definitions of each predictor are given in Table S5.

Missing data

Variables with more than 30% missing data were excluded, and for continuous variables, outliers and apparent contradictions were treated as missing values. However, none of the variables included in this study had missing rates exceeding 30%, outliers, or contradictory values. Multiple imputations were performed using the ‘mice’ package for handling missing values (m = 13, maxit = 100, method = ‘pmm’, seed = 500).¹³ Missing values were imputed using all predictors, outcomes and other covariates.

Statistical analyses

Patient background factors are described as the mean (standard deviation) for continuous variables with a normal distribution, the median (interquartile range) for continuous variables with a non-normal distribution and the number and percentage for categorical variables. Comparisons between groups were made using the t-test or the Mann–Whitney U test for continuous variables as appropriate and the chi-square test for categorical variables.

Five predictive models were created using a logistic regression analysis. We incorporated all clinically useful factors, based on past literature and clinical experience, as prognostic factors into Model 1.

Models 2 to 4 were created by reducing the number of included factors based on the authors' judgement and experience. The prognostic factors for Model 5 were determined using backwards stepwise regression, a method that sequentially removes variables from Model 1 to identify the optimal combination of explanatory variables. This model included the following factors: age; sex; ISS; AAST-OIS 2018 spleen grade; presence or absence of extravasation on initial contrast-enhanced CT; presence or absence of thoracic, abdominal and cardiovascular injuries; cryoprecipitate dose per weight; and NOM content. The details of the predictors incorporated into each prediction model are presented in Table 2.

To assess the validity of factors included in the models, we conducted interaction tests and evaluated multicollinearity. Variance inflation factors were employed to detect multicollinearity among the predictors, with values exceeding 10 considered indicative of its presence. In cases where multicollinearity was identified, we removed one of the correlated variables from the model to maintain its integrity.

As the proportion of splenic pseudoaneurysm events in the database used for the analysis in this study was small (41 cases), it was difficult to create a validation set; therefore, the predictive models and their performance were evaluated using a resampling dataset with the bootstrap method (1000 iterations). The predictive performance of the models was assessed not only by the receiver operating characteristic curve (ROC) and the area under the ROC (AUROC), but also by the precision-recall curve (PRC) and the area under the precision-recall curve (AUPRC), as the incidence of the outcome was <10%. Calibration performance was evaluated using a calibration plot and the Brier score. A comparison of the predictive performance between the models was performed using the DeLong test, was with a significance level set at p < 0.05. Furthermore, the net benefit values of the models were calculated, and decision curves were plotted to assess their clinical usefulness using decision curve analysis (DCA).

All analyses were performed using the commercial software R (version 3.6.3; R Foundation for Statistical Computing, Vienna, Austria) and EZR (Saitama Medical Centre, Jichi Medical University, Saitama, Japan), a modified version of the R Commander designed to add statistical functions frequently used in biostatistics.

Study population and baseline characteristics

The SHIPPS study included 1480 cases from 83 facilities. After applying the exclusion criteria, 1046 cases were excluded, resulting in 434 cases of isolated splenic injury being analysed (Figure 1).

Among these, 41 (9.4%) were found to have pseudoaneurysm formation 24 h after admission. The pseudoaneurysm formation group had a higher median age (11.7 vs. 9.9 years, p = 0.004) and weight (41.8 vs. 34.8 kg, p = 0.005), the proportion of grade IV or higher injuries according to the AAST-OIS spleen 2018 grade (46.4% vs. 19.1%, p = 0.001), and Interventional Radiology (IVR) rate (65.8% vs. 26.2%, p < 0.001) compared to the non-formation group. There was no significant difference in blood transfusion amount or haemostatic agent use between the groups (Table 1).

Details of pseudoaneurysm formation were shown in Table S10.

Predictive models by logistic regression analysis

To estimate the risk of pseudoaneurysm formation after NOM for isolated blunt splenic injuries in children, five predictive models were developed (Table 2). The regression odds ratios for the predictors incorporated into each model are presented in Table S6. Akaike's Information Criterion (AIC), a measure of the predictive model's fit, was 164.4, 163.8, 162.3, 156.5 and 141.6 for Models 1–5, respectively, with Model 5 having the lowest AIC (Table S7).

Predictive performance of each predictive model

Figure 2 presents the ROC and AUROC values for the five logistic regression models. No significant differences were observed between the AUROC values of Models 3 and 5 (p = 0.095). However, significant differences were found in the comparisons among the other models, with Model 1 having the highest AUROC, followed by Model 2.

The results of the PRC, which are useful for evaluating the performance of models on unbalanced data, are shown in Figure 3. The AUPRC values of Models 1 and 5 were significantly higher than those of the other models, and the AUPRC of Model 1 was significantly higher than that of Model 5. However, all models exhibited low AUPRC values, with the precision decreasing rapidly as the recall increased. Model 1 showed the slowest decline compared to Models 2–5 and demonstrated the best discrimination performance among the five models when considering AUPRC values. Table S8 presents a comparison of AUROC and AUPRC for each logistic regression models.

Figure 4 illustrates the relationship between the predicted probabilities and actual observations. All five models demonstrated good calibration performance. The Brier score, which compares the predicted and actual probabilities, further confirmed the good calibration performance of all models (Table S9).

The DCA plotting the net benefit (true positive rate minus false positive rate) of the five prediction models against the threshold probability showed that at low threshold probabilities (below 20%), all models had higher net benefits, with no significant differences between them (Figure 5). However, in a certain range of threshold probabilities (20%–40%), Models 1 and 5 exhibited relatively higher net benefits than the others, suggesting that they were better predictive models.

Interpretation

To date, there have been no studies on predictive models similar to those developed in this study; therefore, we cannot directly compare the usefulness of our models with others. Hence, in this study, we created and compared multiple predictive models using standard logistic regression analysis as a foundation for the future development of predictive models for delayed pseudoaneurysm formation. The ROC and AUC values, which evaluated the discriminative ability of the five predictive models, were approximately 0.8, indicating a relatively good predictive ability for delayed pseudoaneurysm formation. However, in cases of imbalanced data, where the outcome occurrence rate is <10%, predictive models may overestimate their true predictive abilities.^{14, 15} In such cases, the discriminative ability of predictive models should be evaluated using a PRC.^{14, 15} The highest PRC among the models developed in this study was 0.358, indicating moderate discriminative ability; however, it was not sufficiently high. Developing predictive models with low outcome occurrence rates, as in this study, is challenging and requires careful performance evaluation using methods such as PRC. Similar to our study, other studies in the emergency and critical care fields with low outcome occurrence rates might have overestimated their predictive performance when evaluated with the ROC.¹⁶ While the performance of the predictive models created in this study was not excellent, this study is the first to report predictive models for pseudoaneurysm formation in children with blunt splenic injuries and provides foundational knowledge for future research. To further improve predictive performance, it is necessary to explore new factors, increase the number of cases and utilise machine learning methods to further improve the discriminative ability.

The predictors incorporated into the prediction model in this study, particularly the presence of contrast extravasation on initial contrast-enhanced CT and high-grade splenic injury according to the spleen 2018 grade, have been reported to be associated with delayed pseudoaneurysm formation in previous studies, thus supporting the clinical validity of our model. Katsura et al.¹⁷ reported that in children with blunt splenic and hepatic injuries, those with contrast extravasation had a significantly higher rate of delayed pseudoaneurysm formation than those without contrast extravasation on initial contrast-enhanced CT. Regarding the severity of splenic injury, Mauro Podda et al.⁷ demonstrated that according to the WSES, splenic injury classification and grade III or higher were associated with an increased risk of vascular-related complications, including delayed pseudoaneurysm. Although Model 5 in our study incorporated the AAST-OIS spleen 2018 classification grade as a predictor, the WSES and AAST-OIS spleen 2018 grade were closely related, and both identified high-grade injuries as grade III or higher. Therefore, our results were consistent with those reported by Mauro Podda et al., supporting the importance of injury severity as a predictor of delayed pseudoaneurysm formation. Our model may provide clinically valid predictions by incorporating these factors.

Implications

To date, follow-up strategies after NOM for delayed pseudoaneurysms have often been left to the discretion of the physician in charge, partly due to the lack of standardised guidelines and evidence. Although the predictive models for delayed pseudoaneurysm formation created in this study did not exhibit high predictive performance, they represent the first attempt to develop potentially clinically applicable models. As demonstrated in models predicting paediatric patients with pneumonia, which can reduce the frequency of chest X-ray imaging,¹⁸ the construction of models with good predictive performance could allow for the identification of patients at a high risk of delayed pseudoaneurysm formation. This would enable targeted imaging tests, such as contrast-enhanced CT, only for the appropriate patient group, potentially reducing radiation exposure and shortening the hospital stay, as well as the burden on patients and their families. Furthermore, it is expected to support efficient and evidence-based decision-making in clinical practice in Japan.

Limitations

This study had some limitations. First, it had a small sample size. This suggests that a robust model construction may not have been achieved during the developmental stage of the predictive model. Second, the definition of the outcome occurrence was ambiguous. This was a retrospective study, and due to the lack of a unified evaluation protocol, there may be cases where CT follow-up is not performed unless a pseudoaneurysm has formed and ruptured, which may have led to the misclassification of outcome occurrence. Additionally, the main study did not distinguish between pseudoaneurysm formation requiring treatment and that not requiring treatment at the time of data collection, we were unable to set pseudoaneurysm formation requiring treatment as an outcome in this study. Instead, we defined the formation of pseudoaneurysms themselves as the outcome, regardless of whether treatment was required, and the purpose of this study was to construct an initial model for predicting the formation of pseudoaneurysms requiring follow-up imaging in the follow-up strategy after NOM management for traumatic splenic injury. Third, there is an issue caused by the low outcome occurrence rate. Generally, evaluating the reliability of a predictive model for imbalanced data with a low outcome occurrence rate is challenging. Hence, evaluation using methods such as the PRC curve or SMOTE is recommended because the ROC curve may overestimate predictive performance.^{14, 15} Fourth, important factors may not have been incorporated into the predictive model, which in this study were based only on clinical and examination findings at the time of hospitalisation. However, there was a trade-off in the selection of predictors for the construction of predictive models because data up to a certain point had to be used. Finally, the logistic regression analysis may not have been appropriate for constructing predictive models in this study, because it struggles to capture the complex, nonlinear relationships often present in emergency medicine. The use of methods, such as machine learning, may overcome these limitations.