1.1 Research Problem

This research aims to prepare, test and evaluate the use of ML models to identify which factors may contribute to admission to an acute MH ward for those presenting to ED with MH as the primary presenting problem, particularly suicidality.

1.2 Background

1.3 Research Setting and Context

This paper presents research outcomes within a coastal regional local health service in New South Wales (NSW), Australia, located just north of Sydney. The health service encompasses two public hospitals with EDs, catering to a population of 346 596 residents across a land area spanning 1681 km² (Australian Bureau of Statistics 2021). The public MH service delivery programme includes acute inpatient and community-based care. These services are organised into three categories: child and youth MH (spanning ages 0–12 and 12–17), adult MH (ages 18–65) and older persons MH (ages 65+). Notably, the health service distinguishes itself from many other NSW EDs by providing specialised MH nurses and medical staff for assessments at the EDs around the clock, 7 days a week. The health service operates across two sites: Site A comprises 24 acute beds and six high-dependency MH beds, while Site B offers 25 acute beds, 10 high-dependency beds and 15 beds specifically designed for older persons (65+). Additionally, Site B houses a Psychiatric Emergency Care Centre (PECC) equipped with four beds and adjacent to the ED. Neighbouring health districts provide access to subacute, non-acute and rehabilitation beds for individuals requiring extended or specialised MH care. It is important to note that the service lacks dedicated MH beds for adolescents aged 12–17. Instead, referrals for adolescent inpatient services are directed to specialised inpatient units in neighbouring districts (Central Coast Local Health District 2017).

2.1 Design and Data Collection

The primary data collection involved reviewing existing data from 1 January 2016 to 31 December 2021. The ED data were selected for inclusion based on specific terms related to the presenting problem. A second data collection was performed to determine whether the individual presentation was associated with current or past care and is defined in the research as ‘MH history’. Ethical approval for the study was granted by the Hunter New England Human Research Ethics Committee, with reference number 2022/ETH01597, and the approval was given on 18 November 2022. Site-specific approval was also obtained from the Central Coast Local Health District on 14 December 2022. Before any analysis, the medical record numbers were encrypted, and subsequently, they were completely removed from the data set. The study examines the following fields within the data set pertinent to MH-related ED presentation which are presented in Table 1.

For the initial data investigation, we utilised several tools and programming languages. Structured query language was used for managing and querying databases, while Python, a versatile programming language popular for data analysis and ML tasks, played a central role in our analysis. Excel, a spreadsheet application commonly used for data organisation, analysis and visualisation, helped in structuring and reviewing the data. Additionally, Seaborn, a Python library designed for creating informative and attractive statistical graphics, was employed to visualise the data effectively. Further information on the data collection and extraction methods can be found in Higgins et al. (2024b). Using Python, the data were brought into a pandas data frame (McKinney 2010) and modelled with Interpretable Machine Learning (InterpretML) (Nori et al. 2019), discussed later in this document. A complete list of variables used is located in Table 1. The initial review of the model noted that the ‘site’ feature contributed significantly to admission (Figures 1 and 2). Therefore, the data were split, and separate models were developed for each hospital site.

2.2 Data Definitions

The MH context of the ED presentation is established through the use of the presenting problem. In the electronic medical record, the ‘presenting problem’ refers to a field that records the consumer's description of their symptoms or medical condition during triage at the ED (Bureau of Health Information 2022). Despite calls for standardisation of this input (Berendsen Russell, Dinh, and Bell 2017), the literature lacks clear definitions or explanations. However, previous works have set a precedent in employing the presenting problem field for specific purposes (Sara et al. 2022; Sperandei et al. 2022; Wand et al. 2015). For the purpose of this discussion, an admission is derived by the value of ‘Admitted to the ward/inpatient unit, not a Critical Care Ward’ in the ‘Mode of separation’ field of the dataset (NSW Health 2017). ‘MH history’ is defined as any involvement with a Central Coast Local Health District Mental Health Service before the date of presentation, within the time period from 1 January 2015 to 31 December 2021.

2.3 InterpretML

The prepared data set used in Higgins et al. (2024b) was further analysed using the InterpretML python package to determine the most significant factors contributing to admission. InterpretML is a framework for building interpretable models to help clinicians make informed decisions about consumer care. Traditional ML models, such as deep neural networks, can be very accurate but are often considered ‘black boxes’ because it can be challenging to understand how they arrive at their predictions (Rai 2019). This lack of transparency can be a significant obstacle in healthcare, where decisions can have adverse consequences, and consumers have the right to understand how and why decisions are made (Molnar 2020). One of the key features of InterpretML is the ability to visualise the model and its predictions using various techniques, such as feature importance plots, partial dependence plots and individual conditional expectation plots (Nori et al. 2019). These plots provide an understanding of the impact each feature contributes to the outcome and can identify areas where improvements can be made in consumer care (Molnar 2020). Using InterpretML with tools such as Log Odds Ratio, we can build transparent, accurate and effective models to improve consumer outcomes.

2.4 InterpretML Mean Absolute Score

InterpretML helps identify complex interactions, and the overall importance graph is one of the methods it provides for interpreting ML models (Nori et al. 2019). InterpretML calculates overall importance based on the mean absolute score (MAS), which measures the impact of each feature on the model's predictions (InterpretML Documentation 2023). To calculate the MAS, InterpretML performs a series of experiments where the value of each feature is randomly permuted across the data set. The MAS is then calculated for each feature by comparing the model's predictions before and after the feature has been permuted (InterpretML Documentation 2023). The larger the MAS for a given feature, the more important that feature is in the model's predictions. The overall importance graph in InterpretML visualises the MASs for all the features in the model, ranking them in order of importance (InterpretML Documentation 2023). The graph provides a way to quickly identify which features have the most significant impact on the model's predictions and can be a valuable tool for interpreting the model's behaviour. To better understand each element, InterpretML provides a visual interpretation of each element and its relationship to admission using log odds. Log odds is a way of representing the probability of an event as a logarithmic function of the odds of that event occurring (Tabachnick, Fidell, and Ullman 2013). Log odds ratio (OR) is commonly used to compare the odds of an event in two different groups, such as a treatment group and a control group. The Log OR is simply the natural logarithm of the odds ratio, this means that if the odds of an event are 4:1, the log odds would be (4/1) = 1.39 (Tabachnick, Fidell, and Ullman 2013). By incorporating Log OR into the model, we can compare the odds of the outcome between different groups, assess the impact of each predictor on the outcome and compare the probability of different outcomes on a single scale.

3.1 Age Category

The MAS presents that age is a significant contributing factor to an admission from an ED presentation. Figure 4 shows the overall Log OR score in the top half; a positive score implies an increase in the model's weight towards admission, and a negative value implies that they would not be admitted. The bar graph at the bottom of Figure 4 shows the density of the population samples in the data set, with a high negative correlation to admission for presentations under 18 and an increase towards a positive correlation at the age of 65. Note that the outliers are for the under 10's and over 85's groups. When comparing the two sites, there is a considerable difference in the age-related correlations. Site A (Figure 5A) shows a positive correlation in the 5–9.75 age that drops down to a negative before starting to increase in the late teens, flattening through the 20–30s, then slowly climbing as the age increases. Site B (Figure 5B) shows a very different profile. The 0–16.4 range shows a significant negative correlation towards admission, rising steadily through 18 years, where it starts to level off. At approximately 30, the correlation starts to move towards an increased likelihood of admission, with a clear and definite step occurring at 65 years. The ML model identified that increasing age significantly contributed to the likelihood of admission (Figure 1). However, given the lack of data relating to socioeconomic and other factors, age becomes broad and may represent a more generic characterisation of the consumer and their health (Caruana et al. 2015); an older person is more likely to have higher health needs and comorbidities that necessitate admission. However, age is a critical factor that may explain the model (Caruana et al. 2015). There are points in the age data that show a clear ‘step’ in the correlation relationship (Figure 4).

3.2 Triage Category

Triage is similarly weighted to age and is the only element deemed a clinical assessment. The overall figure shows a high and stepped correlation with admission, with a triage code of 1 showing a positive correlation down to 5 having a negative correlation, as would be expected, with both sites exhibiting very similar characteristics (Figures 6 and 7A,B).

3.3 Presenting Problem Category

Overall, ‘Altered Mental Status’ (0.3737), ‘Behavioural Disturbance’ (0.3065), ‘Eating Disorder’ (0.4245), ‘Hallucinations’ (0.3826) and ‘Memory Impairment’ (0.2667) all have a positive correlation to admission. ‘Anxiety’ (−0.5135), ‘Depression’ (−0.0613), ‘Self Harm’ (SH) (−0.0997) and ‘Suicidal Ideation’ (SI) (−0.0945) have a negative correlation (Figure 8). It should be noted that while SI and SH have a negative correlation, their overall influence on the model is minimal. Both sites present very similar correlations in the presenting problem data. Site B (Figure 9B) has a slightly higher positive correlation for ‘Hallucinations’ than Site A (Figure 9A). Interestingly, while both sites have a negative correlation for SI and SH for admission, the weight they contribute is inverse. SI has a stronger negative correlation at Site A than Site B and SH at Site A compared to Site B (Figure 9A,B).

3.4 First Nations Category

According to the model, a person's ‘Indigenous status’ plays a minor role in determining a person's likelihood of admission. However, it should be noted that the subcategory of ‘Neither Aboriginal nor Torres Strait Islander origin’, people that do not identify as First Nations, is by far the largest group and has a flat correlation profile for admission. The next significant subcategory is ‘Aboriginal but not Torres Strait Islander origin’, which has a minor correlation with admission. Both sites show very similar results; however, it is noted that there is a stronger negative correlation at Site A (Figure 10A) for those that identify as ‘Aboriginal but not Torres Strait Islander origin’ compared to Site B (Figure 10B).

3.5 Service User

There is a positive correlation between a presentation being an active service consumer and resulting in admission. In comparison, those that were not active at the time of presentation have a minimal negative correlation.

3.6 Marital Status

The ML model found a positive correlation with admission if the person stated they were ‘Divorced’, ‘Married (including de facto)’, ‘Separated’, ‘Unknown’ or ‘Widowed’. ‘Never married’ and ‘Declined to respond’ had a negative impact on admission. However, it should be noted that these data included people under the age of 16 and may not be a true reflection of the effect of ‘Never married’ or ‘Declined to respond’.

3.7 Month, Day of the Week and Time of the Day

The ML model found a minimal correlation towards the winter months; however, the volume of presentations is lower during this time. When comparing the admitted/not admitted groups, it can be observed that there is a consistent volume of data in presentations over 5 years that resulted in an admission. By contrast, there are significantly higher presentations to ED occurring in the summer months, with the number of people that present to ED changing over the year. However, the number of people admitted remains consistent despite these fluctuations (Figure 11). Even though the number of presentations fluctuates over the period in question, the admission rate remains flat. Overall, according to the ML model, the time of day that a person presents to the hospital has only a small impact. A slightly high positive correlation occurs between 5:00 AM and 8:00 AM, and a small negative correlation occurs between 10:00 PM and 3:00 AM.

3.8 Mode of Arrival

The ML model found a high correlation for ‘Air ambulance’, ‘Internal Ambulance’, ‘Internal Bed/Wheelchair’ and ‘Other’ for Mode of arrival. However, the incidence of these occurring in the data was very small. ‘State Ambulance’ had a small positive correlation, and ‘Private vehicle’ had a small negative correlation.

4.1 Triage

In NSW, ED presentations are rated using the NSW Health triage scale. People who are critically ill and require immediate treatment are classified as Code 1 (NSW Health 2013) and someone who requires less urgent care, such as minor illness or symptoms that have been present for more than 1 week, is classified as Code 5 (NSW Health 2013). The triage code data in this data set aligns with the NSW Health guidelines (NSW Health 2013). There is a positive correlation for admission with Code 1, progressing to a negative correlation for Code 5. Additionally, triage and ED access are local, and it is difficult to compare results in context, even in NSW, as models of triage, preparation and education of emergency nurses are highly variable, and auditing of triage practice occurs infrequently (Varndell, Hodge, and Fry 2019). Furthermore, the importance and responsibility for timely and accurate data entry by the triage nurse should be noted. Data integrity is a fundamental nursing skill that nurses are accustomed to collecting at the bedside (e.g., vital signs), but an enhanced appreciation of the ramifications of accurate data collection is paramount to understanding the importance and value of this data entry. The collection needs to be done with precision integrity due to clinical outcomes associated with the data and will allow for more accurate ML models to be built in the future.

The triage of people with MH problems requires high-level competencies in terms of knowledge, skills and attitudes in unpredictable and complex patient situations (Stigter-Outshoven et al. 2024). Within Australia, triage is an autonomous role predominantly conducted by specialist emergency nurses (Varndell, Hodge, and Fry 2019) and initiates the assessment process for a person who presents to the ED with an MH problem. Timely and accurate assessment, followed by appropriate referral and response, depends on the clinician's understanding of the MH triage and acuity (Broadbent, Moxham, and Dwyer 2020). The triage process is further impacted by conflicting orientations between the two nursing groups with different professional cultures (Broadbent, Moxham, and Dwyer 2020). ED triage nurses have expressed fear, frustration, lack of knowledge and confidence when dealing with people who present with MH problems (Perrone McIntosh 2021). Hall et al. (2016) demonstrated that extended MH professional development training is beneficial for ED triage nurses; however, the nurses in the study acknowledge that their role is primarily driven by time constraints and the volume of people that present to the ED. The resulting gap that emerges in this context is significant, MH triage is complex and requires an extensive understanding of MH (Broadbent, Moxham, and Dwyer 2020). It represents a significant challenge to ED models of care (MoC) that are challenged to support the volume of people seeking MH care, many of whom leave without treatment (Higgins et al. 2024a, 2024b). Given the complexity of the MH presentations, introducing the MH nurse earlier at the point of triage may benefit the ED triage nurse by reducing their workload and, more importantly, provide the person seeking MH care with a clinician who possesses the required MH knowledge and training to understand their needs. Furthermore, Australia's current comprehensive pre-registration nursing education curriculum is perceived as inadequate for preparing nurses for work in MH settings (Lakeman et al. 2024). Therefore, specialist MH nurses with post-graduate qualifications are best suited to supporting MH nursing triage. Addressing triage issues for MH in the ED MoC will require the development of an integrated MH triage MoC that would incorporate the required MH knowledge and alternative pathways to ED, and provide a clear path for the MH needs of the people who need care.

4.2 Access

The findings for age reporting are important in the context of service delivery and the subsequent differences between the two sites. Admission is ultimately a function of bed availability, and it stands to reason that if a particular bed type is available, it is likely to result in proximal admission. The density of the young people presenting is significantly higher than the older age groups. The lack of Child and Adolescent Mental Health Service (CAMHS) beds and the high density of under 18 presentations are likely combined to facilitate the negative correlation in admission (Figure 4). Therefore, there is a need to examine access to acute MH care for young people in the context of ED presentations. Recent research recommendations by Suicide Prevention Australia call for co-designed youth-specific environments to provide young people with medical assistance and compassionate care (Suicide Prevention Australia 2022). This care should be delivered in a way that gives clear processes, improves accessibility and is supported by specific suicide response staff, including youth peer workers (Suicide Prevention Australia 2022). While significant steps have been made with non-clinical spaces, there is still a significant need to integrate these new models into the current MoC and extend the services beyond the current limitations to provide 24/7 support (Shams 2023).

4.3 Young Peoples Access to Mental Health Care

In Australia in 2022, 77 deaths by suicide occurred for young people (aged 17 and below), with 83.1% occurring in the 15–17 age bracket (Australian Institute of Health and Welfare 2023a). This is an increasing concern, as rates of death by suicide among 15- to 17-year-olds ranged between a low of 3.2 per 100 000 population in 2004 and a high of 8.9 in 2021 (Australian Institute of Health and Welfare 2023a). Figure 4 shows a large volume of young people that present to the ED and a significant negative correlation to admission. The healthcare research setting in question does not have any dedicated CAMHS beds; however, State-based CAMHS beds exist in neighbouring local health districts.

However, even if a small number of beds were added, the volume of presentations would still exceed capacity and not address the depth of the need. For services without beds, there is a need to better understand the presentation cohort and provide services that align with the need. Borschmann et al. (2018) found that the bulk of admissions (78.2%) to a Melbourne CAMHS IPU were for suicidality, indicating the high need for young people's safety in times of being unwell. If there are no beds or even a low number of them, services should focus on suicidality and keeping young people safe as a whole of health response. Second, if units are built, the design needs to focus on the needs of young people and not simply be a satellite of general paediatric or adult MH beds. Environments need to be built considering young people's requirements and designed to promote physical security while supporting interactions between users and systems that promote relational security (Wilson, Hutton, and Foureur 2023). The users of the space should be able to rely on the built environment to ensure their physical security and psychological safety and be designed to accommodate a wide range of diversity and acuity (Wilson, Hutton, and Foureur 2023). The aesthetics of the space should align with the promotion of recovery in the context of person-centred and trauma-informed MoC (Wilson, Hutton, and Foureur 2023). The design principles should be conscious of the cultural safety of young First Nations people with cultural safety frameworks, such as those used in Australia (McGough et al. 2022), being incorporated to improve the access and quality of health care for First Nations people and to improve disparities in health care outcomes.

4.4 Presenting Problem

‘Presenting problem’ is the most comprehensive MH data available in this data set and is collected at the point of triage before an MH assessment has been completed. The presenting problem description is comprised of the following options for MH; Altered Mental Status, Anxious, Behavioural Disturbance, Depression, Eating Disorder, Hallucinations, Memory Impairment, Mental Health Problem, Self-Harm and Suicidal Ideation. The ML model rated this item third overall for importance overall and across both sites. ‘Altered MH status’, ‘Behavioural disturbance’ and ‘Hallucinations’ are reported to correlate positively with admission. ‘Eating disorder’ and ‘Memory impairment’ also have a positive correlation but a very low density in the data. These problems are symptoms associated with psychotic-related mental illness (World Health Organization (WHO) 2021). SI negatively correlates to admission, yet it represents a high volume of presentations, accounting for 38.19% (Higgins et al. 2024b).

Both sites have a higher than average (per 100 000) death-by-suicide rate (Site A 12.9 and Site B 16.2) compared to the NSW state 11.2 and 12.6 for Australia for the period of 2016–2020 (Australian Institute of Health and Welfare 2023b). SI at Site B has an almost negligible influence on admission according to the model, which may be indicative of the difficulty in assessing people with this problem. However, this is present with low admission tendencies. There are likely to be contributing factors and relationships in the SI data beyond the scope of this paper.