2.1 Study design and participants

The study included 51 patients diagnosed with bipolar disorder (F31 according to ICD-10 classification). Table 1 gathers the main sociodemographic and clinical characteristics of the study sample.

Inclusion criteria were: adults aged between 18 and 65 years, who gave their informed consent and agreed to use the smartphone daily during the study period. The study was conducted at the Institute of Psychiatry and Neurology in Warsaw (IPiN further on) and at the center specializing in clinical trials. Study participants were enrolled from both inpatient and outpatient settings starting in September 2017 and ending in December 2018. Exclusion criteria were: serious hearing problems, and speech disorders (e.g., dysarthria, aphasia, etc.), which in the researcher's opinion might hinder the patient's participation in the study. Figure S1 presents a diagram illustrating the flow of participants in the study. The study obtained the consent of the Bioethical Commission at the District Medical Chamber in Warsaw (agreement no. KB/1094/17).

The psychiatric assessments were carried out by researchers (physicians) during the personal visits every 3 months. All of them had a speciality in psychiatry and experience confirmed with references in managing patients with bipolar disorder. Physicians were recruited by an external Site Management Organization from two BD treatment centers: IPiN in Warsaw and the Medical University of Poznań. During each visit, the mental state was assessed using the 17-item version of Hamilton depression rating scale (HDRS) and Young Mania rating scale (YMRS). The patient's mood state was also assessed using fortnightly phone-based interactions with the researcher using the telephone questionnaire form (see Figure S2). If a change was suspected, the patient was invited for a personal visit and a clinical assessment was conducted. The BD state was classified into one of the four phases (bipolar depression, hypomania/mania, euthymia, and mixed state). The detailed characteristics of the study design are provided in our previous work.⁹ In total 196 psychiatric assessments were reported for the study sample that confirmed 145 mood state transitions between consecutive psychiatric interviews as summarized in Table 1. Out of 145 transitions, in 74 (51%) cases, the affective state changed from the previous one and 71 (49%) remained the same. The most frequent change was from euthymia to depression state (17 such cases) and from depression to euthymia (14 cases). Further characteristics of the intensity of depressive and manic symptoms for the study sample (n = 51) and its female and male subgroups are gathered in Table S5.

In this article, we focus on the analysis of speech features extracted from the voice recordings from smartphones. The study participants installed an application (BDmon app) that runs in the background and turns on whenever the participant makes or receives a phone call. Thus, data are collected in naturalistic settings during daily activities. Patients used their phones and the possible influences of different models on any data collected were jointly treated as background noise. The application collects three types of data: speech features from the patients' daily conversations, behavioral data concerning smartphone usage and self-assessment data of the mood state. The self-assessment and behavioral data results are presented elsewhere.⁹ The patients were instructed not to use speakerphones to prevent recording of their interlocutor utterances.

2.2 Acoustic features extracted from speech using smartphones

Patient's voice signal was processed in real time to extract its physical descriptors, which were transferred to a secure server. The cached data were permanently deleted from the smartphone after feature extraction. Therefore, the conversation recordings did not leave the patient's phone. The matching between patients and their numerical IDs was kept separately in different institutions to anonymize the speech parameters sent to the server. Due to the limitations of the phone battery and data transfer, the app recorded the speech parameters only for the first 5 min of the call. We used the camera microphone to collect data and extracted parameters for the speech signal divided into 20 ms frames (for each frame the signal was assumed stationary). We discarded the first 500 and last 500 frames as nonrepresentative (extraction of some parameters require the calculation of moving average over the first 100 frames), shortening each recording by 20 s in total. Hence, we considered only these calls, in which the duration of the patient's speaking time exceeded 30 s so that there was enough data to extract voice features. We used the camera microphone to collect data. The sound signal contained background noise recorded during pauses and conversational partner's speaking times. We performed an automatic noise removal using Otsu's method³² and selected only frames exceeding a loudness threshold. Finally, we used the filtered data to compute the arithmetic average of each parameter for each call, thus obtaining the features utilized in predictive modeling.

Table S1 gathers further descriptions of the acoustic parameters considered in this study.

2.3 Main hypotheses

Based on the literature and clinical experience, three main hypotheses regarding changes in voice parameters depending on the severity of affective symptoms were formulated at the beginning of this prospective study. We summarize them in Table 2.

The main contributions of this work include the assessment of the correlation of speech features with clinically assessed depressive and manic symptoms, as well as the development of the model to predict phase in BD patients using an identified subset of the most relevant acoustic parameters.

2.4 Statistical analyses

We applied linear mixed-effects regression models with two levels, following the works of Fauholt-Jepsen et al.¹³ and Dominiak et al.⁹ When assessing the severity of symptoms, either the HDRS or YMRS scores are used as the response vectors. Additionally, we consider specific items connected to voice, the total score of Question 8 on the HDRS-17 scale (retardation—slowness of thought, speech, and activity) and the total score of Question 6 on the YMRS scale (speech—rate and amount) as response vectors. The fixed effects consist of coefficients related to acoustic parameters extracted from phone calls (e.g., loudness) and a constant term. We build separate models for different acoustic parameters (l = 89), symptoms (HDRS sum, YMRS sum, HDRS Q8, and YMRS Q6), cutoff points (lower vs. higher), and sex (female vs. male). The random effects occur at the patient level (level two) and are patient-specific random intercepts. The age of patients was considered as a fixed effect. Model assumptions were checked with the residual analysis and visually using the quantile–quantile plots. All p-values presented in the linear mixed-effects regression models are calculated assuming the normality of error distributions and were adjusted for simple multiple-testing procedures according to Benjamini and Hochberg using “multtest” package. All analyses were done using the R programming language. Linear mixed-effects models were calculated using packages “lme” (for the restricted maximum likelihood estimation) and “lmerTest” (p-values and model diagnostics) available at the CRAN repository for R language.³⁹ The statistical threshold for significance for the adjusted p-values was 0.05.

2.5 Model development for BD prediction using acoustic features

The methodology presented in Section 2.4 aimed to explore the relations between acoustic features and psychiatric assessments. In this section, we focus on the BD prediction and the goal is to develop one model (regardless of sex) aiming to discriminate between euthymia and affective states. We formulated this BD phase prediction problem as a two-stage process. First, we developed two linear mixed-effects regression models adjusted for sex to solve the regression problems, in which the goal was to predict the intensity of symptoms on HDRS and YMRS scales, respectively, similarly to the work of Matthews et al.⁴⁰ Second, we postprocessed the predicted scores with the following standard cutoff points: for bipolar depression HDRS ≥8 and YMRS <6, for hypomania/mania: HDRS <8, YMRS ≥6, for the mixed state: HDRS ≥8, YMRS ≥6, and for euthymia: HDRS <8, YMRS <6 to assess the performance of both models. As secondary analysis, we applied higher cutoff thresholds previously used in the study of Faurholt-Jepsen et al.¹² in the classification of affective states: bipolar depression HDRS ≥13 and YMRS <13; mania: HDRS <13 YMRS ≥13; mixed state: YMRS ≥13 and for euthymia: HDRS <13, YMRS <13.

To obtain reference values of the predictive power, apart from the generalized mixed-effects regression (GLM) models, we tested out-of-the-box versions of the following well-established statistical and machine learning algorithms: random forest (RF), decision tree (rpart), and penalized linear regression models (ridge and lasso). Random forest was calculated using package “randomForest” (number of trees was set to 50), decision trees were calculated with the “rpart” package and penalized linear regression models (ridge and lasso) with Gaussian or Poisson link functions were calculated with the “glmnet” package available at the CRAN repository for R language.³⁹ These algorithms were included in the 10-fold cross validation, in which randomization was done on the level of individual phone calls.

The Spearman rank correlation coefficients were calculated to assess relationships between the explanatory variables (HDRS scores, YMRS scores) and the considered two subsets of acoustic features according to both the automatic feature selection using RFE algorithm and the expert selection.

We treat speech features obtained from 7 days before to 2 after the patient's visit as corresponding to the HDRS and YMRS values assigned during that psychiatric assessment. The choice of such a time window is based on our previous research⁹ and literature.^{12, 13, 30} The lack of symmetry is related to the fact that a visit can result in a modification of treatment.

Finally, experiments were run to evaluate the classifier's performance in the detection of the symptoms of BD phase. Phone calls were aggregated to patient-days. Accuracy and sensitivity were applied to evaluate the performance. Sensitivity is calculated as the ratio of true positives (e.g., 80% sensitivity = 80% of patients who have the affective state will test positive). Due to the low number of observations from the mixed state, especially for the higher cutoff points, the manic states and mixed states were treated collectively.

3.1 Correlation of voice parameters with depressive and manic symptoms

Changes in acoustic parameters by sex in relation to the intensity of symptoms of mania and bipolar depression are depicted in Figure 1. Table S2 presents detailed values of the respective coefficients and confidence intervals. Additionally, we analyzed in both sexes a correlation of all voice parameters studied with the most speech-specific items on the YMRS and HDRS scales, that is the 6th item of YMRS (the manner of speaking) and 8th item of the HDRS scale (the psychomotor retardation). The results of these analyses are depicted in Figure 2 and detailed in Table S3.

3.2 Prediction of BD phase using acoustic features

Table 3 shows the comparative analysis of the performance of the general GLM models for expert and RFE subsets of features. Additionally, the selected well known state-of-the-art classifiers (decision tree, random forest, lasso regression, and ridge regression) are included in comparison.

We measure performance with accuracy (the sum of true positives and true negatives divided by total) and sensitivity, that is, the fraction of correctly classified observations (true positives/total) computed separately for the considered affective states, namely euthymia, depression, and the mania/mixed state. As observed, the GLM either with an expert or RFE subset of variables (see Table 3) achieves the highest accuracy of 0.71 regardless of the cutoff points (higher or lower). The naive classifier that always assigns the most common class, i.e., euthymia, leads to 0.39 for the lower cutoff points and 0.61 for the higher cutoff points. For the lower cutoff points, the GLM with expert and RFE features achieve comparable sensitivities for predicting depression that amount to 0.61 and 0.62, respectively. Thus, we can conclude that 61%–62% of patients who have depressive state will be correctly classified by the developed model. At the same time, the sensitivities for predicting manic/mixed states amount to 0.52 and 0.56, respectively. For higher cutoff points, the sensitivities for predicting euthymia state are the highest and amount to 0.83 and 0.71 for the expert and RFE models, respectively. However, sensitivities for other affective states are much lower and range between 0.32 and 0.49 for GLM model. Interestingly, for the higher cutoff points, the decision tree model delivers sensitivity for manic/mixed state of 0.82 apart from the overall moderate accuracy (0.64). Further details illustrating the actual versus predicted intensity of depressive and manic symptoms are gathered in Figures S3 and S4. We conclude that the differences in performance metrics for both variants of the GLM models are relatively small. Table 4 shows the confusion matrix for the GLM model with the expert features.

The size of sample in affective states changes depending on the cutoff points. For the higher cutoff points, over half of the observations (368) are considered as coming from the euthymic state.

4.1 Correlation of voice parameters with depressive and manic symptoms

4.2 Prediction of BD phase using acoustic features

Our results indicate that voice data can be used to predict phase change in BD. As far as we know, this has not yet been demonstrated. In this study, better sensitivity was achieved for bipolar depression than for mania (61% vs. 51% for the lower cutoff points). The accuracy of the predictive generalized mixed-effect model ranges between 70.9% and 71.4%. Additionally, from a broad set of voice features, we identified a set of voice parameters that are effective in predicting BD phase. This set consists of prosodic, spectral and voice quality-related features. The same set of voice parameters can be used for males and females, noting that the direction in which these parameters change in bipolar depression and mania varies for individuals. This finding is an added value to the literature of this study.

Faurholt-Jepsen et al.^{12, 13} indicate that voice features enable a more accurate classification of manic or mixed states as compared to depressive states. In our study, better sensitivity is achieved for the prediction of depression regardless of the cutoff points. The best sensitivity was achieved for euthymia (83%) according to the higher cutoff points. However, such metrics need to be always interpreted with caution and taking into account the sample size and distribution of classes within it. Other studies on voice-based prediction were for major depression and PTSD⁴⁶ and major depression in a multilingual study.³⁶ A study by Place et al.⁴⁶ assessed symptoms of depression and PTSD using behavioral indicators including voice parameters (speaking rate, prosody, intonation, and voice quality). In that study, pitch-related parameters predicted depressed mood (cross-validated area under the curve AUC 0.74). Interestingly, Kiss and Vicsi³⁶ proved that predicting major depression based on speech analysis is relevant regardless of the language spoken by the patient.