The neural network basis of altered decision-making in patients with amyotrophic lateral sclerosis

Participants

Patients with ALS were recruited at the Department of Neurology at Nagoya University (March 2015 to December 2018). All patients fulfilled the El Escorial revised criteria for probable laboratory-supported or probable ALS.¹⁶ A board-certificated neurologist (W.H.) and trained speech therapist (O.R.) conducted semi-structured clinical interviews concerning the FTD criteria.^{17, 18} Two patients with ALS met the criteria for behavioral variant FTD.

We also evaluated cognitively normal participants who did not show any cognitive impairment (Addenbrooke's Cognitive Examination Revised [ACE-R] ≥ 89) or neurological and psychological diseases. Based on the Fazekas hyperintensity rating system, we could not find white matter abnormalities characterized by hyperintensities more severe than grade 2 in T2-weighted MR images.¹⁹

All participants in the present study had adequate vision (with or without eyeglasses) to perform the PRL task.

This study was conducted according to the Ethical Guidelines for Medical and Health Research Involving Human Subjects endorsed by the Japanese government and approved by the Ethical Review Committee of Nagoya University Graduate School of Medicine. We obtained written informed consent from patients with ALS and cognitively normal participants.

Cognitive assessments

To assess general cognitive function, the Mini-Mental State Examination (MMSE), ACE-R,^{20, 21} Frontal Assessment Battery (FAB), Stroop test, digit span (forward and backward), and word fluency (letter and semantic) were performed to assess detailed executive function in patients with ALS.

PRL task

The PRL task comprised 120 trials. In each trial, the participants were presented with two abstract line drawings on the left and right sides of the screen. The presentation side was randomized for each trial. The participants were asked to select one drawing by pressing a key within 1000 ms, and were consequently presented with a reward or loss. If the participant did not select a stimulus within the presentation time window, the message “Time-up” appeared and the next trial was initiated. During the first 60 trials, one drawing had an advantageous option; the reward/loss frequency ratio was 80:20. The other drawing had a disadvantageous option; the reward/loss frequency ratio was 20:80. After 60 trials, the contingencies were reversed without any instruction to the participants (Supplementary Data 1). Before starting the examination, we set up a practice task to see if the participants understood the task. After the examination, we asked the participants how they had made their choice and reconfirmed that they had understood the task. The paradigm used here was based on previous studies.²²

We excluded participants whose score during the acquisition phase was over 1.5 standard deviations below the mean score of cognitively normal participants because they might not have understood the task. We also excluded cognitively normal participants whose score of the reversal phase was over 1.5 standard deviations below the mean score to only select participants who performed the task successfully.

Learning models

We used Q-learning, win–stay, lose–shift (WSLS), and random choice models to classify choice behavior.⁸ When analyzing decision-making behavior through RL, the elimination of subjects who randomly select their choices is generally used.^{23, 24} In decision-making tasks, including the PRL task, it is well known that some participants make decisions based on the WSLS model instead of the RL model. The WSLS model is only sensitive to the outcome of the previous choice, and the mathematical formulas used for analysis are different.^{13, 25, 26} It is also necessary to exclude cases who adopt the WSLS model when analyzing decision-making behavior.

The Q-learning model is one of the most famous computational RL models. This model updates action values based on the Rescorla–Wagner model.^{13, 25} After the participant chooses a stimulus and feedback is presented, the estimated action values are updated as follows:

where is the estimated action value for stimulus s on trial t, and R(t) is the reward value of the choice on trial t. Given the estimated values are set, the probability of choosing stimulus 1 in the next trail is determined as the soft-max function below:

where a(t) represents the participant’s choice on trial t.

The probability of choosing stimulus 2 is calculated by the following equation:

We adopted the maximum-likelihood approach to fit the parameter set of alpha and beta to the participant’s choice behavior. The log-likelihood for the entire trial is as follows:

The optima function in R was employed to find the parameter set that produced the highest log-likelihood.

The WSLS model in the present study had two parameters. The first parameter represented the probability of staying with the same option on the next trial if a reward was provided:

The probability of switching to another option following a win trial was . The second parameter represented the probability of shifting to another option on the next trial if a reward was not provided:

The probability of staying with an option following a loss trial was .²⁶

In the random choice model, each choice of options was made randomly and in equal frequency.

To clarify which strategy the participant adopted, we compared the goodness-of-fit of the three models with the best-fit parameter set. We computed Akaike’s information criterion (AIC), as shown by the following equation²⁷:

where k is the number of free parameters (two for the standard Q-learning model, two for the WSLS model, and zero for the random choice model). We regarded the model with the smallest AIC value as the strategy the participant adopted, and targeted participants who employed the Q-learning model.

Logistic regression analysis

Logistic regression analysis of alpha and beta values showed the extent to which the participant’s choice behavior was characteristic to ALS compared with the control group. This estimated logistic regression equation was designated as “P_αβ” in the current study. By using receiver operating characteristic (ROC) analysis, we calculated the value of P_αβ that most efficiently discriminated between the ALS and control groups. Based on the value of P_αβ, patients with ALS were classified into two groups: patients with anomalous choice behavior and patients with normal choice behavior. We compared ALS patients who had anomalous choice behavior with those who had normal choice behavior, and assessed the clinical features in ALS patients with anomalous choice behavior.

Analysis of RS-fMRI

RS-fMRI was used to identify the underlying neural networks linked to specific choice behavior in patients with ALS. All MRI scans were performed using a Siemens Magnetom Verio (Siemens, Erlangen, Germany) 3.0-T scanner with a 32-channel head coil at the Brain and Mind Research Center. High-resolution T1-weighted images (T1-WI) were acquired using the following parameters: repetition time (TR) = 2.5 s, echo time (TE) = 2.48 ms, 192 sagittal slices with 1-mm thickness, field of view (FOV) = 256 mm, 256 × 256 matrix size, and an in-plane voxel resolution of 1 × 1 mm². Eight-minute closed eyes resting-state functional images were obtained with echo-planar imaging using the following parameters: TR = 2.5 s, TE = 30 ms, 39 transversal slices with a 0.5-mm interslice interval and 3-mm thickness, FOV = 192 mm, 64 × 64 matrix dimension, flip angle = 80°, and 198 volumes.

Functional MRI data were preprocessed using the default pipeline as the standard procedure implemented in Conn Functional Connectivity Toolbox version 18b,²⁸ using MATLAB. The first five volumes of each participant’s data were discarded to remove initial image inhomogeneity. Then, each participant’s images were realigned, unwarped to remove dynamic EPI distortions,²⁹ slice-time corrected, co-registered to the bias corrected T1-WI, segmented and normalized to Montreal Neurological Institute coordinates, resampled to an isotropic voxel resolution of 2 × 2 × 2 mm³, and smoothed using an 8-mm FWHM Gaussian filter. Moreover, the principal component-based noise-correction CompCor approach was applied to remove the BOLD signal noise associated with white matter and cerebrospinal fluid. Band-pass filtering was performed with a frequency window of 0.008–0.09 Hz. Artifact detection tools (ART)-based outlier detection and scrubbing were performed to eliminate the effects of head motion.

ICC analysis

In graph theory, “degree centrality” represents the number of direct connections between a given region and the rest of the brain. By measuring degree centrality, ICC analysis detects hub regions.¹⁴ We calculated the ICC-power (ICC-p) value that could be interpreted as a “weighted” degree centrality, a network measure representing the number of connections between a given voxel and the rest of the brain, obtained with the connectivity threshold set to 0. We performed correlation analysis between the ICC-p value and P_αβ using the CONN toolbox considering age, sex, and education history as nuisance covariates. Statistical significance was set at a height threshold of P < 0·005, FDR-corrected cluster-size threshold of P < 0·05, and one-sided negative contrast to reveal network dysfunction. We calculated the mean ICC-p value of the cluster with significantly different ICC-P values.

We subsequently performed seed-to-voxel analysis to identify the origin of the alterations in degree centrality. First, all regions with significantly different ICC-p values were extracted as the regions of interest for seeds. The average filtered BOLD signal in each seed was evaluated, and its bivariate correlation with the BOLD signal in all other voxels in the brain was computed. Then, we conducted correlation analysis on the calculated connectivity and P_αβ. The results were assessed with a cluster-forming height threshold of P < 0.005 and FDR-corrected cluster-size threshold of P < 0.05.

Statistical analyses

Clinical backgrounds and cognitive examinations were compared using the Mann–Whitney or chi-squared test. The threshold of statistical significance was set at P < 0.05. Statistical analyses were performed using the Statistical Package for the Social Sciences (SPSS) version 24 (SPSS, Inc., Chicago, IL, USA).

Specific choice behavior in patients with ALS

We performed the PRL task in 127 cognitively normal people and 90 patients with ALS. Clinical backgrounds and cognitive examinations are shown in Supplementary Data 2. Although there were no significant differences in the scores of the PRL tasks and ratio of strategies classified by AIC, we found some ALS patients demonstrated a unique choice behavior (Figure 1), that is, they rarely changed their choice, with few switches in the early stage of the acquisition and reversal phases. Few cognitively normal participants showed such a pattern. Thus, we selected cognitively normal participants with typical choice behavior as control subjects and compared them with ALS patients, to reveal the choice behavior features of ALS patients with unique choice behavior.

First, we excluded 19 cognitively normal participants who scored below 1.5 standard deviations from the mean score during the acquisition and/or reversal phase (score at acquisition phase < 28, score at reversal phase < 20) since they could not perform the PRL task appropriately. Second, we classified their strategies of choice behavior into Q-learning, WSLS, and random choice models, using AIC. Since the unique choice behavior of patients with ALS had the lowest AIC value of Q-learning among the three models, we selected 62 from 90 patients with ALS and 66 of 108 control participants who adopted RL (Q-learning model) and evaluated the parameters in the Q-learning model. Finally, we excluded three control participants who showed abnormal inverse temperature with scores over 3 standard deviations from the mean score (Supplementary Data 3).

Demographic and cognitive features of participants with the Q-learning model

Choice data from 62 patients with ALS and 63 control participants were best fitted using the Q-learning model. Clinical backgrounds and results of the PRL task are shown in Table 1. Although there was no significant difference in age, the ALS group exhibited significantly lower education and MMSE and ACE-R scores. In the PRL task, there were no significant differences in total score, score at acquisition phase, or score at reversal phase between the ALS and control participants. Although the alpha value tended to be lower in the ALS group, it was not significantly different from the control group. The mean PRL beta value was significantly higher in patients with ALS compared to controls.

Features of choice behavior in patients with ALS

Figure 1 shows a representative specific choice pattern, with low exploration and high exploitation, observed in patients with ALS, with high beta values, but relatively variable alpha values. However, ALS patients who had a mild to moderate increase in beta values (from 5 to 10) showed lower alpha values than controls. Besides, there was no significant correlation between alpha and beta values in patients with ALS. Thus, we applied logistic regression analysis to predict the probability of ALS (P_αβ) based on the two parameters concerning choice behavior. According to AIC, P_αβ yielded stronger explanatory power than those using alpha or beta values alone (AIC alpha = 174.69, AIC beta = 163.78, AIC alpha and beta = 162.12). A higher P_αβ well represented the unique choice pattern observed in patients with ALS (Figure 2). A P_αβ cutoff value of 0.512 provide optimal separation between ALS patients and controls in ROC analysis (area under the ROC curve, 0.666 [95% confidence interval 0.571–0.761]; sensitivity, 0.565; and specificity, 0.762). There were no significant correlations between P_αβ and age, education, disease type, disease duration, disease severity (ALS Functional Rating Scale Revised [ALSFRS-R]), and other scores of cognitive tests in patients with ALS (Supplementary Data 4).

Clinical characteristics and ALS-specific choice behavior

Using a P_αβ cutoff value of 0.512, 35 out of 62 ALS patients were classified into the ALS anomalous choice behavior group. In a comparison of ALS patients who had anomalous choice behavior with those who had typical choice behavior, there were no significant differences in clinical backgrounds and conventional cognitive examinations. In the PRL task, ALS patients with anomalous choice behavior exhibited significantly higher scores at the acquisition phase, lower alpha values, and higher beta values (Supplementary Data 5).

In the 35 patients with ALS who had anomalous choice behavior, the total score of the PRL task was significantly correlated with alpha, MMSE, and ACE-R. Furthermore, the score at the acquisition phase was significantly correlated with MMSE. The score at the reversal phase was significantly correlated with alpha and MMSE (Supplementary Data 6).

Functional connectivity changes associated with ALS-specific choice behavior

We performed RS-fMRI in 34 patients with ALS and 33 age- and sex-matched control participants. There were no significant differences in clinical backgrounds, PRL scores, parameters, and P_αβ of ALS patients who underwent MRI and those who did not undergo MRI. ALS patients who underwent MRI had higher ACE-R scores than those who did not (Supplementary Data 7).

ICC analysis demonstrated that P_αβ was associated with decreased degree centrality in the region of the anterior cingulate gyrus and frontal pole (Figure 3). Only the ALS group showed a significant correlation between the degree centrality Z-score in this region and P_αβ score (r = −0.712, p < 0.001; Figure 4). Seed-based analysis from the region of the anterior cingulate gyrus and frontal pole revealed that patients with ALS had decreased functional connectivity with the paracingulate gyrus, frontal medial cortex, anterior cingulate gyrus, frontal pole, subcallosal cortex, and superior frontal gyrus (Supplementary Data 8).

RL, Q-learning model, inverse temperature, and learning rate

The PRL task was designed to assess RL capacity, but as Worthy et al³⁰ demonstrated, participants often use heuristic-based models, including the WSLS model, during these kinds of tasks. In particular, when working memory load was high, participants were more likely to adopt WSLS strategies than RL.³¹ In the WSLS model, they were more likely to stay-with or shift-to options with higher expected values than options with lower expected values (i.e., RL). The combined WSLS-RL dual-process model may provide a superior fit to the data relative to the WSLS model alone.³² When different learning parameters could be associated with positive and negative prediction errors^{33, 34} and participants had a high learning rate and high inverse temperature, decision-making based on Q-learning was close to that based on the WSLS model.

However, individual choices in the Q-learning model are not made in isolation, but are embedded in a series of past experiences, decisions, and outcomes. Conversely, the WSLS model is only sensitive to the outcome of the previous choice. The mathematical formulas used for analysis are also different.^{13, 25, 26} We also thought the WSLS model was a different strategy to RL. Random choice behavior must also be treated as a different strategy from the RL strategy because random choice behavior indicates the failure of learning. Since the WSLS model and random choice are different strategies to the RL model, and the Q-learning model should be applied to the participants who adopted RL, we used AIC to clarify which strategy (i.e., the RL, WSLS, and random choice models) was the most appropriate model and excluded participants classified as WSLS and random choice model. There were no significant differences between the proportion of WSLS to random choice between patients with ALS and control participants. We found that the unique choice behavior of ALS patients could be classified using the Q-learning model. These findings supported the idea that the choice model is determined independently of disease state.

The Q-learning model is the most commonly used RL algorithm for model-free analysis of choice behavior. In the PRL task, the Q-learning model provides two crucial parameters, that is, inverse temperature (beta) and learning rate (alpha). The beta value determines the randomness of choice behaviors. Patients with ALS had high beta values, indicating that they hardly shifted their choice to another alternative, especially in a good situation.^{12, 13} A significantly higher beta value could indicate that they were likely to select the higher value option, which might reflect stereotypic and compulsive behavior.

The alpha value has a complex relationship with performance in RL tasks; different learning environments afford distinct optimal learning rates. However, a low alpha value indicates difficulty in updating old values despite newly acquired information, and subsequently no shift in choice. In ALS patients with the unique choice pattern, the alpha value was significantly correlated with the total score and score at the reversal phase (Supplementary Data 6). This could be attributed to optimization of their internal model, indicating more flexible, goal-directed actions due to representation of the contingencies of the task and less accelerated exploration, in addition to lower attention to the alternative choice after the situation changed.^{35, 36}

P_αβ indicates anomalous decision-making behavior in patients with ALS

The balance between “exploration” for searching for a better choice and “exploitation” for obtaining a large reward is said to be one of the major issues in RL. ALS patients did not change their choice as frequently as controls (Figure 1), which means the change of the balance of exploration vs. exploitation trade-off. ALS patients barely chose the option that they thought was disadvantageous, although they often changed their choice when they thought the value of the two alternatives had become equal. Their choice behavior seemed to be strictly guided by action values that are estimated through RL (such as Q-learning). This anomalous choice behavior is the key point in the change in choice behavior in patients with ALS.

In the current study, we used P_αβ, which was a better measure of anomalous PRL behavior in patients with ALS. The use of only alpha or beta values could not completely explain the uniqueness of choice behavior in patients with ALS. AIC showed P_αβ had stronger explanatory power than alpha or beta values alone. Interestingly, ICC analysis showed that P_αβ was significantly correlated with decreased degree centrality in the anterior cingulate gyrus and frontal pole in patients with ALS. This region was reportedly associated with the decision-making process based on perceptual cues and reward values.³⁷ Furthermore, patients with ALS frequently show pathological changes in these regions.^{3, 38} These findings support the idea that P_αβ, induced by both alpha and beta values, can represent the anomalous decision-making behavior related to the involvement of the prefrontal cortex in patients with ALS.

Network alterations and altered decision-making in patients with ALS

Seed-based analysis from the anterior cingulate gyrus and frontal pole revealed changes in functional connectivity to regions that consist of the paracingulate gyrus, frontal medial cortex, anterior cingulate gyrus, frontal pole, subcallosal cortex, and superior frontal gyrus. The medial prefrontal cortex monitors whether choice behavior is reliable and enforces the switch from exploitation to exploration.^{37, 39} The functional connectivity of the frontal polar cortex alters when someone decides to switch to the alternative behavior, since the frontal pole plays a role in monitoring the alternative course of action.^{37, 40} The reorganization of these networks might lead to a change in the balance between exploitation and exploration, or influence the switch to the alternative behavior, subsequently producing the anomalous choice behavior in patients with ALS.

Influence of motor dysfunction in patients with ALS on the PRL task

Upper and lower motor neurons are affected in patients with ALS, which could influence the selections made by pressing a key within a short period of time. Motor deficits might make ALS patients have a stronger choice perseverance tendency than control participants. However, we carefully assessed whether ALS patients performed the PRL task appropriately. We postulated that motor deficits could not significantly influence the uniqueness of choice behavior in ALS patients. There were no significant differences in the scores and strategies of the PRL task between ALS and control participants. Furthermore, the anomalous choice behavior linked with a high P_αβ required the selection of the advantageous stimulus, which was randomly presented on either the left or right side in each trial. Moreover, there was no significant correlation between P_αβ and disease duration, type, and severity (ALSFRS-R). Thus, changes in choice behavior were not mainly due to motor dysfunction.