Individual Prediction of Remission Based on Clinical Features Following Electroconvulsive Therapy: A Machine Learning Approach

J Clin Psychiatry 2022;83(5):21m14293

To cite: Nakajima K, Takamiya A, Uchida T, et al. Individual prediction of remission based on clinical features following electroconvulsive therapy: a machine learning approach. J Clin Psychiatry. 2022;83(5):21m14293.
To share: https://doi.org/10.4088/JCP.21m14293

© 2022 Physicians Postgraduate Press, Inc.

^aDepartment of Neuropsychiatry, Keio University School of Medicine, Tokyo, Japan
^bMelbourne Neuropsychiatry Centre, Department of Psychiatry, University of Melbourne, Melbourne, Australia
‡These authors contributed equally to this work.
*Corresponding author: Jinichi Hirano MD, PhD, Department of Neuropsychiatry, Keio University School of Medicine, 35 Shinanomachi, Shinjuku-ku, Tokyo 160-8582, Japan ([email protected]).

Major depressive disorder is a highly prevalent and debilitating psychiatric disorder that is a leading cause of disability worldwide.¹ Globally, approximately 350 million individuals have depression. Despite extensive trials of psychotropic medications, up to 30% of patients with depression respond poorly to these treatments.² These patients are classified as patients with difficult-to-treat depression³ and are considered candidates for electroconvulsive therapy (ECT).

Although ECT is the most effective and safe treatment for severe depression,^4–6 it requires frequent general anesthesia and is associated with often transient cognitive side effects.^7–9 Further, ECT is associated with a degree of stigma, fear, and aversion in some individuals.¹⁰ The mechanisms underlying the actions of ECT remain elusive,^11,12 but recent evidence suggests the importance of neuroplastic changes induced by ECT.^13–16

In previous studies, clinical factors such as the presence of psychotic symptoms, older age,¹⁷ and shorter episode duration¹⁸ were associated with a superior response to ECT. However, association and predictions are different,^19,20 and there is currently no established model to predict outcomes following ECT in individuals with depression. Thus, predicting individual therapeutic outcomes would be clinically useful to encourage patients and their relatives to accept ECT as a treatment option.

Machine learning approaches have received growing interest in the field of psychiatry. Machine learning enables the prediction of treatment outcomes for each individual patient. For example, a previous study²¹ predicted suicidality in patients with approximately 90% accuracy using a random forest algorithm. Several machine learning models have been developed to predict the therapeutic effects of ECT. Previous prediction models using neuroimaging data^22–29 have provided preliminary but promising results. Nevertheless, widespread implementation of these models in real-world clinical settings for patients with severe depression who require ECT has been challenging.

The aims of the present study were (1) to build a clinically useful prediction model for ECT remission based solely on clinical information that can be applied in real-world clinical settings as a cost-effective and easy-to-use prediction model and (2) to identify influential features in the model.

METHODS

Participants

This retrospective chart review study involving human participants was in accordance with the ethical standards of the institutional and national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. The ethics committee of the Keio University School of Medicine approved the study. The database included patients who received ECT to treat their depressive episodes at Keio University Hospital, Tokyo, Japan. In this study, we used data registered between April 2012 and March 2019. Data from patients with a DSM-IV-TR diagnosis of major depressive disorder or bipolar disorder were included. For cases in which the same patient received several ECT courses during the study period, the first ECT course was selected for analysis.

Electroconvulsive Therapy

Patients were treated with bitemporal ECT using a half-age method with a brief-pulse (0.5 ms) square-wave ECT device (Thymatron System IV devices; Somatics, Inc; Lake Bliff, Illinois). ECT was performed 2–3 times a week until a stable response was obtained. The number of ECT sessions was determined clinically by each attending psychiatrist. General anesthesia was induced by intravenous administration of sodium thiopental (3–5 mg/kg), propofol (1 mg/kg), or sevoflurane. Succinylcholine (0.5–1.0 mg/kg) was used to induce muscle relaxation. A 2-channel electroencephalograph (EEG) was monitored to ensure adequate seizure duration. The patients were restimulated at a higher intensity (ie, a 50% increase) when the seizure duration was less than 20 s.

Clinical Measurements

Baseline severity and ECT outcomes were evaluated retrospectively using the 7-point Clinical Global Impressions–Severity of Illness scale (CGI-S) and 4-point clinical note CGI-Improvement scale (c-CGI)^30,31 independently by 3 board-certified psychiatrists (A.T., T.U., and J.H.). The c-CGI was scored as follows:

1. Excellent: The patient chart demonstrated dramatic benefit from ECT. Examples of this level of response include rapid discharge after treatment, reduction in need for medications, resolution of target symptoms, and statements such as “dramatic response” or “greatly improved.”
2. Good: The patient chart indicated that the patient responded well. Examples of this level of response include referral for maintenance ECT, significant reduction in severity of target symptoms, and statements such as “responded well” or “good response.”
3. Moderate: The patient chart indicated that the patient had some amount of benefit. Examples of this level of response include slight to moderate reduction in severity of target symptoms and statements such as “improved somewhat” or “partial response.”
4. Poor: The patient chart indicated that the patient had minimal to no benefit. Examples of this level of response include treatment stoppage after 1 to 2 sessions due to adverse effects and documentation of no benefit to the patient with statements such as “no symptom changes” or “no improvement noted.”

Disagreements were resolved through discussion. Remission was defined as a score of 1 on the c-CGI.

We collected clinical demographics and information including age, sex, past medical history, onset age, duration of the current episode, weight, height, body mass index (BMI), family history of psychiatric illness, concurrent psychotropic medications, and ECT information (ie, number of ECT sessions, mean EEG seizure time, and mean postictal suppression index).

The defined daily dose (DDD) of each psychotropic agent (antipsychotics, antidepressants, and benzodiazepine) was calculated. DDD is defined by the World Health Organization (WHO) as the estimated average maintenance dose per day for a drug used for its main indication in adults (https://www.whocc.no/atc_ddd_index/). Each drug is assigned an ATC code based on the Anatomical Therapeutic Chemical (ATC) classification system with drug nomenclature defined by the WHO. Psychotropic medications were classified into 3 groups: antipsychotics (ATC code: N05A), antidepressants (ATC code: N06A), and benzodiazepine derivatives (ATC code: N003AE, N05B, N05C). When patients received 2 or more psychotropic medications in each category, the summed DDD was calculated.

Statistical Analysis

SciPy (https://www.scipy.org), which was supported by Python version 3.4., was used for all statistical analyses. Demographic and clinical variables were compared using a 2-tailed t test, Mann-Whitney U test, or χ² test. Statistical significance was defined by a 2-tailed P value < .05.

We built a prediction model for clinical outcomes following ECT (remitted or not remitted: binary values) as follows. We used a light gradient boosting machine (LightGBM), a gradient boosting framework using a tree-based learning algorithm, as the classifier.³² LightGBM grows leaf-wise trees, transforms continuous variables into a histogram, and uses the bin of this histogram to grow a decision tree. A random seed is fixed to generate the classifier. In this study, we conducted a 5-fold cross-validation. For a 5-fold training/test split, the model was fit to the training data, and the predictive value was assessed using the test data over all splits (5 times). For optimal accuracy, a hyperparameter was tuned using LightGBMTunerCV, which is a software program optimized for hyperparameter tuning using cross-validation and is a function available in Optuna (https://optuna.org). Sensitivity, specificity, positive predictive value, and negative predictive value were calculated to determine the overall results. Statistical significance was empirically estimated by performing permutation tests (1,000 iterations).³³

The SHapley Additive exPlanations (SHAP) value was introduced to improve the interpretability of the machine learning model.³⁴ SHAP is a game-theoretical approach for explaining the output of any machine learning model. A large absolute SHAP value exerts a strong influence on prediction. In this study, clinical features with positive and negative SHAP values were associated with remission and non-remission, respectively.

RESULTS

Clinical Characteristics

Data were collected from 177 patients. After ECT, 112 patients (63%) met the remission criteria. Clinical characteristics are summarized in Table 1.

Model Performance

The accuracy of prediction for remission after ECT with LightGBM was 71.2% (sensitivity, 85.7%; specificity, 46.2%; positive predictive value, 73.3%; negative predictive value, 65.2%). The outcomes of our model are presented in Table 2. The accuracy of our prediction model was significantly higher than the baseline remission rate (63%) (P = .02).

Feature Importance

Figure 1 shows the mean SHAP value in 5-fold cross-validation. The influential features for our prediction model were duration of the current episode, baseline severity (CGI-S score), antidepressant medication dose before ECT, refractory to antidepressant medications before ECT, number of prescribed psychotropics before ECT, disease duration, and BMI.

Figure 2 shows the SHAP values for 3 features with the top-ranked contribution to remission after ECT. In our prediction model, shorter duration of the present episode, lower baseline severity, and higher dose of antidepressant medications before ECT were associated with remission. Other important features included shorter duration of disease, smaller number of prescribed psychotropic medications before ECT, non-refractory to antidepressant medications, and lower BMI.

DISCUSSION

This study is the first to build a machine learning–based model to predict individual ECT responses based solely on clinical information and to identify influential features predictive of remission. Our prediction model demonstrated 71.2% accuracy in terms of therapeutic effects of ECT. Notably, the model includes only clinical information, which is easily obtained in routine clinical settings, thereby enabling its application in real-world practice.

Machine learning is an approach that makes prediction models in a data-driven manner and provides a new understanding that cannot be captured by the statistical approach. Specifically, a nonlinear model such as LightGBM used in this study enables finding of important features such as BMI that were not well known in previous studies that used statistical methods. The new perspectives gained on the therapeutic effects of ECT by machine learning will subsequently contribute to the understanding of the mechanism of ECT. In addition, statistical models often indicate the existence of risk factors, but cannot predict the individual result (ie, treatment response). On the other hand, a machine learning model that is validated for precision can predict the individual result, which would be truly useful in real-world clinical practice where precision medicine is required.

Previous prediction models for ECT outcomes have been based predominantly on a linear support vector algorithm.^{22–29,35,36} In contrast, our prediction model is the first to use LightGBM,³² a gradient-boosting algorithm that creates decision trees in a fast and lightweight manner. Given that the interpretability of machine learning has emerged as a critical factor in the medical field,³⁷ we extracted SHAP values to identify clinical features that were important for outcome prediction. Our model identified several important clinical features (eg, episode duration) that were similar to those in previous reports based on statistical analyses, whereas other features (eg, BMI) have not been previously identified. Accordingly, our data may provide novel insight into the clinical relevance of these features. In addition, the majority of previous prediction models for ECT outcomes were validated using leave-one-out cross-validation (LOOCV). However, LOOCV may cause overfitting, which can hamper model generalizability.³⁸ To overcome this issue, we adopted a 5-fold cross-validation method.

In our model, the most important feature for predicting ECT responses was a shorter duration of the current episode (Figure 1), which is consistent with a previous meta-analysis.¹⁸ The relationship between the duration of the current episode and clinical response appears non-linear, and there may be a cutoff point or threshold at the duration of approximately 12 months (Figure 2). Our results suggest that trials of ineffective treatments longer than a year may reduce the chance of remission following ECT. Of note, the duration of the current episode has been associated with clinical outcomes of other antidepressant treatments, including antidepressant medications³⁹ and repetitive transcranial magnetic stimulation (rTMS).^40,41 This evidence collectively suggests that a longer duration of the current episode may be associated with poorer clinical outcomes regardless of the treatment modality. In this regard, stratification of patients with depression may be critical to avoid ineffective treatments and to match ideal treatments for individual patients earlier in the treatment course rather than employing a trial-and-error approach.

Our prediction model identified both a shorter duration of the current episode and a shorter duration of disease as important features for predicting remission. Previous studies^39,42,43 have reported that individuals with depression with an early age at onset (ie, longer duration of disease) present with greater treatment resistance to antidepressant medications. MacQueen et al⁴⁴ reported that a longer depressive episode duration was associated with a reduction in hippocampal volume. Although speculative, a long disease duration may lead to reversible or irreversible structural changes in the brain that may contribute to poorer clinical outcomes.

The second most important predictive feature was the severity of depressive symptoms. A meta-analysis¹⁷ reported that remission following ECT demonstrated a trend to be less likely in patients with higher depression severity scores, although the effect was not statistically significant, whereas patients with more severe depressive symptoms were more likely to respond to ECT. Previous studies have also demonstrated that depression severity may influence therapeutic outcomes for other treatments. Fitzgerald et al⁴¹ reported that patients with less severe depressive episodes exhibited better responses to rTMS. In addition, Grammer et al⁴⁵ reported that remission rates were higher in patients with mild or moderate depression than in patients with severe depression who received rTMS. In a previous meta-analysis,⁴⁶ antidepressants were effective only in patients with severe depression. In contrast, recent research⁴⁷ has suggested that selective serotonin reuptake inhibitors are as effective in patients with non-severe depression as in those with severe depression. The relationship between the severity of depression and treatment outcomes may depend on the definition of outcomes (eg, response or remission) used in each study. Patients with mild depression may be more likely to meet the remission criteria (eg, a score < 7 on the Hamilton Depression Rating Scale [HDRS]), whereas patients with severe depression may be more likely to meet the response criteria (eg, 50% reduction in clinical scores from baseline) regardless of treatment modality.

The current study identified a relationship between higher doses of antidepressant medications and therapeutic outcomes of ECT. Sackeim et al⁴⁸ reported that concomitant antidepressant treatments during the ECT course enhanced short-term efficacy compared to placebo, highlighting the potential synergistic effects of ECT and antidepressant medications. On the other hand, our results revealed that a large number of baseline psychotropic medications and medication refractoriness before ECT were associated with non-remission. A previous meta-analysis¹⁸ demonstrated that medication failure (ie, nonresponse to at least one adequate antidepressant medication trial during the current episode) was associated with a poorer response to ECT. A major indication for ECT in real-world clinical settings is the failure to respond to several medications.⁶ Collectively, these results suggest that rather than trialing several types of ineffective psychotropic medications for patients with difficult-to-treat depression, clinicians should recommend ECT for patients and families earlier in the course of treatment.

The current study identified lower BMI and related clinical features (eg, weight and total amount of diet) as important features for predicting clinical remission after ECT. BMI had not been identified as a clinical feature predictive of ECT responses until recently. A recent mega-analysis utilizing a machine learning approach²⁸ reported BMI as an important predictor of ECT responses. There are also studies suggesting a neurobiological relationship between BMI and depression. A recent mega-analysis⁴⁹ reported that increased BMI modulated increases in subcortical gray matter volume following ECT: higher BMI was associated with a lower increase in subcortical gray matter volume. Although BMI was not directly correlated with clinical outcome following ECT, BMI significantly moderated the association between change in subcortical gray matter volume and clinical symptom improvement (ie, gray matter volume increase in the thalamus following ECT was associated with clinical improvement in patients with BMI within normal limits, but not in those with higher BMI). In addition to ECT, a recent study by Xiao et al⁵⁰ reported that patients with depression with lower BMI exhibited higher remission rates with antidepressants compared to overweight patients, and changes in HDRS₁₇ total scores decreased with increasing BMI. A systematic review also reported that a high BMI was associated with poor outcomes after antidepressant treatment.⁵¹ Similarly, increased body weight was associated with poorer responses in the acute phase of selective serotonin reuptake inhibitor treatment.⁵² These results suggest that an atypical subtype of depression, which is associated with a higher BMI, may be associated with poorer clinical outcomes. In contrast, Dreimüller et al⁵³ reported that overweight patients demonstrated the best response to antidepressant treatment. Although these findings suggest that BMI is associated with clinical outcomes in patients with depression, the nature of the relationship remains unclear.

In our study, age did not contribute much to the prediction of remission, which was inconsistent with findings of a previous meta-analysis,¹⁷ but consistent with those of a recent large retrospective study.⁵⁴ Additionally, recent studies^55,56 have shown that the relationship between age and ECT outcome was mediated by clinical symptoms, such as psychotic and psychomotor symptoms. The relationship between age and clinical outcome should be further investigated in future studies.

Some limitations should be acknowledged. First, the data were collected retrospectively, and the outcome (ie, remission) was evaluated retrospectively using the c-CGI. However, we minimized potential bias by recruiting 3 board-certified psychiatrists to evaluate clinical outcomes. A prospective study using gradient rating scales, such as HDRS or Montgomery-Asberg Depression Rating Scale (MADRS), should be conducted in future. Second, the number of samples was modest for machine learning application but was comparable to those in previous studies in the field of psychiatry. Indeed, many machine learning studies on mood disorders have employed fewer than 100 samples.³⁶ Future studies that include larger datasets from multiple sites should be conducted to validate our findings. Third, the results regarding the dose of the antidepressants should be interpreted with caution. In this study, we mixed patients with major depressive disorder and bipolar disorder for analysis, but these diseases have different strategies of pharmacotherapy.

In conclusion, our machine learning model predicted ECT remission with an accuracy of 71.2%. Given that the model included only clinical information, application in real-world clinical settings is highly feasible. Based on the set of features identified in the current study, introducing ECT earlier in the treatment course may lead to improvements in clinical outcomes.

Submitted: October 17, 2021; accepted April 4, 2022.
Published online: August 24, 2022
Author contributions: Drs Nakajima, Takamiya, and Hirano were critically involved in the data analysis and wrote the first draft of the manuscript. Drs Uchida, Kudo, Minami, Yamamoto, and Nishida contributed to the interpretation of the data. Drs Mimura and Yamagata supervised the entire project and were critically involved in the design and interpretation of the data. All authors contributed to and approved the final manuscript.
Relevant financial relationships: The authors declare no conflicts of interest.
Funding/support: This study was supported by the Japan Agency for Medical Research and Development (AMED) under Grant Number JP21dm0307102h0003, by the Japan Society for the Promotion of Science (JSPS) KAKENHI under Grant Numbers 17K10315 and 21K07551, and by the Daiwa Securities Health Foundation.
Role of the sponsor: The supporters had no role in the design, analysis, interpretation, or publication of this study.
Acknowledgments: The authors thank A. Hamada, MD; M. Inaoka, MD; M. Isobe, MD; S. Katayama, MD; Y. Kikuchi, MD; S. Kurose, MD; N. Nitta, MD; H. Oi, MD; M. Sakuma, MD; K. Sawada, MD; Y. Shimomura, MD; A. Shiomi, MD; S. Takasu, MD; M. Wada, MD; N. Waki, MD; R. Watanabe, MD; M. Kimura, MD; and T. Yatomi, MD, for data collection. All are from the Department of Neuropsychiatry, Keio University School of Medicine, Tokyo, Japan, and have no conflicts of interest to declare.

CLINICAL POINTS

• There is currently no established model to predict therapeutic effects of electroconvulsive therapy (ECT) on depressed patients based solely on clinical information.
• The current study provides a predictive model for the efficacy of ECT with 72% accuracy, using clinical information collected in routine clinical practices.
• Duration of the current episodes and baseline severity were identified as important features in predicting remission after ECT.