Predictors of 6- and 12-Month Relapse After Stopping Electroconvulsive Therapy: Critical Considerations, Including Overfitting in Regression and Confounding in Follow-up Studies

J Clin Psychiatry 2021;82(4):21f14174

To cite: Andrade C. Predictors of 6- and 12-month relapse after stopping electroconvulsive therapy: critical considerations, including overfitting in regression and confounding in follow-up studies. J Clin Psychiatry. 2021;82(4):21f14174.
To share: https://doi.org/10.4088/JCP.21f14174

© Copyright 2021 Physicians Postgraduate Press, Inc.

See letter by Lambrichts et al

Patients with major mental illness need to continue their antidepressant, antipsychotic, and/or mood stabilizer and other medications, usually for years, and often for a lifetime. This is because discontinuation of effective pharmacotherapy exposes them to the risk of relapse. This is true for electroconvulsive therapy (ECT), as well, which is usually discontinued, not continued, after a successful course of treatment. In a systematic review and meta-analysis¹ of modern era studies in which patients received continuation pharmacotherapy after a successful course of ECT, the cumulative relapse rates after stopping ECT were 27.1% at 3 months (11 studies; 95% confidence interval [CI], 0.21%–0.35%), 37.7% at 6 months (17 studies; 95% CI, 30.7%–45.2%), and 51.1% at 12 months (8 studies; 95% CI, 44.7%–57.4%).

One of the reasons why the post-ECT relapse rates are so high despite continuation pharmacotherapy is that ECT is most commonly prescribed to patients who did not respond well to pharmacotherapy trials; so, if medications did not work well before ECT, there is no compelling reason to believe that they will work well after ECT. Keeping patients well after a successful course of ECT is therefore a challenge. Adding lithium is a possible pharmacologic strategy in such situations; for example, in a randomized controlled trial (RCT)² of 84 depressed patients who had remitted with ECT, the 24-week relapse rate with the nortriptyline-lithium combination was 39%; this was significantly lower than the relapse rates with nortriptyline (60%) or placebo (84%) monotherapy.

A general expectation in psychopharmacology is that whatever worked to elicit response and remission from illness should work to help the patient remain well. This is why effective pharmacotherapy for an episode of illness is continued as continuation and maintenance pharmacotherapy. Continuation and maintenance ECT (hereafter referred to as maintenance ECT [M-ECT], for convenience) is based on the same principle. M-ECT is effective in improving outcomes after a successful course of ECT. For example, in a systematic review and meta-analysis, Elias et al³ found that fewer depressed patients relapsed by 6 months if they had received M-ECT combined with pharmacotherapy than if they had received pharmacotherapy alone (4 RCTs; risk ratio [RR], 0.52; 95% CI, 0.28–0.97). Fewer patients relapsed with M-ECT plus pharmacotherapy at a 1-year follow-up, as well (3 RCTs; RR, 0.64; 95% CI, 0.41–0.99). Readers may note that, in this article,³ there were discrepancies in the results presented across abstract, text, and forest plots and that the values presented here are from the forest plots, which are most likely to be correct.

Two recent studies, with reasonably large samples by ECT research standards, examined relapse after ECT. One examined relapse and predictors thereof 12 months after remission of depression following a course of ECT.⁴ The other examined relapse and predictors thereof 6 months after the abrupt, unplanned discontinuation of M-ECT.⁵ Both studies are important for many reasons and are discussed in the sections that follow.

Relapse After ECT: A 12-Month Follow-up Study⁴

Patients who remit after treatment of a major depressive episode may relapse, months or years later. In this context, Jelovac et al⁴ examined 12-month relapse and predictors thereof in patients who had experienced a major depressive episode and who had remitted with ECT.

This was a prospective observational study. The sample comprised 61 patients who had remitted after receiving bitemporal ECT at 1.5 times the seizure threshold (1.5í threshold) or right unilateral ECT at 6í threshold. The mean age of the sample was 62 years. The sample was 64% female and 100% white Irish in ethnicity. Three quarters of the sample was unipolar, a quarter had psychotic features at pre-ECT baseline, and two thirds had failed at least 1 adequate antidepressant trial for the index depressive episode (mean number of adequate trials, 1.2).

After ECT, patients received individualized pharmacotherapy; no patient received M-ECT; 44% received lithium. Relapse was defined as an increase (from baseline) in Hamilton Depression Rating Scale-24 (HDRS-24) score by at least 10 points, with a minimum score of at least 16 points, maintained for at least 2 consecutive weeks. Deliberate self-harm, a new course of ECT, or hospital admission were also considered as relapse.

Patients were followed up by phone or in person at 9 specified time points across 12 months. Seven predictor variables were examined in a multivariable Cox proportional hazards regression model. These variables included age, illness polarity, number of previous depressive episodes, presence of psychotic features before the index course of ECT, medication resistance at the index depressive episode in least 1 adequate trial, electrode placement during the index ECT course, and HDRS-24 score after the last ECT.

Important findings from the study are presented in Table 1. In summary, 19 (31%) of the 61 patients in the cohort relapsed within 6 months, and 24 (39%) within 12 months. Lithium, used by 44% of patients, was associated with a substantially reduced risk of relapse. A bipolar II diagnosis and a larger number of previous depressive episodes were associated with higher risk of relapse, whereas older age and the presence of psychotic symptoms at the pre-ECT baseline were associated with a lower risk of relapse.

Relapse After ECT: Strengths and Limitations of the Study⁴

The study by Jelovac et al⁴ had many strengths and limitations. The most important strengths were the modestly large sample of patients who had remitted with ECT (n = 61) and the long duration of prospective follow-up (12 months) during which only 1 patient withdrew (while still in remission), with data available for all the rest (n = 60).

The study had limitations that the reader must consider not just with regard to this study but with regard to other studies of a similar nature. There was no primary outcome explicitly stated; rather, several relationships were examined across different, essentially exploratory analyses with no protection against a Type 1 error. This means that some or many of the significant relationships identified may have been false positive findings.⁶

In the main analysis, the authors studied 7 predictors of relapse. Of these, the inclusion of at least 1 may have been unnecessary; for example, it is debatable whether remission elicited by bitemporal vs right unilateral ECT should be expected to influence 12-month relapse rates, especially when residual depressive symptoms are also adjusted for in the analysis. More to the point, the authors failed to include variables that could a priori have been expected to be strongly associated with relapse: the use of lithium, the use of second generation antipsychotic (SGA) drugs, or the overall appropriateness and adequacy of the maintenance pharmacotherapy regimen.

Why are lithium and SGA medications important? Because there is a large body of literature, especially in patients with bipolar depression, demonstrating their value in relapse prevention. And, it goes without saying that adequacy of maintenance pharmacotherapy, defined in any appropriate manner that considers drugs as well as doses, would also be expected to influence the risk of relapse during follow-up (note: if lithium and SGA medications are studied, then adequacy of pharmacotherapy should not be included because it is a superset of the lithium and the SGA variables).

As further justification of this suggestion, readers may note that after the main analysis, the authors presented a Cox regression analysis in which they found that lithium was indeed a significant predictor of relapse. However, they did not conduct a Cox regression to examine the role of SGA medications. This could and should have been considered post hoc because antipsychotics were used in the maintenance pharmacotherapy of 61% of the patients, and particularly because the better outcomes in patients with psychotic depression and those with bipolar I depression may have been because these patients had received SGA (and not because psychotic depression and bipolar I depression are protective against relapse).

The better outcome in patients who had had psychotic depression was unexpected; so was the failure of index episode antidepressant-resistance to predict relapse. The former result may have been due to confounding; that is, a higher frequency of use of SGA medications in such patients. The latter result may have been because antidepressant resistance was not strictly defined; patients were required to have failed only 1 previous adequate trial to be classified as resistant.

Relapse After M-ECT: A 6-Month Follow-up Study⁵

Patients with major mental illness are at high risk of relapse after abrupt discontinuation of maintenance pharmacotherapy. ECT is the only treatment for major mental illness that is abruptly discontinued after the efficacy target is met. This may explain why relapse after discontinuation of ECT is highest in the weeks to months immediately following discontinuation. For example, in the systematic review and meta-analysis¹ referred to in an earlier section, relapse rates after a successful course of ECT were 27% at 3 months, 38% at 6 months, and 51% at 12 months; that is, the highest density of relapse was in the first 3 months after discontinuation of ECT.

All discontinuation of M-ECT is necessarily abrupt; it is hard to taper and withdraw a treatment that is already widely spaced out in its administration. So, one might expect a high rate of relapse after the discontinuation of M-ECT, as well. Moderate to high rates of relapse have indeed been reported. For example, in a retrospective study, Huuhka et al⁷ presented a 1-year follow-up of 45 patients who had discontinued M-ECT because they had been in remission for at least 6 months (82% of the sample), because they did not wish to continue with M-ECT (11%), or because there were medical reasons for discontinuation (7%). The relapse rate at 1 year was 44%, with almost three-quarters of the relapses occurring within 3 months of M-ECT discontinuation. In another retrospective study, Cabelguen et al⁸ described 16 patients receiving 18 courses of M-ECT; the 6-month relapse rate was 44%, after stopping M-ECT.

In this context, Lambrichts et al⁵ presented prospective observational data from a moderately large cohort of patients whose maintenance ECT (M-ECT) was abruptly discontinued, force majeure, at the start of the COVID-19 pandemic.

The sample comprised 83 patients whose M-ECT had been abruptly discontinued; 1 patient was lost to follow-up, M-ECT was resumed at request in 1 patient, and 81 were followed up for 6 months. The mean age of the sample was 69 years. The sample was 59% female. Clinical diagnoses were major depressive disorder (63%), bipolar disorder (20%), psychotic disorder (15%), and others (2%). Clinical syndromes that constituted the indication for ECT were major depressive episode without psychotic features (40%), major depressive episode with psychotic features (38%), psychosis (11%), catatonia (10%), and mania (1%).

These patients had received a median of 25 M-ECT treatments, at a frequency of once a week to once in 6 weeks (median, once in 3 weeks), across a median of 59 weeks. About half of the patients had been receiving suprathreshold right unilateral ECT, and the rest, bitemporal ECT.

After M-ECT was stopped, patients continued with their individualized pharmacotherapy with antidepressants, mood stabilizers, and/or antipsychotic drugs and were followed weekly to monthly for 6 months by their treating psychiatrists. Relapse was defined as change of pharmacotherapy (excluding rescue medication, such as for anxiety or insomnia), re-introduction of ECT, deliberate self-harm, or hospital admission.

Five predictor variables were examined in 2 multivariable Cox proportional hazards regression models. These variables included age, number of previous courses of ECT for acute illness, number of M-ECT treatments received, and interval between M-ECT treatments at the time M-ECT was stopped. The fifth variable was clinical diagnosis in one regression and syndromal indication for ECT in the other regression.

Important findings from the study are presented in Table 2. In summary, in elderly patients who were receiving maintenance ECT (M-ECT) for mostly mood disorders, the abrupt discontinuation of M-ECT was associated with a 44% risk of relapse across 6 months, with half of the relapses occurring within 8 weeks of M-ECT discontinuation. Psychotic disorder, the need for more frequent M-ECT (indicating need for more intense intervention), and a larger number of previous courses of ECT were all associated with a higher risk of relapse.

Relapse After Abrupt Discontinuation of M-ECT: Strengths and Limitations of the Study⁵

The study by Lambrichts et al⁵ had many strengths and limitations. The most important strengths were the modestly large sample of patients (n = 83) and the prospective follow-up of 6 months during which only 2 patients withdrew, with data available for all the rest (n = 81).

A special strength is that the study was a natural experiment. Usually, discontinuation of M-ECT is a planned and shared decision and is effected when it is judged that the patient has a good chance of remaining free from recurrence of illness. In most of such patients, M-ECT is well spaced out. Sometimes, however, practical considerations or patient choices drive the decision to stop M-ECT. Thus, reasons to stop treatment are heterogeneous, and stopping biases may exist. In this study, there were no stopping biases; treatment was impartially stopped for all. This, however, is also a special limitation of the study because the findings may be hard to generalize to clinical contexts where stopping is a shared, planned decision.

The study had other limitations, too, some of which were similar to those in the study by Jelovic et al.⁴ There was no primary hypothesis that the authors set out to test. Furthermore, several relationships were examined across different, essentially exploratory analyses with no protection against a Type 1 error. In the Cox regressions, the authors failed to include variables that could a priori have been expected to be strongly associated with relapse; examples of such variables are the use of lithium, the use of antipsychotic drugs, or the overall appropriateness and adequacy of the maintenance pharmacotherapy regimen.

Limitations peculiar to this study were that the authors ran the same regression twice, with one variable substituted, the second time. In neither regression did they state the reference category for categorical explanatory variables. In neither regression did they state how multiple testing of one variable against another was conducted for categorical variables that had more than 2 categories.

General Comments: Overfitting in Cox Regression

In both of the studies^4,5 reviewed in this article, an important requirement for Cox regression was not met. The requirement, as most commonly stated, is that there needs to be 10–15 times as many events recorded in the study as there are predictor variables in the regression, and this value may exceed 20 when several low prevalence predictors are included; here, “event” refers to the less frequent of 2 outcomes.^9,10 In both of the studies reviewed in this article, relapse (as compared with no relapse) was the less frequent event. Jelovac et al⁴ had 24 relapse events and 7 predictors, for an events per variable (EPV) value of 3.4, and Lambrichts et al⁵ had 36 relapse events and 5 predictors, for an EPV value of 7.2.

When the EPV requirement is not met, overfitting is likely to occur. Overfitting is the situation in which the presence of an excessive number of predictor variables results in the regression presenting a spuriously better “fit” than is justified by the data. Babyak⁹ demonstrated this graphically using simulated models (n = 10,000 per model) with EPVs of 3.3, 6.6, 10, and 13.3. For each model, values for 15 independent variables and 1 dependent variable were randomly drawn. Thus, the independent variables were effectively noise variables; despite this, the models explained a progressively larger proportion of the variance in the dependent variable as the EPV decreased from 13.3 to 3.3.

Babyak explained overfitting simply: “If you put enough of predictors in a model, you are very likely to get something that looks important regardless of whether there is anything important going on in the population.”^9(p415) Another way to explain overfitting is to say that when there are too many predictor variables in the model, the model fits not only the true effects but also the statistical noise that is unique to the sample.¹¹ Whereas overfitting essentially describes the overall fit of the model, it follows that if the overall model is defective, the coefficients of the predictor variables could also be wrong.

What is the solution? Authors can use fewer explanatory variables, but this could be unsatisfactory when it results in the omission of important predictors and important confounders, Or, authors must recruit larger samples that contain enough of events to accommodate all their predictor variables; this is obviously the better option but is much harder to do in real world research. But who said that good research is easy to perform?

The bottom line is that the results of the Jelovac et al⁴ and the Lambrichts et al⁵ studies should be interpreted as being possible rather than probable.

The Elephants in the Room: Confounding, and Interactions Between Variables

Patients were not randomized into relapsed and non-relapsed categories; so, say, in the study by Jelovac et al,⁴ the analyses could have been compromised by inadequately measured, unmeasured, and unknown confounds. As possible examples of important unmeasured confounds, a higher episode density before ECT, higher acute and chronic stress levels after ECT, poorer post-ECT pharmacotherapy adequacy, and poorer post-ECT medication adherence could all be expected to increase the risk of relapse. There is no assurance that these variables were well balanced between, say, patients who received lithium and those who did not. As a result, the finding obtained for lithium in the sample may not be true for the population; the same conclusion applies to the other significant predictors, as well.

As a further note, regressions should but rarely include interaction terms between variables. Thus, for example, in order to know whether stress has a disproportionately greater effect on relapse rates in patients who are poorly adherent to medications, a stress í adherence interaction term would need to be included in the regression. In the study by Jelovac et al (2021), the interaction between lithium use and illness polarity (unipolar vs bipolar) is an interaction that is of interest. Similarly, the stress-support dimension is different across diagnoses and patients, and stress could affect patients with different diagnoses in different ways, so a stress í diagnosis interaction is of interest in follow-up studies that search for predictors of relapse.

Confounding, and interactions between variables, are issues that generalize to all follow-up studies in psychiatry and are not peculiar to the studies reviewed in this article. A problem is that adding confounding variables and adding interaction terms increases the sample size requirement, making it harder to perform a study that has good internal validity. As a final note, readers may note that postrandomization confounding may occur within RCTs, as well.¹²

General Summary

With regard to the 1-year follow-up study of patients who had received a course of ECT for major depressive episode,⁴ it is disheartening that, despite individualized maintenance pharmacotherapy, nearly a third of patients relapsed within 6 months; however, lithium appeared to substantially reduce the risk of relapse. The results of this study suggest that younger patients, those who had many previous depressive episodes, and those who have a bipolar II disorder diagnosis may require close attention during maintenance therapy, post-ECT, because they may be at higher risk of relapse.

With regard to the M-ECT discontinuation study,⁵ it is reassuring that only 44% of the patients relapsed across 6 months despite abrupt, unplanned discontinuation of treatment and despite experiencing the social and other stresses associated with the COVID-19 pandemic. This does suggest the interesting possibility that about half of patients who receive M-ECT may not need to continue the treatment. It is noteworthy that half of the relapses occurred early; that is, within 8 weeks of M-ECT discontinuation; so, this is the period during which clinicians, patients, and caregivers need to be especially alert for early warning signs of relapse. This study also suggests that, if patients are receiving M-ECT for psychotic disorder, if patients received many courses of ECT in the past, or if patients are so fragile as to need M-ECT at more frequent intervals, M-ECT should not be suddenly stopped.

Finally, investigators who conduct follow-up studies require to recruit larger samples to ensure that the sample size and the number of events of interest meet the requirements for the statistical models that are employed in the analyses. Investigators also need to include in their plans the study of clinically relevant explanatory variables, related to the risk of relapse, that may confound the hypotheses being examined. These variables include variables related to the appropriateness and adequacy of the maintenance pharmacotherapy regimen, adherence to maintenance pharmacotherapy, variables related to the stress-support dimension, and interactions between important explanatory variables.

Published online: August 10, 2021.

Each month in his online column, Dr Andrade considers theoretical and practical ideas in clinical psychopharmacology with a view to update the knowledge and skills of medical practitioners who treat patients with psychiatric conditions.

Department of Clinical Psychopharmacology and Neurotoxicology, National Institute of Mental Health and Neurosciences, Bangalore, India ([email protected]).
Financial disclosure and more about Dr Andrade.