Research Design and Overfitting: Reply to Jelovac and McLoughlin

J Clin Psychiatry 2022;83(3):21lr14371a

To cite: Andrade C. Research design and overfitting: reply to Jelovac and McLoughlin. J Clin Psychiatry. 2022;83(3):21lr14371a.
To share: https://doi.org/10.4088/JCP.21lr14371a

© Copyright 2022 Physicians Postgraduate Press, Inc.

^aDepartment of Clinical Psychopharmacology and Neurotoxicology, National Institute of Mental Health and Neurosciences, Bangalore, India
*Corresponding author: Department of Clinical Psychopharmacology and Neurotoxicology, National Institute of Mental Health and Neurosciences, Bangalore 560 029, India ([email protected]).

See letter by Jelovac and McLoughlin and article by Andrade

To the Editor: I appreciate the interest of Drs Jelovac and McLoughlin¹ in my article² on predictors of relapse after electroconvulsive therapy (ECT). In that article,² I had stated that their study³ was observational because the interventions of interest, bitemporal (BT) vs high-dose right unilateral (RUL) ECT, stopped before the period of interest, the 12-month follow-up. During this follow-up, the authors merely observed patients who received individualized rather than standardized continuation pharmacotherapy from treating clinicians; the authors did not intervene. Individualized pharmacotherapy could have improved outcomes in disadvantaged patients, blurring possible differences in long-term efficacy between the ECT groups to which the patients had originally been randomized. Furthermore, because only patients (n = 61 out of 138) who had remitted in their respective ECT groups were eligible for study, it is unlikely that the nature of the original randomization would have been preserved in the follow-up sample. Finally, the maintenance of treatment blind stated by Drs Jelovac and McLoughlin¹ is good research practice but does not have relevance to the conceptualization of research design as observational vs interventional or as randomized vs nonrandomized.⁴ In short, what they conducted³ was a prospective observational cohort study of outcomes in ECT remitter groups to which patients had not been randomized, and not a randomized controlled study of an intervention.

I had suggested that the inclusion of the BT vs RUL ECT variable in the regression may have been unnecessary not because the ECT grouping variable was not significant in the earlier, interventional phase of the study but because it seemed to me that if an intervention is withdrawn after remission has been brought about, it does not matter what brought about the remission when studying what influences relapse. However, I accept that it could equally well be argued that remission brought about by different forms of ECT could differ in duration and therefore merits study.

I agree with Drs Jelovac and McLoughlin that in studies such as theirs, the a priori selection of known or expected prognostic factors is standard practice, and the choice of covariates is not easy. However, overfitting happens when the model includes more covariates than the sample size permits. This risks the creation of a model that reduces residual variance in the regression and explains what fits the sample rather than what fits the population.⁵ I had suggested that their analysis may have been limited by overfitting.

Lithium and second-generation antipsychotic drugs have been found effective, some as monotherapy and some as antidepressant augmentation therapy, in patients with major depressive disorder and bipolar depression, and some among these have been found effective in maintenance therapy, as well. So, regardless of how the remission from depression was brought about, it could be reasonable to include the use of lithium and second-generation antipsychotics as regression covariates. Unfortunately, and this is something regarding which I sympathize with the authors,³ to do this could worsen overfitting. In this connection, I appreciate the clarification that, at least in univariate analysis, maintenance treatment with second-generation antipsychotics was not significantly associated with relapse.¹

On a parting note, the purpose of my article² was not to devalue the studies that I reviewed but to provide teaching on issues related to conducting, reading, and understanding research.

Published online: April 18, 2022.
Potential conflicts of interest: Please see https://www.psychiatrist.com/jcp/about-dr-andrade.
Funding/support: None.