A Randomized Survey of the Public’s Expectancies and Willingness to Participate in Clinical Trials of Antidepressants Versus Psychotherapy for Depression

A Randomized Survey of the Public’s Expectancies and Willingness to Participate in Clinical Trials of Antidepressants Versus Psychotherapy for Depression

Brandon A. Gaudiano, PhDa,b,*; Stacy R. Ellenberg, BSa,b; Casey A. Schofield, PhDc; and Lara S. Rifkin, BSd

ABSTRACT

Background: Expectancies and treatment preferences are known to affect the outcomes of patients enrolled in clinical trials for depression, but there is little research on their influence when the public is considering participation in these trials.

Method: We conducted an online survey (May 2013) in which participants (N = 615) were randomly assigned to read hypothetical descriptions of clinical trials for depression based on 1 of the following study designs: medication versus placebo, medication versus medication, psychotherapy versus placebo, or psychotherapy versus psychotherapy. Afterward, individuals rated willingness to participate in the trial, logic and credibility of the treatments, and expected success and improvement in symptoms.

Results: There were no differences in expectancies for ratings of credibility and logic or success and improvement among clinical trial designs. However, self-reported willingness to participate in the study was rated significantly higher in the 2 psychotherapy trial designs (active-comparator and placebo-controlled) compared with the active-comparator medication design (P < .05). Psychiatric treatment history, general treatment preferences, and depression severity were positively correlated with willingness to participate primarily in the active-comparator medication design.

Conclusions: Consistent with the broader treatment preference literature, individuals reported a greater willingness to participate in psychotherapy compared with antidepressant studies. Thus, people’s perceptions of different treatments are likely to influence not only the outcomes of clinical trials for depression but also decisions to participate in these trials in the first place.

Prim Care Companion CNS Disord 2016;18(1):doi:10.4088/PCC.15m01879

© Copyright 2016 Physicians Postgraduate Press, Inc.

aButler Hospital, Providence, Rhode Island

bAlpert Medical School of Brown University, Providence, Rhode Island

cDepartment of Psychology, Skidmore College, Saratoga Springs, New York

dMcLean Hospital/Harvard Medical School, Belmont, Massachusetts

*Corresponding author: Brandon A. Gaudiano, PhD, Butler Hospital, 345 Blackstone Blvd, Providence, RI 02906 ([email protected]).

Rutherford et al1 state that "prognostic" expectancies refer to when "a patient estimates the probabilities associated with various future scenarios, including anticipated positive or negative effects of treatment."(p1) These treatment-related expectancies have been shown to predict outcomes of patients participating in clinical trials for depression, sometimes explaining more variance in improvement than the treatment itself.1 For example, Krell et al2 found that 90% of patients who expected the study medication to be "very effective" responded to treatment, whereas only 33% of those who believed it would be "somewhat effective" responded to treatment. Relatedly, treatment preferences also can affect treatment outcomes. Kocsis et al3 found that patients favoring psychotherapy or antidepressant treatment at study entry achieved higher remission rates if they received their preferred treatment in a large randomized trial (N = 429) of medication, psychotherapy, or their combination for chronic depression.

One of the factors that may influence both willingness to participate and expectancies for improvement is the design of the study. Previous research4 has shown that participants rate their expected improvement higher after being presented with descriptions of active-comparator (drug vs drug) relative to placebo-controlled (drug vs pill placebo) study designs. Furthermore, meta-analyses5,6 indicate that response rates are better in active-comparator versus placebo-controlled antidepressant trials, perhaps on the basis of differential expectancies for improvement related to the probability of receiving pill placebo or an "inactive" treatment. Recently, Rutherford et al7 found that expectancies measured before and after randomization during a clinical trial were lower in those assigned to the placebo-controlled trial.

Gaudiano et al8 conducted a study of 55 hospitalized patients diagnosed with depressive disorders who rated their expectancies following the presentation of 3 hypothetical clinical trial descriptions: antidepressant versus pill placebo, antidepressant versus antidepressant, or psychotherapy versus psychotherapy (in counterbalanced order). Patients reported greater expectancies for improvement and willingness to participate in the psychotherapy trial design compared with either of the medication trial designs. Patients also reported greater expectancies for improvement in the active-comparator antidepressant design compared with the placebo-controlled design,8 which is consistent with past research.4

Given that all participants in our previous vignette study8 were exposed to psychiatric treatment and diagnosed with severe depression at the time, we wished to examine expectancy effects in a broader public sample in the current study. It is important to examine the general public’s perceptions of various study designs and the treatments offered in them to understand the generalizability of findings from clinical trials. Examination of treatment expectancies and preferences in a broader public sample compared with traditional psychiatric samples is also particularly relevant to primary care settings where these individuals are likely to first be seen. Differences in expectancies, acceptability, and willingness to participate in clinical trials of psychotherapy versus medication have important treatment implications. For example, it would be particularly important to consider matching lower acceptability treatments to patients’ preferences to obtain optimal clinical results. Furthermore, more acceptable treatments should be considered the "first-line" evidence-based treatments to be offered to improve treatment adherence and efficiency relative to less acceptable treatments (if they have similar efficacy).

• The public’s expectancies and willingness to participate in clinical trials can affect the generalizability of results.
• Participants reported a greater willingness to participate in psychotherapy studies than medication studies for depression.
• Findings from psychotherapy trials may be more generalizable to primary care settings, and patient preferences should be carefully considered.

In the current study, we randomly assigned participants to different study vignettes to examine the specific effects of our trial design manipulation on expectancies. Thus, we implemented a fully factorial experiment that included an additional vignette condition describing the psychotherapy versus placebo design to directly compare to the antidepressant versus placebo design. We hypothesized that participants would report more positive expectancies and willingness to participate for (1) psychotherapy compared with medication study designs and (2) active-comparator compared with placebo-controlled study designs. We also examined the relationship between expectancies and individual differences, including depressive symptoms and psychiatric treatment history.

METHOD

Sample

Participants were 615 individuals from the United States who responded to a request to complete a brief online survey to "understand people’s decisions to participate in studies of treatments for depression." Advertisements were delivered through a separate service hosted by Amazon.com called Mechanical Turk (mTurk). mTurk is an Internet-based service that allows researchers to identify people willing to participate in online research. Participants recruited through this service have been shown to be representative of the general population and have been used extensively in psychiatric and psychological research.9 Data were collected in May 2013.

Inclusion criteria were (1) age > 18 years, (2) ability to read and write English sufficiently to complete the survey, and (3) a record of at least a 95% approval rate from previous mTurk tasks. Exclusion criteria included lack of a computer or Internet access to complete the survey. Due to an initial survey programming error, demographic data were not collected for the first wave of participants across all vignette conditions; 424 respondents provided demographic information once this error was fixed. However, it is unlikely that the demographic characteristics of these initial participants differed meaningfully from those of participants collected later, because data were missing similarly across all conditions over a similar time period. The mean age of the sample was 34.8 (SD = 13.0) years. A total of 63.0% were women, 81.6% were white/non-Hispanic, 58.9% were employed full- or part-time, 13.9% were students, and 20.5% had a high school degree or less (Table 1).

Click figure to enlarge

Study Vignettes

Study vignettes were originally developed and adapted for psychotherapy on the basis of those used by Rutherford et al4 for medication treatments. Academic psychiatrists and psychologists involved in clinical trials of medication, psychotherapy, and combined treatment reviewed vignette language. They provided feedback to ensure consistency across vignettes, reduce potential bias in treatment descriptions, and ensure comparability with commonly used study consent language. Participants were randomly assigned by the survey software to read and rate 1 of 4 brief vignettes of clinical trials for the treatment of depression: (1) antidepressant versus pill placebo, (2) antidepressant A versus B, (3) psychotherapy versus psychological placebo, and (4) psychotherapy A versus B. No specific medications or psychotherapies were mentioned so as not to bias participant responses. The vignettes also described other general information typically presented during the informed consent process that was kept standardized across vignette conditions to isolate the variable of interest (randomized condition): (1) randomization to treatment, (2) blinding from treatment allocation, (3) study duration, (4) number of treatment visits, (5) treatment provision free of charge, and (6) poststudy debriefing and unblinding. Table 2 provides sample drug versus placebo study vignette language. In the other vignettes, psychotherapy was described as follows: "Psychotherapy refers to ‘ talk therapy’ or counseling." In addition, psychological placebo was described as follows: "A placebo is designed to be an inactive treatment (for example, providing support and encouragement but not providing specific techniques)."

Click figure to enlarge

Measures

Modified Credibility/Expectancy Questionnaire. The modified Credibility/Expectancy Questionnaire (CEQ)10 was adapted for the current study to assess respondents’ treatment expectancies after reading the study vignette: (1) how logical and credible the study seemed to the person (1 = "not at all logical" to 10 = "very logical"), (2) how successful study participation was anticipated to be for reducing symptoms (1 = "not at all useful" to 10 = "very useful"), and (3) the percentage of symptom improvement the person expected to achieve by the end of the study (0%-10% to 100%; later converted into 10-point Likert scale format similar to the other items). Additionally, we modified the CEQ by adding a question to assess willingness to participate in the trial (1 = "definitely not willing to participate" to 10 = "definitely willing to participate") as used in our previous vignette study.8 The CEQ has demonstrated reliability and validity in clinical and nonclinical samples.4,8,10 The CEQ is not a unitary scale and measures logic and credibility as well as expectancy for improvement. Also, we only administered 3 items (out of 6) from the original scale. Therefore, we analyzed each modified CEQ item individually (and not as a total score).

Beck Depression Inventory-II. The Beck Depression Inventory-II (BDI-II)11 is a 21-item self-report measure of depression symptom severity with high reliability and validity demonstrated in clinical and nonclinical samples.12 The suicidality item was excluded from the measure due to inability to contact participants directly should they endorse this item; therefore, the total score was based on 20 items.

Treatment Options Questionnaire. The Treatment Options Questionnaire13 (TOQ; available from C. Battle and I. W. Miller, who own copyright, upon request) provides descriptions of different treatments for depression and assesses the likelihood that participants would be willing to receive (or continue to receive) these treatments (1 = "definitely would not" to 10 = "definitely would").14 We assessed psychotropic medications and psychotherapeutic interventions in the current study. Items also assessed whether the participant received psychotropic medication or psychotherapy (0 = no, 1 = yes).

Procedures

All study procedures were approved by the local institutional review board. After clicking on the link to the survey, participants read a statement of informed consent online and confirmed their consent by clicking on the corresponding button. Participants first completed demographic questions. Next, participants read 1 of 4 randomly assigned vignettes and then responded to questions about the vignette on the basis of the modified CEQ and completed other measures (eg, BDI-II). Participants were asked to rate CEQ items on the basis of the overall clinical trial description and not separately for the individual treatment conditions offered within the study (to obtain total ratings for each vignette). Overall ratings of the trials were collected because participants were informed that they could be randomized to either of the treatments offered in the trial. In other words, participation in a clinical trial requires an "all-or-nothing" judgement, which we wanted to capture in our clinical trial ratings. General acceptability ratings for psychotherapy and medication treatments were collected separately on the basis of the TOQ. As is typical of mTurk projects, participants were compensated a small amount of money for completing the brief survey ($0.50).

Statistical Analyses

Variables showing baseline differences among conditions were used as covariates in subsequent analyses. We conducted multivariate analyses of covariance (MANCOVAs), entering the specific CEQ items as the group of dependent variables and study vignette type as the independent variable. Only significant omnibus MANCOVAs were followed up by individual ANCOVAs (and appropriate post hoc tests) examining each specific CEQ item to further reduce type I error.

RESULTS

Preliminary Analyses

Demographics. No significant differences were found among the vignette conditions on demographic variables (P = not significant).

Depression severity. The mean total BDI-II score was 18.4 (SD = 13.3), and scores did not significantly differ among the vignette conditions. The BDI-II severity ranges for the entire sample were as follows: minimal range (0-13) = 42.0%, mild range (14-19) = 14.6%, moderate range (20-28) = 19.6%, and severe range (29-63) = 23.8%. These results demonstrated good variability in depressive symptoms and are typical of those found in other general community samples.11

Psychiatric history. A total of 45.6% reported a history of psychotropic medication use (21.1% current), 47.7% reported a history of psychotherapy (12.2% current), and 56.4% reported a history of either medications or psychotherapy (24.2% current). Thus, approximately half the sample had a history of psychiatric treatment consistent with the BDI-II severity ratings described previously. There was a significant difference between those receiving psychotropic medications or psychotherapy versus no treatment history among vignette conditions (χ23 = 8.38, P = .039): drug versus placebo = 58.3%, drug versus drug = 46.8%, psychotherapy versus placebo = 62.3%, and psychotherapy versus psychotherapy = 58.3%. Therefore, we controlled for psychiatric treatment history in the subsequent analyses.

Primary Analyses

General treatment preferences. First, we examined general treatment preferences in the sample (TOQ ratings). Overall, participants reported a significantly greater preference for psychotherapy (mean = 5.2, SD = 1.8) over psychotropic medications (mean = 4.3, SD = 2.2) for treating depression (t610 = 8.70, P < .001, Cohen d = 0.42 [medium effect]). This finding remained significant even after controlling for psychiatric treatment history (P < .001).

Vignette ratings by clinical trial design. Descriptive statistics for CEQ items are displayed in Table 3. Ratings were normally distributed and did not show significant (ie, > ± 1.0) skewness or kurtosis. On average, participants found the different trial designs moderately to strongly acceptable as demonstrated by relatively high mean CEQ ratings.

Click figure to enlarge

The overall MANCOVA* was significant when conducted on individual CEQ ratings of logic, success, improvement, and willingness to participate in vignettes, controlling for treatment history (Wilks λ = 0.964, F12, 1553.35 = 1.83, P = .040, ηp2 = 0.012). Follow-up ANCOVAS showed no significant differences on ratings of logic and credibility, success, or improvement across vignette conditions. However, an ANCOVA showed that the CEQ rating of willingness to participate was significantly different across conditions (F3, 590 = 4.39, P = .005, ηp2 = 0.022). Post hoc comparisons revealed that ratings of willingness to participate in the therapy versus therapy (P = .001) and therapy versus placebo (P = .014) conditions were significantly higher compared with the drug versus drug condition. The difference between the therapy versus therapy and drug versus placebo conditions was only marginally significant (P = .055) but in the same direction as the other findings (Figure 1).

Click figure to enlarge

Correlations among vignette ratings and other study variables. Given the differences across conditions on ratings of willingness to participate in the trial, we examined their relationship with other study variables (Table 4). Willingness to participate ratings (CEQ) were significantly positively correlated with general medication and psychotherapy treatment preference ratings (TOQ) primarily for the medication conditions (drug vs placebo or drug vs drug) in contrast to the psychotherapy conditions (therapy vs placebo or therapy vs therapy). The only exception to this was a significant positive correlation between willingness to participate and therapy treatment preference ratings in the therapy versus placebo condition alone. Also, willingness to participate scores were positively correlated with medication and psychotherapy treatment history and current depression severity (BDI-II) only in the drug versus drug condition.

Click figure to enlarge

DISCUSSION

The current results partially replicated and extended the findings of our previous survey conducted in a clinically depressed sample receiving psychiatric treatment,8 as well as other research investigating expectancies in nonclinical samples.4 As predicted, the general public reported greater preference when presented with an active-comparator psychotherapy design compared with a similar active-comparator medication design. Contrary to predictions, participants also reported greater willingness to participate in the placebo-controlled psychotherapy design compared with the active-comparator medication design. Respondents did not report differences in perceived logic and credibility or expected treatment success and improvement among these different trial designs.

Having a psychiatric medication and psychotherapy treatment history, higher preference and acceptability ratings for psychotherapy and antidepressants in general, and higher current depression severity correlated with higher willingness to participate ratings only in the drug versus drug design. Previous research has shown that various historical factors, including psychiatric treatment, are related to antidepressant response and treatment resistance in clinical trials.15 No significant correlations were found between willingness to participate in the psychotherapy versus psychotherapy design and the other study variables, which may have been due to the high level of psychotherapy acceptability reported in the sample. Overall, individuals found participation in depression trials in which they were guaranteed to receive psychotherapy (ie, psychotherapy vs psychotherapy and psychotherapy vs psychological placebo designs) the most acceptable, regardless of factors such as treatment history or depression severity. Our findings are consistent with those of McHugh et al,16 who conducted a meta-analysis of treatment preference studies and found that psychotherapy or counseling was preferred over psychotropic medication at a ratio of 3:1 in the public.

In the current study using a general public sample, no differences were found in expected improvement on the basis of the study design. In contrast, participants in our past study8 may have had preexisting biases about treatment on the basis of their own treatment participation and the clinically depressed nature of the sample. What is particularly intriguing is that participants still preferred psychotherapy studies even though they reported similar levels of credibility and expected improvement in antidepressant trials, suggesting that other factors (eg, medication side effects, perceived differences between depression etiology and treatment approach) may make antidepressant trials less appealing, which will require further study.

In addition, only a marginally significant difference was found between the active-comparator psychotherapy design compared with the placebo-controlled medication design in terms of willingness to participate ratings. Numerically, scores were somewhat lower for the placebo-controlled medication study compared with both psychotherapy study designs, but scores were even lower for the active-comparator medication study (see Figure 1). Thus, it is possible that we detected a type of "dose-response" relationship, such that greater likelihood of receiving active medication in the sample was perceived as least acceptable, whereas greater likelihood of receiving "active" psychotherapy was perceived as most acceptable. However, given the unexpected nature of these findings, it will be important for future research to replicate and examine this more closely.

Several limitations should be considered when interpreting current findings. First, participants who were not currently depressed were instructed to imagine that they were feeling depressed when rating the study vignettes presented to them. Therefore, results may not generalize to depressed individuals’ actual decisions to participate in a trial. However, research conducted in clinically depressed samples suggests that results from survey studies are similar to expectancies observed in actual clinical trials.7 Second, our vignettes presented very brief information about the treatments and clinical trial design. Thus, such vignettes may best inform us about people’s initial perceptions of the treatments offered in clinical trials, and these perceptions may change later if more extensive information is provided. Third, effect size differences were small to medium in magnitude, which may reflect the unselected nature of the current sample. Effect size differences in expectancies were medium to large in our previous vignette study8 using clinically depressed patients. Fourth, as is typical of online survey research, it was not possible to determine the number of potential participants who viewed the survey request but chose not to participate in order to calculate the true participation rate. Finally, participants were asked to rate the trial overall instead of the individual treatments offered within the trial. Future research should also collect individual treatment ratings within each trial to clarify overall trial ratings.

Previous research17,18 has shown that mTurk methods produce the same or higher psychometric quality as data from published research using traditional and other Web-based samples. In addition, the mTurk participant pool represents a more diverse population than typical Internet and college samples.9,17 Potential problems related to social desirability or malingering seem less likely in the current sample given the anonymity of the survey and lack of incentive for "faking bad" given our unselected sample. However, it is possible that mTurk respondents find research participation in general more acceptable than the general population. Thus, it will be important to replicate results in a more traditional community sample to verify the current findings.

There also are several important implications of the current study for research and clinical practice, particularly related to primary care settings. First, the current study contributes to the literature demonstrating that expectancies and preferences can affect the acceptability of clinical trials for depressed and nondepressed individuals. The greater overall preference for psychotherapy versus antidepressants for treating depression in the public could bias participation in medication trials more than psychotherapy trials, as participants in the former type of study may not be as representative of the overall population. Thus, psychotherapy trial results may be more generalizable to primary care and other nonpsychiatric patients than antidepressant trials. Furthermore, clinical trial designs may need to better take expectancies into account by incorporating the public’s preferences into the randomization procedure to reflect real-world practice.19,20 Lin et al21 found that primary care patients matched to their preferred treatment (psychotherapy vs antidepressant) in the context of a randomized trial achieved superior outcomes compared to those mismatched to treatments. This study shows that it may be important to consider past treatment failures or successes when recruiting participants into clinical trials for depression.

In conclusion, this study reinforces the idea that the generalizability of results from drug trials to clinical practice settings can be problematic as more patients may not find study participation acceptable. Psychotherapy trials may be less subject to this selection bias on the basis of greater overall acceptability of psychotherapy as a treatment for depression. Clinicians should consider patient preferences when discussing various treatment options for depression. In particular, they should carefully assess the acceptability of antidepressants for depression when offered to minimize potential problems later on with adherence and outcomes. Combined treatment with psychotherapy should also be considered for patients already receiving medication if preferred by the patient and drug response is found to be suboptimal.

Submitted: August 31, 2015; accepted November 12, 2015.

Published online: February 25, 2016.

Potential conflicts of interest: None reported.

Funding/support: This study was supported by internal funds provided to Dr Schofield by Skidmore College, Saratoga Springs, New York.

Role of the sponsor: The funding source had no involvement in the study design; collection, analysis, and interpretation of data; or writing of the manuscript.

REFERENCES

1. Rutherford BR, Wager TD, Roose SP. Expectancy and the treatment of depression: a review of experimental methodology and effects on patients outcome. Curr Psychiatry Rev. 2010;6:1-10. PubMed doi:10.2174/157340010790596571

2. Krell HV, Leuchter AF, Morgan M, et al. Subject expectations of treatment effectiveness and outcome of treatment with an experimental antidepressant. J Clin Psychiatry. 2004;65(9):1174-1179. PubMed doi:10.4088/JCP.v65n0904

3. Kocsis JH, Leon AC, Markowitz JC, et al. Patient preference as a moderator of outcome for chronic forms of major depressive disorder treated with nefazodone, cognitive behavioral analysis system of psychotherapy, or their combination. J Clin Psychiatry. 2009;70(3):354-361. PubMed doi:10.4088/JCP.08m04371

4. Rutherford BR, Rose SA, Sneed JR, et al. Study design affects participant expectations: a survey. J Clin Psychopharmacol. 2009;29(2):179-181. PubMed doi:10.1097/JCP.0b013e31819a9181

5. Rutherford BR, Sneed JR, Roose SP. Does study design influence outcome? the effects of placebo control and treatment duration in antidepressant trials. Psychother Psychosom. 2009;78(3):172-181. PubMed doi:10.1159/000209348

6. Sneed JR, Rutherford BR, Rindskopf D, et al. Design makes a difference: a meta-analysis of antidepressant response rates in placebo-controlled versus comparator trials in late-life depression. Am J Geriatr Psychiatry. 2008;16(1):65-73. PubMed doi:10.1097/JGP.0b013e3181256b1d

7. Rutherford BR, Marcus SM, Wang P, et al. A randomized, prospective pilot study of patient expectancy and antidepressant outcome. Psychol Med. 2013;43(5):975-982. PubMed doi:10.1017/S0033291712001882

8. Gaudiano BA, Hughes JA, Miller IW. Patients’ treatment expectancies in clinical trials of antidepressants versus psychotherapy for depression: a study using hypothetical vignettes. Compr Psychiatry. 2013;54(1):28-33. PubMed doi:10.1016/j.comppsych.2012.06.003

9. Buhrmester M, Kwang T, Gosling SD. Amazon’s mechanical turk: a new source of inexpensive, yet high-quality, data? Perspect Psychol Sci. 2011;6(1):3-5. PubMed doi:10.1177/1745691610393980

10. Devilly GJ, Borkovec TD. Psychometric properties of the Credibility/Expectancy Questionnaire. J Behav Ther Exp Psychiatry. 2000;31(2):73-86. PubMed doi:10.1016/S0005-7916(00)00012-4

11. Steer RA, Ball R, Ranieri WF, et al. Dimensions of the Beck Depression Inventory-II in clinically depressed outpatients. J Clin Psychol. 1999;55(1):117-128. PubMed doi:10.1002/(SICI)1097-4679(199901)55:1<117::AID-JCLP12>3.0.CO;2-A

12. Beck AT, Steer RA, Garbin RA. Psychometric properties of the Beck Depression Inventory: 25 years of evaluation. Clin Psychol Rev. 1988;8:77-100.

13. Battle C, Miller IW. Treatment Options Questionnaire. Brown University: Providence, Rhode Island; 2008 (unpublished).

14. Battle CL, Salisbury AL, Schofield CA, et al. Perinatal antidepressant use: understanding women’s preferences and concerns. J Psychiatr Pract. 2013;19(6):443-453. PubMed doi:10.1097/01.pra.0000438183.74359.46

15. Souery D, Oswald P, Massat I, et al. Group for the Study of Resistant Depression. Clinical factors associated with treatment resistance in major depressive disorder: results from a European multicenter study. J Clin Psychiatry. 2007;68(7):1062-1070. PubMed

16. McHugh RK, Whitton SW, Peckham AD, et al. Patient preference for psychological vs pharmacologic treatment of psychiatric disorders: a meta-analytic review. J Clin Psychiatry. 2013;74(6):595-602. PubMed doi:10.4088/JCP.12r07757

17. Behrend TS, Sharek DJ, Meade AW, et al. The viability of crowdsourcing for survey research. Behav Res Methods. 2011;43(3):800-813. PubMed doi:10.3758/s13428-011-0081-0

18. Shapiro DN, Chandler J, Mueller PA. Using Mechanical Turk to study clinical populations. Clin Psychol Sci. 2013;1(2):213-220. doi:10.1177/2167702612469015

19. Brewin CR, Bradley C. Patient preferences and randomized clinical trials. BMJ. 1989;299(6694):313-315. PubMed doi:10.1136/bmj.299.6694.313

20. Long Q, Little RJ, Lin X. Causal inference in hybrid intervention trials involving treatment choice. J Am Stat Assoc. 2008;103(482):474-484. doi:10.1198/016214507000000662

21. Lin P, Campbell DG, Chaney EF, et al. The influence of patient preference on depression treatment in primary care. Ann Behav Med. 2005;30(2):164-173. PubMed doi:10.1207/s15324796abm3002_9