Cognitive Debriefing Interviews

To pretest and refine the draft BEDS item pool, 2 iterative sets of face-to-face cognitive debriefing interviews were conducted with 13 participants self-reporting BED characteristics in 2 US locations. Interview participants, recruited and screened by qualitative research firms in each location, were aged ≥ 18 years with a normal or greater body mass index (BMI ≥ 19) based on self-reported height and weight. Each firm recruited from their established database of general community residents previously agreeing to be contacted for research opportunities.

Participants were asked to answer and provide feedback on the draft BEDS items using a “think-aloud” process to identify any problems with question phrasing or response options. The interviewers also asked participants how they would revise the BEDS items (if needed) to make them clearer and easier to answer. The initial pool of 20 items (tested in the first set of interviews) addressed 13 unique concepts. A reduced item pool of 15 items was tested in the remaining interviews; alternative items were tested for several concepts. Results from these interviews were used to create a 13-item draft version of the BEDS for quantitative evaluation and further refinement. The 13 draft items represented each DSM-5 diagnostic criterion for BED.

Quantitative Evaluation

Data were collected from 97 participants in a multisite, cross-sectional, prospective, noninterventional study to finalize the content and develop a scoring algorithm for the screener.

Sample

Similar to the pool of general community residents in the cognitive debriefing interviews, participants with and without self-reported BED characteristics were recruited and screened by qualitative research firms in 4 US locations. Participants were aged ≥ 18 years with a normal or greater BMI (≥ 19). Due to the low prevalence of BED, individuals more likely to meet diagnostic criteria for BED were overrecruited. Specifically, screening items consistent with the DSM-5 criteria for BED1 were developed and used to target a study sample with half of the participants self-reporting all BED characteristics.

Procedure

Individual data collection sessions consisted of a semistructured diagnostic interview administered by 1 of 2 PhD-level clinical psychologists (D.B.D. and T.M.B.) (to diagnose BED) and administration of the 13-item BEDS pilot version, the Binge Eating Scale (BES),9 and the RAND 36-Item Short-Form Health Survey (RAND-36).10 Interviewers were blinded to participants’ responses to the recruiting items and self-report instruments. The order of the diagnostic interview and instrument administration was counterbalanced.

Given the absence of any known clinical interviews updated to reflect the DSM-5 criteria (at the time of this study), the BED criteria from the Structured Clinical Interview for DSM-IV-TR Axis I Disorders-Research Version, Non-Patient Edition (SCID-I/NP)11 were updated. Study participants were diagnosed as having BED (yes) or not having BED (no) using the modified BED portion of the SCID-I/NP. As part of the informal training to conduct the diagnostic interview, both interviewers diagnosed an initial set of participants and discussed any differences related to the use and interpretation of interview feedback until consensus was obtained.

The 16-item BES measures the presence and severity of binge-eating behaviors that may be indicative of an eating disorder. Total scores range from 0 to 46 points. BES cut scores proposed by Timmerman12 were used to classify study participants into 3 severity subgroups: none or mild (≤ 17), moderate (18 to 26), and serious (≥ 27). The RAND-36, a 36-item generic HRQL measure, assesses 8 concepts and yields subscale scores for each, ranging from 0 to 100, with higher values denoting more favorable health.

Analysis

The observed item-level response distributions of the 13 BEDS items by BED diagnosis were reviewed to guide the selection of an optimal algorithm for identifying participants’ BEDS screening status (positive or negative).

A reference BEDS classification algorithm was defined by a specific response pattern to the 13 items on the basis of DSM-5 criteria. Numerous alternative classification algorithms were also formed by altering the responses required for the 13 items under the reference algorithm and eliminating nondiscriminating items (between those who did and did not meet the BED diagnostic criteria from the clinical interview). Each classification algorithm resulted in 4 diagnostic accuracy subgroups (true positive, false positive, true negative, false negative) on the basis of structured clinical interview (BED diagnosis: yes or no) and BEDS algorithm screening status.

The accuracy of the reference and alternative BEDS screening status subgroups in predicting BED diagnosis (ie, sensitivity and specificity) was tabulated to compare algorithm performance. The optimal algorithm was expected to produce a reasonable number of false positives while minimizing the number of false negatives so that most or all individuals with BED are detected for follow-up and few, if any, are missed.

RESULTS

Initial BEDS Item Pool

The preliminary BEDS item pool developed for clinical expert review included 18 items addressing 12 unique concepts relevant to the diagnosis of BED (alternative items were developed for 6 concepts).

While all 3 clinical experts deemed these 18 items to be appropriate candidates for a BED-specific screener, several improvements resulted, including minor modification of several items and the addition of new items addressing the concept of distress (due to bingeing). The experts also recommended the implementation of a stopping rule at the beginning of the screener if respondents indicated a lack of binge eating in the past 3 months. The resulting pool of 20 items addressed 13 unique behaviors pertaining to binge eating and the lack of recurrent compensatory behaviors (eg, vomiting) and included alternative items for 7 of these concepts.

Cognitive Debriefing Interviews

The cognitive debriefing participants (n = 13) had a mean age of 41.6 years (range, 21-59 years) and a mean BMI of 36.2 (range, 23.3-53.2), and 53.8% (n = 7) were female. Approximately two-thirds (69.2%, n = 9) were white, and 92.3% (n = 12) had completed at least some college. While interview participants reported many BED characteristics, none had received a diagnosis of BED from a health care professional.

Interview participants easily understood the BEDS instructions, questions, and response scales. As noted previously, alternative items were tested; the items deemed clearer or easier to answer by interview participants were retained for further testing.

On the basis of results of the 2 iterative rounds of cognitive debriefing interviews, and representing all DSM-5 BED criteria, a 13-item pilot version of the BEDS was developed (Table 2). Items 1 and 2 establish the presence of participants’ excessive overeating during the past 3 months. If a participant reports at least 1 episode of excessive overeating per week during that time period, he or she is directed to items 3-13. If a participant reports no episodes of excessive overeating (no to both items 1 and 2), the participant is directed to stop.

Quantitative Evaluation

Ninety-seven adults participated in the BEDS quantitative evaluation. Table 3 shows the participant characteristics by BED diagnosis status and overall. Of the 97 participants, the ratio of females to males was relatively similar—approximately 60%:40%—across BED diagnosis status. Nearly half (n = 47, 48.5%) endorsed all the recruitment screening items consistent with the DSM-5 criteria for BED (data not shown), and 16 (16.5%) of these participants were ultimately diagnosed with BED via the clinical interviews. The proportions of females and males diagnosed with BED were similar (10/62 female, 16%; 6/35 male, 17%). While the small sample size prohibits generalization, compared to those not diagnosed with BED, participants with BED tended to be slightly older (median age: 46 years), have higher BMIs (median: 34.7), be black or Hispanic, be employed part-time, and have participated in a weight-loss program.

The mean and median item scores tended to be similar in magnitude, irrespective of the order of the diagnostic interview and instrument administration. Table 2 shows the item-level response distributions of the 13-item pilot BEDS.

BEDS Classification Algorithms

The left side of Table 4 shows the item response pattern of the reference BEDS classification algorithm. The right side of Table 4 shows that only 3 of the 16 participants diagnosed with BED during the clinical interview responded in a manner wholly consistent with the reference algorithm, yielding a low sensitivity of 18.8% (true positive rate) and an unacceptably high false-negative rate of 81.2% (ie, 13 of 16). However, a specificity of 89.3% indicates that the majority of participants not diagnosed with BED on the basis of the clinical interview were also not identified as probable BED by the reference algorithm.

Because a primary goal was to develop a screener that would identify all (or nearly all) individuals with probable BED, instrument developers considered discarding items that reduced sensitivity, while retaining items that maximized specificity. Originally, item 1 (excessive overeating in last 3 months) and item 2 (excessive overeating at least once per week in last 3 months) were viewed as key items because their content is important in a clinical interview. However, the inclusion of item 2 in the BEDS-7 algorithm (for a total of 8 items) yielded a sensitivity of 87.5% and specificity of 56.0%. All 16 participants diagnosed with BED responded yes to item 1 and only 14 responded yes to item 2. While the specificity of the algorithm including item 2 was moderately better than the BEDS-7 algorithm (38.7%), sensitivity was reduced by 12.5%. Measurement error associated with item 2 was also a concern, because respondents must recall the total number of episodes during the past 3 months and divide by 12 to ensure an accurate answer. Ultimately, the developers decided to remove item 2 rather than sacrifice some of the screener’s sensitivity by retaining a question with a significant potential for inaccurate response.

Researchers explored over 15 alternative BEDS classification algorithms without including responses on item 2 (data not shown); item 1 and item 3 (distress) were included to maintain maximum sensitivity (100% using only items 1 and 3). Response distributions of the remaining 10 items were reviewed, and each item was added to gauge its impact on specificity. The response patterns for item 5 (eat faster than normal), item 6 (eat until uncomfortably full), and item 7 (continue eating) were similar with overlapping content. To create the briefest screener, only item 7 was retained. Additionally, while the items addressing compensatory behaviors (items 10, 11, 12, and 13) did not contribute substantially to the screener’s predictive value, to maintain face validity, item 10 (vomiting) was retained.

Ultimately, 7 items (final BEDS-7 items shown in Supplementary Appendix 1) were retained in the algorithm that maximized sensitivity and obtained the highest possible specificity. The response pattern of the BEDS-7 required for a diagnosis of BEDS is shown in the left-hand side of Table 4. The BEDS-7 algorithmic scheme yielded 100% sensitivity and 38.7% specificity when applied to the study sample. Specifically, with use of the BEDS-7, 100% of the participants who received a diagnosis of BED via the clinical interview screened positive (true positives). Among those without a diagnosis of BED (n = 75), 38.7% (n = 29) screened negative on the BEDS-7 (true negatives). Forty-six of the 62 participants who screened positive using the BEDS-7 (74.2%) were not diagnosed with BED via clinical interviews. Table 4 shows that the sensitivity (at 100%) and specificity (33.3%) of the 16-item BES subgroup12 ranges were similar to those observed using the BEDS-7.

Table 5 shows select characteristics for participants identified by the BEDS-7 algorithm as true positives, true negatives, and false positives (there were no false negatives due to the 100% sensitivity). A higher proportion of women (71%, 44 of 62) than men (51%, 18 of 35) were identified with probable BED. Of the participants identified with probable BED, those who were not diagnosed with BED tended to be younger (median age: 33.5 years) and have a lower BMI (median: 30.9) than those diagnosed with BED (median age: 46.0 years, median BMI: 34.7). Notably, participants who were classified as unlikely to have BED had the lowest median BMI (29.9). Participants who were correctly identified tended to have higher BES scores and lower mean RAND-36 subscale scores, both indicating poorer health status, than participants identified as true negatives.

DISCUSSION

The goal of this study was to develop a brief, valid, patient-reported screening tool for use in primary care and general psychiatry settings to identify individuals most likely to have BED and to facilitate further evaluation or referral to specialists. The 13-item BEDS pilot version, developed with a review of existing instruments and input from BED clinical experts, was included in a cross-sectional study of 97 participants, including a semistructured clinical interview to diagnose BED. Although the presence of all 13 concepts addressed in the BEDS pilot version is required to clinically diagnose BED (per DSM-5 criteria), analyses were conducted to investigate the feasibility of eliminating items for the briefest screening tool possible without sacrificing its face and content validity or predictive ability. Maximizing the sensitivity of the BEDS was considered critical to ensure that future respondents with BED receive further evaluation and care. Additionally, a moderate level of specificity was considered reasonable because the consequences of further evaluating a modest number of individuals without BED are outweighed by those of missing individuals needing treatment.

The sensitivity and specificity of numerous candidate classification algorithms were evaluated. The BEDS-7 maximized sensitivity (100%) while closely preserving the content of the DSM-5 criteria. In comparison to the 16-item BES, the BEDS-7 is shorter with comparable sensitivity and specificity. Contributing to the high rate of false positives from the BEDS-7 was a large proportion of individuals who did not meet diagnostic criteria for BED during the clinical interview but reported that they were regularly engaging in excessive overeating episodes, had a lack of control over this behavior, and were distressed about the behavior.

A potential weakness of this study is the small number of individuals diagnosed with BED (n = 16) from the clinical interviews. Obtaining a larger sample of individuals diagnosed with BED was the original intent, but differences between individuals’ initial self-report of food and eating behaviors and perceptions via the phone screening versus those obtained in the clinical interview proved meaningful. As such, consistent with the high rate of false positives from the BEDS-7, many individuals not diagnosed with BED self-reported BED behaviors and beliefs.

A strength of this study is the rigorous design, which allowed for the development of a brief screening tool based on existing instruments, clinician and patient input, and quantitative evaluation to inform item reduction and scoring in a sample of participants including a subset diagnosed with BED. Additionally, the BEDS-7 would appear to have broader utility in clinical settings beyond that of primary care and general psychiatry, for example, in the allied health practitioner clinic or office setting. Implementation of the BEDS-7 in real-world clinical practice is expected to promote better understanding of BED characteristics and help health care professionals identify patients who may have BED for appropriate follow-up.

Submitted: October 16, 2015; accepted February 1, 2016.

Published online: April 28, 2016.

Potential conflicts of interest: Dr Herman is an employee of Shire Development, LLC. Ms Deal is currently an employee of Pfizer, Inc but was employed at Shire Development, LLC during the conduct of this project. Drs DiBenedetti, Nelson, Fehnel, and Brown are employees of RTI Health Solutions.

Funding/support: Funding for this study and article was provided by Shire Development, LLC. RTI Health Solutions was contracted by Shire to design and implement this project.

Role of the sponsor: Authors from Shire Development, LLC, the sponsor of this study, made substantial contributions to the study design and interpretation of data, as well as the drafting and revision of the manuscript.

Previous presentation: Content from this original research was presented at the following annual meetings and conferences: American Psychiatric Association 168th Annual Meeting, Toronto, Canada, May 16-20, 2015 ▪ XXIst Annual Meeting of the Eating Disorders Research Society, Taormina, Sicily, September 17-19, 2015 ▪ Canadian Psychiatric Association 65th Annual Conference, Vancouver, Canada, October 1-3, 2015.

Acknowledgments: Special thanks are extended to Susan G. Kornstein, MD (Virginia Commonwealth University, Richmond, Virginia); Rebecca Puhl, PhD (University of Connecticut, Storrs, Connecticut); and Denise E. Wilfley, PhD (Washington University, St Louis, Missouri) for reviewing and providing feedback on the initial version of the Binge-Eating Disorder Screener. They are all paid consultants to Shire Development, LLC.

Additional information: Shire develops and manufactures treatments for psychiatric disorders, including binge-eating disorder.

Supplementary material: See accompanying pages.

REFERENCES

1. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. Fifth Edition. Washington, DC: American Psychiatric Association; 2013.

2. Kessler RC, Berglund PA, Chiu WT, et al. The prevalence and correlates of binge eating disorder in the World Health Organization World Mental Health Surveys. Biol Psychiatry. 2013;73(9):904-914. PubMed doi:10.1016/j.biopsych.2012.11.020

3. Hudson JI, Coit CE, Lalonde JK, et al. By how much will the proposed new DSM-5 criteria increase the prevalence of binge eating disorder? Int J Eat Disord. 2012;45(1):139-141. PubMed doi:10.1002/eat.20890

4. Trace SE, Thornton LM, Root TL, et al. Effects of reducing the frequency and duration criteria for binge eating on lifetime prevalence of bulimia nervosa and binge eating disorder: implications for DSM-5. Int J Eat Disord. 2012;45(4):531-536. PubMed doi:10.1002/eat.20955

5. Ágh T, Kovács G, Pawaskar M, et al. Epidemiology, health-related quality of life and economic burden of binge eating disorder: a systematic literature review. Eat Weight Disord. 2015;20(1):1-12. PubMed doi:10.1007/s40519-014-0173-9

6. Masheb RM, Grilo CM. Quality of life in patients with binge eating disorder. Eat Weight Disord. 2004;9(3):194-199. PubMed doi:10.1007/BF03325066

7. Padierna A, Quintana JM, Arostegui I, et al. The health-related quality of life in eating disorders. Qual Life Res. 2000;9(6):667-674. PubMed doi:10.1023/A:1008973106611

8. Sallet PC, de Alvarenga PG, Ferrão Y, et al. Eating disorders in patients with obsessive-compulsive disorder: prevalence and clinical correlates. Int J Eat Disord. 2010;43(4):315-325. PubMed

9. Gormally J, Black S, Daston S, et al. The assessment of binge eating severity among obese persons. Addict Behav. 1982;7(1):47-55. PubMed doi:10.1016/0306-4603(82)90024-7

10. Hays RD, Sherbourne CD, Mazel RM. The RAND 36-Item Health Survey 1.0. Health Econ. 1993;2(3):217-227. PubMed doi:10.1002/hec.4730020305

11. First MB, Spitzer RL, Gibbon M, et al. Structured Clinical Interview for DSM-IV-TR Axis I Disorders-Research Version, Non-Patient Edition (SCID-I/NP). New York, NY: Biometrics Research, New York State Psychiatric Institute; 2002.

12. Timmerman GM. Binge Eating Scale: further assessment of validity and reliability. J Appl Biobehav Res. 1999;4(1):1-12. doi:10.1111/j.1751-9861.1999.tb00051.x