The New Hampshire Hospital Screening and Referral Algorithm (NHHSRA) for Substance Use in People With Serious Mental Illness

Prim Care Companion CNS Disord 2023;25(3):22m03410

To cite: Stanciu CN, Healey CJ, Emeny RT, et al. The New Hampshire Hospital Screening and Referral Algorithm (NHHSRA) for substance use in people with serious mental illness. Prim Care Companion CNS Disord. 2023;25(3):22m03410.
To share: https://doi.org/10.4088/PCC.22m03410

© 2023 Physicians Postgraduate Press, Inc.

^aDepartment of Psychiatry, Dartmouth’s Geisel School of Medicine, Dartmouth-Hitchcock Medical Center, Lebanon, New Hampshire
^bNew Hampshire Hospital, Concord, New Hampshire
^cDepartment of Psychiatry, Dartmouth Institute of Health Policy and Clinical Practice, Dartmouth-Hitchcock Medical Center, Lebanon, New Hampshire
*Corresponding author: Corneliu N. Stanciu, MD, CPE, MRO, FASAM, FAPA, Dartmouth’s Geisel School of Medicine, Dartmouth-Hitchcock Medical Center, Department of Psychiatry, One Medical Center Dr, Lebanon, NH 03756 ([email protected]).

A staggering 21 million Americans experience substance use, yet only 10% of sufferers receive any intervention, presenting a large treatment gap associated with significant financial and human costs.¹ Health care organizations worldwide emphasize the importance of screening for substance use in all treatment settings, followed by early targeted interventions.² Current data demonstrate tremendous cost savings to health care systems when substance use is identified and appropriately addressed,^3–6 and hospitalizations present a critical window of opportunity for substance use treatment engagement.⁷ With a lack of screening and referrals for interventions in place, individuals accumulate greater burden or morbidities and have longer lengths of hospital stay compounded by higher readmission rates.⁸

Individuals with serious mental illness, such as schizophrenia, bipolar disorder, and major depressive disorder, experience high rates of comorbid substance use,⁹ which are often undetected or not appropriately detected in routine clinical settings.¹⁰ Although several substance abuse screening instruments exist, most are intended for primary care settings¹¹ or for individuals with mild anxiety and depression, and all rely heavily on patient self-report. None have been validated in a seriously mentally ill population. Substance use screening is challenging in people with serious mental illness and relies heavily on clinical judgment to decide who should be referred for interventions. Without standardized tools for systematic screening and referral to care, there is great variability in this process, resulting in poor identification of those patients who might benefit from treatment.

Misclassification of seriously mentally ill patients screened for interventions to address substance use may have negative impacts through under- or overestimating treatment needs.¹² Underestimation may lead to no referral for treatment, or it may result in referral for treatment that is insufficient in intensity or duration. Alternatively, overestimation may lead to an excessive level of treatment intensity, which risks noncompliance by patients, or to a wasteful utilization of resources.

Given the large unmet need for a seriously mentally ill–tailored substance use screening instrument, our team has developed, validated, and piloted a novel screening and referral algorithm for psychiatrically hospitalized individuals with serious mental illness. This algorithm uses laboratory and other data available to clinicians and does not rely on patient self-report. The purpose of this report is to present results from this effort as well as an evaluation of the feasibility of this algorithm to identify individuals who are at risk or who already have a substance use disorder and to facilitate referrals for targeted interventions in an inpatient psychiatric setting.

METHODS

The current study involved 2 aims: (1) the development and validation of a novel screening algorithm for this population and (2) a pilot implementation study. We first selected individual items from existing validated substance use screening instruments to create a new screening instrument that fits the needs of an inpatient psychiatric hospital context and hospitalized patient populations with serious mental illness. After validation, we pilot tested the algorithm in a sample of 20 hospital admissions by comparing patient assignment results from the novel screening algorithm with individual expert evaluations (gold standard) by an addiction psychiatrist. Dartmouth-Hitchcock’s Institutional Review Board (STUDY02001059) determined this study qualified for exempt not human subject research status.

Development of a Screening and Referral Instrument

Given that no instruments exist that would be appropriate for a psychiatrically hospitalized patient population with serious mental illness, due to severe psychiatric symptoms and cognitive impairments that are common in this group, we were able to identify relevant substance abuse domains and items from existing instruments that could be incorporated in developing a new instrument tailored for this patient group and setting.

As a first step in the process, we performed a literature review and assessed all existing substance use screening instruments for objective elements. We then compiled a comprehensive list of all identified objective domains while removing duplicates. These include cannabis toxicology screen result, non-cannabis toxicology screen result, substance use disorder diagnosis, current medication regimen, and legal or mandated encounter related to substance use extracted from the patient’s medical records (Table 1). Each positive item was scored as 1, and a total continuous score was calculated by summing the 5 individual items for a total score ranging from 0 to 5.

The next step in this process was to validate these selected items in our targeted population based on patient-level data from medical records extracted from a convenience sample within the study setting.

Study Setting

New Hampshire Hospital (NHH), a 202-bed inpatient psychiatric facility located in the Northeastern US with an average census of 170 patients, was the study site. NHH cares primarily for patients with serious mental illness admitted under involuntary status from emergency departments, other inpatient psychiatric units and psychiatric holding facilities, and the community. Patients carry diagnoses of schizophrenia spectrum conditions, bipolar disorders, and severe major depression. On admission, patients are allocated beds in 1 of 8 units depending on the anticipated length of stay, complexity, and overall treatment needs with patients in each unit managed by either a general psychiatrist or a nurse practitioner.

Routine Assessment of Need for Addiction Services Interventions and Data Collection

In the current study, as part of routine care at NHH, an addiction psychiatrist evaluated patients referred to the Addiction Services for any substance use interventions and treatment recommendations. This expert addiction psychiatrist evaluation is considered the gold standard for determination of patient need for substance use services. Referrals are made at any point during the duration of the inpatient treatment if the treating general psychiatrist or nurse practitioner determines that the Addiction Services may have an impact in their care. Thus, for this study, for each referred patient, an expert addiction psychiatrist conducted a formal consultative evaluation and noted whether any active intervention was recommended or implemented over the course of the patient stay. This expert evaluation entails an assessment of the patient’s reported substance use and psychosocial history, a review of the chart and laboratory test results as well as the controlled substance database, and collaboration/consultation with family and psychiatrist for collateral information. Active interventions were defined either as a pharmacologic recommendation for relapse prevention or if any meaningful one-on-one motivational enhancement work was beneficial and needed.

The study sample includes a total of 302 patients, aged 18 to 65 years, who were referred to the Addiction Services by standard clinical processes between July 15, 2019, and February 18, 2020. In parallel to the expert addiction psychiatric evaluations, our team applied the 5-item instrument—the novel New Hampshire Hospital screening and referral algorithm (NHHSRA)—to patient medical records of all the clinician-based referrals and recorded how each specific question was answered.

Algorithm Development

Development of the NHHSRA. The association between the expert addiction psychiatric evaluation for substance use interventions and calculated need based on the 5 individual instrument items for each patient referred to the Addiction Services was examined using the Pearson correlation coefficient (r). Results of the correlation matrix showed that there was low correlation (all r < 0.40) between individual items. Overall variability in the direction of association was observed. The strongest positive association was observed between patients with a diagnosis of substance use disorder and receipt of medication for treatment or relapse (Q3 and Q5, r = 0.401, P < .001), and the strongest negative correlation was observed between a positive non-THC or > 0% blood alcohol level (Q1) and a cannabis (Q2) toxicology screen (r = −0.277, P < .001; see Table 2). On the basis of the non-linear associations between individual items, we established a decision tree algorithm. Figure 1 depicts the decision tree algorithm guiding the indication of a feasible intervention by the Addiction Services.

This algorithm creates a process of screening individuals starting with the items most highly correlated with the outcome and, if those items are not present, then going down the algorithm to items less correlated with outcomes until one would screen out.

Validation of the NHHSRA. As part of the validation process, we evaluated the construct and predictive validity of the algorithm:

Construct validity. Construct validity of the NHHSRA was evaluated using descriptive statistics to assess the distribution of the summed 5-item score and comparing each item between patients who were determined by expert addiction psychiatrist assessment to need active intervention and those who were not. Independent-sample t tests were used to examine differences between the mean scores of the continuous NHHSRA sum score between those patients who were determined to need an intervention and those who were not. Chi-square tests were used to examine differences in categorical responses to individual items between the 2 groups.

Predictive validity. Predictive validity of the NHHSRA was evaluated using the receiver operating characteristic (ROC) curve to determine how well all 5 items could distinguish between patients who were determined by expert addiction psychiatrist assessment to need active intervention and those who did not. The area under the ROC curve (AUROC) of all 5 NHHSRA screening items is reported as a criterion to measure the test’s discriminative ability regarding need for intervention compared to the expert clinician assessment. A logistic regression model was used to determine the association of each item of the NHHSRA with the dependent variable assignment to active intervention (yes or no). Model 1 included only the 5 NHHSRA items, and Model 2 was additionally adjusted for age and sex (see Table 4). Maximum likelihood effect estimates, standard error, and χ² P values are reported. For all analyses, P values less than .05 were considered to be statistically significant. All analyses were performed using the statistical software package SAS (Version 9.4; SAS Institute Inc; Cary, NC).

Pilot implementation of the NHHSRA. Following the algorithm development and validation, we conducted a pilot study of the NHHSRA to determine its ability to correctly identify patients in need of substance use interventions based on a convenience sample of patient admissions. An expert addiction psychiatrist evaluated 20 consecutive admissions to the hospital to determine whether each patient would benefit from substance use interventions provided by the Addiction Services. The NHHSRA was applied in parallel to this expert addiction psychiatrist evaluation, and results were confidentially noted. The patients were then tracked during their admission to each of the 8 treatment units, and the authors noted whether or not a referral to the Addiction Services was subsequently made. Comparisons were then made by the authors between the admitted patients identified through the NHHSRA using objective data from the medical record and the patients subsequently referred by clinicians from the 8 units. Descriptive comparisons and conclusions were drawn regarding how these 3 approaches identified the group of patients deemed to be in need of substance use interventions through the initial consultative assessment on admission.

RESULTS

Item Analysis and Construct Validity

The study sample (n = 302) was 62.3% male with a mean ± SD age of 35.4 ± 11.5 years. There was no statistical difference in either the age or gender distribution of patients who received an intervention (no intervention: 37.1 ± 11.8 years, intervention: 34.6 ± 11.3 years, P = .08; women/men with intervention: 71.1%/70.2%, P = .88). The mean ± SD score on the NHHSRA was significantly higher in patients who received an intervention (3.00 ± 0.87) compared to those who did not (1.40 ± 0.63; P < .001; Table 3).

The distributions of patients with documentation confirming scores on the 5 items are listed in Table 3. The highest concordance with receiving an intervention following referral to the Addiction Services was with question 3 (Q3), followed by Q1 and Q5 (see items listed in Table 1). As the NHHSRA showed, among patients who received an intervention, 98.59% had a substance use diagnosis (Q3) compared to 33.71% among patients who were not referred, 85.45% had a positive toxicity (non-THC) or blood alcohol test (Q1) compared to 41.57% among patients with no referral, and 52.58% had documented receipt of medication-assisted treatment or for relapse preventions in their records (Q5) compared to 4.49% of patients who were not referred by clinician evaluation (Table 3).

The AUROC for the 5-item NHHSRA was 0.96, indicating high overall sensitivity (Figure 2). Q1 (positive non-THC toxicity test or blood alcohol), Q3 (substance use disorder diagnosis), and Q5 (medications for treatment or relapse prevention) were significantly associated with the likelihood of receiving an intervention (maximum likelihood estimate ± SD = 3.61 ± 0.60, 4.89 ± 0.73, and 2.78 ± 0.67, respectively; all P values < .001). Neither Q2 (+THC toxicity test) nor Q4 (revocation or violation of conditional discharge terms based on substance use) achieved statistical significance. See Table 4 for unadjusted (5 items only) and age- and gender-adjusted models.

Pilot Implementation Study of the NHHSRA

Six of the 20 admitted patients assessed were deemed to benefit from interventions by the expert addiction psychiatrist evaluation. The same 6 patients were also identified for need of intervention by the electronic health record data-driven NHHSRA. During hospitalization of these 20 patients, 6 were eventually referred to the Addiction Services by standard (non-expert) clinician-based referral. Only 2 of these patients were among the 6 identified by both expert and NHHSRA evaluation. For the other 4 patients who were referred over the course of their hospitalization, none were identified by the NHHSRA or by the expert addiction psychiatrist evaluation. Standard “clinician-based referral” hence failed to refer 4 individuals for whom interventions would have been feasible and erroneously referred 4 patients who were not in need of Addiction Services based on expert clinical assessment.

DISCUSSION

The current study showed that an objective screening and referral algorithm (NHHSRA) had high sensitivity, allowing for broad yet accurate identification of those in need of substance use interventions independent of severity of substance use and level of motivation for treatment.

Psychiatric disorders and substance use frequently co-occur. Epidemiologic surveys show illicit use of substances is proportionally related to the severity of one’s mental illness, and it predicts the need for inpatient psychiatric hospitalization.¹³ As such, best practice supports addressing both the substance use and the psychiatric condition simultaneously on identification rather than in isolation or sequentially.¹⁴ In the inpatient psychiatric setting, when patients with substance use are referred for targeted interventions they are twice as likely to engage in ongoing substance use treatment on discharge.¹⁵ The first step in substance use interventions is identifying those who would benefit through screening, as frequency of treatment is correlated with use of screening tools.¹⁶ Existing screening instruments are not appropriate for the seriously mentally ill population, and current processes for identifying patients who would benefit from substance use interventions are clinician dependent, resulting in a referral process that is highly variable and affected by many contributing factors.

The NHHSRA is the first instrument developed and validated for the seriously mentally ill population and fills a gap in the availability of screening instruments applicable to psychiatrically hospitalized individuals with serious mental illness. In our validation study, the analysis demonstrates that the items in the NHHSRA have high construct validity and correlate well with expert addiction psychiatric evaluations of patient need for substance use interventions. The NHHSRA also demonstrated high predictive validity with a high sensitivity for identifying patients whom expert addiction psychiatric clinicians would refer for intervention. The NHHSRA has the potential to impact the care of psychiatrically hospitalized patients with serious mental illness. Through implementation of this algorithm, clinicians working with the seriously mentally ill patient population can rapidly identify those in need of targeted substance use interventions much earlier in their psychiatric admission process with high accuracy. Such a streamlined process of identification and referral can reduce variability and eliminate human error and biases when combined with patient self-report screening. Use of the objective and automated NHHSRA in hospital admissions additionally has the potential to reduce wasteful utilization of resources, leading to increased access to and efficiency of addiction services.

Early substance use screening instruments were developed to determine drug dependence, particularly to specific substances. Over time, the goal shifted to identifying risky and hazardous use much earlier among those at risk.¹⁷ All currently available substance use screening instruments rely on subjective input from the patient, either orally or through written questionnaires, and most were designed for the primary care outpatient setting.¹⁸ There are also instruments crafted for classification accuracy with a particular target population in mind (age, race, language, etc), time allocated for assessment, and whether it is clinician- or self-administered. The studies in which these instruments were validated excluded patients with psychiatric disorders. Furthermore, acutely ill psychiatric patients have been shown to be unable to complete such structured interviews through self-reports or interview instruments.¹⁹ Additionally, many of the subjective items of these instruments can be confusing to seriously mentally ill individuals, leading to poor utility of these instruments for the seriously mentally ill population.²⁰ The Dartmouth Assessment of Lifestyle Instrument (DALI),²¹ which screens specifically for alcohol, cannabis, and cocaine use in the seriously mentally ill group, relies on subjective elements acquired through a clinician interview. It has shown superiority over other validated instruments not designed specifically to focus on the seriously mentally ill population. Limitations of the DALI, however, include its reliance on self-report, which restricts it to the higher functioning end of the serious mental illness spectrum and to patients who are forthcoming about their substance use and who are willing to participate in the screening. Unlike all existing screening instruments, the NHHSRA is tailored for the serious mental illness population and its needs, being able to identify patients who would merit substance use interventions involving use of any substance based strictly on objective elements.

Future research directions involve developing additional instruments to stratify the type of substance use intervention each individual patient may need. Interventions offered by the Addiction Services are heterogeneous and include motivational enhancement interventions delivered by a recovery coach or substance abuse counselor, relapse prevention skills and other forms of talk therapy delivered by a substance abuse counselor, and pharmacologic modalities for relapse prevention treatment by an addiction psychiatrist. The NHHSRA does not distinguish between each patient’s candidacy for any or all of these interventions.

One of the limitations of this study is the small sample size in the pilot implementation study component. Although 20 consecutively admitted patients were evaluated, increasing the sample size was beyond the scope of this current study. Additionally, the study involved only a single site, an inpatient state hospital caring for patients with serious mental illness; hence, this algorithm is not validated for use outside of this treatment setting and patient population for the purpose of identifying those who would benefit from further substance use interventions. Widespread applicability of the NHHSRA for substance use interventions may require systematic implementation efforts. Implementation may, for example, be hindered by the lack of availability of addiction specialists, since of the 836 board-certified addiction psychiatrists in the entire US, the majority are concentrated in the Northeast, while 92.8% of the nation’s counties do not have any.²² Worthy of mentioning here is that our study used a single addiction psychiatry expert as the gold standard. Lastly, the NHHSRA was validated for patients between ages 18 and 65 years and relies on elements and information to be available prior to a patient’s admission to the hospital (ie, urine toxicology and documentation of a substance use disorder), which some facilities may not have access to and also some patients may not be willing to cooperate with. Additionally, the substance use disorder diagnosis present in the record is not always free from human error and subjective judgment. Future research could evaluate adaptations to the NHHSRA as it is implemented in new psychiatric settings (eg, community inpatient psychiatric units, emergency rooms, mobile crisis services) with new patients.

CONCLUSION

The NHHSRA is the first screening and referral algorithm for substance use services designed specifically for individuals with serious mental illness in an inpatient psychiatric setting. Identification of patient need via the NHHSRA is superior to standard clinical-based referrals to the Addiction Services that occur with great variation and inaccuracy during patient hospitalization. The algorithm uses objective data, is brief and easy to use, and has high specificity and sensitivity in the population tested. Future research should focus on a multisite trial using more rigorous experimental procedures to evaluate the effectiveness and generalizability of the algorithm.

Submitted: September 1, 2022; accepted December 8, 2022.
Published online: May 18, 2023.
Relevant financial relationships: The authors have no conflicts of interests or disclosures relevant to the content of this manuscript.
Funding/support: None.
Acknowledgments: The authors would like to thank Gregory J. McHugo, PhD, Emeritus Professor of Psychiatry at Dartmouth’s Geisel School of Medicine, for all the guidance and mentorship provided on this project. The authors would also like to acknowledge the contribution made by Karen Goodman, MSLIS, MA, medical librarian at the Dorothy M. Breene Memorial Library at New Hampshire Hospital, as she assisted with the literature search and procuration of the articles needed for this article. Dr McHugo and Ms Goodman have no relevant financial relationships relevant to the subject of this article. The preparation of this report was financed under a contract with the State of New Hampshire Department of Health and Human Services, with funds provided in part by the State of New Hampshire.

Clinical Points

• Substance use among patients with serious mental illness is challenging to identify using screening instruments that rely on self-report.
• The authors developed and validated an entirely objective substance use screening algorithm, the New Hampshire Hospital screening and referral algorithm (NHHSRA), which in a small pilot study was found to be superior to standard clinical-based identification.
• The NHHSRA can improve objective and timely identification of substance use in a serious mental illness population, helping to facilitate treatment.