Module 2 Chapter 4: Locating Intervention Evidence

This chapter is about identifying approaches and resources for evidence used in making social work practice decisions. As you learned in earlier modules, evidence-based practice incorporates practitioner experience and expertise, client preferences and circumstances, and the best possible research evidence available. The aim of this chapter is to answer the question: “If social workers are committed to using the best available evidence, where do they locate it?” General approaches and resources for locating evidence to answer practice questions are discussed.

In this chapter you read about:

• limitations of literature reviews
• systematic reviews
• scoping reviews
• meta-analysis reviews
• locating individual research studies.

Limitations of Literature Reviews

The introduction to an empirical article often includes a literature review conducted by the authors. In addition, literature reviews sometimes appear in the published literature. These may provide important information for practitioners to consider, however they are seldom sufficient by themselves. Bronson and Davis (2012) summarized the problems facing social work practitioners who rely on a literature review to inform their practice:

• a great deal of the research literature lacks sufficient intervention detail to inform practitioners about howto implement the best practices reported in the articles;
• outcome studies often present the practitioner with contradictory results;
• authors’ reviews often lack sufficient background concerning underlying assumptions and biases that may influence the credibility and relevance for practice decision-making, particularly within specific practice settings or with specific types of clients.

A literature review is only as thorough as the search for literature; an extensive review is more informative than a search limited by factors such as:

• the discipline where the literature appears, ignoring literature from other professions and disciplines;
• the time frame covered in the search (e.g., limited to just a few years);
• the nation where the work was conducted (e.g., an “Americentric” search might miss important contributions from other countries);
• only reviewing what is easily accessed from a library.

Generic literature reviews also may be conducted in an unsystematic fashion, leaving a reader uncertain as to the extensiveness of the coverage. This leads to discussing the qualities of more systematically conducted types of reviews: systematic, scoping, and meta-analysis reviews.

Systematic Reviews

The topic of systematic reviews was briefly introduced in our earlier course. In this chapter, you are provided with greater detail about these particularly useful tools for practitioners in many disciplines. Systematic review is a methodical process of summarizing evidence that results in a product: the systematic review report. What the systematic review method provides is:

• “An attempt to include all relevant research, including published and unpublished studies and those with contradictory findings;
• A systematic method of collecting data from the existing studies to allow for meaningful synthesis;
• Explicitly stated inclusion and exclusion criteria for identifying relevant research; and
• Transparent search methods to allow for replication”(Bronson & Davis, 2012, p. 15).

The application of systematic review methodology is how systematic reviews differ from general literature reviews. Systematic review reports are published in various dissemination outlets, including:

• professional journals;
• government agencies and government resources, such as:
  • various institutes of the National Institutes of Health/NIH (National Institute of Mental Health/NIMH, National Institute on Alcohol Abuse and Alcoholism/NIAAA, National Institute on Drug Abuse/NIDA, National Institute of Child Health and Human Development/NICHD, National Institute on Aging/NIA, National Institute of Environmental Health Sciences/NIEHS, and National Institute on Minority Health and Health Disparities/NIMHD),
  • Substance Abuse and Mental Health Services Administration/SAMHSA,
  • Agency for Healthcare Research and Quality (AHRQ),
  • Office of Justice Programs/OJP,
  • Office of Juvenile Justice and Delinquency Prevention,
  • Child Welfare Information Gateway at childwelfare.gov/topics/responding/child-protection/evidence/,
  • National Registry of Evidence-based Programs and Practices, NREPP at https://www.samhsa.gov/nrepp,
  • U.S. Department of Education’s What Works Clearinghouse/WWC at https://ies.ed.gov/ncee/wwc/and,
  • Centers for Disease Control and Prevention/CDC;
• United States Preventive Services Taskforce (USPSTF) at www.uspreventiveservicestaskforce.org;
• Campbell Collaboration at www.campbellcollaboration.org/;
• Cochrane Collaboration database at http://us.cochrane.org/and the Cochrane Community Database of Abstracts of Reviews of Effects (DARE);
• the patient information resource, Informed Health Online, at www.informedhealth.org/;
• PubMed database of systematic reviews at the U.S. National Library of Medicine in NIH at https://www.ncbi.nlm.nih.gov/pubmed/.

Many of these sources address medical practice questions but may also include behavioral health questions of interest to social work practitioners. The Campbell Collaboration may be of particular interest to social work practice and policy (https://www.campbellcollaboration.org/), including umbrella topics such as:

• crime and justice
• disability
• education
• international development
• social welfare
• knowledge translation and implementation

One way to locate systematic reviews about a specific practice topic or question is to place the phrase “systematic review” in your electronic search terms. Let’s consider a practice example where a social worker needed to find evidence to inform practice decisions concerning the practice question: what is the best intervention for promoting development in young children with autism? The social worker wanted to inform a family about options for early and intensive intervention to improve outcomes for their 4-year old child with autism. So, the social worker entered the search terms: “systematic review parenting education intervention” into Google Scholar. The result appears in Figure 3-1:

Figure 3-1. Result of a systematic review search for autism interventions

Reading the abstract for this review indicated that the objective was a good match for the practice question, and the social worker learned generally how the review was conducted (methods). While this looked promising, the results and the conclusions of the published review were somewhat disappointing because the strength of the evidence the reviewers identified was generally low. Two intervention approaches showed benefits in some aspects of development for some children (the Lovaas-based intervention and the Early Start Denver Model), but which children did and did not benefit was poorly understood (Warren et al, 2011); parent intervention showed some short-term gains, but long-term durability was unclear. So, the social worker did not yet have a strong evidence-based answer to the practice question and turned back to the search for systematic reviews related to autism.

The next article that came up reviewed nutritional and dietary intervention strategies (Figure 3-2).

Figure 3-2. Next result of a systematic review search for autism interventions

In reading the abstract, the social worker was not as certain about the goodness-of-fit with the practice questions regarding promoting development in young children with autism but continued reading through the contents of the abstract. Again, the strength of the evidence was low, the results were inconsistent and were not demonstrated to persist over time (Sathe, Andrews, McPheeters, & Warren, 2017). The social worker still had little evidence to inform practice decisions.

Next the social worker considered the possibility that perhaps the search question was too broad and general. Autism is a complex problem with many variants and theories about etiology and factors that influence its course, thus it has many possible avenues for intervention. The social worker then considered what the family described as what they were most interested in pursuing with this child at this time. They were interested in behavioral approaches rather than medication. The next reference that the social worker located was the Systematic Review of Early Intensive Behavioral Interventions for Children with Autism (Howlin, Magiati, & Charman, 2009). This review applied inclusion criteria that the study’s population had to be children under the age of six years, which made it a good fit for the 4-year old family member. The review results indicated that there exists evidence to support the effectiveness of the Early Intensive Behavioral Intervention (EIBI) approach initially developed at UCLA and offered through various programs elsewhere. The review also specified that the observed effectiveness was not consistent across all children, and observed effects typically appeared within the first 12 months of intervention. The results of the review were similar to those for a similar systematic review conducted by another team, and this confirmation served to enhance the social worker’s confidence in the results. This information gave the social worker a starting place to seek local programs based on this approach in referring the family, and a timeline for evaluating the impact of the intervention (12 months). However, the social worker might also want to extend the literature search to see whether more recent evidence exists about this or other relevant interventions.

The results of the EIBI autism review were based on over 640 published studies, making the single review highly useful to a practitioner who does not have time to locate, read, critically analyze, and draw conclusions from so many pieces. Furthermore, individual practitioners may find it difficult to be systematic in the process of critically reviewing many pieces of literature. It helps to understand what goes into a systematic review to know how to trust the authors’ critical review process in place of one’s own critical analysis.

Process of a Systematic Review

Authors of a research skills workbook delineated 12 stages in the systematic review process they called “meta-synthesis” (Corcoran & Secret, 2013). Here is an overview of what they outlined, integrated with content from other resources about systematic reviews (Bronson & Davis, 2012; Higgins & Green, 2011; Littell, Corcoran, & Pillai, 2008).

• Develop specific, answerable research question. The project begins with clearly formulated objectives and the research question is specific. The COPES or PICO approach you learned in Chapter 2 might be applied at this stage.
• Form a review team. The team should include a diverse skill set, covering a range (substantive on the content area and relevant methodological/technical expertise).
• Determine explicit inclusion/exclusion criteria. These criteria concerning which reports ti include in the review should be specific about the conditions, circumstances, populations, types of intervention, intervention settings, outcome indicators, and study design/approach.
• Develop a protocol. This is a set of written guidelines about procedures and methods the team will use in the review process. This is done prescriptively (before the work begins), it is not meant to be a post hoc description after the work is completed.
• Implement a reproducible strategy for study identification. This strategy isdefined in the protocol and incorporates the inclusion/exclusion criteria that were established in prior stages. The strategy needs to result in identification of all relevant studies, including those that might be unpublished. The review should include outcomes that were adverse, beneficial, or “no observed difference” on the meaningful (primary) outcomes.
• Screen titles and abstracts. The team applies their pre-determined criteria and protocol to determine which of the previously identified studies are potentially relevant to the review.
• Retrieve potentially relevant study reports. The team utilizes strong library and electronic search approaches to gain access to the reports (including unpublished reports) identified as potentially relevant. A strong system of organizing the secured reports is important at this stage, as it can reduce duplicated effort.
• Team members judge each study. As in the case of coding qualitative data, two or more individuals should independently determine whether each report represents an eligible study—based on the predetermined eligibility criteria. Reviewers document their decisions, and discrepancies are resolved by the team.
• Data from eligible studies are extracted. For each study deemed eligible, data about the study are copied into standardized forms (created as part of the protocol). As in the judging phase, this is done by at least two team members, and disagreements are resolved by the team.
• Rate study quality. For each study deemed eligible, at least two raters critically appraise its quality (scientific integrity and design adequacy) and its qualities (sampling, data collection methods, results, and conclusions). As before, this is documented on standardized forms and disagreements are resolved by the team.
• Describe key features of included studies. Tables are used to document descriptions of these elements, created as part of the review protocol. Common categories are: author’s/authors’ purpose(s)/study aims; study design and approach; sample information; measures/data collection; data analysis/results; conclusions.
• Synthesize results. Team members work with the data accumulated in the prior steps to draw conclusions, answering the initial research question.

Strength of Evidence in a Systematic Review

Systematic reviews differ, to some extent, in terms of eligibility criteria concerning strength of evidence (scientific rigor) of the studies included. Study replication, where different investigations lead to the same observed results, increases confidence in reliability of the intervention study findings. Also, interventions found to be effective in different contexts (with different populations, in different settings, and/or by different service providers) are considered especially meritorious in terms of strength of the evidence.

In many instances, the only studies included in a systematic review are randomized control trials (RCTs) since these are believed to represent the strongest evidence possible (internal validity). This is the case for the Cochrane review system, for example. A table describing levels of evidence summarizes a prevailing belief system concerning the strength of evidence different studies present (adapted from Ackley, Swan, Ladwig, & Tucker, 2008, p. 7 and the Winona State University library https://libguides.winona.edu/c.php?g=11614&p=61584). In this framework, the “best” evidence for informing practice is associated with the top of the hierarchy (Level 1), the “weakest” evidence is at the bottom (Level 7), progressing from “green means go” toward “orange means caution.” This table follows the decision rules presented in the Oxford guidelines concerning levels of evidence (Centre for Evidence-Based Medicine, https://www.cebm.net/2009/06/oxford-centre-evidence-based-medicine-levels-evidence-march-2009/).

Scoping Reviews

“The scoping review has become an increasingly popular approach for synthesizing research evidence” (Pham et al., 2014, p. 371). This approach is used when the existing literature in an area is wide ranging, complex, and diverse in terms of study approaches, methods, and levels of evidence available. Scoping reviews apply systematic procedures and review criteria, resulting in a “map” depicting an area of research—particularly an emerging area where the approaches to gathering evidence remain diverse.

• the research question tends to be broader, less well-defined in a scoping review;
• greater “tolerance” allowed in the inclusion/exclusion criteria related to quality or level of evidence—this allows reviewers to include evidence of greater diversity (Arksey & O’Malley, 2005).

Meta-Analysis Reviews