Module 2 Chapter 4: Reviewing Empirical Articles

After carefully reviewing the source you have located, it is time to critically review the piece itself. In this chapter, you will read about:

• Steps in reviewing different sections of an empirical article
• The importance of maintaining a critical perspective on what you are reading—being an “active reader”

The seven steps considered in this chapter relate to the structure of typical journal articles published in social work and allied discipline journals. The structure is familiar to anyone who works with the American Psychological Association (APA) guide to how journal articles are written and structured, thePublication Manual of the American Psychological Association(APA, 2009 for the sixth edition).

Step 1. What is in a Name? Reviewing the title

Authors vary tremendously in their approach to titling their work, much as parents differ markedly in their approach to naming their babies. Ideally, a title is sufficiently precise and specific to tell a reader what the article is about. Sometimes titles have attention catching phrases added at the front or back end. Ideally, a title is also not overly elaborate and lengthy.

For example, the following article titles clearly and succinctly communicates what each article is about.

• “Race and ethnic differences in early childhood maltreatment in the United States” (Lanier, Maguire-Jack, Walsh, & Hubel, 2014).
• “Tracking the when, where, and with whom of alcohol use: Integrating ecological momentary assessment and geospatial data to examine risk for alcohol-related problems” (Freisthler, Lipperman-Kreda, Bersamin, & Gruenewald, 2014).
• “Meeting them where they are: An exploration of technology use and help seeking behaviors among adolescents and young adults” (Cash & Bridge, 2012).
• “A systematic review of the relationship between internet use, self-harm and suicidal behaviour in young people: The good, the bad and the unknown” (Marchant et al, 2017). [Note the word “behavior” is spelled “behaviour” in the United Kingdom, but without the “u” in the United States.]

Examples of (hypothetical) article titles that are non-communicative or miscommunicate include:

• “The problem of drugs in America.” This title is not sufficiently specific for an empirical article. First, a reader does not know if this is about how the pharmacology industry manufactures drugs (quality control), an epidemiology report about the scope of the problem, an etiology report about theories related to the causes of the problem, a test of a sociological, psychological, or biological theory. It is more likely a title about an opinion piece, a general book chapter, or even an entire book.
• “The effectiveness of PFI and MI in AOD treatment.” This title is swimming in jargon, making it difficult to interpret without looking into what each acronym means, and also makes it difficult to locate in the literature. (PFI is Personal Feedback Intervention, MI is motivational interviewing, and AOD is about alcohol and other drugs.) In addition, there exists a problem of ambiguity with acronyms: BPD could refer to either bronchopulmonary dysplasia, a lung complication common among prematurely born infants, borderline personality disorder, or bipolar disorder (both are mental disorders).
• “The effects of self-esteem on high school student retention and drop-out.” This title is only good if the study design and methods actually allow for a causal inference. If the study design only allows for conclusions about the existence of a relationship between these two variables, the title is a poor choice—the word “effects” implies causality and misrepresents the study.

    Step 2. What is it about? Reviewing the abstract

Authors present a summary of their manuscript in a brief abstract that appears at the start of a published article. Abstracts are also published in a number of indexing and abstracting resources, making them relatively easy to access. Journals limit the length of abstracts, usually to somewhere between 150-250 words depending on the journal. This makes it challenging to explain the important aspects of the manuscript, with enough detail to be clear, but without the luxury of unlimited space to present nuances. An abstract should address the following points:

• Study’s purpose, research aims, questions, and/or hypotheses
• Study approach
• Study design and methods (including study participants and measures)
• Data analysis and key results
• Key implications of the study results.

As a reader, the abstract should provide enough information for you to determine whether it is relevant for you to pursue the full article. An article abstractis not sufficient information for you to evaluate the evidence, even the evidence that appears in the description of results! To evaluate the evidence, you need to acquire and review the full article.

This point is so important that it warrants repeating:

An article’s abstract is not sufficient information for you to evaluate the evidence, even the evidence that appears in the description of results! To evaluate the evidence, you need to acquire and review the full article.

            A Note about research abstracts. An excellent search of literature will often turn up published research abstracts. In this case, there will not be a full article to locate. The abstract describes a conference presentation, and these are the precursors to publishing an article about a research study. Many professional organizations publish these abstracts in a special journal issue, sometimes a “supplement” to the journal. The next step for your search, if the title and abstract seem relevant, will be to determine whether a paper was ever published based on the study described in the abstract. This is a place where you would search by author name(s) rather than by subject or topic alone.

Step 3. What is the rationale and background knowledge? Reviewing the introduction

Once you have acquired an article that seems interesting and relevant (based on its title and abstract), you will next encounter its introduction. The purpose of an introduction is to provide the reader with a background orientation to the study that was conducted. This might include background information regarding the scope of the problem being addressed by the research. It should certainly provide the reader with a review of literature related to the topic and research questions. This might include an overview of the theory or theories related to the research that was conducted. In the end, the reader should understand the following:

1. Why was the study was undertaken, why was it important, why does it matter?
2. What was known from the literature that informed the study’s development?
3. What knowledge gap or gaps did the study aim to fill, or what did the study aim to contribute to the body of knowledge?
4. What research questions did the investigators address in their study?

The introduction often also informs readers about the study’s approach (e.g., qualitative, quantitative, mixed methods) and type of study that was implemented (e.g., exploratory, descriptive, experimental). After reviewing the introduction, you should have an even better idea of whether the article is relevant for your purposes.

Step 4. What happened? Reviewing the methods

If it was not made clear in the introduction, the study approach and type of study should be made explicit in the methods section of the article. The methods section, at a minimum, needs to explain who participated in the study and how data were collected. In a quantitative study, the study design is also described in the methods section; in a qualitative study, the type of study is described. There are three basic sub-sections in the methods section of an empirical article: study participants, study measures, and study procedures.

Study participants. A methods sub-section describes who actually participated in the study, including numbers and characteristics of the study participants, as well as the pool from which these participants were drawn. The purpose of this sub-section in describing a quantitative study is to inform readers about generalizability of the study’s results and inform other investigators about what they would need to do to replicate the study to determine if they achieve similar results. Authors may present some of the description material in the form of tables with information about numbers and proportions reflecting categorical variables (like gender or race/ethnicity) and distribution on scale/continuous variables (like age). The method of selecting these participants should be clear along with any inclusion or exclusion criteria that were applied. The participant response rate might also be calculated as the number of participants enrolled in the study divided by the number of persons eligible to be enrolled (the “pool”), multiplied by 100%. Very low response rates make a study vulnerable to selection bias—the few persons who elected to participate might not represent the general population. In a qualitative study, the study participants section again describes how the individuals were selected for participation, and details describing these individuals are provided. Generalizability is not a goal in qualitative studies, but information about study participants should provide an indication to a reader of how robust the results might be. Robust descriptions come from participants who exhibit a range of defining and/or experiential characteristics. Finally, regardless of study approach, authors typically make evident that the study was reviewed by an Institutional Review Board for the inclusion of human participants.

Study Measures.Another methods sub-section explains how the data were collected. For quantitative studies, the data collection instruments used to measure each study variable are described. If the tools used for data collection were previously published, authors cite the sources of those tools and published literature about them—their reliability and validity, for example. The authors may also summarize literature concerning how the measures are known to perform with the specific type of study participants involved in the study—for example, different ages, diagnoses, races/ethnicities, or other characteristics. For qualitative studies the interview protocols or questions asked of participants are described in detail. In observational data collection studies, the approach to recording and scoring/coding observed behavior are described. In any case, the approach to data collection or measurement is described in sufficient detail for a reader to critically appraise the adequacy of the data collection approach and conclusions that can be drawn from the data collection process, and for other investigators to be able to replicate the study should they wish to confirm the results.

Study procedures.Sometimes study procedures is a separate methods sub-section and sometimes this content is incorporated into the participants and measures sub-sections. This sub-section includes information about activities in which the study participants engaged during the study. In a quantitative, experimental study, the methods utilized to assign study participants to different experimental conditions might be described here (i.e., the randomization approach used). Additionally, procedures used in handling data are usually described. In a quantitative study, investigators may report how they scored certain measures and what evidence from the literature informs their scoring approach. In a qualitative study, details about how data were coded are reported here. Procedures for ensuring inter-observer or inter-rater reliability and agreements will also be reported for either type of study. Regardless of the study’s research approach, a reader should come away with a detailed understanding of how the study was executed. As a result, the reader should be sufficiently informed about the study’s execution to be able to critically analyze the strength of the evidence developed from the methods that were applied.

Step 5. What was found? Reviewing the results

The results section is where investigators describe the data they collected, how it was analyzed, and what was observed in the data.The structure and format of the results section varies markedly for different research approaches.

 Qualitative methods results description.The nature of qualitative research questions and methods leads to data that are richly descriptive. The results derived from the data are, therefore, generally descriptive in nature. There may be a great deal of direct quotes, presenting information in participants’ own words. Descriptions may include thematic or concept maps constructed by the study investigators as a means of “sense making” from the data. If statistics are included, they tend to be of a descriptive nature—perhaps demonstrating the frequency with which certain results were observed in the data. The results may include tables or figures representing results. Ideally, a reader can identify the way that reported results relate to the research aims or questions originally asked by the study.

Quantitative methods results description. The nature of quantitative research questions and methods leads to numeric data that can be summarized using various forms of statistical analyses. The results section of a quantitative study report will describe which statistical analyses were utilized, and should indicate the rationale for selecting those analyses, as well as discussing how well the data were suited to those analyses. Analyses that involve hypothesis testing will indicate the statistical support for conclusions drawn from the data (i.e., descriptive statistics, test statistics, significance levels, and confidence intervals). The results may be presented in a combination of text descriptions, tables, and figures. An informed reader should be able to determine the appropriateness of the statistical approaches used in the analyses and the conclusions drawn from those analyses. Ideally, study results are presented in association with each study question or hypothesis as it is answered. Problems encountered with any specific analyses are also reported here, such as when data were not suitably distributed, sample sizes were inadequate, or assumptions underlying specific types of analytic approaches were violated.

Mixed methods results description. As mixed methods approaches continue evolving, so are creative ways of presenting results of studies that integrate qualitative and quantitative approaches. Results for a study employing mixed methods are often presented question-by-question. Where qualitative questions were addressed, the descriptive results will be presented as outlined above. Where quantitative questions were addressed, the numeric and statistical results will be presented as outlined above.

Regardless of study approach, a reader should have a clear understanding of the way data were analyzed and results of those analyses. This is not a place where authors have drawn conclusions about the implications of those results—that belongs in the article’s discussion section.

Step 6. What was concluded? Reviewing the Discussion

In the end, the authors will offer their interpretation of the evidence described in their Results section. This discussion should include several elements:

• A brief overview summary of the key results.
• Discussion of how each key result relates to the study aims, questions, and/or hypotheses.
• Discussion of how the observed results relate to the previous existing literature (are they mutually confirming or contradicting), if the study results were completely new contributions, or if they were ambiguous and no conclusions could be drawn.
• Discussion of the study’s methodological or analysis/results limitations.
• Practical implications of the study results for practice and for future research.

It is important to remember that the discussion is the authors’ own interpretation of the results. This, again, is a place where readers must apply their own critical analysis to the study implications. For example, sometimes authors get a bit carried away with their interpretation and make suggestions that are not supported by the evidence in their studies. Or, they may not have gone far enough, and you see potential implications that they did not.

Step 7. Where are other relevant pieces? Reviewing the reference list

As you search for relevant literature, you might want to review the reference list to an article that you found to be relevant. Sometimes your own search methods and search terms might have missed some important items that the article’s authors were able to identify. This review will provide you with titles to consider, and possibly you will recognize the names of key scholars in the topic area. You can then pursue these background resources as part of your own search.