6-Evaluating Sources

2. Evaluating for Relevancy

Relevant sources are those that pertain to your research question. You’ll be able to identify them fairly quickly by reading or skimming particular parts of sources and maybe jotting down little tables that help you keep track. We’ll show you how below, including where to look in specific kinds of sources and what questions to ask yourself as you do.

One thing to consider early on as you make inferences about relevancy is the effect that timeliness– called a source’s currency–should have on deciding whether a source is relevant. Sometimes timeliness has a lot to do with relevancy; sometimes it is less important. Your research question and your discipline will determine that.

For instance, if your research question is about the life sciences, you probably should consider only the most recent sources relevant for citing because the life sciences are changing so quickly. There is a good chance that anything but the most recent sources may be out of date. So it’s a good idea to aim for life sciences sources no more than 5 years old. (An example of a discipline that calls for even newer sources is computer security.)

Sometimes emergencies change the schedule of what is recent enough. For instance, when the Covid-19 pandemic started, it was incredibility important for scientists to share their research information as quickly as possible. At that time, scientific information about Covid-19 could become outdated in weeks or months–before the peer review process was barely started.

Lives were at stake and for that reason, scientists started publishing their new research results on Covid-19 as preprints—publications of results that had not yet been peer-reviewed–in an attempt to have them be useful faster. Nonetheless, after preprint publication, the peer review process continued for much of that research.

But pre-prints didn’t start with the Covid pandemic. Around for more than 30 years and now at Cornell University, arXiv is a free distribution service and an open-access archive for more than two million scholarly articles first published as preprints in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics. Materials on the site are not peer-reviewed by arXiv itself. (arXiv is pronounced archive.)

Before using preprints as sources, talk with your professor about whether she or he recommends their use in your situation.

Many sciences other than life sciences primarily use newer content under 10 years old. But not always. That’s because the history department is not alone in valuing older content. For instance, mathematics is a discipline that makes heavy use of older content. So how important the currency of your sources is will depend on your research question and your discipline. Your professor can guide you about your own situation.

In most cases, it’s best not to use a hard and fast rule about how current your sources have to be. Instead, consider your discipline and research question and do some critical thinking. For example, suppose your research question is about the Edo Period in Japan (1603-1868) or about Robert Falcon Scott, who explored the Antarctic from 1901-1913. In these cases, an item from 1918 might be just as useful as an item from 2018 (although new information may have been found in the 100-year gap). But something from 1899 about Antarctica or from 1597 about Japan would not be current enough for these research questions.

These examples also give you two more clues about how to treat the timeliness or currency of sources as you consider relevance:

  • Because of how long ago they lived or occurred, it would be unusual for many sources on Robert Scott or the Edo Period to have been published very recently. So, unlike sources for the life sciences, whether a source is very recent should probably not determine its relevancy to research questions about Scott or the Edo Period.
  • Primary sources might be considered especially relevant to many humanities and other non-science research questions. For disciplines in the humanities, the phrase primary sources refers to sources created at the same time as something under study—in this case, things such as Scott’s diaries and expedition photographs, as well as paintings, literature, clothing, and household items from the Edo Period. They go a long way to explain faraway people and times. (See Primary, Secondary, & Tertiary Sources.) On the other hand, when science disciplines use the phrase primary source, they usually mean where they primarily find the information they consider valid—in research journals.

EXAMPLE: TED Currency

Check out how currency is handled on TED. This site provides videos of speakers talking about new ideas in technology, entertainment, and design. (That’s what TED stands for.) Some videos are labeled “Newest Talks,” and TED tells when every video was recorded. That’s because currency matters with TED Talks.

For your own sources for which timeliness matters, see the section below called Where to Look, which includes where to look in websites, articles, and books for information about a source’s currency.

Time-Saving Tips

Instead of thinking you have to read all of every source in order to figure out whether it’s relevant, read or skim only parts of each source. If you’re looking at the right parts, that should give you enough information to make an educated guess about relevancy and currency.

But what should you be looking for as you do that reading and skimming? One way to figure that out is to first parse your research question so that you can figure out its main concepts. (This is like identifying main concepts in your research question in order to search precisely, as we advise in Chapter 4.)

For instance, suppose your research question is: How does having diverse members in a group increase the critical thinking of the group?

What are this question’s main concepts? Our answer is: group diversity and critical thinking.

So when trying to judge which sources are relevant to these main concepts, you would assess whether each source you’ve found pertains to at least one of these main concepts. We recommend you jot down a little table like the one in the example below to keep track of which sources address each main concept.

To be considered relevant to your research question, a source wouldn’t necessarily have to cover all of your main concepts. But finding sources that do is ideal. Otherwise, you just have to make do with what you’ve got. Don’t forget that each source would have to pass the currency test, too, if the currency is important to your research question. So it saves time to record your decisions about the sources’ currency on your tables, too.

EXAMPLE: Sources’ Main Concepts and Currency

Research question: How does having diverse members in a group increase the critical thinking of the group?

Sources and Currency
Currency Okay Group Diversity Critical Thinking
Source A title X X
Source B title X
Source C title X X X

The table in this hypothetical example indicates that both Sources A and C are relevant because each pertains to at least one main concept from the research question. Currency doesn’t seem to matter much to our research question, so all three sources were marked current. But since currency is all that Source B has to offer, it is not relevant for this project.

If you do make little tables for relevance, it’s probably a good idea to hang on to them. You might find them helpful later in your research process.

Where to Look in Websites, Articles, and Books

The information below tells where to look and what questions to ask yourself to assess the relevancy of articles, books, and websites. The name of a source seldom tells you enough about its relevance, so whatever you do, don’t stop evaluating after looking only at a website’s name or the title of another source.

Save time by looking in particular places in sources for information that will help you figure out whether the source is relevant to your research project. Much of our advice below comes from “Speedy Reading” in The Craft of Research, second edition, by Wayne Booth, Gregory Colomb, and Joseph Williams, University of Chicago Press, 2003, pp. 108-109.

On a website, check the name of the website and its articles for clues that they contain material relevant to your research question. Consider whether time should have an impact on what information can be considered relevant to your research question. If so:

  • Skim any dates, datelines, What’s New pages, and press releases to see whether any website content works with the time considerations you need.
  • Check for page creation or revision dates that you find. What you’ve already learned from other sources can also help. For instance, you may know that the information covered by a particular website, which seems relevant, is no longer considered the latest thinking. In that case, you could mark it irrelevant on your little table.
  • Skim any site map and index on the website for key words related to your research question.
  • Try the key words of your research question in the search box. Do you see enough content about your keywords to make you think parts of the website could be helpful?

For a research journal article, magazine article, or newspaper article, think about the title. Does it have something to do with your research question? Consider whether time should have an impact on what sources can be considered relevant. If so:

  • Is the publication date of any of these three kinds of articles within your parameters?
  • Skim the abstract of a journal article to see whether the article works with the time considerations you need. For instance, if there is a time period in your research question, does the article address the same time period or was it created during that time period?
  • Look at the abstract and section headings in a journal article or the early parts of a newspaper or magazine article to locate the problem or question that the article addresses, its solution, and the outline of the article’s argument for its main claim. Can those help answer your research question? Do they make it seem as if the article will give you information about what others have written about your research question? Do they offer a description of the situation surrounding your research question?
  • Do the journal article’s introduction and conclusion sections help you answer your research question and/or offer a description of the situation surrounding your question so you can explain in your final product why the question is important?
  • Check whether the journal article’s bibliography contains keywords related to your research question. Do the sources cited by the bibliography pertain to your research question? (Bibliographies are especially good places to look for sources.)
  • If you decide the newspaper or magazine article is relevant, look at sources quoted or otherwise identified within it. Those may be additional sources for you.

For a book (perhaps in its library catalog listing), check whether the title and/or subtitle indicates the book could be about your research question. You can find a lot of such information about the book from its listing in a library catalog. Consider whether time should have an impact on what sources can be considered relevant.

  • Is the publication date or copyright date (usually listed in the library catalog or on the back of the book’s title page) too early or late for any time constraints in your research question? Maybe it’s just right.
  • Skim some of the preface and introduction to see whether the book works with the time considerations you need.
  • Check the bibliography to see whether the sources cited are about your research question.
  • Skim the book’s table of contents and any summary chapters to locate the problem or question that the book addresses, its solution, and the broad outline of the book’s argument for its main claim. Will any of that be helpful in answering your research question?
  • Skim some of the preface and introduction to see whether the book works with the time considerations you need.
  • Do those sections give you information about what others have written about your research question?
  • Do they offer a description of the situation surrounding your research question?
  • Look for your key words in the bibliography. Do the sources cited pertain to your research question?
  • Skim the index for topics with the most page references. Do the topics with the most page references pertain to your research question?

ACTIVITY: Follow a Title’s Clues for Relevance

Instructions: This quiz asks you to use logic, the titles of sources, and their publication dates, to identify the source most likely to be relevant to each research question. (Outside of this quiz, sources are not actually in competition with one another to be relevant. But this seemed like a good way to have you practice your skills at assessing relevance.) Many titles and dates below are fictitious, but that doesn’t affect their relevance within the quiz. Book, journal, website, and newspaper titles are italicized; chapter and article titles are in quotes.

  1. For each, read the information about the research question and each source.
  2. For each, record your judgments on a little table that you jot down like those illustrated earlier.
  3. For each, mark your answer, which should be the most relevant source according to the little table you completed for the question.
  4. Check your answers with our feedback.

 

ACTIVITY: Connecting the Dots Beyond the Title

Instructions: You always need to go beyond the title of a source when judging relevance. In the previous activity, you evaluated the titles of sources for currency and relevance. For this activity, you will investigate beyond the title to see whether one of the (hypothetical) articles named in the last activity is indeed relevant to meeting your information needs.

  1. Read the abstract of the article below, using your critical thinking skills to try to identify the information needs of your project it could help you meet.
  2. Then answer the questions about which information needs the source can help you meet. (Mark all that apply.)
  3. If there is at least one need it can help meet, you should judge the article relevant. Don’t forget to compare your answers with our feedback.

Your research question is: How does “prospect theory” in behavioral economics help explain medical doctors’ decisions to favor surgery or radiation to cure cancer in patients?

As usual, your information needs are:

  • To learn more background information.
  • To answer your research question.
  • To convince your audience that your answer is correct or, at least, the most reasonable answer.
  • To describe the situation surrounding your research question for your audience and explain why it’s important,
  • To report what others have said about your question, including any different answers to your research question.

Abstract:

“Cancer Treatment Prescription–Advancing Prospect Theory beyond Economics,” in Journal of The American Medical Association Oncology, June, 2022. (This article and abstract are fictitious but the journal and its form for abstracts are real.)

Importance Cancer treatment is complex. We expect oncologists to make treatment decisions according to definitive standards of care. Finding out that prospect theory demonstrates that they react very much like most other people when deciding to recommend surgery or chemotherapy for their patients indicates that more self-reflection on oncologists’ part could help patients make better decisions. (Prospect theory describes how people choose between alternatives that have risk when the probability of different outcomes is unknown.)

Objective To show whether prospect theory applies to how oncologists framed their recommendations for surgery or chemotherapy for patients in good condition and bad condition.

Design, Settings, and Participants Records of 100 U.S. oncologists were examined for the years 2019 and 2020, which documented patient conditions and the way oncologists framed their recommendations regarding surgery or chemotherapy. Records of nine thousand patients were involved. Thus, a quasiexperimental ex post facto design was used for the study.

Main Outcomes and Measures This study explored the relationship between the way in which the oncologists “framed” the choice of surgery or chemotherapy as they made recommendations to patients, the patients’ conditions, and the choice actually made. Those results were compared to what prospect theory would predict for this situation.

Results Physicians seemed to present their recommendation of surgery or chemotherapy in a loss frame (e.g., “This is likely to happen to you if you don’t have this procedure”) when patients’ conditions were poor and in a gain frame (e.g., “By having this procedure, you can probably dramatically cut your chances of reoccurrence”) when their conditions were less poor. These results are what prospect theory would have predicted.

Conclusions and Relevance This study opens up the possibility that, as described by prospect theory, a person’s choice of framing behavior is not limited to how we naturally act for ourselves but includes how we act for other people, as the oncologists were acting on behalf of their patients. More research is necessary to confirm this line of evidence and determine whether oncologists’ decision making and framing is the most effective and entirely according to the best standards of care.

Which information needs could this source help you meet if your research question was: How does “prospect theory” in behavioral economics help explain medical doctors’ decisions to favor surgery or radiation to cure cancer in patients?

 

definition

License

Icon for the Creative Commons Attribution 4.0 International License

Choosing & Using Sources: A Guide to Academic Research Copyright © 2015 by Teaching & Learning, Ohio State University Libraries is licensed under a Creative Commons Attribution 4.0 International License, except where otherwise noted.

Share This Book