Racer Library Tutorial - MPA Program (POL 670, 671, 681, etc.)

Evaluating Resources

Academic writing requires you to build your research upon the work of others—which means referring to scholarly sources. But failing to select the right sources can result in shoddy research and can hurt your credibility. To help you evaluate sources, we suggest using the C.R.A.P. test—that’s Currency, Reliability, Authority, and Purpose. Applying the C.R.A.P. test to your sources will show you which ones are appropriate to use in academic research. None of these filters are foolproof, but they’re a good starting point.

C is for Currency

When was the source published?

• Are you writing about the history of a subject/discipline/topic or about more recent developments?  If you are writing about the history of an event or discipline (including any field of study, not just history), then you must take into account the difference between primary and secondary sources.
  Primary sources were written at the time of the event in question, usually by someone involved. Secondary sources are written later, often by historians, critics, or others who received the information secondhand. The Diary of Anne Frank is a good example of a primary source on World War II. At the same time, a scholarly article about the diary is a good example of a secondary source. In a literary study, the primary source is the text written by the original author, while secondary sources are texts written about the work by critics.
• How recently was the source published? Depending on your discipline, it may be permissible to use a source published many years ago. In many fields, though, including those in the sciences, sources should be recent (unless you’re writing about the history of your field, as discussed above). Ask yourself if the information has changed since this source was published. If not, then it’s probably okay to use. When in doubt, ask your professor or advisor about the “expiration date” on sources related to your topic.
• If it’s an online source, is it current? Are all of its hyperlinks up to date? Out-of-date websites should be a red flag. Check to ensure the site has been updated recently (i.e., no broken links or outdated information).
   

R is for Reliability

How credible and applicable is the information?

• Is the source evaluated by experts in that particular field (also referred to as “peer-reviewed” or “refereed”)? You can limit your database searches to include only refereed articles. If unsure, check for the journal title in the online version of Ulrich’s Periodical Directory (known as Ulrichsweb), which will help you determine if the article is refereed. Non-refereed sources (like newspapers) are usually not considered as credible as refereed sources.
• Where did the source originate? If it was published by a dependable organization, like the government or a university, you’re probably better off than if an independent publisher put it out. If it is online, check the domain name. Websites ending in .edu and .gov tend to be the most trustworthy. Sites ending in .org are often unbiased, but remember that some organizations have a specific agenda. Sites ending in .com are usually trying to sell you something and are likely to be the least reputable.
• Does the source provide references? If so, it’s probably more trustworthy. These references can also be helpful to you if you are looking for more sources to consult.
   

A is for Authority

What do you know about the author(s) of the source?

• Who is the author(s)? Works put out by corporations/organizations (like universities) are sometimes more authoritative than those put out by individuals. For instance, consider the authority of an article from a physician doing research with the National Aeronautics and Space Administration (NASA) versus one from a doctor in private practice. However, it can go the other way: a report from an international oil company probably wouldn’t be considered unbiased if it were on the topic of fracking.
• Are the author(s)’ credentials readily accessible and appropriate? Look for authors who have expertise in their discipline—an advanced degree, awards, recommendations from others in the field, affiliation with a distinguished organization or university, and so on. For example, a professor with a Ph.D. and years of specialized experience would be a more authoritative source on an academic subject than a journalist or celebrity.
• What if there is no author? Sometimes, works put out by corporations or organizations do not list authors. If that is the case, evaluate the credibility of the organization itself. How many resources does it have at its disposal? How renowned is it? What would it gain or lose from the information presented?
• Who published the information? Is it a reputable publisher, or did the author pay to have the book printed? Does the publisher have a vested or conflicting interest in the work? For instance, drug studies published by pharmaceutical companies are not as credible as those published by independent researchers.
   

P is for Purpose

What is the source trying to accomplish?

• Is it unbiased, or does it have an agenda (to persuade, entertain, etc.)? Be wary of publications and websites that contain advertisements—they’re likely to have an agenda designed to sell you something.
• Is the source seeking to persuade or to inform? If the author is seeking to persuade, the piece may be biased and, therefore, less credible than an informative source. If you can’t tell whether the author approves or disapproves of the topic, that’s a good sign that it’s a reputable academic source. However, even informative pieces can be biased. Look for sources that present both sides of the story and that support claims with solid evidence that is cited—not a vague statement that “researchers have found this” but a citation that points back to the research.
• Who is the intended audience? This will give you a clue to the author’s intent. Is the author attempting to sell something or entertain you? Or are they trying to understand, explore, educate, or inform?  Watch out for satirical publications (i.e., The Onion), and be careful not to take their information at face value.

Anyone can publish information online and there is no overall fact-checker of the Internet. Therefore, it is important to evaluate the information you find for your assignments or your own personal interests. The ABC³ method of website evaluation highlights important points to consider as you research online sources: Authority, Bias, Content, Currency, and Consistency. 

Authority
Authority refers to who wrote the information and why you should trust them. Check to see if the author has credentials on the subject matter in terms of a degree and/or extensive experience. This helps verify that they are qualified to write about the topic. The author’s name is typically listed under the title and is commonly hyperlinked. Be aware that sometimes author does not refer only to one person but it can be a group of people, an organization, a company, and so on.

Locating the author:

• Usually located under the title of the article
• Sometimes located at the end of the article
• Contact us webpage

Evaluating authority:

• Click the hyperlinked author’s name (if applicable)
• Read the author’s bio
• Check the website’s About Us section
• Google the author
• Check Social Media  
• How easy is it to contact the author in case you have questions?

Bias
Bias refers to the level at which the author’s personal beliefs, opinions, or interests are displayed in the information. Is the author trying to make you feel a certain way? Are the author’s words emotionally charged? Can you tell how the author feels about the subject? Also, take a look at the sponsors to see who is funding the information to ensure they are not swaying the objectivity of the message being conveyed. Lastly, consider your own bias or experiences that may impact how you interpret the information you find.

Locating Bias:

• Words used in the article 
• Check the ads
• Check the mission statement of the website

Evaluating bias:

• Is the article written in the first person? (This is not necessarily a deal-breaker but may indicate a limited view on the topic.)
• Does the article contain a high level of emotionally charged language?
• Do you have preconceived opinions about the topic?

Content
Content refers to the information contained in the article. Does it make sense? Is it appropriate for your intended audience? What was the editing process? How is the information researched? What is the content type (entertainment, education, research, informational, etc.)? 

Locating Content:

• The text of the article or web entry
• Check methodology section (for peer-reviewed articles)

Evaluating Content:

• Check your assignment to verify the types of sources allowed (e.g., websites, peer-reviewed publications, newspapers, etc.)
• Check for editorial guidelines and/or the About Us page
• Check references, works cited or hyperlinks
• Check the length of the article
• Check for spelling or grammatical errors
• Verify that people quoted or companies mentioned actually exist

Currency
Currency refers to when the information was written or last updated. How regularly is the website maintained?

Locating Currency:

• The publication date is usually located underneath the title or at the end of the article.
• Many websites have a last updated or copyright date at the bottom of their webpages. 

Evaluating Currency:

• Check your assignment to ensure the publication date of the information falls within any stated timeframes.
• Check to see if there have been any new developments on your topic that were published after your article. (This can be accomplished by checking to see if any articles have cited your article, performing a google search on your topic and restricting the timeframe, or checking the library’s databases.)

Consistency
Many times your instructor will require you to include multiple sources in your research. This is to ensure you have a consensus on your topic and that you are not taking just one source’s word for it. Verify, verify, verify.

Locating consistency:

• This is done by reviewing other websites.

Evaluating consistency:

• Check your assignment for the number and types of sources required.
• Check other sources to see if they have the same information on your topic.

Understanding Networked Information

Traditional ways of conveying information – through books, journals, newspapers, broadcast news,  film, and even conversations among friends – have been undergoing enormous changes. Technology platforms developed in the past twenty years have changed how people find, share, and create information. This not only affects what information is available, it has shaken up the business models and cultures around information.

Google was not the first internet search engine, but after its founding in 1998 it quickly dominated the market because of its ability to crawl the web quickly and efficiently and because its page rank algorithm provided better relevance ranking than its competitors at the time. Likewise, Facebook (founded in 2004) was not the first social network, but it grew fast and now claims over two billion active users. Both companies have expanded their reach through the acquisition of other companies and growth into new areas. Google invested in mobile phone technologies, artificial intelligence, and purchased YouTube. Facebook bought Instagram, Oculus VR, and WhatsApp, a popular messaging service. Both companies have joined Apple, Microsoft, Amazon, and two Chinese giants, Tencent and Alibaba, on the list of the top ten public corporations by market capitalization. It’s a sign of their power that seven of the top ten are tech companies.

In the meantime, the way news is produced and shared has changed dramatically. Newspapers (the “first draft of history”) have lost subscribers and reporters. Newsrooms employ half the reporters they did in the 1990s, while a number of new digital startups have been created, many of them partisan, some of them satirical, some downright fraudulent. Network news and cable news make segments available on YouTube and news is largely encountered on phone apps and social media. A significant percentage of the advertising revenue news organizations rely on goes to the platforms people increasingly use to discover news – platforms like Google, Facebook, Apple News, and YouTube, companies that don’t produce news but benefit from digital advertising dollars that news producers generate. Books continue to be published in print even as Amazon dominates the e-book and audiobook markets, thanks to acquiring the companies that initially developed them. Even scientific and scholarly journals are going through profound changes. Once only available on paper to those who subscribed, journal production went digital, throwing up paywalls to protect profitable subscriptions. New open access publishing models are under development, but in the meantime impatient activists have launched sites where millions of papers are uploaded in violation of copyright, arguing the prevailing publishing model inhibits sharing knowledge.

It is not only easy for disinformation and propaganda to be created and spread through information networks; it is financially rewarding for creators and for the networks themselves. 

Four Moves and Habit

Being able to prove or disprove factual information is not the only way we can assess information or evaluate an argument, but facts are a key building-block of knowledge. Given the volume of factual claims we encounter every day in social media, in news stories, in political speeches, in arguments with That Opinionated Uncle, it may seem impossible to tackle the problem. Who has the time?

What people need most when confronted with a claim that appears to be factual but may not be 100 percent true is simple strategies to get closer to the truth. It doesn’t have to take a lot of time and work to take one or two key fact-claims and check them for accuracy.

These four moves (or tactics) can get you started with the fact-checking process. Try them in order.

• Check for previous work: Look around to see if someone else has already fact-checked the claim or provided a synthesis of research on the claim. Fact-checking sites are good sources for this move, particularly if the fact you’re checking has been getting a lot of attention. When you encounter a claim you want to check, your first move might be to see if sites like Politifact, Snopes, or even Wikipedia have researched the claim.
• Go upstream: Most web content is not original. Get to the original source to understand the trustworthiness of the information. This may be the first newspaper article that reported the claim; it may be a published journal article reporting original research or a government document. If the claim is about a research finding, try to find the original article. If the claim is about an event, try to find the original reporting.
• Read laterally: (This useful phrase was popularizedby Sam Wineburg and his colleagues at the Stanford History Education Group.)  Once you get to the source of a claim, make a quick check to see what other people think about the source (the publication, the author, etc.). Maybe you get lucky and the source is something you already know is reputable, such as the journal Science or the newspaper The New York Times If so, you can stop there. If not, dig around to see what you can find out about the source. Increasingly Wikipedia includes articles about publications with basic background about them. You can also make a quick check to see how others have addressed the same topic. Is this claim widely believed? Embraced by a particular group? Or is it contradicted by most sources? The truth is in the network.

• Circle back: If you get lost, hit dead ends, or find yourself going down an increasingly confusing rabbit hole, back up and start over knowing what you know now. You’ve learned something about the claim. Now you’re likely to take a more informed path with different search terms and better decisions. And if at any point you fail–if the source you find is not trustworthy, complex questions emerge, or the claim turns out to have multiple sub-claims–then circle back to step one, pick another fact-claim, and start over.

In general, try these moves in sequence. If you find success at any stage, your work might be done.

But How Do I know What Sources are "Reputable"?

This is a fraught question today. A sizeable percentage of Americans distrust all “mainstream media.” A recently published book, Network Propaganda: Manipulation, Disinformation, and Radicalization in American Politics by Yochai Benkler, Robert Faris, and Hal Roberts (Oxford, 2018) conducted extensive network analysis of news consumption and social sharing. They concluded our news media have bifurcated. There’s an entire self-referential media ecosystem centered on Fox News and embracing many right-wing and far-right websites that amplify and affirm one another while discrediting all news sources that fall outside their circle as “biased,” “left-wing,” or even “fake.” This is what the authors of the book call a “propaganda feedback loop.” In this media sphere, adherence to a partisan belief system and cultural identity is more important than being factually correct. A claim that is demonstrably false is more acceptable than stating something that challenges the overarching political narrative.

Doesn’t this happen on the left-wing too? There are hyperpartisan sites on the left that traffic in rumors and lies, but the difference is their audiences tend to balance their consumption of those sites with mainstream news organizations that generally attempt to adhere to journalistic norms set out in the Society of Professional Journalists Code of Ethics (Links to an external site.). In this sphere, it’s harder to maintain a story or a position based on “facts” that can be disproved or that even seem unlikely. Traditional journalistic processes are valued as much as or more than alignment with a set of political beliefs, and fewer people on the left dismiss the mainstream media’s reporting out of hand.

In this course, healthy skepticism is encouraged, but blanket distrust of mainstream news organizations based only on alleged political bias is not supportable. Part of the job of becoming a savvy information consumer is figuring out what sources of information you can, for the most part, trust to get it right much of the time and strive to correct mistakes.

Some generally-respected news organizations include . . .

• The New York Times
• The Wall Street Journal
• The Los Angeles Times
• The Washington Post
• PBS NewsHour
• NPR
• BBC

In addition to the moves, there’s a habit to cultivate: Check your emotions.

When you feel strong emotions – happiness, anger, pride, vindication – and that emotion pushes you to assume something is true and share a “fact” with others, stop and think. These are exactly the claims that you must fact-check.

Why? Because you’re already predisposed to analyze things that put you in an intellectual frame of mind. But things that make you angry or overjoyed, well… our record as humans is not good when it comes to being influenced by our emotions.

Our normal inclination is to agree with the content that we already feel is correct. (Psychologists call this “confirmation bias.” We are inclined to accept things that align with what we already believe and reject or ignore things that don’t.) Moreover, researchers have found that content that causes strong emotions (both positive and negative) spreads the fastest through our social networks. This shouldn’t surprise us; social media platforms are designed to maximize “stickiness.” They do everything they can to keep users engaged, because engagement is profitable. Savvy activists, advocates, and marketers take advantage of platform design and human psychology, getting past our filters by posting material that goes straight to our hearts.

Not only that, but whoever writes a headline is likely trying to catch your attention, and that can distort the meaning of a story. “Clickbait” is designed to provoke an emotion so it can spread. Resist the urge to share something based only on a headline. If you read the story – even if you simply skim it quickly – you will have a better sense of what it’s really about and whether it’s something you want to share. (Incidentally, authors of news stories usually don’t write their own headlines, so don’t blame them if they are ridiculously off base.)

Use your emotions as a checkpoint. Strong emotions should become a trigger for your new fact-checking habit. Every time content you want to share makes you feel rage, laughter, ridicule, or even a heartwarming buzz, spend 30 seconds fact-checking. You might avoid misleading others and embarrassing yourself by passing along something that’s simply not true.

In some cases an emotionally-charged piece of news may be factually true, but more complicated than it appears at first. By learning the context and the complexities through a more nuanced source of information, you can share valuable information without all the emotional baggage. You might even help cool down an over-heated argument.

For more information about the emotional and psychological components to information, see “The Psychology of Fake News” by Tania Lombrozo (NPR, March 27, 2018).

"ABC3 Criteria." Valencia College Library. Valencia College, Updated 22, July 2021. Web. 3 Aug. 2021. 

"Evaluating Scholarly Sources." Texas A&M University Writing Center. Texas A&M University, n.d. Web. 3 Aug. 2021.

“Evaluating Sources: Use the C.R.A.P. Test!” Mercer University Libraries. Mercer University, n.d. Web. 7 Dec. 2014.

“CRAP Test: Evaluating Websites.” South Mountain Community College Library, South Mountain Community College. Updated 25 July 2014. Web. 7 Dec. 2014.

“The CRAP Test.” Ohio University Libraries. Ohio University, Updated 23 March 2012. Web. 7 Dec. 2014.