Internet Related Items

Evaluating Internet Research Sources

October 19, 2020

82554

“The central work of life is interpretation.” –Proverb

Introduction: The Diversity of Information

Information is a Commodity Available in Many Flavors

Think about the magazine section in your local grocery store. If you reach out with your eyes closed and grab the first magazine you touch, you are about as likely to get a supermarket tabloid as you are a respected journal (actually more likely, since many respected journals don’t fare well in grocery stores). Now imagine that your grocer is so accommodating that he lets anyone in town print up a magazine and put it in the magazine section. Now if you reach out blindly, you might get the Elvis Lives with Aliens Gazette just as easily as Atlantic Monthly or Time.Welcome to the Internet. As I hope my analogy makes clear, there is an extremely wide variety of material on the Internet, ranging in its accuracy, reliability, and value. And unfortunately, the same applies to many of the products of the traditional information media (books, magazines, organizational documents, television, film). There has been an increasing amount of suspect information in the information space in general, making the evaluation of information to determine its reliability more and more challenging. It’s your job as a searcher, then, to evaluate what you locate, in order to determine whether it suits your needs.

Information Exists on a Continuum of Reliability and Quality

Information is everywhere on the Internet, existing in large quantities and continuously being created and revised. This information exists in a large variety of kinds (facts, opinions, stories, interpretations, statistics) and is created for many purposes (to inform, to persuade, to sell, to present a viewpoint, and to create or change an attitude or belief). For each of these various kinds and purposes, information exists on many levels of quality and reliability. It ranges from very good to very bad and includes every shade in between.

Adopting a Skeptical (or Cautious) Attitude

You might have heard of the term information warfare, the use of information as a weapon. Now, as I just said above, there is a lot of high quality information available through the Internet. Unfortunately, however, there is also a large amount of misinformation (honest people mistakenly spreading false information), together with the information warfare ammunition: disinformation (dishonest people knowingly spreading false information), half truths, distortions, urban legends, fallacies, exaggerations, “fake news,” and just plain old lies. So you need to adopt a skeptical–or at least a cautious–attitude toward all knowledge claims. You don’t have to believe everything you read, hear, or see. (Nor do you have to become completely cynical and disbelieve everything.) Just be very cautious. Instead of putting each knowledge claim into your “accept” or “reject” pile, put it in a “this is claimed, but I neither accept nor reject it right now” pile. Seek confirmation and disconfirmation from other sources and your own thinking. Remember that a Web site that makes up facts, quotes people who don’t exist, and cites imaginary sources could look just as professional and slick as a highly reliable and credible site. Don’t be deceived by appearances. Develop your critical thinking skills.

Understand the Nature and Prevalence of Fake News

It is essential to understand that some Web sites exist that present what is now being called fake news. Fake news comes from several different sources. First, there are sites which publish satires. Satiric articles present phony news containing hyperbole (exaggeration) and employ irony, in order to criticize the subject or events being covered. For example, if, in real life a general favors sending more troops to a troubled country, the satiric site might produce an article supposedly quoting the general saying that we need to increase our combat fatalities. Satire is not intended to be understood as literally true, but many people take it that way. Second, as the saying is, anyone and his dog can have a Web site or a Twitter account. Many people with political agendas invent news accounts of events that never happened. Their goal is to influence your belief about a person or group, negatively or positively. For example, it is believed that Russia has a goal of destabilizing Western democracies, the US in particular. Fake news accounts designed to increase distrust of American institutions and hostility among its citizens are created and distributed through social media. The traditional name for this type of fake news is disinformation, information the creator knows is false but which he wants you to believe is true. Third, remember that visits to web sites, YouTube videos, and so on, are often monetized: that is, the more people who click on them, the more money the creators make. This fact incentivizes people to create fake news with often extreme content in order to get hits. Fourth, be cautious about information that too easily supports something you support. Fake news creators know about confirmation bias, the habit of seeking information that agrees with what we already think, so they create stories that support various beliefs, sometimes simplistically. When we see “proof” of our opinion, we can’t resist and give in to the “click bait.”

Getting Started: Screening Information

Pre-evaluation

The first stage of evaluating your sources takes place before you do any searching. Take a minute to ask yourself what exactly you are looking for. Do you want facts, opinions (authoritative or just anyone’s), reasoned arguments, statistics, narratives, eyewitness reports, descriptions? Is the purpose of your research to get new ideas, to find either factual or reasoned support for a position, to survey opinion, or something else? Once you decide on this, you will be able to screen sources much more quickly by testing them against your research goal.
If, for example, you are writing a research paper, and if you are looking for both facts and well-argued opinions to support or challenge a position, you will know which sources can be quickly passed by and which deserve a second look, simply by asking whether each source appears to offer facts and well-argued opinions, or just unsupported claims.

Select Sources Likely to be Reliable

Becoming proficient at selecting sources will require experience, of course, but even a beginning researcher can take a few minutes to ask, “What source or what kind of source would be the most credible for providing information in this particular case?” Which sources are likely to be fair, objective, lacking hidden motives, showing quality control? It is important to keep these considerations in mind, so that you will not simply take the opinion of the first source or two you can locate. By thinking about these issues while searching, you will be able to identify suspicious or questionable sources more readily. With so many sources to choose from in a typical search, there is no reason to settle for unreliable material.
For example, if you are looking for reliable statistics on American population data (marriages, divorces, college degrees, auto accidents, incomes by state, etc.) then a site with a .gov extension should be among the reliable sources you select because .gov sites represent the official information of the American government.

But Wait a Minute

Remember that to locate fair, objective material, you must be fair and objective, too. A major error that too many researchers make is to look only for sources whose ideas, findings, or arguments they already agree with. It’s fine to have a sense of where you think you are going, but you should be open to opposing ideas and not discount them just because you don’t like them or because they conflict with your planned direction. The best researchers usually don’t start out “to prove X.” Instead, they start out “to find out about X.” Be careful not to fall into that circular reasoning trap by thinking, “Books expressing that view are unreliable.”

Source Selection Tip:
Try to select sources that offer as much of the following information as possible:
Author’s Name
Author’s Title or Position
Author’s Organizational Affiliation
Date of Page Creation or Version
Author’s Contact Information
Some of the Indicators of Information Quality (listed below)

Evaluating Information: The Tests of Information Quality

Reliable Information is Power	You may have heard that “knowledge is power,” or that information, the raw material of knowledge, is power. But the truth is that only some information is power: reliable information. Information serves as the basis for beliefs, decisions, choices, and understanding our world. If we make a decision based on wrong or unreliable information, we do not have power–we have defeat. If we eat something harmful that we believe to be safe, we can become ill; if we avoid something good that we believe to be harmful, we have needlessly restricted the enjoyment of our lives. The same thing applies to every decision to travel, purchase, or act, and every attempt to understand.
Source Evaluation is an Art	Source evaluation–the determination of information quality–is something of an art. That is, there is no single perfect indicator of reliability, truthfulness, or value. Instead, you must make an inference from a collection of clues or indicators, based on the use you plan to make of your source. If, for example, what you need is a reasoned argument, then a source with a clear, well-argued position can stand on its own, without the need for a prestigious author to support it. On the other hand, if you need a judgment to support (or rebut) some position, then that judgment will be strengthened if it comes from a respected source. If you want reliable facts, then using facts from a source that meets certain criteria of quality will help assure the probability that those facts are indeed reliable.
The CARS Checklist	The CARS Checklist (Credibility, Accuracy, Reasonableness, Support) is designed for ease of learning and use. Few sources will meet every criterion in the list, and even those that do may not possess the highest level of quality possible. But if you learn to use the criteria in this list, you will be much more likely to separate the high quality information from the poor quality information.

The CARS Checklist for Information Quality

Credibility	Because people have always made important decisions based on information, evidence of authenticity and reliability–or credibility, believability–has always been important. If you read an article saying that the area where you live will experience a major earthquake in the next six months, it is important that you should know whether or not to believe the information. Some questions you might ask would include, What about this source makes it believable (or not)? How does this source know this information? Why should I believe this source over another? As you can see, the key to credibility is the question of trust.There are several tests you can apply to a source to help you judge how credible and useful it will be:
Author’s Credentials	The author or source of the information should show some evidence of being knowledgeable, reliable, and truthful. Here are some clues: Author’s education, training, and/or experience in a field relevant to the information. Look for biographical information, the author’s title or position of employment Author provides contact information (email or snail mail address, phone number) Organizational authorship from a known and respected organization (corporate, governmental, or non-profit) Author’s reputation or standing among peers. Author’s position (job function, title) Note: Mere fame is not an indicator of credibility. A famous actor or athlete presenting arguments about, say, the ozone layer, should be considered an ordinary layman unless evidence is presented that the person has attained expetise in the area.
Evidence of Quality Control	Most scholarly journal articles pass through a peer review process, whereby several readers must examine and approve content before it is published. Statements issued in the name of an organization have almost always been seen and approved by several people. (But note the difference between, “Allan Thornton, employee of the National Oceanographic and Atmospheric Agency, says that a new ice age is near,” and “The National Oceanographic and Atmospheric Agency said today that a new ice age is near.” The employee is speaking for himself, whereas a statement in the name of NOAA represents the official position of NOAA.)Evidence of quality control of Internet material includes these items: Information presented on organizational web sites On-line journals that use refereeing (peer review) by editors or others Postings of information taken from books or journals that have a quality control process Note: Appearances can be deceiving. Don’t assume that a great-looking Web site is automatically credible. Very professional and sophisticated Web page templates are available for a few dollars, so that anyone and his pet skunk can put up a site that looks expensive and authoritative. Good looks are not evidence of credibility.
Metainformation	Metainformation is information about information. Information workers (sometimes called knowledge workers) all over the world are constantly poring over, processing, and evaluating information–and making notes. As the challenges produced by the increasing quantity of information continue, access to high quality metainformation will become increasingly important. Metainformation can take many forms, but there are two basic types, summary and evaluative.Summary metainformation includes all the shortened forms of information, such as abstracts, content summaries, or even tables of contents. This type of metainformation gives us a quick glance at what a work is about and allows us to consider many different sources without having to go through them completely. Evaluative metainformation includes all the types that provide some judgment or analysis of content. This type includes recommendations, ratings, reviews, and commentaries. Even the search results order of pages from a search engine like Google represents a type of evaluative metainformation, since pages are ranked in part by the number of other pages linked to them (and hence “voting” for them in some sense). And, of course, these two types can be combined, resulting in the best form of metainformation, providing us with a quick overview and some evaluation of the value. An examples would be a World Wide Web yellow pages or directory which describes each selected site and provides evaluations of its content. Tip: If you are thinking about using a printed book in your research paper, Google the book title with the word “review” appended and read what others think of the book.
Indicators of Lack of Credibility	You can sometimes tell by the tone, style, or competence of the writing whether or not the information is suspect. Here are a few clues: Anonymity Lack of Quality Control Negative Metainformation. If all the reviews are critical, be careful. Bad grammar or misspelled words. Most educated people use grammar fairly well and check their work for spelling errors. An occasional split infinitive or comma in the wrong place is not unusual, but more than two or three spelling or grammar errors is cause for caution, at least. Whether the errors come from carelessness or ignorance, neither puts the information or the writer in a favorable light. Emotional earnestness accompanied by exaggeration or absolutes. Even in very controversial areas (gun control, global warming, abortion, capital punishment) and promotional contexts (product claims and evaluations) we expect reasons, data, and emotional restraint. Articles where the writer’s feelings have clearly taken over from thinking make us wonder if we are reading ideology instead of information and arguments that might persuade us. Breathless, sweeping generalizations should set off your baloney detector. For example, “Did you know that none of the vitamins and supplements sold in stores work correctly with your body chemistry? Only SuperDuperVite has been formulated to blah blah blah.” Claims of unique, secret information (which is now on the Web site) or claims of such dramatic implications that you should expect widespread discussion. For example, “The CIA was responsible for the assassination of President Kennedy.” Conspiracy theories in general, because they run counter to official reports and often counter to reason, should be met with great caution.

Accuracy	The goal of the accuracy test is to assure that the information is actually correct: up to date, factual, detailed, exact, and comprehensive. For example, even though a very credible writer said something that was correct twenty years ago, it may not be correct today. Similarly, a reputable source might be giving up-to-date information, but the information may be only partial, and not give the full story. Here are some concepts related to accuracy:
Timeliness	Some work is timeless, like the classic novels and stories, or like the thought provoking philosophical work of Aristotle and Plato. Other work has a limited useful life because of advances in the discipline (psychological theory, for example), and some work is outdated very quickly (such as technology news). You must therefore be careful to note when the information you find was created, and then decide whether it is still of value (and how much value). You may need information within the past ten years, five years, or even two weeks. But old is not necessarily bad: nineteenth-century American history books or literary anthologies can be highly educational because they can function as comparisons with what is being written or anthologized now. In many cases, though, you want accurate, up-to-date information.An important idea connected with timeliness is the dynamic, fluid nature of information and the fact that constant change means constant changes in timeliness. The facts we learn today may be timely now, but tomorrow will not be. Especially in technology, science, medicine, business, and other fields always in flux, we must remember to check and re-check our data from time to time, and realize that we will always need to update our facts. Note: Many Web pages display today’s date automatically, regardless of when the content on the page was created. If you see today’s date on a page other than from a news site, be extra careful.
Comprehensiveness	Any source that presents conclusions or that claims (explicitly or implicitly) to give a full and rounded story, should reflect the intentions of completeness and accuracy. In other words, the information should be comprehensive. Some writers argue that researchers should be sure that they have “complete” information before making a decision or that information must be complete. But with the advent of the information age, such a goal is impossible, if by “complete” we mean all possible information. No one can read 20,000 articles on the same subject before coming to a conclusion or making a decision. And no single piece of information will offer the truly complete story–that’s why we rely on more than one source. On the other hand, an information source that deliberately leaves out important facts, qualifications, consequences, or alternatives may be misleading or even intentionally deceptive. Tip: Enter your topic into the search box of Google or another search engine and append one of the following words or phrases: controvesy, dispute, disagreement, alternate views, debate, arguments for and against. The results will give you an indication about how comprehensive your source is and whether it has omitted some important details.
Audience and Purpose	For whom is this source intended and for what purpose? If, for example, you find an article, “How Plants Grow,” and children are the intended audience, then the material may be too simplified for your college botany paper. More important to the evaluation of information is the purpose for which the information was created. For example, an article titled, “Should You Buy or Lease a Car?” might have been written with the purpose of being an objective analysis, but it may instead have been written with the intention of persuading you that leasing a car is better than buying. In the latter case, the information will most likely be biased or distorted. Such information is not useless, but the bias must be taken into consideration when interpreting and using the information. (In some cases, you may need to find the truth by using only biased sources, some biased in one direction and some biased in the other.) Be sure, then, that the intended audience and purpose of the article are appropriate to your requirements or at least clearly in evidence so that you may take them into account. Information pretending to objectivity but possessing a hidden agenda of persuasion or a hidden bias is among the most common kind of information in our culture.
Indicators of a Lack of Accuracy	In addition to an obvious tone or style that reveals a carelessness with detail or accuracy, there are several indicators that may mean the source is inaccurate, either in whole or in part: No date on the document Vague or sweeping generalizations Old date on information known to change rapidly Very one sided view that does not acknowledge opposing views or respond to them

Reasonableness	The test of reasonableness involves examining the information for fairness, objectivity, moderateness, and consistency.
Fairness	Fairness includes offering a balanced, reasoned argument, not selected or slanted. Even ideas or claims made by the source’s opponents should be presented in an accurate manner. Pretending that the opponent has wild, irrational ideas or arguments no one could accept is to commit the straw man fallacy. A good information source will also possess a calm, reasoned tone, arguing or presenting material thoughtfully and without attempting to get you emotionally worked up. Pay attention to the tone and be cautious of highly emotional writing. Angry, hateful, critical, spiteful tones often betray an irrational and unfair attack underway rather than a reasoned argument. And writing that attempts to inflame your feelings to prevent you from thinking clearly is also unfair and manipulative.
Objectivity	There is no such thing as pure objectivity, but a good writer should be able to control his or her biases. Be aware that some organizations are naturally not neutral. For example, a professional anti-business group will find, say, that some company or industry is overcharging for widgets. The industry trade association, on the other hand, can be expected to find that no such overcharging is taking place. Be on the lookout for slanted, biased, politically distorted work.One of the biggest hindrances to objectivity is conflict of interest. Sometimes an information source will benefit in some way (usually financially, but sometimes politically or even emotionally or psychologically) if that source can get you to accept certain information rather than the pure and objective truth. For example, many sites that sell “natural” products (cosmetics, vitamins, clothes) often criticize their competitors for selling bad, unhealthy or dangerous products. The criticism may be just, but because the messenger will gain financially if you believe the message, you should be very careful–and check somewhere else before spending money or believing the tale.
Moderateness	Moderateness is a test of the information against how the world really is. Use your knowledge and experience to ask if the information is really likely, possible, or probable. Most truths are ordinary. If a claim being made is surprising or hard to believe, use caution and demand more evidence than you might require for a lesser claim. Claims that seem to run against established natural laws also require more evidence. In other words, do a reality check. Is the information believable? Does it make sense? Or do the claims lack face validity? That is, do they seem to conflict with what you already know in your experience, or do they seem too exaggerated to be true? “Half of all Americans have had their cars stolen.” Does that pass the face validity test? Have half of your friends had their cars stolen? Is the subject on the news regularly (as we might assume it would be if such a level of theft were the case)?It is important, of course, to remember that some truths are spectacular and immoderate. Over the past few decades, Michel Lotito, a French performer with the stage name of Monsieur Mangetout (French for “eats everything”) has actually eaten 18 bicycles, several TV sets, a few shopping carts, and a small airplane by first having them ground into a fine powder and sprinkling a few teaspoonfuls on his breakfast cereal each morning. So do not automatically reject a claim or source simply because it is astonishing. Just be extra careful about checking it out.
Consistency	The consistency test simply requires that the argument or information does not contradict itself. Sometimes when people spin falsehoods or distort the truth, inconsistencies or even contradictions show up. These are evidence of unreasonableness.
World View	A writer’s view of the world (political, economic, religious–including anti-religious–and philosophical) often influences his or her writing profoundly, from the subjects chosen to the slant, the issues raised, issues ignored, fairness to opponents, kinds of examples, and so forth. World view can be an evaluative test because some world views in some people cause quite a distortion in their view of reality or their world view permits them to fabricate evidence or falsify the positions of others. For some writers, political ideology or political agenda takes precedence over truth and sometimes even over fairness. If you are looking for truth or a whole picture, such sources are not the best. Do note, however, that everyone, including you and I, has a worldview through which we interpret reality and find meaning in events. Many writers can and do control the biases that accompany their world view, so simply discovering a writer’s world view does not automatically disqualify the information or even render it suspect. A book or article about Middle East politics by a Muslim, a Jew, a Christian, or an atheist can still be quite reasonable and a valuable source.
Indicators of a Lack of Reasonableness	Writers who put themselves in the way of the argument, either emotionally or because of self interest, often reveal their lack of reasonableness. If, for example, you find a writer reviewing a book he opposes by asserting that “the entire book is completely worthless claptrap,” you might suspect there is more than a reasoned disagreement at work. Here are some clues to a lack of reasonableness: Intemperate tone or language (“stupid jerks,” “shrill cries of my extremist opponents”) Overclaims (“Thousands of children are murdered every day in the United States.”) Sweeping statements of excessive significance (“This is the most important idea ever conceived!”) Conflict of Interest (“Welcome to the Old Stogie Tobacco Company Home Page. To read our report, ‘Cigarettes Make You Live Longer,’ click here.” or “The products our competitors make are dangerous and bad for your health.”)

Support	The area of support is concerned with the source and corroboration of the information. Much information, especially statistics and claims of fact, comes from other sources. Citing sources strengthens the credibility of the information. (Remember this when you write a research paper.)
Source Documentation or Bibliography	In the art world, several paintings by Vincent van Gogh have sold for more than 50 million dollars each. At such a price, there muist be the temptation for unscrupulous people to paint forgeries. So how can the buyers be confident that they are getting a genuine Van Gogh? The answer is provenance. Provenance is a list of previous owners, tracing back to the original buyer from Van Gogh himself. Provenance answers the critical question, “Where did this painting come from?” Similarly, when you wonder if the source you are reading is relaible, a powerful question to ask is, “Where did this information come from?” In this case, the “provenance” questions include, What sources did the information creator use? Are the sources listed? Is there a bibliography or other documentation? Does the author provide contact information in case you wish to discuss an issue or request further clarification? What kind of support for the information is given? How does the writer know this? It is especially important for statistics to be documented. Otherwise, someone may be just making up numbers. Note that some information from corporate sites consists of descriptions of products, techniques, technologies, or processes with which the corporation is involved. If you are careful to distinguish between facts (“We mix X and Y together to get Z”) and advertising (“This protocol is the best in the industry”), then such descriptions should be reliable.
Corroboration	See if other sources support this source. Corroboration or confirmability is an important test of truth. And even in areas of judgment or opinion, if an argument is sound, there will probably be a number of people who adhere to it or who are in some general agreement with parts of it. Whether you’re looking for a fact (like the lyrics to a song or the date of an event), an opinion (like whether paper or plastic is the more environmentally friendly choice), or some advice (like how to grow bromeliads), it is a good idea to triangulate your findings: that is, find at least three sources that agree. If the sources do not agree, do further research to find out the range of opinion or disagreement before you draw your conclusions.What you are doing with corroboration, then, is using information to test information. Use one source, fact, point of view, or interpretation to test another. Find other information to support and reconfirm (or to challenge or rebut) information you have found. Corroboration is especially important when you find dramatic or surprising information (information failing the moderateness test, above). For example, the claim that a commonly used food additive is harmful should be viewed with skepticism until it can be confirmed (or rebutted) by further research. The claim may be true, but it seems unlikely that both government and consumer organizations would let the additive go unchallenged if indeed it were harmful.
External Consistency	While the test of corroboration involves finding out whether other sources contain the same new information as the source being evaluated, the test of external consistency compares what is familiar in the new source with what is familiar in other sources. That is, information is usually a mixture of old and new, some things you already know and some things you do not. The test of external consistency asks, Where this source discusses facts or ideas I already know something about, does the source agree or harmonize or does it conflict, exaggerate, or distort? The reasoning is that if a source is faulty where it discusses something you already know, it is likely to be faulty in areas where you do not yet know, and you should therefore be cautious and skeptical about trusting it.
Indicators of a Lack of Support	As you can readily guess, the lack of supporting evidence provides the best indication that there is indeed no available support. Be careful, then, when a source shows problems like these: Numbers or statistics presented without an identified source for them Absence of source documentation when the discussion clearly needs such documentation You cannot find any other sources that present the same information or acknowledge that the same information exists (lack of corroboration)

Summary of The CARS Checklist for Research Source Evaluation

Credibility	trustworthy source, author’s credentials, evidence of quality control, known or respected authority, organizational support. Goal: an authoritative source, a source that supplies some good evidence that allows you to trust it.
Accuracy	up to date, factual, detailed, exact, comprehensive, audience and purpose reflect intentions of completeness and accuracy. Goal: a source that is correct today (not yesterday), a source that gives the whole truth.
	fair, balanced, objective, reasoned, no conflict of interest, absence of fallacies or slanted tone. Goal: a source that engages the subject thoughtfully and reasonably, concerned with the truth.
Support	listed sources, contact information, available corroboration, claims supported, documentation supplied. Goal: a source that provides convincing evidence for the claims made, a source you can triangulate (find at least two other sources that support it).

Living with Information: The CAFÉ Advice

	Here is one last piece of advice to help you live well in the world of information: Take your information to the Café (Challenge, Adapt, File, Evaluate).
Challenge	Challenge information and demand accountability. Stand right up to the information and ask questions. Who says so? Why do they say so? Why was this information created? Why should I believe it? Why should I trust this source? How is it known to be true? Is it the whole truth? Is the argument reasonable? Who supports it?
Adapt	Adapt your skepticism and requirements for quality to fit the importance of the information and what is being claimed. Require more credibility and evidence for stronger claims. You are right to be a little skeptical of dramatic information or information that conflicts with commonly accepted ideas. The new information may be true, but you should require a robust amount of evidence from highly credible sources.
File	File new information in your mind rather than immediately believing or disbelieving it. Avoid premature closure. Do not jump to a conclusion or come to a decision too quickly. It is fine simply to remember that someone claims XYZ to be the case. You need not worry about believing or disbelieving the claim right away. Wait until more information comes in, you have time to think about the issue, and you gain more general knowledge.
Evaluate	Evaluate and re-evaluate regularly. New information or changing circumstances will affect the accuracy and hence your evaluation of previous information. Recognize the dynamic, fluid nature of information. The saying, “Change is the only constant,” applies to much information, especially in technology, science, medicine, and business.