Fact Checking Project Instructions
Click here to go back to the Fact Checker Application
Thank you so much for your interest in helping keep our work as scientifically rigorous as possible! Here’s the situation: Dr. Greger has co-authors taking his video work and transforming it into popular press book form. So they’re writing their interpretation of Dr. Greger’s interpretation of some body of science. You can imagine how something might get lost in translation or how easily inadvertent “telephone game”-type errors could be introduced (not to mention Dr. Greger or one of his researchers flubbing it up in the first place!). It’s not about egos; it’s about getting the science right.
Why is this job so important?
One major error, and the entire book could be (understandably!) dismissed out of hand and the credibility of NutritionFacts.org called into question. Blogs can be corrected or retracted and videos re-recorded, but circulating books are forever. Yes, I’ll keep a “Things I Got Wrong in the Book” blog updated, but this is not like a book about knitting where the worst an error can do is ruin a sweater. Books about health carry the heavy burden of affecting people’s lives. We have to try to get it perfect.
What will the job primarily entail?
Should you get the job, we’d email you a chapter in Microsoft Word along with a folder containing PDFs of all the sources cited in the chapter and your job would be to go sentence by sentence and make sure that every specific statement of fact is cited and accurately represents what the cited source actually says. Then you’d highlight the statement and, using the Track Changes function in Word, document your findings in a comment bubble.
Can you give an example?
Sure! If the chapter said “there appear to be dozens of Emporer penguin colonies on Antarctica{1}” citing {1} Fretwell P T and Trathan PN. Penguins from space: faecal stains reveal the location of emperor penguin colonies. Global Ecol Geogr 2009;18:542–52, you might say in your comment “Well as of the time they did the study the number apparently checks out, see for example on p. 543 “We have identified colony locations of emperor penguins at a total of 38 sites,” but the study was published years ago, and so you may want to clarify. Also, you spelled emperor wrong, miscapitalized it—oh, and you got the name of the journal wrong and it starts on page 543, not 542.”
Even if the statement was 100% accurate, you’d still want to acknowledge that with a comment bubble, justifying your assertion by copy/pasting the relevant bits from the source. So if instead the quote said “Researchers have found evidence of dozens of emperor penguin colonies on Antarctica” and cited the source correctly this time, then the comment would just be something like “Checks out as per p. 543: “We have identified colony locations of emperor penguins at a total of 38 sites”
Do I have to be a nutrition expert?
The job does not require mastery of the subject matter, just basic scientific literacy, the right attitude, and exquisite attention to detail. If one is not used to reading journal articles in a particular field it can be difficult to interpret whether or not an author’s assertion is accurately supported by the source. Quotes and are easy; a source either says it or not. But when one paraphrases or summarizes a source it can sometimes be difficult for the fact checker to substantiate. Never fear! Should you run into this situation you’d just tag the assertion with a simple comment like “I’m not sure if this is supported or not.” This will be a signal to the writing team that they better have someone who knows what they’re doing double-check it.
So as the factchecker, you should not be spending more than a few minutes on each point. Comments such as “I don’t see this right off the bat” are encouraged to speed things up. It should always be easier for the author to find whatever it is they originally found than the factchecker. Factcheckers should always feel comfortable delegating anything not immediately apparent back to the writing team to expedite the process, since the manuscript is due in April and so we only have about a month to finish the thing.
What do you mean by “the right attitude”?
The proper mind-set when approaching fact checking is that of a soulless tobacco industry hack that gets paid big bucks for every tiny little thing they can find even the slightest bit off in some American Lung Association brochure. In other words, every sentence is guilty until proven innocent. One must maintain a ruthless cynicism and never fall into the fatal fact checking trap of giving the author any benefit of the doubt.
What are some common errors to look out for?
Absolutist statements
One of the most common problems one sees translating scientific papers into lay publications is overstating the definiteness of findings. For example, a source might say, “bruising could indicate possible trauma,” but appear in the book as “bruising indicates trauma.” Or the source suggests that X may be one of the causes of Y, but appear in the paper as simply X causes Y.
Those examples are clearly egregious errors to be avoided, but to take this one step further, we should try to stay away from absolutist or generalized statements period—even when we can support them with good sources. Consider this scenario: Take a reasonable-sounding statement such as: “People in situation X suffer from unusually high rates of Y”z (where z is a study or review article precisely supporting that statement). While this would not be considered an “error” per se, it would have to be true in every case for it to be an accurate statement, which is one reason why some kind of qualifier is preferred. For example: “People in situation X have been found to suffer from unusually high rates of Y.”z or “People in situation X seem to suffer from unusually high rates of Y.”z
Thus, even for statements/facts not open to dispute, we should be conservative in our wording—even for something as uncontested as “Cigarette smoking causes cancer.” Even if we cite an authoritative review that says exactly that, it is still an absolutist statement. Instead we could say something like “Cigarette smoking has been shown to cause cancer” or (if the source supports it) “Cigarette smoking is considered undeniably to be a cause of cancer.” Notice the subtle difference: This shifts the burden of responsibility from us to the source by using words like “found,” “suggested,” “reported,” or “considered.” Remember, the book is just trying to report the best available balance of evidence, not independently asserting anything itself. Highlighting the wording with a simple comment bubble of “Absolutist language” can suffice.
Unattributed statements
Unless a statement of fact is so common-sense and uncontroversial as to make a reference unnecessary (like “The world is round”), every specific assertion of substance should be cited (i.e. if it isn’t, you should highlight it and comment bubble “unattributed” or some such). You should find citations at the end of each sentence if everything in the sentence comes from that source or at the end of a supported clause if only part of the sentence is supported by the source. For example, if source X reported that dogs prefer white toys and source Y reported that dogs prefer round toys, it could be cited in a number of ways:
- When it comes to toy color, dogs were found to prefer white.X When it came to shape, dogs were found to prefer round.Y
- Researchers found that dogs prefer toys that were whiteX and toys that were round.Y
- Researchers found that dogs have toy preferences.X,Y
If within the same paragraph, consecutive sentences or clauses come from the same source, then the book can just cite the last occurrence (so if all the sentences in one paragraph are supported by a single source, the book can just put a single citation mark at the end of the last sentence in the paragraph. Even if all the sentences in two consecutive paragraphs are from the same source, though, each paragraph should have its own citation).
Misquoted Quotes
Another common error is quotes bring transcribed incorrectly from the source. Quotes are sacred. We must carry over quotes word for word, letter for letter, with the exact punctuation, the exact spellings, etc. This may be harder than one might think. For example, Microsoft Word may inadvertently autocorrect foreign spellings (such as “behaviour,” a legitimate British spelling), so it is critical to carefully recheck every quote that is transferred.
In informal writing, “I thought his house was nice” could be changed to “I thought [George’s] house was nice.” But in scientific writing (which the book should emulate), if one wanted to do the same thing in the sentence, one would have to account for the removal of the “his,” so it would be: “I thought…[George’s] house was nice” or more simply: I thought George’s house was “nice.”
Also, please make sure no quotes are taken out of context, and be on the lookout for inadvertent plagiarism, where entire phrases are lifted from a source without being paraphrased or placed in quotation marks.
Selective Citation
Cherry-picking outlying studies to support one’s thesis while failing to even mention a large body of contradictory evidence is rife within the popular press on nutrition. It can be difficult for a fact checker to pick up on this, but researchers typically try to place their findings in context. For example if the book says “An apple a day has been found to be associated with a 20% lower risk of earlobe cancer” and cites a study that clearly says right up front in the Abstract: “Eating an apple every day was associated with a 20% lower risk of cancer of the earlobe,” rather than just noting that the fact checks out and moving on, it’s worth a quick scan of the Discussion section in case the whole thing is about how surprising the finding is considering that the only other dozen studies on apples found higher cancer risk. In that context, the 20% lower statement may be accurate but highly misleading without acknowledging the discrepancy.
Misspelled Names
Frequently, proper nouns are misspelled. Anytime a word is introduced that spell-check may not pick up, like a person, institution or location, please double-check it.
Secondary Citations
Whenever possible we should avoid using secondary citations if we are citing specifics. So, if the book says “Jones found X,” but the book cites a Smith paper discussing Jones’ work instead of the primary Jones source, the book authors are blindly assuming that Smith reliably reported Jones’s work. The authors should always attempt to get the original work (i.e., the “primary” source as opposed to the “secondary” source Smith). If that is not possible, they need to add a qualifier like “Jones reportedly found X.”
Please post any questions in the comment section here.
