It’s hard to say things that are interesting, simple, and true
Why it took two years to fact check What We Owe the Future
Slaveholding Quakers and my epistemic crisis
In 2022 I found myself doing a deep dive into rates of slave-ownership among 18th-century US Quakers. This didn’t seem particularly strange because I was working as a research assistant for Will MacAskill while he wrote What We Owe the Future. Naturally that should include researching premodern Quaker slavery practices.
We were trying to show that slaveholding declined among American Quakers during Benjamin Lay’s lifetime. Some contemporary accounts testify to Lay’s impact on his peers. But it sure would be nice if we could back this narrative with data.
Such data proved elusive. The simple fact is that it wasn’t collected; the census, for example, only started in 1790.
Or so we thought, until we found historian Jean Soderlund’s 1985 blockbuster Quakers and Slavery. That book included a table that reported slave ownership rates among the leaders of the Philadelphia Quaker’s Yearly Meeting, citing “PYM mins, 1681-1780; probate records; tax assessment lists”.
The table was helpful, but also seemed bit strange. It reported slave-ownership for irregular periods of 24, 24, 22, and 26 years. Plus, the number of Quakers accounted for in each period ranged from just 30 to 176. Not much to work with.
It would be great if Will could have written something pithy like “During Lay’s life, Quaker slave-owning rates fell from x% to y%”.
But to stick more closely to the available data, he went with the clunkier “Lay’s activism coincided with [a time of moral change.] In the period of 1681 to 1705, an estimated 70 percent of the leaders of the Quaker’s Yearly Meeting owned people; for the period 1754 to 1780 that figure was only 10 percent.”
(And even so, looking back at the book while writing this post I noticed a mistake still slipped in! Will wrote that these figures were for the “Quaker’s Yearly Meeting”, but really they were only for the Philadelphia Quaker’s Yearly Meeting. Ugh.)
The interesting, simple, and true trilemma
Imagine repeating this process over and over (and over and over and over) again and you’ll understand how we collectively managed to spend almost two years fact-checking that book.
It was a mostly fascinating, sometimes frustrating, effort. Many great facts were tried, convicted, and deleted for the crime of not being exactly “true”. Over time, I noticed a general pattern.
Ideally we’d say things that are interesting, simple, and true.
But often we achieve just two of these.
We can say things that are true and interesting, but they won’t be simple (like trying to describe how Quaker slave-holding rates changed through the 1700s).
We can say things that are true and simple, but they won’t be very interesting (like noting that while many Philadelphia Quaker leaders owned slaves in 1681, today none of them do).
Or we can say things that are simple and interesting, but they might not be true (like saying Lay’s activism coincided with a decline in American Quaker slave ownership in his lifetime).
Noticing how often simple and interesting claims turn out to be false has made me much more skeptical of the non-fiction I read. And noticing how hard it is to resolve the interesting, simple, and true trilemma has changed the way I write.
Let’s look at some more examples of the trilemma in action.
Do fungi cover 8% of the earth?
Even thinking about going back to look for more errors in WWOTF causes me immense suffering, so I’m going to pick on somebody else for publishing a simple, interesting, but sadly untrue fact.
(I’ve also wanted to write about this story for ages.)
In Entangled Life, biologist Merlin Sheldrake writes that “lichens encrust much as eight percent of the planet’s surface” (p. 80).
It’s a great fact. Definitely simple and interesting. Just, sadly, not true. As one friend said, “I live on the planet’s surface and that doesn’t seem right.”
Perhaps missing the thrills of WWOTF fact-checking, I decided to verify this fact recreationally. I followed a citation trail back to a 1987 article in Bibliotecha Lichenologica: “The absorption and release of water by lichens”. I can’t find a digital copy of that paper, but every paper that cites it says the same thing: that it claims 8% of earth’s “terrestrial surface” is covered by “lichen-dominated ecosystems.”
So the first mistake Entangled Life makes is oversimplifying a mildly interesting fact. 8% of the terrestrial surface became 8% of the planet, and “lichen-dominated ecosystems” became, simply, lichen.
But Sheldrake made one more mistake: he didn’t follow the citation trail all the way down.1 If he had, he would have found that even the more complicated original fact is completely made up.
I emailed Dr. Douglas Larson (author of the 1987 paper) he told me that I “would not believe the story behind this statistic” (direct quote). In short, as a student in 1974 Larson heard someone at a departmental party claim that “8% of the earth’s terrestrial ecosystems were lichen dominated”. It seemed legit. He casually repeated this claim at conferences and other lichen fan meetups. Along the way it got mentioned in a keynote, which turned into a publication, which now gets cited all the time in lichenology. We have no idea how much of the earth’s surface is covered by lichen, but it’s surely less than 8%.
The Entangled Life version of this fact is simple and interesting, but not true. The real-world version about lichen-dominated ecosystems may be true and is kind of simple, but it’s not interesting.
Other examples I remember
So many other bits were cut from WWOTF. Either the interesting and true versions weren’t simple enough, or the interesting and simple version weren’t true.
For example:
It’s often claimed many Chinese emperors died from drinking immortality elixirs. But after hours of searching I didn’t find any accounts that cite a primary source.2
Some people, including Steven Pinker, have called the An Lushan rebellion the worst atrocity in human history. But both the death count of the rebellion, and China’s population when it occurred, are wildly uncertain.3
During the pandemic I often heard that during the Black Death, Isaac Newton went into isolation and had a genius “miracle year” (annus mirabilis). But this didn’t happen.
Implications and lessons learned
I’m a skeptical reader
I’d bet WWOTF is more thoroughly cited than pretty much any comparable general non-fiction book. Its 1000 endnotes total more than 60,000 words.4 Some of the appendices may yet be published as standalone papers. A common criticism was that this seems like overkill. For example, deploying so much evidence makes the vignettes that open each chapter seem too long.
But I think a simpler version of the book — one without labyrinthine footnotes or book-length appendices — would contain a lot of fudge-y truths. And a version that hadn’t been fact-checked for two person-years would contain a lot of outright falsehoods.
That’s because an astonishing proportion of the time, our fact-checking found that the claims used to support the arguments in the book were importantly wrong or distorted. I’d say something like 30-50% of the time, we had to cut or revise important facts. Often a narrow, conditional claim had become a sweeping generalization. This was true even when those facts were pulled from seemingly-reliable sources like expert-authored books and articles.
Perhaps this is because there are few commercial or professional incentives to ensure one’s claims conform exactly to the available evidence. Or perhaps it’s because most authors just aren’t able to dedicate so many resources to researching and fact-checking.
Regardless, I’m now much more skeptical of everything I read. In particular, I’m suspicious of interesting, simple claims — it’s about as likely as not that some important nuance has been lost.
Historical claims were particularly hard to fact-check. I was repeatedly surprised by how uncertain we are about anything that happened before, like, 1950. It was hard to say interesting, simple, and true things about issues of major historical importance: where and when slavery was prevalent and how many enslaved people there were in various countries, or even what general living standards were like and how they varied over time in different places. Data are hard to come by: either they’ve disintegrated over time or were never collected. As a result, I’m very skeptical about historical claims.
(Historians are probably rolling their eyes right now, but this was a big learning for me!)
It’s not just ancient history that suffers from this problem. The Paraguayan War of 1864-70, for example, was probably the most intense war of the last 200 years. Some estimates suggest that two-thirds of Paraguay’s population was wiped out.
But, despite the war occurring less than 200 years ago, there is enormous uncertainty among even the best estimates. Censuses were only conducted in 1846 and 1886, leaving a gap of forty years. That’s all there is to go on; there are various ways of trying to fill in population stats for the years in between, but these generate death toll estimates that range from “just” 7% of Paraguay’s population to 69%.
As a result, the death count for by some measures the bloodiest event in centuries is roughly unknown; a void of understanding, with little prospect of ever filling it.
Beware the urge to simplify
My experience has also changed the way I write. I now notice how subtly the urge to simplify can meaningfully separate analysis from its citations.
I like simple writing. But I now notice how easily this risks distorting my claims to make them potentially untrue. And simplifying while still sticking to truth risks making things uninteresting.
Another quick example: in WWOTF, Will discusses how quickly Hiroshima recovered after being destroyed by the atomic bomb. He wanted to say something like “Hiroshima recovered remarkably quickly, being rebuilt within years.”
But once again our knowledge is too messy for this. It seems like some, but definitely not all, infrastructure was rebuilt quickly. We’re reliant on second-hand reports that were often issued for propagandistic purposes. So the final version of the book includes true, quite simple, but arguably less interesting narrower claims like “power was restored to some areas within a day, to 30 percent of homes within two weeks, and to all homes not destroyed by the blast within four months.” And even so, the text had to be complemented by a nuanced 800 word endnote (6.38 here).
You just have to be really careful to make the substance of what you’re saying accord with the details of what you’re citing. It’s hard. It’s really hard. Communicating uncertainty, or narrowing a claim, feels boring and clunky. But I think that’s just the way it has to be. Otherwise you risk building your argument on a house of cards. If one premise at the bottom gives out, the whole thing falls apart, and people will reasonably downgrade how much they trust your other claims.
There is one concrete habit which I’ve adopted and which helps: when citing, don’t just point to an entire book or paper, even if you include a page number. Instead, chop out the sentences that support your point, and quote the full thing in a footnote. This lets readers make easy spot-checks, which keeps you honest.
Sheldrake cites a 1995 “commentary” by Vernon Ahmadjian in which he writes "According to Douglas W. Larson of the University of Guelph in Ontario, Canada, approximately 8% of the earth's terrestrial surface has lichens as its most dominant life-forms." The citation trail ends there.
The oldest citation I found was to Part III of Volume 5 of Science and Civilization in China (1976) by Joseph Needham, Ho Ping-yu, and Lu Gwei-Djen (pdf link). There they write that “no less [sic] than six Thang emperors” died of elixir poisoning (p. 208). But there’s no further citation.
They were too long for the book. The publisher refused to print them all and we had to put many of them online instead.