No, I won't read your amateur COVID-19 "research review"

Even good arguments in a bad context encourage bad thinking

A few days ago, I stumbled on an impressive-looking “research review” linked to from Hacker News on why the current lockdowns in the US are an overreaction to the COVID-19 threat, and we need to reopen the country as quickly as possible to avoid catastrophic economic damage from which we can’t recover. The paper is about 7500 words long, written in a very authoritative tone, and links to dozens of credible citations.

EDIT: to be fair, the author did later go back and add a disclaimer that they lack any experience or credentials in any relevant fields.

It was only after exploring a bit that I realized that the author is a software engineer, not an economist, epidemiologist, etc.

(Side note: what is it about software engineers that we think we’re qualified to pontificate on anything remotely technical or complex?)

In the comments on Hacker News, the author took a lot of flak for having no experience or expertise in any relevant fields. His rejoinder was that calling him out for a lack of “credentials” was a form of ad hominem attack, and his work should be evaluated on its merits.

High-minded, but not really realistic. The idea that everything should be evaluated on its merits has a couple problems:

1. There isn’t enough time to evaluate everything on the merits. I’m sure there are millions of people out there with arguments and plans on how we should deal with COVID-19. We can’t take a close look at all of that chatter before deciding.

2. Most things can’t be evaluated on the merits by most people, because they lack the expertise to do so. In the case of this paper, the author linked to many research papers by experts, which is good, but then he wove an argument together out of all that research. And since I don’t have any expertise in any of those fields, I don’t know if he’s evaluating the data correctly, or if he’s ignoring other relevant data that doesn’t support his conclusion.

As a result, everyone relies on mental shortcuts or heuristics when evaluating ideas and arguments to decide how much credence to give them.

But some heuristics are better than others.

For example, when evaluating claims in a high-stakes, high-uncertainty, high-complexity area, like the public policy approach to COVID-19, a good heuristic might be “does this person have expertise or experience in a relevant field?”

A bad heuristic would be: “are the claims being made with very high confidence and in an authoritative style?”

Or the most common bad heuristic of all:

“Does this confirm my previously held beliefs?”

If I’m honest with myself, I have to acknowledge that I would have had much less of a problem with this paper if it agreed with what I already thought to be true. I likely would have just mentally filed it away as another piece of “evidence” supporting my position and moved on. In fact, I suspect that everyone who read that paper who was already persuaded to its conclusions did exactly that.

But that’s the problem: the paper might be built on good arguments, but it was presented in a format and context that encourages the use of bad heuristics. The tone was authoritative and confident, the author didn’t prominently disclose their lack of expertise, it’s fairly long and comprehensive, it’s well-organized, etc. The author even wrote the whole thing using “we” instead of “I”, which subtly gives the impression that a group of people collaborated to write this (which is probably a good heuristic for evaluation, if those people are experts).

At first glance, it looks like a comprehensive, evidence-driven report by a group of experts. As a result, the non-expert reader who can’t really evaluate the base claims is left with little in the way of good heuristics upon which to base their evaluation, which leads to them falling back on bad thinking.

This is exactly why the author wrote and presented the paper in such a format.

Doing so is a form of sophistry, where you care more about winning the argument than making your case in good faith. It’s pretty rich to be saying you want your claims evaluated on the merits while also going out of your way to encourage people to do the opposite.

If you are someone who has influence, who has an audience, who tries to persuade people that your ideas are true, you have a responsibility to not only make good arguments, but to provide them in a format and context that encourages the right kind of evaluation of those arguments.

And if you are reluctant to do so because it weakens your case, maybe you should rethink what you’re doing.

Much love,
Ryan