“A study shows…”

There is a growing epidemic of reference-based medicine and confirmation bias. Here’s how to spot it.

In the age of information overload, finding a published study to support a viewpoint has become effortless. The phrase “A study shows…” has become ubiquitous, but what does it truly signify? Is it reliable evidence for the claim(s) made or simply one of many studies aligning with a desired narrative — confirmation bias in full flow? The devil is in the detail.

In 1979, the year of my birth, there were 282,120 publications indexed in PubMed. By 2022, that number had surged to 1,770,974 — a nearly six-fold increase in my lifetime. This exponential growth means that it has never been easier to find a published study and cry out “evidence-based”. However, the chances the study genuinely provides solid evidence for the claim made are likely to be small.

To communicate health research accurately, we must prioritize attention to detail and adhere to the principles of sound and fair critical appraisal. Unfortunately, the reality is that many claiming “evidence-based” lack the necessary skills to conduct appraisals of its strengths and weaknesses. Abilities often extend only to finding studies and, if lucky, commenting on rudimentary aspects of study design.

In medicine and public health, causality and the degree of (un)certainty we can claim from health research are based on critical factors such as the strength of the association and coherence within the existing knowledge base. These principles align with and build upon Bradford Hill’s suggestions for establishing truth through a positivist epistemological approach to science. Importantly, any claims should be based on the best available evidence established through systematic collation of relevant studies and their appraisal.

Regrettably, most communications of research evidence rely on finding any study that supports a predetermined narrative, merely stating “A study shows…” Few, if any, of these communications delve into the essential details required to present a comprehensive, fair and accurate picture of the quality and certainty of evidence. Consequently, we are witnessing a growing epidemic of reference-based medicine (RBM) — everyone in a PubMed race to the bottom, talking past each other on the way down.

In the rare instances when more details are considered, scrutiny is solely on those studies contradicting the main narrative. Biases become apparent when authors scrutinize the methodology of opposing studies to highlight (possibly legitimate) weaknesses in their findings but a simple “A study shows…” approach is taken to studies with more supportive findings. This creates an illusion that evidence for a particular narrative is less reliable.

Even when evidence is critiqued in more detail, still important methodological aspects are frequently overlooked. Evidence often then serves as a means to an end — convince the audience that “the science”, which we must be seen to follow lest our thinking be viewed as residing in the dark ages, supports the message.

The COVID-19 pandemic has exposed the epidemic of reference-based medicine in science communication. The cure lies in paying greater attention to detail and the application of the skills of evidence-based health care and critical appraisal.

The first question I ask when I see “A study shows…” is “How do you know?”. It’s a useful way to find out if claims of “evidence-based” are simply reference-based.