Another example of a reporter — this time at the BBC — completely screwing up a story. In a February 27, 2004 story, Scientists doubt animal research, the BBC reports on a study published the British Medical of Journal which was basically nothing more than an animal rights attack on animal research. But whoever wrote the BBC story screwed up and didn’t have a clue what the BMJ paper actually said. Here’s how the BBC describes the paper,

In reaching their conclusions, the London team carried out a systematic review of all animal experiments which purported to have clinical relevance to humans.

Conducting a review of all animal experiments that had some sort of clinical relevance would be a gargantuan task that would likely take years to accomplish. A lot of studies from the 19th and 20th centuries likely don’t even exist in electronic formats yet.

Which is why the researchers didn’t even attempt to do what the BBC claims they actually accomplished. As the paper makes extraordinarily clear,

We searched Medline to identify published systematic reviews of animal experiments (see bmj.com for the search strategy). The search identified 277 possible papers, of which 22 were reports of systematic reviews. We are also aware of one recently published study and two unpublished studies, bringing the total to 25. Three further studies are in progress (M Macleod, personal communication).

Leave it to the BBC to think that doing a Medline search for studies that systematically reviewed certain animal experiments is the same thing as systematically reviewing “all animal experiments which purported to have clinical relevance to humans.”

This is an especially pernicious error because one of the major problems this study has is one of selection bias. If a drug is supposed to do X but doesn’t, then researchers have a good chance of publishing a review of the animal research to see if there was anything that could have indicated the problem before going to market with the drug (ironically, in the reviews mentioned by the study, the problem was not the animal research itself but rather how the animal research results were used — or more accurately not used — by clinical researchers). Of course if a drug does exactly what it was supposed to do, then good luck getting an analysis of why nothing went wrong published. The researchers did the equivalent of tracking down man bites dog stories and concluding that more people bite dogs than vice versa.

But you’d never know that from the BBC which can’t even accurately describe the study.