Fun With Statistics: surviving the Hunger Games

Is it really “reviving” a blog when you’ve only posted once or twice?

Anyway, here’s my re-entry into science blogging: a delightfully thorough and silly survival analysis for The Hunger Games, written by Brett Keller.  Fantastic.

Posted in fun things, wish I'd thought of that | Leave a comment

ESP, peer review, and extraordinary evidence.

“Extraordinary claims require extraordinary evidence.”  –Carl Sagan

At the beginning of January, the NYT published an article entitled “Journal’s Paper on ESP Expected to Prompt Outrage.” It describes the scientific community’s (by no means monolithic) reaction to a forthcoming publication purporting to show statistical evidence for the existence of extrasensory perception; or, in Professor Bem’s words, “feeling the future.”  Reactions seem to run the gamut from embarrassment, to anger, to amusement, but as far as I can tell, researchers seem to have settled into two camps: those criticizing the peer review process, and those criticizing the statistical methodology.

The original article, set to be published in the Journal of Personality and Social Psychology, may be found here.  It’s an interesting read, and gives exhaustive detail on the experimental methodology, which I’m not actually going to get into here.  Let’s take it on faith that Professor Bem conducted his experiments appropriately and that the data he generated are accurate.

This conflict, at its core, underlines the clash between classical and Bayesian statistics, and for me, this is where things start to get interesting.  This is an old argument, and better minds than mine have written extensively and persuasively in favor of a Bayesian approach (Cohen’s paper, “The Earth Is Round (p<.05)” is excellent, and I highly recommend giving it a read).  Rather than get into an extended description of the two methodologies here- an undertaking that could fill books, let alone a measly blog post- I’ll give things a quick and over-simplified gloss, and reserve the right to come back to the topic later.

Essentially, classical approaches require the statistician to begin each analysis as though he or she has amnesia.  No matter how many studies have been performed in the same area, each new analysis begins from scratch.  Imagine, for example, a pharmaceuticals company testing a new drug.  If over 100 trials, the new drug was no more effective than a placebo, and then in the 101st experiment, a difference was shown, it’s as though the first 100 trials never happened (in fact, the pharma company can take just that one successful trial to the FDA- they don’t have to say anything about the original 100*.  But that’s a post for another day).

A Bayesian approach allows us to incorporate the information from the 100 failed trials into our analysis of the 101st trial.  This information- known as a “prior”- carries a different weight in the analysis, depending on how much we know.  Rather than relying on the decision rules used in classical statistics, an empirical Bayesian analysis offers us the opportunity to adjust what we knew previously in light of new data.

One of the papers written in response to Bem’s article is this excellent Bayesian analysis done by Jeffrey Rouder and Richard Morey.  It contains a useful primer on empirical Bayes analysis.   Rather than dismissing Bem’s findings out of hand, they use the data he provides to calculate a “Bayes factor” of about 40, which is considered to be rather convincing evidence that an effect exists.   Rouder and Morey then offer this elegant solution to the question of how to consider the results of Bem’s experiments: assuming that the skeptical reader believes that the likelihood of ESP being a real phenomenon is somewhere in the region of 1/1,000,000, Bem’s data suggest that this prior knowledge should be adjusted by a factor of about 40.  In other words, the reader should now believe that there is about a 1/25,000 chance of ESP really existing- certainly believing it more likely than before, but probably still not betting on it.  If future trials continue to show similar evidence, that 1/25,000 figure should gradually decrease, reflecting a greater prior knowledge of the phenomenon.

How should we view the publication of Bem’s article, then: as a failure of peer review or as a failure of methodology?  I don’t think this is a case of a bad article getting past a peer reviewer asleep at the switch- the statistical methods he used are the same ones used routinely in the biomedical literature.  Few researchers employ Bayesian methods, and as such, it is rarely suggested as a revision (in fact, a Bayesian approach is more likely to be   challenged by a reviewer than a classical approach).

But in a similar line of reasoning, Bem’s reported statistics are the statistics currently used by most researchers to test whether a new medication works better than a placebo, or to test how smoking or saturated fats or cell phones are associated with cancer risk.  They are, in fact, used all the time to shape major policy decisions. That they are inadequate and misleading is obvious in this case- but only because our prior beliefs about the phenomenon in question are so deeply held.

I’d suggest that it is not an extraordinary claim that requires extraordinary proof, but rather, a claim that may have far-reaching effects on the public health.  Are Bem’s claims outlandish?  Of course.  But they’re also ultimately frivolous. Let Bem and his colleagues publish on ESP to their hearts’ content, if only we begin to require a higher statistical standard for studies that truly matter.

 

*This is less true than it used to be, given the more widespread awareness of clinicaltrials.gov, but publication bias in the biomedical literature is still a serious problem.

Posted in Uncategorized | Leave a comment

A methodologist on a mission.

One of the less wonderful side effects of working as a statistician is that it’s very easy to miss the forest for the trees.  I get lost in the details and sometimes forget to take a step back to admire (or cringe at) the bigger picture.  This is a problem, I think, because the bigger picture is necessary for perspective.

My goal, starting out, is to take a look at how the lay press reports on scientific literature, and to try to evaluate where and how it all fits in- to my field, and to science as a whole.  Lofty goals.  Let’s see how it goes.

Posted in Uncategorized | Leave a comment