Let Free Will Be

We’ve all seen them, those colorful images that show how our brains "light up" when we’re in love, playing a video game, craving chocolate, etc. Created using functional magnetic resonance imaging, or fMRI, these pictures are the basis of tens of thousands of scientific papers, the backdrop to TED talks and supporting evidence in best-selling books that tell us how to maintain healthy relationships, make decisions, market products and lose weight.

But a study published last month in the Proceedings of the National Academy of Sciences uncovered flaws in the software researchers rely on to analyze fMRI data. The glitch can cause false positives — suggesting brain activity where there is none — up to 70 percent of the time.

“It’s impossible to know how many fMRI studies are wrong, since we do not have access to the original data,” says computer scientist Anders Eklund of Linkoping University in Sweden, who conducted the analysis.

This cued a chorus of “I told you so!” from critics who have long said fMRI is nothing more than high-tech phrenology. Brain-imaging researchers protested that the software problems were not as bad nor as widespread as the study suggested.

The dust-up has caused considerable angst in the fMRI community, about not only the reliability of their pretty pictures but also how limited funding and the pressure to publish splashy results might have allowed such a mistake to go unnoticed for so long. The remedial measures some in the field are now proposing could be a model for the wider scientific community, which, despite breathtaking technological advances, often produces findings that don’t hold up over time.

Developed in the 1990s, fMRI creates images based on the differential effects a strong magnetic field has on brain tissue. The scans occur at a rate of about one per second, and software divides each scan into around 200,000 voxels — cube-shaped pixels — each containing about a million brain cells.

The software then infers neural activity within voxels or clusters of voxels, based on detected blood flow (the areas that "light up.") Comparisons are made between voxels of a resting brain and voxels of a brain that is doing something like, say, looking at a picture of Hillary Clinton, to try to deduce what the subject might be thinking or feeling depending on which area of the brain is activated.

But when you divide the brain into bitty bits and make millions of calculations according to a bunch of inferences, there are abundant opportunities for error, particularly when you are relying on software to do much of the work. This was made glaringly apparent back in 2009, when a graduate student conducted an fMRI scan of a dead salmon and found neural activity in its brain when it was shown photographs of humans in social situations. Again, it was a salmon. And it was dead.

This is not to say all fMRI research is hooey. But it does indicate that methods matter even when using whiz-bang technology. In the case of the dead salmon, what was needed was to statistically correct for false positives that arise when you make so many comparisons between voxels.

The authors of the paper on the software glitch found that a vast majority of published papers in the field do not make this "multiple comparison" correction. But when they do, they said, the most widely used fMRI data analysis software often doesn’t do it adequately.

To try to create some consistency and enhance credibility, a lengthy report titled "Best Practices in Data Analysis and Sharing in Neuroimaging Using MRI" was recently published. The intent was to increase transparency through comprehensive sharing of data, research methods and final results so that other investigators could “reproduce findings with the same data, better interrogate the methodology used and, ultimately, make the best use of research funding by allowing reuse of data.”

The shocker is that transparency and reproducibility aren’t already required, given that we’re talking about often publicly funded, peer-reviewed, published research. And it’s much the same in other scientific disciplines.

Indeed, a study published last year in the journal Science found that researchers could replicate only 39 percent of 100 studies appearing in three high-ranking psychology journals. Research similarly hasn’t held up in genetics, nutrition, physics and oncology. The fMRI errors added fuel to what many are calling a reproducibility crisis.

Data-sharing platforms such as OpenfMRI and Neurovault have already been established to make fMRI data and statistical methods more widely accessible. In fact, it was data sharing that revealed the fMRI software glitch. ”If we don’t have access to the data, we cannot say if studies are wrong,” says Anders Eklund. “Finding errors is how scientific fields evolve. This is how science gets done.”
(October 2016)