PAPER: How reliable are the results from functional magnetic resonance imaging?
- Current Citation:
Bennett CM, Miller MB. (in press). How reliable are the results from functional magnetic resonance imaging? Annals of the New York Academy of Sciences.
- Abstract:
Functional magnetic resonance imaging is one of the most important methods for in vivo investigation of cognitive processes in the human brain. Within the last two decades an explosion of research has emerged using fMRI, revealing the underpinnings of everything from motor and sensory processes to the foundations of social cognition. While these results have revealed the potential of neuroimaging, important questions regarding the reliability of these results remain unanswered. In this chapter we take a close look at what is currently known about the reliability of fMRI findings. First, we examine the many factors that influence the quality of acquired fMRI data. We also conduct a review of the existing literature to determine if some measure of agreement has emerged regarding the reliability of fMRI. Finally, we provide commentary on ways to improve fMRI reliability and what questions remain unanswered. Reliability is the foundation on which scientific investigation is based. How reliable are the results from fMRI?
- Downloadable Versions:
[Manuscript PDF]
LOVE Conference Wrapup
The Lake Ontario Visionary Establishment (LOVE) conference just wrapped up and, I have to say, it was a genuinely fantastic experience. I gave a lighthearted presentation on Type I error and reliability in functional imaging, which hopefully made the message a bit easier to swallow. I also got the chance to catch up with longtime friends while making some new aquaintances. Thanks to the organizers Daniel Ansari and Jonathan Fugelsang for having me up to present.
For all those who are interested: you can download a copy of my presentation slides here.
Send me an email if you have any questions or comments. Thanks!
Spring/Summer 2010 Conference Schedule
It is going to be a busy conference season again this spring. I will be at the following professional gatherings over the next few months – send me an email if you will be attending as well and would like to meet up. I’ll buy the first round and we can talk shop.
Lake Ontario Visionary Establishment Conference [LOVE]
February 11-12, Niagra Falls, NY Ontario, Canada
Cognitive Neuroscience Society Conference [CNS]
April 17-20, Montreal, Canada
Association for Psychological Science Convention [APS]
May 27-30, Boston, MA
Organization for Human Brain Mapping Conference [HBM]
June 6-10, Barcelona, Spain
Applied Human Factors and Ergonomics Conference [AHFE]
July 17-20, Miami, FL
Here is some of what I will be presenting:
[LOVE]
• Special topic talk: ‘Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: addressing the multiple comparisons problem in fMRI.’
[APS]
• Invited talk: ‘The development of interoceptive information processing across adolescence.’
[CNS] [HBM]
• Poster: ‘How reliable are the results from fMRI?’
Bennett CM, Guerin SA, Donovan CL, Miller MB
[HBM]
• Poster: ‘A device for simultaneous thermal and tactile stimulation in an MR environment.’
Bennett CM
Quote of the Week – Pashler
“It’s hellishly complicated, this data analysis, and that creates great opportunity for inadvertent mischief.” – Hal Pashler (As seen in Science News)
PAPER: The Principled Control of False Positives in Neuroimaging
- Current Citation:
Bennett CM, Wolford GL, Miller MB. (in press). The Principled Control of False Positives in Neuroimaging. Social Cognitive and Affective Neuroscience.
- Abstract:
An incredible amount of data is generated in the course of a functional neuroimaging experiment. The quantity of data gives us improved temporal and spatial resolution with which to evaluate our results. It also creates a staggering multiple testing problem. A number of methods have been created that address the multiple testing problem in neuroimaging in a principled fashion. These methods place limits on either the familywise error rate (FWER) or the false discovery rate (FDR) of the results. These principled approaches are well established in the literature and are known to properly limit the amount of false positives across the whole brain. However, a minority of papers are still published every month using methods that are improperly corrected for the number of tests conducted. These latter methods place limits on the voxelwise probability of a false positive and yield no information on the global rate of false positives in the results. In this commentary we argue in favor of a principled approach to the multiple testing problem – one that places appropriate limits on the rate of false positives across the whole brain and gives the reader the information they need to properly evaluate the results.
- Downloadable Versions:
[Manuscript PDF]
Holiday Presents for a Neurogeek
I know this post might be a bit late in the season to make much of an impact on your shopping plans, but if your loved ones can’t get enough neuroscience then here are some thoughts for great gifts. Some are specific to neuroscience, while others are more general and appropriate for any academic. Enjoy!
General Neuroscience.
- Book: Functional Magnetic Resonance Imaging
I picked this up a few weeks ago since I heard it had a good section on signal and noise in fMRI. What I found was, far and away, the best single introduction to fMRI that I have run across. If I am ever fortunate enough to run my own lab then I will see to it that all new lab members are handed this book as soon as they step in the door. It’s that good.
- Plush: Neuron or set of Neurons. ~$12-$24
How much cute can a few dollars buy? Quite a bit, apparently. I have a set of plush neurons in my office. The best part is that they can slot into each other, forming neural networks! I love it.
- T-Shirt: I ♥ Brains. ~$20
Don’t hide your love, share it with the world. While there may be other organs in the body , the brain is where it’s at.
- Poster: Think Hard Print (Map of the brain’s surface). ~$18
The folks at Ork Posters are well-known for their city neighborhood posters. In this case they turned their creative talent to the neighborhoods of the brain, and created a great piece of art. It’s even anatomically correct.
- Book: This Is Your Brain on Music
I purchased this book on a whim two years ago and was very pleasantly surprised at how good it is. Music (and dance) are a key part of the human condition. With this book you can learn more about what makes music so special within the brain.
- Book: The Human Brain Coloring Book
What coloring books do your kids have? Disney? Pokemon? Upgrade them to something better – something that even med school students use to help learn neuroanatomy. I purchased my first brain coloring book when I was an undergrad. It was great then, and it remains great now.
- Tool: Atlas of the Human Brain
When you start getting serious about the brain then you are going to need a serious map to help guide you. My personal standby is the Atlas of the Human Brain by Mai, Paxinos, and Assheuer. It is a great reference book with excellent illustrations. As a bonus the atlas comes with a DVD containing PDFs of all the book material. Copy the DVD to your laptop and you will have your atlas with you everywhere you go.
- Tool: Somso Human Brain Model. ~$LOTS
One day someone will explain to me why plastic models of the human brain must cost hundreds of dollars. For now I am a bit lost regarding their exorbitant cost. Still, these models are incredibly handy to have around when discussing brain anatomy or function. The link goes to one example of a human brain model, but there are many variations on the theme available. It is not impossible to spend $1000+ on a really good version.
General Academia.
- Writing Tool: Moleskine
There are times when academics are out there, on the front line. Lab meetings. Department presentations. Lunch with a collaborator. Conferences. In these battles you need the best weapons you can get. Don’t get caught with your pants down – always have solid instruments along with you. It has taken years of careful testing, but I have settled on Moleskine notebooks and the Copic Multiliner SP pen. Get the Moleskine with graph paper, and get the 0.35 mm tip Multiliner. Make sure to get the SP series, because you deserve a rugged aluminum body.
- Writing Tool: Any
Sometimes I long for a typewriter when I am writing a new manuscript. Part of the allure is the romance – feeding the paper in and hearing the click-clack of the hammers striking the page. The biggest advantage though? THERE IS NO INTERNET ON A TYPEWRITER. If you know someone who is as distractible as I am then drop the $15 and buy them a kitchen timer. Tell them to set it for twenty minutes and make sure to work for that length of time. Then, when time has elapsed, you get ten minutes to do whatever you want. This ‘dash’ method has saved my bacon, and it is well worth the small cost to give it a try. Learn more here.
- Presentation Tool: Kensington Wireless Clicker
From the audience it can be a bit humorous when the speaker can’t seem to get their Powerpoint slides to advance. Conversely, it is hell when forty pairs of eyes are watching you fumble around at the podium. If you are presenting in the near future, get a clicker that you can trust. I have found this Kensington model to be worthy. You can get this clicker with a laser pointer
- Book: PhD Comics, first
Let’s get something squared away right off the bat: Jorge Cham saves lives. His creation, PhD comics, details the everyday insanity that every grad student must deal with. Take a few minutes and surf over to the website and read a few panels, just to get a feel for it. If you know anyone who has ever struggled with the soul-crushing madness of grad school then any one of these books will be a cathartic experience. Also check out the PhD Comics online store.
- Software: Papers, the personal research library (Mac OS X). ~$42
I have several thousand PDF files on my computer. Now, suppose I need to find ONE of them. In the bad old days I would have the PDFs organized by topic in a series of folders on my computer. To find the right one I would have to remember what topic it might be under, or else face the time-sucking wrath of the Finder’s search tool. Now, enter Papers, the iTunes of PDF articles. It will properly store and organize all your academic PDF files. Want to see all the articles for a specific author? Done. Want to see all articles you have from a specific journal? Done. Need to build a list of articles that will be useful for your next paper? Done and done. A simple and beautiful program. Try it out for 30 days and decide if it works for you. They even give an academic discount!
- Reading Gadget: Amazon Kindle DX Reader
When the Kindle first came out I quickly dismissed it as a device with a lot of promise, but limited by various hardware and software shortcomings. No longer. With the Kindle DX things start getting really interesting for academics. The device natively support the PDF file format, which means that all of the journal articles we have downloaded can be opened. Further, the screen is large enough to be able to read those articles pretty comfortably. The Kindle might be a unique solution if you are looking to go all-digital.
Quote of the Week – Logothetis
“fMRI is a measure of mass action. You almost have to be a professional moron to think you’re saying something profound about the neural mechanisms. You’re nowhere close to explaining what’s happening, but you have a nice framework, an excellent starting point.” ~ Nikos Logothetis (As seen in Science News)
Live Sectioning of HM’s Brain
The Brain Observatory at UCSD is doing a live feed of the histological sectioning of patient HM’s brain today. The feed will continue for the next two days while they slice through HM’s brain by fractions of a millimeter at a time. You can view the feed yourself at the following link:
http://thebrainobservatory.ucsd.edu/hm_live.php.
The studies done with HM revolutionized our understanding of human memory. His case remains one of the most important in the history of psychology and cognitive science. If you aren’t familiar with patient HM then take a few minutes and go through the Wikipedia article on him:
http://en.wikipedia.org/wiki/HM_(patient)
The War on Fish: False Positive Horror Stories
Citizens of the Interwebs – we are in need of your assistance!
My advisor Mike Miller and I have been asked to write a commentary in a major neuroimaging journal that discusses the importance of protecting against false positives (Type I error) in fMRI. This is essentially an extension of the arguments that we made in the Atlantic Salmon paper. The commentary will be published alongside a similar piece from a separate group of authors that discusses the relative merits of avoiding false negatives (Type II error).
As part of our commentary we are collecting horror stories of what can happen when high rates of false positives are allowed to be present in imaging results. Have you ever looked at an article that immediately sounded your BS alarm? Perhaps you have spent time hopelessly trying to replicate a study that refused to cooperate. There are a thousand ways in which false positives can negatively impact cognitive neuroscience. If you know of a good anecdote I would love to hear about it.
If you would like to share your story online then you can post it below in the comments. Alternatively you can simply email it to me directly. No names, places, or dates are necessary please. We’re looking forward to hearing from you!
Best ~ Craig [Prefrontal]
The Internet Found the Atlantic Salmon
The last 72 hours have seen an incredible increase in traffic here at prefrontal.org. To sum it up in a single sentence: the site has received as many hits in the last three days as it has during the past two years. Yeah, really. My activity graph on the Wordpress Dashboard looks like this:
It seems that late last week a few major neuroscience weblogs discovered the Salmon poster and decided to write up summaries. Those readers then posted to their weblogs, whose readers posted to their weblogs, and so on. By 10am Friday morning the prefrontal.org activity meter was pegged and my inbox was full.
A few important bits of info:
* The current status of the Salmon is that we are trying to publish it as an editorial in a major neuroimaging journal. We are very close to resubmitting, only needing to complete a survey on the prevalence of multiple comparisons correction in the previous neuroimaging literature. We hope that it will be released in the near future.
* If you would like to be sent a copy of the commentary if/when it becomes published just send me an email and I will put you on the list.
* Some sites have played up how difficult it has been for us to get the Salmon published. We have received some, well, interesting feedback by a few editors in the course of our submission. Still, it has not been more difficult than average to get the Salmon commentary published (so far).
* The goal of the Salmon poster was to encourage the minority of researchers who report uncorrected statistics to move forward and begin using basic multiple comparisons correction in their research. The Salmon doesn’t add anything to the technical discussion of how multiple comparisons correction is performed, it is simply a salient reminder of why proper correction is always necessary.
* None of the authors intended for the Salmon to go public in such a big way, especially before the commentary was reviewed and published. We were actually quite content to publish our editorial in a neuroimaging journal and be done with it. We feel that, fundamentally, this is an internal debate within the field of neuroimaging.
Some of the best/notable writeups that I have found:
* http://neuroskeptic.blogspot.com
* http://languagelog.ldc.upenn.edu
* http://lawandbiosciences.wordpress.com
* http://www.mindhacks.com
* http://www.wired.com
* http://www.newscientist.com/blogs/shortsharpscience
* http://blogs.nature.com/news/thegreatbeyond
* http://blogs.discovermagazine.com/discoblog/
* http://chronicle.com/
* http://slashdot.org
Some of the best comments that I have run across:
“The recorded signal is changing due to noise. The point of the experiment is that if you look at enough signals, the noise in one will match the timing of your experimental stimulus, purely out of chance. Another way of looking at it is this: if you choose a statistical threshold of p 0.05 then, statistically, you expect a result that is significant at that level purely out of chance once in every twenty experiments. When you’re analyzing images, or worse volumes, pixel by pixel, you’re doing a LOT of comparisons. If you don’t correct for that you WILL get false positives, no matter what you’re looking at.” – ceoyoyo
“But not everyone uses multiple comparisons correction. This is where the fish comes in – Bennett et al show that if you don’t use it, you can find “neural activation” even in the tiny brain of dead fish. Of course, with the appropriate correction, you don’t. There’s nothing original about this, except the colourful nature of the example – but many fMRI publications still report “uncorrected” results” – Neuroskeptic
“… it seems to me like that their point wasn’t that the fMRI wasn’t sensitive enough, or particular enough. Instead the problem seems to be a problem of statistically expected random noise. Their point seems to be that users of an fMRI should bear in mind that their marvelous magical machine can generate “real” errors, and that basic, common-sense multiple comparison habits should be developed, instead of a take a picture, slap a stat against it approach.” – Nemus
“The entire point the write up was to warn about the danger of false positives. Your attributing of brain activity to random, natural noise is exactly the danger they want to avoid.” – Anonymous
“The trouble is, most scientists are not mathematicians, and have no good theoretical understanding of statistics. Most people pushing buttons in SPSS or SAS (or what have you) are just doing “cargo cult” mathematics. Ask them to justify why their “very conservative” confidence interval for a given test is appropriate when dealing with eleventy billion variables, or why a particular post-hoc test is the proper one to use, and they’ll look at you like you just asked them to prove that the sky is blue.” – Anonymous
“Actually, the voodoo correlations paper is actually talking about performing correlations between the signals we get from fMRI scans (you can read the actual paper instead of the somewhat misleading article here [edvul.com]), and other measurements or scores. This doesn’t do that at all. This is about the danger of false positives in fMRI imaging, because of the large number of statistical tests that are done across the brain. The majority of peer reviewed published fMRI papers do some type of multiple comparisons correction to attempt to adjust for this problem.” – daenris
Some of the more terrible writeups that I have found:
By and large the comments have been quite good. However, there have been a few people arguing that the dead fish is actually still thinking or that we have observed evidence of the ethereal soul. I am not going to quote the comments here, but it has been a bit amusing to see this play out…
The funniest comments so far:
“Of course, Bennett’s group don’t mean to suggest that a post-mortem salmon is capable of perspective-taking. Cod forbid.”
- Kerri
No sir. What it proves is the existence of the sole.
-Jeremi
Yeah, the measurements were right off the scales.
- grcumb
Thank you… he’s here all week. Try the fish.
- theshowmecanuck
I’m wondering if the issue could be resolved if the salmon was smoked and served with cream cheese…
- G
No, what it proves is that while you can tune an fMRI, you can’t tuna fish.
- limekiller4
I would think that a salmon in an MRI would be thinking more along the lines of “HOLY FUCK! I CAN’T BREATHE!”
- geminidomino
Does the scientific method for biologists exclude barbeques?
- value_added
And I, for one, welcome our new zombie salmon overlords!!
- DarkOx
Why wasn’t this published? Maybe the reviewers considered the experiment a bit fishy …
- maxwell_demon
First, the fish wasn’t dead, it was just tenured.
- jesor
I demand that fMRI techniques get a fair herring!
- Bob O’H
Scream if you love the multiple comparisons problem! AAAAAAAAAAAAAAAAHHHHHHHHHHH!
- Jess
… compared with how Vul et al. handled a similar topic, this is a party with clowns and flowers
- powrogers
The joke possibilities are endless but I won’t bother. It’s like shooting fish in a barrel.
- Anonymous
A common mistake made in discussions of taxonomy is overlooking the issue of whether closely related species taste the same. In this case, you omitted the fact that all of them are great when grilled. With a slice of lemon on the side.
- value_added
HOWEVER a fish that has been caught, killed/gutted, frozen, shipped, sold by auction, shipped again, sold again, taken to a hospital and put in an MRI machine is a dead fish. He ain’t pining for human faces, he has passed on. This fish is no more. He has ceased to be. He’s expired and gone to meet his maker. He’s a stuff. Bereft of life, he rests in filets! If you hadn’t glued him to his tank he’d be pushing up the seaweed. Its brainactivity is now history. He’s out of the pond. He’s kicked the tank, he’s shuffled of his mortal coil, run down the river and joined the bleeding choir invisibisble. This is an EX-SALMON!
- Anonymous
Conclusion
I just want to say that it has been great to see the discussion the Salmon has generated in the last few days. Our hope for this work was that it would call new attention to the multiple comparisons problem. I think that we can safely say that it has. Thanks.
~ Craig [Prefrontal].