The Internet Found the Atlantic Salmon
The last 72 hours have seen an incredible increase in traffic here at prefrontal.org. To sum it up in a single sentence: the site has received as many hits in the last three days as it has during the past two years. Yeah, really. My activity graph on the WordPress Dashboard looks like this:
It seems that late last week a few major neuroscience weblogs discovered the Salmon poster and decided to write up summaries. Those readers then posted to their weblogs, whose readers posted to their weblogs, and so on. By 10am Friday morning the prefrontal.org activity meter was pegged and my inbox was full.
A few important bits of info:
* The current status of the Salmon is that we are trying to publish it as an editorial in a major neuroimaging journal. We are very close to resubmitting, only needing to complete a survey on the prevalence of multiple comparisons correction in the previous neuroimaging literature. We hope that it will be released in the near future.
* If you would like to be sent a copy of the commentary if/when it becomes published just send me an email and I will put you on the list.
* Some sites have played up how difficult it has been for us to get the Salmon published. We have received some, well, interesting feedback by a few editors in the course of our submission. Still, it has not been more difficult than average to get the Salmon commentary published (so far).
* The goal of the Salmon poster was to encourage the minority of researchers who report uncorrected statistics to move forward and begin using basic multiple comparisons correction in their research. The Salmon doesn’t add anything to the technical discussion of how multiple comparisons correction is performed, it is simply a salient reminder of why proper correction is always necessary.
* None of the authors intended for the Salmon to go public in such a big way, especially before the commentary was reviewed and published. We were actually quite content to publish our editorial in a neuroimaging journal and be done with it. We feel that, fundamentally, this is an internal debate within the field of neuroimaging.
Some of the best/notable writeups that I have found:
* http://neuroskeptic.blogspot.com
* http://languagelog.ldc.upenn.edu
* http://lawandbiosciences.wordpress.com
* http://www.mindhacks.com
* http://www.wired.com
* http://www.newscientist.com/blogs/shortsharpscience
* http://blogs.nature.com/news/thegreatbeyond
* http://blogs.discovermagazine.com/discoblog/
* http://chronicle.com/
* http://slashdot.org
Some of the best comments that I have run across:
“The recorded signal is changing due to noise. The point of the experiment is that if you look at enough signals, the noise in one will match the timing of your experimental stimulus, purely out of chance. Another way of looking at it is this: if you choose a statistical threshold of p 0.05 then, statistically, you expect a result that is significant at that level purely out of chance once in every twenty experiments. When you’re analyzing images, or worse volumes, pixel by pixel, you’re doing a LOT of comparisons. If you don’t correct for that you WILL get false positives, no matter what you’re looking at.” – ceoyoyo
“But not everyone uses multiple comparisons correction. This is where the fish comes in – Bennett et al show that if you don’t use it, you can find “neural activation” even in the tiny brain of dead fish. Of course, with the appropriate correction, you don’t. There’s nothing original about this, except the colourful nature of the example – but many fMRI publications still report “uncorrected” results” – Neuroskeptic
“… it seems to me like that their point wasn’t that the fMRI wasn’t sensitive enough, or particular enough. Instead the problem seems to be a problem of statistically expected random noise. Their point seems to be that users of an fMRI should bear in mind that their marvelous magical machine can generate “real” errors, and that basic, common-sense multiple comparison habits should be developed, instead of a take a picture, slap a stat against it approach.” – Nemus
“The entire point the write up was to warn about the danger of false positives. Your attributing of brain activity to random, natural noise is exactly the danger they want to avoid.” – Anonymous
“The trouble is, most scientists are not mathematicians, and have no good theoretical understanding of statistics. Most people pushing buttons in SPSS or SAS (or what have you) are just doing “cargo cult” mathematics. Ask them to justify why their “very conservative” confidence interval for a given test is appropriate when dealing with eleventy billion variables, or why a particular post-hoc test is the proper one to use, and they’ll look at you like you just asked them to prove that the sky is blue.” – Anonymous
“Actually, the voodoo correlations paper is actually talking about performing correlations between the signals we get from fMRI scans (you can read the actual paper instead of the somewhat misleading article here [edvul.com]), and other measurements or scores. This doesn’t do that at all. This is about the danger of false positives in fMRI imaging, because of the large number of statistical tests that are done across the brain. The majority of peer reviewed published fMRI papers do some type of multiple comparisons correction to attempt to adjust for this problem.” – daenris
Some of the more terrible writeups that I have found:
By and large the comments have been quite good. However, there have been a few people arguing that the dead fish is actually still thinking or that we have observed evidence of the ethereal soul. I am not going to quote the comments here, but it has been a bit amusing to see this play out…
The funniest comments so far:
“Of course, Bennett’s group don’t mean to suggest that a post-mortem salmon is capable of perspective-taking. Cod forbid.”
– Kerri
No sir. What it proves is the existence of the sole.
-Jeremi
Yeah, the measurements were right off the scales.
– grcumb
Thank you… he’s here all week. Try the fish.
– theshowmecanuck
I’m wondering if the issue could be resolved if the salmon was smoked and served with cream cheese…
– G
No, what it proves is that while you can tune an fMRI, you can’t tuna fish.
– limekiller4
I would think that a salmon in an MRI would be thinking more along the lines of “HOLY FUCK! I CAN’T BREATHE!”
– geminidomino
Does the scientific method for biologists exclude barbeques?
– value_added
And I, for one, welcome our new zombie salmon overlords!!
– DarkOx
Why wasn’t this published? Maybe the reviewers considered the experiment a bit fishy …
– maxwell_demon
First, the fish wasn’t dead, it was just tenured.
– jesor
I demand that fMRI techniques get a fair herring!
– Bob O’H
Scream if you love the multiple comparisons problem! AAAAAAAAAAAAAAAAHHHHHHHHHHH!
– Jess
… compared with how Vul et al. handled a similar topic, this is a party with clowns and flowers
– powrogers
The joke possibilities are endless but I won’t bother. It’s like shooting fish in a barrel.
– Anonymous
A common mistake made in discussions of taxonomy is overlooking the issue of whether closely related species taste the same. In this case, you omitted the fact that all of them are great when grilled. With a slice of lemon on the side.
– value_added
HOWEVER a fish that has been caught, killed/gutted, frozen, shipped, sold by auction, shipped again, sold again, taken to a hospital and put in an MRI machine is a dead fish. He ain’t pining for human faces, he has passed on. This fish is no more. He has ceased to be. He’s expired and gone to meet his maker. He’s a stuff. Bereft of life, he rests in filets! If you hadn’t glued him to his tank he’d be pushing up the seaweed. Its brainactivity is now history. He’s out of the pond. He’s kicked the tank, he’s shuffled of his mortal coil, run down the river and joined the bleeding choir invisibisble. This is an EX-SALMON!
– Anonymous
Conclusion
I just want to say that it has been great to see the discussion the Salmon has generated in the last few days. Our hope for this work was that it would call new attention to the multiple comparisons problem. I think that we can safely say that it has. Thanks.
~ Craig [Prefrontal].
The Story Behind the Atlantic Salmon
The Atlantic Salmon fMRI poster has garnered a fair amount of attention since its presentation at the Human Brain Mapping conference last June in San Francisco. So far the reaction from other researchers has been almost unanimously positive. A sizable number of people stopped by the poster while it was displayed and Rainer Goebel (of BrainVoyager fame) was kind enough to give the fish a shout-out during the closing ceremonies (see photo).
All in all I am quite pleased that the Salmon seems to be generating a fresh discussion of multiple comparisons correction in neuroimaging.But, how did it all begin? I mean, really, why would anybody want to scan a fish? This was one of the top five questions I was asked during the HBM poster session. It it a story that deserves to be told, and a weblog post is perhaps the ideal medium to tell it. So, for all readers who are curious, I have written up the story of the Salmon.
The story begins during my first year in graduate school at Dartmouth College. I was working with Abigail Baird on fMRI studies investigating the maturation of decision-making and we were developing a large number of new MRI protocols to use with adolescents and adults. Not wanting to waste valuable magnet time imaging and reimaging a MRI phantom, we instead challenged ourselves to scan the most curious objects we could find at the local grocery store.
For our first attempt we scanned a pumpkin. One result of this endeavor can be seen here. This is a pretty standard fruit to scan, as just about every imaging center around the country obtains a T1-weighted image of them in late October. Still, it was exciting to us. During the next pilot testing session Abby brought in a Cornish game hen to be scanned. This really upped the ante, as we had now put a dead bird into the head coil. When pondering our next step the comment was made: “we should scan a whole fish”.
I picked up the salmon from our local supermarket early on an early Saturday morning in spring of 2005. The clerk behind the counter was a little shocked to be selling a full-length Atlantic salmon at 6:30 AM, especially when I told her what was about it happen to it. About an hour later we were in the imaging center with the fish wrapped in plastic and securely placed within the head coil. We proceeded to test our entire protocol with the salmon in the magnet. In total, we did an anatomical localizer scan, four functional runs, a T1-weighted anatomical scan, and a diffusion tensor imaging (DTI) scan.
After transferring the data off of the scanner we first took a look at the high resolution anatomical image. It was simply incredible. Slice the fish along the sagittal plane and you could see the fish split right down the middle. Slice the fish coronally and you could see what looked like salmon steaks on the viewer. By far it was our crowning achievement in terms of ridiculous objects to scan. Then, our curiosity satisfied, I socked the salmon data away for the next three years.
In early 2008 I was working with my co-adviser George Wolford on a presentation he was giving regarding the multiple comparisons problem in fMRI. We were discussing false positives in MRI phantom data and I brought up the idea of processing the salmon fMRI data to look for some ‘active’ voxels. I ran the fish data through my SPM processing pipelines and couldn’t believe what I saw. Sure, there were some false positives. Just about any volume with 65,000 voxels is going to have some false positives with uncorrected statistics. Rather, it was where the false positives occurred that really floored me. A cluster of three significant voxels were arranged together right along the midline of the salmon’s brain. If they would have been anywhere else the salmon would have been just a curious anecdote, but now we had a story.
George presented the salmon data at our local fMRI methods group, but nothing much happened for a while after that. George was convinced that we could/should publish the data and that it was an excellent example of the multiple comparisons problem. I was less convinced, remarking about how silly that would be and how terrible it would be for a young postdoc to become known as ‘the fish guy’. For the next year we went back and forth about the issue, until one day in January, 2009. George was out in Los Angeles and came up to UCSB to visit. Over lunch he said that it was time to ‘get the fish out’. I relented, and agreed to start writing the paper.
About a week later the HBM conference poster deadline came around and we decided to submit the salmon as an abstract. We genuinely wanted it to be a part of the conference, but we really doubted that it would be approved. How right we were. Through some sources close to the matter I have learned that the salmon poster was indeed rejected by every reviewer who saw the abstract. Just about everyone thought it was a joke – some rogue student who was playing a prank on the OHBM. It was only when the rejected abstract went before the OHBM Program Committee that it was given approval to stay as part of the conference. I hear that even that vote was contentious.
While the abstract reviewers were busy rejecting the salmon poster my co-authors and I were diligently writing a full-on salmon manuscript. The overall outline of the paper had been in our heads for some time and the writing went rather quickly. By April we had a polished manuscript ready for review and we sent it off to a major neuroimaging journal. Within a week we heard back that it was being rejected on an editorial basis. We heard that there were several major discussions within the journal staff regarding whether to even review the piece. In the end they decided to pass the responsibility, and the trouble, on to another journal.
That brings us to today. The ‘Post-Mortem Atlantic Salmon’ was a strong success at the OHBM conference. It is also under review at a second major neuroimaging journal. The more I think about the affair the more I believe that the fish has the chance to impact the field of neuroimaging in a very positive way. Predefined significance thresholds with a specified cluster extent are a weak control to the problem of false positives in imaging data. Statisticians and methods researchers have argued about the need for multiple comparisons correction for some time. In just one figure the salmon data illustrates exactly why we need stronger controls for the false positive problem in fMRI. I hope it finds a good home in an open-minded journal.
You can find a copy of the ‘Post-Mortem Atlantic Salmon’ poster at this link:
http://prefrontal.org/blog/2009/06/human-brain-mapping-2009-presentations/
Upcoming Talk: Bay Area Memory Meeting
I’ll be giving a short presentation on individual differences and fMRI experimental design at the upcoming Bay Area Memory Meeting (BAMM) on Monday, August 24th. If you are around Genentech Hall at the UCSF Mission Bay campus and have some time available in the late afternoon then you should definitely swing by!
The Middle Ground in Multiple Comparisons Correction
I got a note last week from a longtime colleague seeking advice on some reviewer comments of their latest paper. In their remarks the reviewer requested that the authors revert the corrected statistical threshold back to an uncorrected level of p < 0.001. The authors were left scratching their heads, wondering how they were going to justify their use of a standard statistical technique. I was a bit shocked to hear that the reviewer was so opposed to multiple comparisons correction, but not entirely surprised. As multiple comparisons correction has become increasingly standard there seems to be a subtle push against it from some in the neuroimaging field. We experienced the push ourselves when we got our first rejection on the fish paper. The reviewer said that there needed to be a more in-depth discussion of avoiding both Type I and Type II error in the paper. They added that always controlling for multiple comparisons correction could potentially lead one to ignore legitimate results that fall short of the corrected statistic value.
So, as a researcher, what do you value more? Do you want to ensure that false positives are controlled, or do you want to minimize false negatives? Do you side with roughly 80% of the field and correct for multiple comparisons, guarding against Type I error, or do you side with the other 20% of the field and use uncorrected stats, minimizing the chance of Type II error? [1]
My position: why not do both?
For the last two years I have been an advocate of reporting both uncorrected and corrected statistics in results tables and figures. It just makes sense. An example of this approach is posted below, shown as corrected results overlaid on top of uncorrected results for an inflated cortical surface. Regions surviving an uncorrected threshold of p < 0.001 with an 8 voxel extent threshold are shown in blue and the subset of regions surviving a FDR-corrected threshold of p(FDR) = 0.05 with an 8 voxel extent threshold are shown in yellow/orange.
At a glance you can see what regions survive multiple comparisons correction and which only survive an uncorrected threshold. Most interesting to me is that, for this task, there are some regions that survive an uncorrected threshold but do not survive the FDR threshold. For example, the left and right occipital pole in both hemispheres is only significant for uncorrected statistic values.
In writing the paper for the above data we have refrained from discussing the regions that do not survive multiple comparisons correction. Still, what I consider elegant is that I can give the reader additional information about the data from which they can make their own judgements. Perhaps a vision researcher would be interested in the uncorrected result and could run a more powerful study to investigate this effect, who knows. The point is that our readers, by and large, are very smart people. I don’t mind giving them all the information I can for them to draw their own conclusions and insights from. I also think that it represents a very good balance between those who support multiple comparisons correction and those who, as my colleague encountered, insist on uncorrected thresholds.
[1] – Stats gathered during a forthcoming literature review in the salmon paper.
Neuroimaging Statistics Workshop Videos
The Columbia University Department of Statistics hosted a workshop last month titled “Estimating Effects and Correlations in Neuroimaging Data”. Some great folks stopped by to give talks, including Ed Vul, Nikolas Kriegeskorte, Tor Wager, and Andrew Gelman. They recorded everything into Quicktime movies for those of us who couldn’t stop by – click the link below and check it out:
http://www.stat.columbia.edu/~martin/Workshop/ECWorkshop.html
Quote of the Week – Coggan
“My ignorance of science is such that if anyone mentioned copper nitrate I should think he was talking about policemen’s overtime.” – Frederick Donald Coggan
Atlantic Salmon – MRI Data
There have been some requests for the T1-weighted high-resolution anatomical MRI data of the Atlantic Salmon. Click on the link below and you can download the files in the ANALYZE file format.
Load it up in SPM, take a screenshot, and presto, you have a fancy new desktop picture. If you want to have even more fun then load it up into OsiriX and do a 3D volume rendering, like these:
Salmon-3D-VR.mov
Salmon-3D-MIP.mov
Those QuicktimeVR files are pretty big, weighing in at around 20MB. As a result it might take thirty seconds or so to fully download before they display. Enjoy!
Brain Dissection: Insula Anatomy
We just wrapped up the 2009 Summer Institute on Cognitive Neuroscience here at UCSB. There are a million stories to tell from the two weeks of the Summer Institute, but for this post I just wanted to upload a few pictures from the brain dissection lab that went on during the first week. Mark D’Esposito was our small group leader and took us on an amazing tour of brain anatomy. At the end of the session he allowed us to investigate specific brain structures we had a particular research interest in. I reassembled the axial slices from the right hemisphere and cut away the frontal, parietal, and temporal opercula to reveal the insula, hiding underneath.
After some quick Photoshop work you can isolate the insula from other structures in the photo. You can clearly see the gyri and sulci of the insular cortex. To the left is the posterior insula and to the right is anterior insula.
From there we can overlay some labels that detail major anatomical features of the insula.
After investigating the insula using MRI for the last three years it was a real treat to finally see an example of it in person. Special thanks to Mark D’Esposito, Mike Miller, and Mike Gazzaniga for letting me take part.
Human Brain Mapping 2009 – Presentations
I want to thank everyone at the Human Brain Mapping conference for their excellent comments and insight on my research. I had an unbroken string of amazing conversations with researchers from around the world – it was a real treat. Below you will find copies of both posters that I presented at HBM along with a copy of the slides that I used in my presentation. If you have any questions or would like larger copies of the figures please let me know.
The processing of internally-generated interoceptive sensation
Conference Poster: [PDF] [JPEG]
Presentation Slides: [PDF]
Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction
Conference Poster: [PDF] [JPEG]
Atlantic Salmon Index
This post serves as an index of the articles that reference the ‘Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction’ paper. There have been four posts so far, each listed below:
[1] Human Brain Mapping 2009 – Presentations
http://prefrontal.org/blog/2009/06/human-brain-mapping-2009-presentations/
[2] Atlantic Salmon – MRI Data
http://prefrontal.org/blog/2009/07/atlantic-salmon-mri-data/
[3] The Story Behind the Atlantic Salmon
http://prefrontal.org/blog/2009/09/the-story-behind-the-atlantic-salmon/
[4] The Internet Found the Atlantic Salmon
http://prefrontal.org/blog/2009/09/the-internet-found-the-atlantic-salmon/
~Craig