The Internet Found the Atlantic Salmon
The last 72 hours have seen an incredible increase in traffic here at prefrontal.org. To sum it up in a single sentence: the site has received as many hits in the last three days as it has during the past two years. Yeah, really. My activity graph on the WordPress Dashboard looks like this:
It seems that late last week a few major neuroscience weblogs discovered the Salmon poster and decided to write up summaries. Those readers then posted to their weblogs, whose readers posted to their weblogs, and so on. By 10am Friday morning the prefrontal.org activity meter was pegged and my inbox was full.
A few important bits of info:
* The current status of the Salmon is that we are trying to publish it as an editorial in a major neuroimaging journal. We are very close to resubmitting, only needing to complete a survey on the prevalence of multiple comparisons correction in the previous neuroimaging literature. We hope that it will be released in the near future.
* If you would like to be sent a copy of the commentary if/when it becomes published just send me an email and I will put you on the list.
* Some sites have played up how difficult it has been for us to get the Salmon published. We have received some, well, interesting feedback by a few editors in the course of our submission. Still, it has not been more difficult than average to get the Salmon commentary published (so far).
* The goal of the Salmon poster was to encourage the minority of researchers who report uncorrected statistics to move forward and begin using basic multiple comparisons correction in their research. The Salmon doesn’t add anything to the technical discussion of how multiple comparisons correction is performed, it is simply a salient reminder of why proper correction is always necessary.
* None of the authors intended for the Salmon to go public in such a big way, especially before the commentary was reviewed and published. We were actually quite content to publish our editorial in a neuroimaging journal and be done with it. We feel that, fundamentally, this is an internal debate within the field of neuroimaging.
Some of the best/notable writeups that I have found:
* http://neuroskeptic.blogspot.com
* http://languagelog.ldc.upenn.edu
* http://lawandbiosciences.wordpress.com
* http://www.mindhacks.com
* http://www.wired.com
* http://www.newscientist.com/blogs/shortsharpscience
* http://blogs.nature.com/news/thegreatbeyond
* http://blogs.discovermagazine.com/discoblog/
* http://chronicle.com/
* http://slashdot.org
Some of the best comments that I have run across:
“The recorded signal is changing due to noise. The point of the experiment is that if you look at enough signals, the noise in one will match the timing of your experimental stimulus, purely out of chance. Another way of looking at it is this: if you choose a statistical threshold of p 0.05 then, statistically, you expect a result that is significant at that level purely out of chance once in every twenty experiments. When you’re analyzing images, or worse volumes, pixel by pixel, you’re doing a LOT of comparisons. If you don’t correct for that you WILL get false positives, no matter what you’re looking at.” – ceoyoyo
“But not everyone uses multiple comparisons correction. This is where the fish comes in – Bennett et al show that if you don’t use it, you can find “neural activation” even in the tiny brain of dead fish. Of course, with the appropriate correction, you don’t. There’s nothing original about this, except the colourful nature of the example – but many fMRI publications still report “uncorrected” results” – Neuroskeptic
“… it seems to me like that their point wasn’t that the fMRI wasn’t sensitive enough, or particular enough. Instead the problem seems to be a problem of statistically expected random noise. Their point seems to be that users of an fMRI should bear in mind that their marvelous magical machine can generate “real” errors, and that basic, common-sense multiple comparison habits should be developed, instead of a take a picture, slap a stat against it approach.” – Nemus
“The entire point the write up was to warn about the danger of false positives. Your attributing of brain activity to random, natural noise is exactly the danger they want to avoid.” – Anonymous
“The trouble is, most scientists are not mathematicians, and have no good theoretical understanding of statistics. Most people pushing buttons in SPSS or SAS (or what have you) are just doing “cargo cult” mathematics. Ask them to justify why their “very conservative” confidence interval for a given test is appropriate when dealing with eleventy billion variables, or why a particular post-hoc test is the proper one to use, and they’ll look at you like you just asked them to prove that the sky is blue.” – Anonymous
“Actually, the voodoo correlations paper is actually talking about performing correlations between the signals we get from fMRI scans (you can read the actual paper instead of the somewhat misleading article here [edvul.com]), and other measurements or scores. This doesn’t do that at all. This is about the danger of false positives in fMRI imaging, because of the large number of statistical tests that are done across the brain. The majority of peer reviewed published fMRI papers do some type of multiple comparisons correction to attempt to adjust for this problem.” – daenris
Some of the more terrible writeups that I have found:
By and large the comments have been quite good. However, there have been a few people arguing that the dead fish is actually still thinking or that we have observed evidence of the ethereal soul. I am not going to quote the comments here, but it has been a bit amusing to see this play out…
The funniest comments so far:
“Of course, Bennett’s group don’t mean to suggest that a post-mortem salmon is capable of perspective-taking. Cod forbid.”
– Kerri
No sir. What it proves is the existence of the sole.
-Jeremi
Yeah, the measurements were right off the scales.
– grcumb
Thank you… he’s here all week. Try the fish.
– theshowmecanuck
I’m wondering if the issue could be resolved if the salmon was smoked and served with cream cheese…
– G
No, what it proves is that while you can tune an fMRI, you can’t tuna fish.
– limekiller4
I would think that a salmon in an MRI would be thinking more along the lines of “HOLY FUCK! I CAN’T BREATHE!”
– geminidomino
Does the scientific method for biologists exclude barbeques?
– value_added
And I, for one, welcome our new zombie salmon overlords!!
– DarkOx
Why wasn’t this published? Maybe the reviewers considered the experiment a bit fishy …
– maxwell_demon
First, the fish wasn’t dead, it was just tenured.
– jesor
I demand that fMRI techniques get a fair herring!
– Bob O’H
Scream if you love the multiple comparisons problem! AAAAAAAAAAAAAAAAHHHHHHHHHHH!
– Jess
… compared with how Vul et al. handled a similar topic, this is a party with clowns and flowers
– powrogers
The joke possibilities are endless but I won’t bother. It’s like shooting fish in a barrel.
– Anonymous
A common mistake made in discussions of taxonomy is overlooking the issue of whether closely related species taste the same. In this case, you omitted the fact that all of them are great when grilled. With a slice of lemon on the side.
– value_added
HOWEVER a fish that has been caught, killed/gutted, frozen, shipped, sold by auction, shipped again, sold again, taken to a hospital and put in an MRI machine is a dead fish. He ain’t pining for human faces, he has passed on. This fish is no more. He has ceased to be. He’s expired and gone to meet his maker. He’s a stuff. Bereft of life, he rests in filets! If you hadn’t glued him to his tank he’d be pushing up the seaweed. Its brainactivity is now history. He’s out of the pond. He’s kicked the tank, he’s shuffled of his mortal coil, run down the river and joined the bleeding choir invisibisble. This is an EX-SALMON!
– Anonymous
Conclusion
I just want to say that it has been great to see the discussion the Salmon has generated in the last few days. Our hope for this work was that it would call new attention to the multiple comparisons problem. I think that we can safely say that it has. Thanks.
~ Craig [Prefrontal].
6 Responses to “The Internet Found the Atlantic Salmon”
Steve - September 22nd, 2009
It would be nice to see a write-up of how this findings fits in with the Vul et al. paper.
Steve – I might try to get something like that up next month. At the moment I am busy finishing a review paper on fMRI reliability. Let me know if there is anything specific you would like to see discussed. ~ Craig [Prefrontal]
flip phillips - September 25th, 2009
Perhaps a better lolcats – “I can has Bonferroniz?”
Caution with fMRI studies « NLP New Code Development - September 28th, 2009
[…] fMRI studies on dead Atlantic Salmon and the research poster on the study. […]
Neural Activity Advances - fMRI Researchers Cautioned « NLP for your emergence - September 28th, 2009
[…] fMRI studies on dead Atlantic Salmon and the research poster on the study. […]
Catfish - October 10th, 2009
It is the Salmon of Doubt.
Lencho - September 22nd, 2012
Congratulations on the award. It is very important, even if very specific data with decimal points are profferred by men and women in lab coats with pocket protectors and impressive degrees, there is always room for a healthy scepticism. And this scepticism must be exercised with scientific methods, not by some pseudo-scientific postulate à la creationism.
Leave a Reply
You must be logged in to post a comment.