The ‘Voodoo Correlations’ debate heats up
It hasn’t taken long for the academically-heated exchanges to begin with regard to the recent Vul, Harris, Winkielman, and Pashler paper. You can’t call out such a large group of authors, say their results are practically meaningless, and not have some of them speak up.
One group of authors who were red-flagged as having non-independence errors have responded with a formal rebuttal. The Jabbi, Keysers, Singer, and Stephan PDF can be viewed at http://www.bcn-nic.nl/replyVul.pdf.
Ed Vul has already posted a response to this rebuttal. It can be viewed on his website at http://edvul.com/voodoorebuttal.php.
The author rebuttal does raise some good points, but to some degree it misses the point of Vul’s original paper. While a multiple comparisons correction does correct for the number of false positives that you will find in your data, it does not correct for the inflated correlation values that you will still find. Vul brings this up in his rebuttal, and in my mind it remains a key point.
The were several other arguments in the rebuttal that I just didn’t buy. For instance, the argument that social neuroscience is more concerned with where in the brain correlations occur and not necessarily with how strong the relationship is. While they gave a good example (I have read the Jabbi paper), it is not a good position to defend. I care about the strength of relationship, and I know a whole lot of other scientists who do as well. Are we wrong to want accurate correlation values?
The rebuttal also argued that the questionnaire used by Vul was incomplete and ambiguous. While this may or may not be true, it seems to be the case that the information was enough for Vul et al. to calculate their results. It will be interesting to see if any papers get taken off the ‘red list’ in the long run due to additional information being offered. This will be the true test of the survey.
Two points that I will add to the discussion:
1. I still can’t believe that so many people were reporting the correlation off of a single peak voxel. These authors were certainly capitalizing on chance when reporting their results, with or without the non-independence error.
2. I do not believe for a second that fMRI has a test-retest reliability of 0.98, not even on a good day. The rebuttal cites this number as justification for why correlation values could potentially be higher than Vul et al. suggest. I think that Vul’s original estimate of 0.70 may still be optimistic. For test-retest data from individuals my money would be somewhere in the 0.50 to 0.60 neighborhood.
Ed Vul has been keeping a very good collection of links and such on a webpage dedicated to the Voodoo Correlations paper. You can find it at http://edvul.com/voodoocorr.php. If interesting new arguments come up I may post again here as well.
As an aside, I can’t help but believe that this is going to be the direction of scientific discourse in the age of the Internet. Discussions and debates of hot topics are going to take place on a diverse collection of news sites, weblogs, and personal websites. Exchanges that used to take several publication cycles to occur will now happen in a matter of days or hours. It will likely be more confrontational in nature, but I do believe that it will be good for our scientific knowledge and scientific methods.
One Response to “The ‘Voodoo Correlations’ debate heats up”
‘Voodoo Correlations in Social Neuroscience’ « The Amazing World of Psychiatry: A Psychiatry Blog - January 31st, 2009
[…] Prefrontal.org blog article (also here) […]
Leave a Reply
You must be logged in to post a comment.