The ‘Voodoo Correlations’ saga continues

Saddle up cowpokes, and prepare yourselves, for yet another episode of Voodoo Correlations.

The latest salvo comes in the form of a reply authored by Lieberman, Berkman, and Wager. This is an invited paper that will appear in a future edition of Perspectives on Psychological Science alongside the original Vul, Harris, Winkielman, and Pashler paper.

You can download the new Lieberman, Berkman, and Wager reply here:
http://www.scn.ucla.edu/pdf/LiebermanBerkmanWager(invitedreply).pdf

The authors attack Vul et al. with a number of arguments, starting literally at the beginning with the title of their paper. As before, I think that there are a subset of points that should be regarded as ‘take home’:

voodoocorrelations-lieberman

1. An examination of the mean correlation scores between Vul’s independent and non-independent groups shows that the difference between them is about 0.12. While the correlation scores of a non-independent analysis may be inflated, the degree of inflation is not as extreme as some have suggested.

2. The Vul paper left out correlation data from their initial analysis that skewed their ‘non-independent’ distribution of values higher. Lieberman et al. argue that 54 additional correlations from the papers examined by Vul et al. should also be included. Most of these omitted values work against the findings of Vul’s analysis.

If correlation scores were selectively excluded by Vul et al. then the situation definitely needs some explanation. The very fact that Lieberman et al. found omitted correlation values in the papers examined by Vul et al. is somewhat suspect. I think there needs to be some more information released by Vul regarding how they selected the social neuroscience papers and what their criteria for inclusion was.

Lieberman’s argument regarding the actual degree of inflation in the non-independent correlation values is a strong one. Vul et al. argue that ‘a considerable number’ of the non-independent findings may be illusory, but one look at their original figure of independent and non-independent results shows that the distributions may simply have a small mean difference. This does impact the interpretation of the correlation magnitude, but speaks to the idea of valid results. This is an intuitive perspective that validates both Vul’s findings and the presence of honest effects in the social neuroscience literature.

None of the responses to date have discussed the fact that 44% of the authors who replied to Vul’s survey obtained their correlation value from a single peak voxel. So, within a cluster of 50 or 100 highly correlated voxels they chose to report only the one with the best correlation value in their results. This is capitalizing on chance and it still stands as a major issue. I do not believe that the practice is defensible.

My only wish in the Leiberman response is that they wouldn’t have introduced the ‘range correction’ in their new analysis of the independent and non-independent groups. I was really looking forward to a straight-up test of group differences with the excluded correlation values put back in. Instead, Lieberman et al. introduced the idea that certain correlation values of the independent group might have been underestimated because of the selection method. They then try to correct for this restriction, increasing the mean correlation of the independent group until it was not significantly different from the non-independent group. The problem is that now we cannot be sure what made the groups equal – the included correlations or the range correction.

Does anyone have a pool going with regard to when the next Voodoo Correlations response will be coming out? I’d like to put a few bucks on February 5th.

January 31, 2009 • Posted in: CogNeuro, MRI, Statistics

One Response to “The ‘Voodoo Correlations’ saga continues”

Leave a Reply

You must be logged in to post a comment.