Comments on: Voodoo Correlations in Social Neuroscience

By: Justin

Justin — Fri, 01 Jan 2010 12:35:10 +0000

Hi Craig,

Happy New Year!
Any chance of some constructive feedback on this video I put together on the above study

Regards

Justin

By: prefrontal

prefrontal — Sun, 08 Feb 2009 22:06:28 +0000

Neuroskeptic,

The multiple comparisons problem is still a huge issue in neuroimaging, you are right. I think that part of the problem we are dealing with is an issue of tradition. For the last 10-15 years a significance cutoff of p < 0.001 and an extent threshold of 8 voxels was ‘good enough’ to deal with the multiple comparisons problem. I think it is generally accepted that this is a poor control, but it’s hard to change a well-entrenched habit. That was the magic of the Vul et al. paper: by pointing fingers and naming names they forced the field to debate and confront the problem.

We are in the process of getting a paper out that focuses on the multiple comparisons problem. I will send you a copy when we have an accepted draft – it should be a fun read.

Best,
Craig [Prefrontal]

By: Neuroskeptic

Neuroskeptic — Sat, 07 Feb 2009 11:06:24 +0000

Right – that’s what I thought. Although you’re clearly the expert here! Personally, I think that the “Forman error” is more serious, and more important, than the non-independence error. A result due to the non-independence error may be inflated, but a result due to the Forman error could be entirely illusory.

By: prefrontal

prefrontal — Fri, 06 Feb 2009 23:47:40 +0000

Neuroskeptic,

My grad school adviser and I did a department lecture series on multiple comparisons correction a few years ago. We looked at the Forman et al. (1995) paper with interest since it lays out how the effective significance value changes with varying values of minimum cluster size. The only problem is that Forman did the testing in two dimensions, not three. This makes the utility to fMRI virtually nil since the results do not generalize to 3D volumes of data. That is where Eisenberger got into trouble in the Vul paper – they said their effective significance for their 3D data was p < 0.000001 based on the 2D estimates.

Adding a cluster size threshold does increase the effective significance value, it is just that nobody knows by how much. It is also not a strong control for the multiple comparisons problem, which is what Vul’s argument was. A thorough investigation of the issue is necessary, it just seems like nobody has stepped up to the plate yet.

Cheers,
Craig [Prefrontal]

By: Neuroskeptic

Neuroskeptic — Thu, 05 Feb 2009 09:39:05 +0000

Regarding “Sin #1″ (I like that!) it’s worth noting that Vul et al. specifically discuss a problem with multiple-comparisons testing which they say stems from a failure to read a paper.

If they’re right, many fMRI results could be spurious. But they only name one paper as a victim of this mistake although they say there are others.

This is a completely different argument to their more widely discussed one about “non-independence”. See my post for more…

By: Deception Blog / Voodoo science in fMRI and voice analysis to detect deception: compare and contrast

Mon, 02 Feb 2009 06:03:55 +0000

[...] Correlations in Social Neuroscience [pdf]. If not, you’ll find the detail in coverage all over the psych and neuroblogs by googling the title or simply “voodoo [...]

By: ‘Voodoo Correlations in Social Neuroscience’ « The Amazing World of Psychiatry: A Psychiatry Blog

Sat, 31 Jan 2009 01:09:14 +0000

[...] blog article (also [...]

By: Matt

Matt — Thu, 29 Jan 2009 18:57:18 +0000

We were invited to publish a reply to Vul in the same issue of the journal. Here is a link to what we have submitted. Bottom line – among the many errors in the Vul paper – they mischaracterized the critiqued studies. Nobody runs their analyses the way Vul suggests and they don’t have a non-independence problem.

http://www.scn.ucla.edu/pdf/LiebermanBerkmanWager(invitedreply).pdf