I got quoted in a random neuromarketing article recently. In the flurry of people I have been chatting with about statistics and functional neuroimaging I often neglect to ask what organizations people are associate with. In this case it was Forbes magazine.
In the online version of the article there was a user comment from a neuromarketing company CEO defending the honor of his business and the field in which they operate. He went so far as to compare the launch of neuromarketing with the initial steps of market research in the early 20th century. He further argued that neuromarketing would bring about the next revolution in understanding consumer behavior.
I have to admit, my gut reaction on first reading this statement was one of mild disgust. This got me thinking about why neuromarketing hangs in a cloud of disdain among many scientists. Below are some of the ‘sins’ which I feel currently plague the field of neuromarketing. This is all just my opinion of course, but I do think that it raises some interesting points for discussion.
1) The curtain of proprietary analysis methods limits our knowledge of how effective neuromarketing can be.
Neuromarketing seems to be primarily driven by the private industry, not academia. This is not to say that research into consumer behavior has not occurred at the university level. There has been a lot of good neuroeconomics research in the last several years. Still, it is mostly companies in private industry that are driving the application of these findings to practical consumer behaviors. Because these companies are in competition with each other they are reluctant to give others the recipe to their secret analysis sauce. From the outside this means that the analysis pipeline of all neuromarketing companies is that of a black box, with data going in one end and the results-you-need coming out the other.
My colleagues and I have the position that fMRI research utilizing incorrect statistics can generate a large number of false positives. That is, many of the results will be there simply because of noise. Because so much of the current neuromarketing data is hidden behind the veil of proprietary analysis methods it is impossible to judge how successful their methods actually are, and to what degree their findings are false positives.
2) There is little peer-reviewed literature that is specific to neuromarketing.
Neuromarketing is an emerging discipline that will, in time, give us new insight into human behavior. Unfortunately, little peer-reviewed research has currently been published in this area. Search for ‘neuromarketing‘ in the PubMed database of abstracts (www.pubmed.com) and you will find all of ten publications. This must change for neuromarketing to mature.
Again, without peer-reviewed results on the effectiveness of neuromarketing experiments all we have to rely on are self-reports from the neuromarketing firms themselves. An issue similar to the file-drawer problem then exists. The file-drawer problem is when only positive results get published in journals while negative results sit unpublished in the file drawer. Neuromarketing companies will be likely to report positive results while negative results sit undistributed. Either way, the end result is a biased understanding.
3) Most people’s introduction to neuromarketing is through press releases, not peer-reviewed studies.
In 2006 there was an “instant-science” article released online by Marco Iacoboni et al. revealing their analysis of fMRI date obtain while subjects were watching Super Bowl advertisements. The much-discussed post, entitled “Who Really Won the Super Bowl?”, tried to determine the most effective commercial by judging which one activated regions involved in reward and empathy to the greatest degree. They determined that a commercial from Disney fared the best when evaluated by these measures. Many neuroscientists shook their heads and moved on.
In 2009 the same group published an op-ed in the New York Times detailing the results of scanning 20 individuals while looking at pictures and videos of leading political candidates. They drew conclusions on candidate evaluations by examining activity in areas like the amygdala and anterior cingulate. For example, they concluded that amygdala activity indicated a state of anxiety and cingulate activity indicated cognitive conflict. These oversimplifications were so well publicized and widely distributed that a number of leading neuroscientists were compelled to publish a letter in the New York Times calling the Iacoboni results into question.
Let’s put it this way, when many of the top minds in neuroimaging feel compelled to assemble a letter to the New York times regarding your non-peer-reviewed neuromarketing/neuropolitics results then the field has a problem.
There are a handful of peer-reviewed neuromarketing papers that do deliver. One recent paper by Michael Schaefer was a very interesting investigation into the representation of brand associations. However, these type of studies are typically rare, and it remains that the signal-to-noise ratio of information in the press is very low.
4) Neuromarketing methods are not immune to subjectivity and bias.
One of the most highly touted aspects of neuromarketing methods is that they are free from subjectivity and bias on the part of the participant. For example, asking a subject what they thought of a particular brand introduces the muddying waters of conscious consideration. The person’s response will be colored by a complex web of tangential cognitive factors and contextual biases. The promise of neuromarketing is that you can bypass these confounding factors to get at the heart of the matter – the real representation of the brand. While this is true to a degree, an entirely new set of confounding factors is introduced during the analysis of neuromarketing data.
While many neuromarketing measures are indeed more objective than verbal reports, I must disagree with the observation that they are unfiltered, true reports of the underlying representation. While the signals are not filtered by the consciousness of the research subject, a great deal of manipulation and filtering of the data is done by the researcher. This does introduce the potential for bias, simply by a different avenue.
Small changes in processing pipelines can have a huge impact on the power of fMRI to detect relevant signals. Some excellent papers by Stephen Strother come to mind with regard to this point. With no knowledge of what is going on we have no idea how objective the analyses by these companies can be.
5) The value per dollar of neuromarketing methods has yet to be determined.
Neuromarketing studies are expensive. The Forbes article says that an average EEG or fMRI marketing study costs in the neighborhood of $50,000. Immediately this number can trigger a ‘more expensive = better’ response, especially if you have a large budget to support such studies. What rarely gets discussed is what kind of value you obtain in return for the huge amount of money that is spent.
The key question in neuromarketing is what information can you get with EEG / fMRI / eye tracking / biometrics that you cannot obtain using other methods. If I can spend $1000 to do a traditional market study that gets me 85% of what a $50,000 fMRI study does then the return on my neuromarketing investment is not great. Thinking about it another way, how much less or more could I get across 50 traditional studies relative to the value of one neuromarketing study.
Many companies are not limited by the extreme cost of neuromarketing studies, and a significant fraction of them are not afraid to take the risk to try something new. Perhaps part of the motivation is also the fear of being left behind – that a competitor will take the risk and gain a competitive advantage in consumer understanding. Whatever the motivation, there will always be a market for neuromarketing methods. Still, we must still acknowledge that the value of neuromarketing is an open question.
6) People are rushing the field to make a quick buck, and not everyone is trustworthy.
The emergence of neuromarketing represents a modern day gold rush in terms of buzz and promises. Brilliant researchers will be attracted to this opportunity and will significantly advance the field of neuromarketing. Morally questionable individuals will also be drawn to the opportunity, and will end up giving the field a black eye. Reputations will build up over time and trustworthy companies will emerge from the fray, but the current situation is more akin to the wild west than a civilized exchange.
7) The true value of neuromarketing is obscured by the above-mentioned problems.
I thought I would end on a high note. There is certainly significant value to using neuromarketing methods in consumer research. Why else would companies like Nielsen Holdings be investing in neuromarketing firms like NeuroFocus? One of the biggest problems is that the true value of these methods is obscured by those who treat it as a gimmick and have the loudest voice. The next ten years will represent a true shakedown of the neuromarketing industry. Companies that are able to provide real value to their customers will live on while those who simply seek to make pretty pictures will fall by the wayside. It will be a fascinating time to be an observer of the business and politics in this emerging field.
The above points ignore many other issues facing neuromarketing. I have completely bypassed a discussion of the ethics of neuromarketing. Many people worry that technologies like fMRI will help marketers find the ‘buy button’ in the brain, stripping away people’s free will in product choice. I am not terribly worried about that discussion, perhaps because I am ignoring the problem or perhaps because I know too much about brain function or neuroimaging methods. Regardless, there are other issues and hurdles that neuromarketing must address to grow as a field.
In the end I do wish neuromarketing great success. I simply fear that those individuals who are seeking to profit on the popularity will tarnish the reputation of neuromarketing before it is able to legitimize itself.
I had a great time at the Cognitive Neuroscience Society meeting in San Francisco this week. My only complaint was that there wasn’t enough time in the day to catch up with all the people I wanted to see! Beyond that there were some excellent sessions on long-term memory and executive control that I got a lot out of. Overall, I felt like it was a very strong year for CNS.
Below are links to PDF and JPEG copies of the posters that were presented at the conference:
It has been a long road, but our multiple comparisons paper including the salmon has been published. See below for more details, including the abstract and a link to the download page of the journal. If you have any questions or comments please post them below or send me an email directly.
Neural Correlates of Interspecies Perspective Taking in the Post-Mortem Atlantic Salmon: An Argument For Proper Multiple Comparisons Correction
Craig M. Bennett(1), Abigail A. Baird(2), Michael B. Miller(1) and George L. Wolford(3)
1)Department of Psychology, University of California at Santa Barbara, Santa Barbara, CA 93106
2)Department of Psychology, Blodgett Hall, Vassar College, Poughkeepsie, NY 12604
3)Department of Psychological and Brain Sciences, Moore Hall, Dartmouth College, Hanover, NH 03755
Journal of Serendipitous and Unexpected Results, 2010. 1(1):1-5
Early Access: Oct 20, 2010
With the extreme dimensionality of functional neuroimaging data comes extreme risk for false positives. Across the 130,000 voxels in a typical fMRI volume the probability of at least one false positive is almost certain. Proper correction for multiple comparisons should be completed during the analysis of these datasets, but is often ignored by investigators. To highlight the danger of this practice we completed an fMRI scanning session with a post-mortem Atlantic Salmon as the subject. The salmon was shown the same social perspective-taking task that was later administered to a group of human subjects. Statistics that were uncorrected for multiple comparisons showed active voxel clusters in the salmon’s brain cavity and spinal column. Statistics controlling for the family-wise error rate (FWER) and false discovery rate (FDR) both indicated that no active voxels were present, even at relaxed statistical thresholds. We argue that relying on standard statistical thresholds (p < 0.001) and low minimum cluster sizes (k > 8) is an ineffective control for multiple comparisons. We further argue that the vast majority of fMRI studies should be utilizing proper multiple comparisons correction as standard practice when thresholding their data.
See the JSUR early access page to download the article and supplementary material.
Just wanted to take a second to thank the kind folks in the Psychology Department at UC Riverside for hosting me this afternoon. I gave a neuroimaging stats talk for their cognitive brown bag series, and it was a really great time!
For anyone who is interested a copy of the slides from my presentation can be downloaded at the link below. If you have any questions or comments feel free to email me – I would love to chat more. Take care UCR!
I have been remiss in uploading copies of my spring conference posters. October seems like a fine month to rectify that. Below are links to the research I presented at the Cognitive Neuroscience Society meeting in Montreal and at the Organization for Human Brain Mapping meeting in Barcelona. Both meetings were fantastic – I got to meet a lot of new people and experience all the awesomeness that Montreal and Barcelona have to offer.
Great post from Doug Hull at MathWorks on the Top 10 MATLAB programming practices that only lead to consternation and the regret.
Personally, I have often been guilty of #9 and #3…
I have wanted to attend the Association for Psychological Science annual convention for a number of years, but I was always frustrated by the number of other conferences I had to attend during the spring. All that changed early this year when I was offered the opportunity to give a presentation on interoceptive development. I suddenly had a very good reason to free up some time and hop on a plane!
I want to thank everyone who attended my address this morning. After untold amounts of airline trouble getting to Boston it was a real pleasure to have the chance to talk about the insula and interoceptive development.
If you are interested you can download a copy of my presentation slides here. Send me an email if you have any questions or comments. Thanks!
- Current Citation:
Bennett CM, Miller MB. (in press). How reliable are the results from functional magnetic resonance imaging? Annals of the New York Academy of Sciences.
Functional magnetic resonance imaging is one of the most important methods for in vivo investigation of cognitive processes in the human brain. Within the last two decades an explosion of research has emerged using fMRI, revealing the underpinnings of everything from motor and sensory processes to the foundations of social cognition. While these results have revealed the potential of neuroimaging, important questions regarding the reliability of these results remain unanswered. In this chapter we take a close look at what is currently known about the reliability of fMRI findings. First, we examine the many factors that influence the quality of acquired fMRI data. We also conduct a review of the existing literature to determine if some measure of agreement has emerged regarding the reliability of fMRI. Finally, we provide commentary on ways to improve fMRI reliability and what questions remain unanswered. Reliability is the foundation on which scientific investigation is based. How reliable are the results from fMRI?
The Lake Ontario Visionary Establishment (LOVE) conference just wrapped up and, I have to say, it was a genuinely fantastic experience. I gave a lighthearted presentation on Type I error and reliability in functional imaging, which hopefully made the message a bit easier to swallow. I also got the chance to catch up with longtime friends while making some new aquaintances. Thanks to the organizers Daniel Ansari and Jonathan Fugelsang for having me up to present.
For all those who are interested: you can download a copy of my presentation slides here.
Send me an email if you have any questions or comments. Thanks!
It is going to be a busy conference season again this spring. I will be at the following professional gatherings over the next few months – send me an email if you will be attending as well and would like to meet up. I’ll buy the first round and we can talk shop.
Lake Ontario Visionary Establishment Conference [LOVE]
February 11-12, Niagra Falls,
NY Ontario, Canada
Cognitive Neuroscience Society Conference [CNS]
April 17-20, Montreal, Canada
Association for Psychological Science Convention [APS]
May 27-30, Boston, MA
Organization for Human Brain Mapping Conference [HBM]
June 6-10, Barcelona, Spain
Applied Human Factors and Ergonomics Conference [AHFE]
July 17-20, Miami, FL
Here is some of what I will be presenting:
• Special topic talk: ‘Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: addressing the multiple comparisons problem in fMRI.’
• Invited talk: ‘The development of interoceptive information processing across adolescence.’
• Poster: ‘How reliable are the results from fMRI?’
Bennett CM, Guerin SA, Donovan CL, Miller MB
• Poster: ‘A device for simultaneous thermal and tactile stimulation in an MR environment.’