My first NIH grant proposal
My postdoc adviser Mike Miller and I spent several weeks last April writing a grant proposal requesting ARRA economic stimulus money to do some aging research. Mike has written several grants before, so he is a bit of a pro. In contrast I have never needed to submit a NIH grant proposal. I had steady funding all through grad school and my postdoc, so I never needed to ask for money from Uncle Sam. The problem I am facing is that one day I will absolutely need to be adept at writing grant proposals So, I decided to start learning now. Here are some things I picked up along the path of co-writing my first RC1 grant:
* Download all grant-related documents from the NIH and save them in a ‘Instructions’ folder or something like that. In fact, grab as much information as you can from any source you can get your hands on. This means you should get a copy of the original Request for Applications (RFA), SF424 R&R instructions, PHS398 instructions, data sharing guide, and the like. You will be referring to these documents often as you write the proposal.
* Save a day at the beginning just to read the instructions in the RFA. Even though many grants have standard requirements there always seem to be a few unique twists for the one that you are chasing after. Also, the NIH is very specific about how your proposal should be formatted and assembled. Try to bend the rules at your own peril.
* Before you start writing buy the book Guide to Effective Grant Writing: How to Write a Successful NIH Grant Application by Otto Yang and read it. The book will give you a good foundation for how to approach NIH grant proposals. As you start writing buy the book Writing the NIH Grant Proposal: A Step-by-Step Guide by William Gerin and refer to it as necessary. If you get stuck on a section this book will walk you through the problem with excellent comments and examples.
* Iron out a rough sketch of the overall research plan and project budget before you begin. We had to change the number of study participants halfway through and it was a huge pain going through every document to alter a few numbers. Also, understand the differences between direct, indirect, and total costs. This is what caused us to change the number of participants.
* Make a ridiculously detailed outline before you start writing. I try to make an outline of every large writing project before I begin, but never have I gotten so much mileage out of this early step. Use it as a scaffold and just build the application up from there.
* You will go into the writing process thinking that the research plan will take the most time to complete. The reality for us was that all the other documents, like the summary, narrative, human subjects protection, and the like, took a great deal more time to get right. Part of the problem is that they were my responsibility to assemble and it was my first go-round. Still, my subjective impression is that these documents are every bit as important as what you are going to do with the money. Get them done right.
* After you have written the above documents for the first time you will be able to recycle them quite easily for any new grant you write. That is why it is so important to get that first submission out of the way, even if it never gets funded. You will have the template ready to go for any new ideas that come along later. That is also why it is important to write them well the first time.
* Don’t go to Hawaii or make big travel plans two weeks before the grant is due. When you return from your travels you will only have a week left and it will be crunch-time mode for real. I found this out the hard way, although I did get a lot written on the plane.
* Tell your family that you love them before you start writing. This is important. You are about to fall off of the radar of the world for a while.
* Don’t worry too much if you start dreaming about budgets and other aspects of the proposal. My wife told me that I would talk about the grant in my sleep at times. I must have been setting up an animal lab, because I very lucidly said, “No, we have to purchase the cages before we buy the monkeys”. Your mileage may vary…
* Finally, enjoy it. This was the largest writing project I have completed since I finished my dissertation one year ago. Like the dissertation it was long, hard, completely depressing at times, and felt absolutely incredible when we got it submitted. Also, you get to work with others in trying to complete the proposal, which makes things a hell of a lot easier. Misery loves company.
From here we wait around for several months and see what happens. Our chances aren’t very good. There are around 200 funded RC1 positions and the NIH was estimating 20,000-30,000 applications chasing after them. Yikes. Still, Mike and I put in a very strong proposal for a high-priority topic. I think that at the very least we are going to make it past the first cut. Time will tell – here is hoping that good science and bit of luck will make all the difference.
Voodoo Perspectives on Psychological Science
I received my May 2009 issue of Perspectives on Psychological Science in the mail last week. This is the issue containing the original Voodoo Correlations paper along with responses from, well, just about everybody. Man, is it chock-full of debate. I don’t know if I have ever seen a journal volume published with more commentary than original articles, but such is the case here.
It has been quite fascinating to read the original Vul et al. paper once more with the added context of the comments. Ed Diener made a very good point in his Editor’s Introduction: by publishing the article online before it appeared in an APS journal Vul et al. effectively short-circuited the comment process, giving his article a window of time without published criticism or opposition. The comments add a new, and I believe necessary, dimension to the discussion.
If you are still interested in the debate then I highly recommend sitting down for a while and reading through the comments. Also, don’t forget to check out the new Kriegeskorte, Simmons, Bellgowan, and Baker (2009) article in Nature Neuroscience and a new commentary by Poldrack and Mumford (2009) on Voodoo Correlations in SCAN.
The full online table of contents from the APS:
http://www.psychologicalscience.org/journals/pps/4_3.cfm
Articles in Perspectives related to Voodoo/Puzzling debate:
Editor’s Introduction to Vul et al. (2009) and Comments
Ed Diener
Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Edward Vul, Christine Harris, Piotr Winkielman, and Harold Pashler
Commentary on Vul et al.’s (2009) “Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition”
Thomas E. Nichols and Jean-Baptist Poline
Correlations in Social Neuroscience Aren’t Voodoo: Commentary on Vul et al. (2009)
Matthew D. Lieberman, Elliot T. Berkman, and Tor D. Wager
Discussion of “Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition” by Vul et al. (2009)
Nicole A. Lazar
Correlations and Multiple Comparisons in Functional Imaging: A Statistical Perspective (Commentary on Vul et al., 2009)
Martin A. Lindquist and Andrew Gelman
Understanding the Mind by Measuring the Brain: Lessons From Measuring Behavior (Commentary on Vul et al., 2009)
Lisa Feldman Barrett
Reply to Comments on “Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition”
Edward Vul, Christine Harris, Piotr Winkielman, and Harold Pashler
Quote of the Week – Sagan
“The truth may be puzzling. It may take some work to grapple with. It may be counterintuitive. It may contradict deeply held prejudices. It may not be consonant with what we desperately want to be true. But our preferences do not determine what’s true. We have a method, and that method helps us to reach not absolute truth, only asymptotic approaches to the truth — never there, just closer and closer, always finding vast new oceans of undiscovered possibilities. Cleverly designed experiments are the key.” – Carl Sagan, “Wonder and Skepticism”
The Dangers of Double Dipping (Voodoo IV)
A new article discussing non-independence errors has arrived on the scene, and it is quite good. Nikolaus Kriegeskorte, Kyle Simmons, Patrick Bellgowan, and Chris Baker have authored a Nature Neuroscience paper called ‘Circular analysis in systems neuroscience: the dangers of double dipping‘. It is the same fundamental argument as the original Voodoo Correlations paper (now renamed Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition), but in a more generalized and effective form.
Kriegeskorte leads off with a large-scale evaluation of fMRI studies in Nature, Science, Nature Neuroscience, Neuron, and Journal of Neuroscience. They were looking specifically at the prevalence of non-independent analyses in these journals. Out of 134 fMRI papers 42% had at least one non-independent analysis. That is a sizable number of studies in the highest-impact journals and demonstrates right away that circular analysis is a widespread issue.
The authors later provide several good examples of why circular, non-independent analysis is dangerous. The first example is through the lens of pattern-analysis in fMRI. They describe how it is common practice to do training and testing of classifiers on separate data sets because of overfitting. Overfitting is picking up on noise in the data during training of the classifier. When using a single randomized dataset for training and testing a linear classifier can approach 100% accuracy. When trained on one random data set and tested on another this percentage falls to approximately chance (50%). The same effects that lead to classifier overfitting can influence non-independent analyses.
The second example was a ROI analysis of simulated fMRI data. They demonstrated (as shown by the figure above) that noise at the fringes of an ROI can influence the shape of the ROI, and therefore the statistics used to later characterize it. The noise almost always leads to improved statistics and inflated effect sizes. This was the most intuitive figure I have seen describing the effect of non-independence.
One strength of this paper is that it discusses shortcomings that are common to functional imaging without being heavy handed. The authors spend a fair amount of text at the beginning and end listing how many systems neuroscience methods engage in circular analysis, such as the identification of neurons with specific response properties or gene selection in microarray studies. They also never called anyone out by name, instead using their own data to provide examples of non-independent analysis.
In my mind the best part of the paper is the decision tree the authors provide for determining if you are conducting a circular, non-independent analysis. It leads you through your analysis, step-by-step, to help guide you toward methods of keeping your analyses independent. Brilliant.
If you do grab a copy of this paper to read then, by all means, also download a PDF of the supplementary materials. It is even better than the paper, trust me. There is a 32-point, FAQ-style discussion of data independence that represents the absolute best tutorial on the topic that I have seen. Some parts are quite technical, but the majority of their answers are straightforward in providing a ‘yes’ or ‘ no’.
In conclusion, the authors argue that non-independent analyses should not be acceptable in scientific publications and that a large number of papers may need to be reanalyzed or replicated. This is the same conclusion of Vul et al. in the Puzzlingly High Correlations article. In comparing the two papers I think that Kriegeskorte was more effective in arguing in its favor. Puzzlingly High Correlations was a real wake-up call for functional neuroimaging and had all the drama of naming names and pointing fingers at social neuroscience. Still, I am drawn to Kriegeskorte’s approach. With far less consternation he describes how many fields have circular analysis problems, identifies what issues circular analysis causes, and provides a large amount of detail on how to avoid it in the future.
Matlab finite() function warnings
The latest version of Matlab deprecated the finite() function in favor of isfinite(). This is all fine and dandy in terms of improving the scripting language, but this change currently causes a crapalanche of warnings to be thrown as you use SPM. Usually along the lines of:
Warning: FINITE is obsolete and will be removed in future versions. Use ISFINITE instead.
Here is the line of code to turn those warnings off:
warning off MATLAB:FINITE:obsoleteFunction
Peace, serenity, and a clean terminal window to you my neuroimaging friend.
Pacific Rim Neuroimaging 2009 Review
I just recently returned from the “New Horizons in Human Brain Imaging: A Focus on the Pacific Rim” conference in Waikoloa, Hawaii. Here are some of the highlights:
– Kang Cheng showed some amazing results from high-resolution fMRI of visual cortex. He described some of the technical difficulties that he and his team have encountered as they engage in sub-millimeter resolution functional imaging. For instance, the distortions caused by blood flow in very small capillaries around the calcarine sulcus can seriously impact their measurements. They have trained their researchers to artfully select slices to minimize the impact.
– All I have to say about Michael Breakspear’s talk was that I understood very little of what he said and that it was one of the most awesome things I have seen in recent memory.
– The focus of Andrew Stenger’s talk was on new parallel imaging techniques being developed at the University of Hawaii. As the field strength of MRI moves higher there are a series of new problems that must be addressed. Issues of field inhomogeneity and motion artifacts that could be ignored at 1.5 or 3 T become much bigger issues at 7 T. One aspect that Stenger is working on is the signal dropout in the orbital areas of prefrontal cortex. They are working on 4-channel (and soon 8-channel) RF transmit systems that address this issue. My subjective impression of this work is that this is where we are going to reap the most rewards in terms of improving MRI in the near term. We can’t just default to higher field strength as a way to increase our signal to noise ratio – improving on RF transmission and reception are going to be key. Cool stuff.
– Alan Evans gave a talk that focused on issues of multi-center collaboration in neuroimaging. However, as a developmental researcher I was much more interested in the data he presented, which was from The MRI Study of Normal Brain Development. From their longitudinal database of hundreds of anatomical MRI scans he was able to show not just how the cortical surface changed during development but also at what rate cortex was changing across development. It was beautiful data.
– We got into a spirited debate over neuroimaging file formats on Wednesday evening. Prompted by Alan’s collaboration talk, we discussed about why there wasn’t an imaging file format that allowed for the handling of study metadata and provided a record of image processing. My argument was that there should be a means to store what each researcher did to the data in terms of the processing steps and then retain all parameters that a researcher enters for the final analysis, such as study design stimulus onsets. Having this information available would enable another researcher to fully replicate a study without further input from the original investigator. There were two responses to this:
1) Current imaging formats, like MINC, can already handle this type of information.
2) You don’t want to get involved in the holy wars of file formats.
That is fair enough, but we are going to need this form of interoperability in the future. One day in the not-too-distant future the NIH is going to require us to submit our data to a warehouse for sharing. What information do you want in your image headers when that day arrives?
Other notes:
– The Hilton Waikoloa Village Resort felt a bit like Disneyland. There was an amazing number of things to do and places to explore, but felt rather engineered at the end of the day. It was good to get out and explore the island after a while.
– Was less than impressed with Kona International Airport. Get the USDA and TSA inspection stations re-engineered people.
– On Tuesday night we ascended Mauna Kea to go stargazing. It was probably the most amazing view of the sky I have ever seen. Guides pointed out constellations using laser pointers and telescopes were available to look at stars and planets. I got to see the rings of Saturn, which was pretty damn cool.
– Volcano National Park was also a highlight. The scale of the place just boggles your mind – there are lava flows from recent eruptions everywhere. Seeing the seismometers tremble as the volcano churned underground was a treat. When you are there you really have the feeling that the ground underneath your feet is truly alive.
The OHBM has been good to me
Just got word that I was given a travel award for the Human Brain Mapping conference in June:
http://ric.uthscsa.edu:9000/TravelAwards/awardees.html
This year’s HBM conference is shaping up to be rather epic. I have two abstracts accepted for poster presentations, one of which was chosen for an oral presentation later that weekend. Now they are even giving me a few bucks to show up!
I have said it before, but HBM is by far my favorite conference of the year. I can’t wait.
Using Caret for fMRI Visualization
Caret is an incredibly useful program for the calculation and visualization of brain data. Like most incredibly useful programs it is also quite complex and a bit daunting to approach. While looking at three-views is a staple of fMRI research, sometimes it is easier to get the ‘big picture’ from a cortical surface rendering. I have had several requests for a basic fMRI->Caret tutorial over the last few months and I have finally gotten around to writing it. I make no guarantees that this is the most efficient or correct process, but it is how I was trained by an expert in Caret.
Read the rest of this post »
Quote of the Week – Curie
“I am among those who think that science has great beauty. A scientist in his laboratory is not only a technician: he is also a child placed before natural phenomena which impress him like a fairy tale.” – Marie Curie
Cognitive Neuroscience Society 2009 Review
I had a great time at the 2009 meeting of the Cognitive Neuroscience Society (CNS) last weekend. This is a conference that I try to attend every year, and I have been successful in that goal for six years now. Below is a list of highlights from the conference. I want to take just a second to thank my postdoc adviser Mike Miller for letting me go. I also want to thank the Institute of Collaborative Biotechnology for funding the trip.
SUNDAY
• One of the best sessions on Sunday was a symposium discussing the integration of genetics, cognitive neuroscience, and psychology. It was hosted by an old friend of mine from grad school, Adam Green, and featured several top researchers in the field of cognitive neurogenetics. The findings that they were presented were quite interesting in terms of relating genetic variability to brain and behavior, but the most important part of the session was the feeling that genetics was going to be one of the ‘next big things’ in cognitive neuroscience. In the Miller Lab we have been able to identify roughly 50% of the variability in human episodic memory using behavior and functional imaging. Hearing now that the remaining 50% of variance may be genetic really makes me believe that this is something worth pursuing further.
Before this session there really hasn’t been a lot of discussion of genetic factors at CNS. It will be interesting to see how this evolves in the next several years. My prediction is that next year will see an uptick in the number of posters and presentations addressing genetic variability. Then, in two years there will be an explosion of posters and presentations incorporating genetic data.
Thought-provoking tidbit from the genetics session: Andreas Papassotiropoulos discussing the need to look for interactions in genetic data. Essentially, his point was that most current research is focused on finding direct gene to behavior correlations. He argued that there is the need to look at the influence of gene clusters to better understand how genetic factors interact.
• Marcus Raichle received the George A. Miller prize in cognitive neuroscience this year. He is a huge figure in the field of cognitive neuroscience – it is hard to think of a researcher more deserving of this honor. I was particularly excited to attend this lecture because I hadn’t seen Raichle speak before. I think it would be accurate to say that he is the father of ‘default mode’ research, a topic that is proving to be a rich area of inquiry. He did discuss default mode research in his talk, but the more interesting aspect of his lecture was the timeline he presented of how cognitive neuroscience and the cognitive neuroscience society came into being. He told stories of how the idea for the society was formed in the back of a taxicab, and how the startup funding for the society came from Mike Gazzaniga’s credit card. Cognitive neuroscience is a relatively young field, but that doesn’t mean that it lacks history. Raichle’s talk was a rare opportunity to learn how it came together many years ago.
MONDAY
My favorite session on Monday was the Emotion slide presentations, hosted by Kevin Ochsner. The slide presentations are a new addition to CNS that are designed to give outstanding research a wider audience. I think that it is a great change, and I appreciate that the CNS organizers continue to help the conference evolve to better serve the community.
• One excellent presentation was by Virginie Czernecki, a researcher with Inserm at Salpetriere Hospital in Paris. She showed data from a group of patients with insular damage demonstrating that they had deficits in unpleasant, irritating, and gustative emotion identification. The lesion focus of each patient was variable, limiting the spatial identification of what brain structure was causing the changes in behavior. Still, they showed the effects of insular damage across a range of tasks involving facial affect, odor identification, and working memory. Good stuff.
• Another presenter I wanted to mention was Tor Wager. He spoke on a re-analysis of a placebo analgesia dataset using a new multivariate analysis designed to identify functional networks of brain regions. I wanted to mention his talk primarily because it represented the best integration of technical imaging methods applied to an empirical question that I saw at the conference. Don’t get me wrong, there were many presentations that had worthy scientific investigations and many posters that had excellent new methods, but it is rare to see the two so well joined together. I think that his approach is quite analogous to how I hope to conduct my own research: a fusion of advanced methods applied to interesting questions.
• On Monday evening I went around the poster session doing a count of how many researchers used multiple comparisons correction in their analyses. George Wolford and I are wrapping up the final drafts of a new paper arguing in favor of correction and we wanted to get a feel for what percent of people are using it. Of the 42 posters that used a whole-brain, general linear model (GLM) approach only 9 posters used any kind of multiple comparisons correction. That’s rather sad. Worse was that some researchers who used techniques like FDR and FWE to do correction would sometimes revert to uncorrected stats if no activity was detected. I am trying to decide which is worse – not doing correction at all or doing correction only when it shows what you want. I think that our paper is going to hit at just the right time – now we just need to get it out.
TUESDAY
• The highlight of the day (personally) was the presentation of my poster at the morning session. Since it was a methods poster on the last day of the conference I wasn’t sure how many people would stop by. There is nothing more lonely than standing beside your research for two hours with nobody giving your title a second glance. I was fortunate this year in that I had a steady stream of folks stopping by, with some individuals quite interested. The research I presented had to do with the impact of experimental design on the ability to detect individual differences in fMRI activity. It turns out that some experimental designs may be better than others for getting at what makes us all different. Because experimental design is a topic of general interest I think the poster was able to pull in a diverse array of researchers who are looking to investigate individual differences. By any measure it was a very successful poster. Now, as always, the goal is to get it written up as a manuscript.
• The second highlight of the day was a talk on the representation of body postures by another friend from grad school, Emily Cross. She gave a great presentation on the separable roles of the mirror action system and the extrastriate body area. Also, she was one of the only researchers I saw all weekend who used a repetition suppression fMRI design in their research. Kudos to her for a great presentation.
OTHER NOTES
– The best part about attending conferences like CNS is the opportunity to catch up with friends and acquaintances from other academic institutions. Nobody in academia ever seems to stay put for very long, which makes reunions like CNS all the more special. The first night of the conference is a frenzy of people seeing each other for the first time in a long time. I love it.
– San Francisco is an amazing town. I have probably visited the city a dozen times and it never gets old. It is probably my favorite big city in the US. Which is good, because I have two more conferences there this spring.
– If you happen to find yourself in San Francisco you should eat at the House of Nanking, situated between Chinatown and North Beach. Simply amazing Asian/American food. I would also recommend Chow on Church Street in the Castro district. They have a real focus on healthy dining with an eye toward responsible cooking. The kicker is that the food is absolutely amazing. Check them out if you are around.