The Problem With Peer Review

"If this is the best of all possible worlds, what are the others?"

- Voltaire, Candide, Chapter 6

Peer review is a lousy way to score research proposals; it just seems to be the best one we have found to date. NIH is altering its application and review processes, attempting to improve things. Unfortunately, the major problem cannot be fixed.

What major problem? The peer reviewers.

For example (derived from my own experience and that of my colleagues):

A proposal is submitted to study biological process A in disease X. A model of disease X is proposed, and standard techniques will be used to examine the role of process A. On initial submission, the reviewers say that process A appears to be important in disease X, but “enthusiasm would be stronger” if further preliminary data were presented. Furthermore, pathway Y contributes to process A, so perhaps it should be included in the study.

As PI, you consider these suggestions and do further experiments to augment your preliminary data. You review studies of pathway Y and conclude that it should be included in your proposal. You rewrite you grant, and send it back to Bethesda.

This time, you have a novel reviewer. Those who saw the first version are satisfied, but the latest one feels that biological process B is of far more importance than process A, even though you have data and published literature from other labs supporting process A. Perhaps the latest reviewer believes you should study disease Z rather than disease X. Or maybe a different model should be used, even though you have been using this model for all other work in your lab. Or, perhaps, s/he wants all of the above changed.

By the way, the reviewers also criticize your productivity because, while generating the preliminary data requested, you have not continued to publish as many peer-reviewed journal articles.

This wackalunacy must stop.

It is one thing to suggest additional experiments or a better technique. These suggestions may strengthen the work.

Telling the PI to completely change the nature of the study is inappropriate:

• Look at process B even though your preliminary data and published studies support process A in your hypothesis
• Disease Z is more important than disease X, even though you have never studied disease Z

Proposals get toasted with this sort of criticism in each round of reviews. If we could fund the top 25-30% of proposals(which are getting scored in the very good-excellent range), some of these applications would still fly; in the current funding drought, any little glitch (no matter how trivial) can drop the score below the pay line.

I do not have the answer. I beg any reviewers reading this to play fair. Judge what is proposed; no study can examine every biological process! If your favorite pathway is omitted, fear not! Someone else may be asking that question; let this PI examine his/her idea! Occasionally I have seen someone who proposes to examine the role of process A in disease X, but is using a model of disease Z. This is a major flaw, and a change of model or disease is in order. It is NOT in order if you, the reviewer, find disease Z more interesting than disease X. Would you want someone to tell you to study disease X? I thought not…

NIH has shortened its application process and changed the scoring system. Somehow I don’t believe this will change reviewer behavior, but I will find out early in 2010.

In the meantime, I will just keep imagining worse worlds of review. I finally appreciate Candide.

Art courtesy of PhotoXpress.