The Personal Paradigm and Peer Review

RenalWeek 2009 in San Diego is underway. I just enjoyed a forum on conflict of interest in medicine. One talk, by Rober M. Califf, MD, of Duke University, discussed bias in clinical trial reporting. Most of the speakers focused on financial conflicts of interest. Dr. Califf described a number of nonmonetary conflicts of interest as well.

One of these nonfinancial conflicts involved scientific hubris. For example, you review a paper or a grant and its (potential) results shoot down your own research and ideas. Do you accept the possibility that you are wrong? This work has the potential to make your prior work irrelevant, as well as decreasing your ability to get funding and promotion. The document is well-written and convincing, at least to the other reviewers. Do you allow it to proceed, or do you demand further work to reconcile it with your current hypothesis? Data that may not be feasible, if your personal paradigm is wrong?

We scientific types supposedly worship at the alter of truth and knowledge. We are human, though, and humans like to have explanations for everything. As nature abhors a vacuum, our brains hate missing information or pieces. In other words, we are biologically predisposed to develop hypotheses and patterns for everything around us. We also seem predisposed to defending views in which we are invested.

This source of conflict is rampant in peer review. Qualified peers must have knowledge of the subject area in question, and they will have pre-formed hypotheses about the material. Most reviewers can accept new experiments that advance their ideas, even if the world moves in a new direction. When those pre-formed ideas quash legitimate questions, then they become a source of conflict of interest.

What can one do? One can exclude reviewers based on probable bias. This practice is commonplace for manuscript submission. If a member of your study section could be trouble, you can request that they not review your grant. You should send your stuff to lots of different funding agencies as well; the same proposal may be funded by one agency and triaged by another, just because peer review is subjective and biased in this way.

Human nature simply will not allow all sources of bias to be removed from the world. I hope we in science can consider our own intellectual prejudices and overcome this conflict of interest.

Photos courtesy of PhotoXpress.

The Problem With Peer Review

"If this is the best of all possible worlds, what are the others?"

- Voltaire, Candide, Chapter 6

Peer review is a lousy way to score research proposals; it just seems to be the best one we have found to date. NIH is altering its application and review processes, attempting to improve things. Unfortunately, the major problem cannot be fixed.

What major problem? The peer reviewers.

For example (derived from my own experience and that of my colleagues):

A proposal is submitted to study biological process A in disease X. A model of disease X is proposed, and standard techniques will be used to examine the role of process A. On initial submission, the reviewers say that process A appears to be important in disease X, but “enthusiasm would be stronger” if further preliminary data were presented. Furthermore, pathway Y contributes to process A, so perhaps it should be included in the study.

As PI, you consider these suggestions and do further experiments to augment your preliminary data. You review studies of pathway Y and conclude that it should be included in your proposal. You rewrite you grant, and send it back to Bethesda.

This time, you have a novel reviewer. Those who saw the first version are satisfied, but the latest one feels that biological process B is of far more importance than process A, even though you have data and published literature from other labs supporting process A. Perhaps the latest reviewer believes you should study disease Z rather than disease X. Or maybe a different model should be used, even though you have been using this model for all other work in your lab. Or, perhaps, s/he wants all of the above changed.

By the way, the reviewers also criticize your productivity because, while generating the preliminary data requested, you have not continued to publish as many peer-reviewed journal articles.

This wackalunacy must stop.

It is one thing to suggest additional experiments or a better technique. These suggestions may strengthen the work.

Telling the PI to completely change the nature of the study is inappropriate:

• Look at process B even though your preliminary data and published studies support process A in your hypothesis
• Disease Z is more important than disease X, even though you have never studied disease Z

Proposals get toasted with this sort of criticism in each round of reviews. If we could fund the top 25-30% of proposals(which are getting scored in the very good-excellent range), some of these applications would still fly; in the current funding drought, any little glitch (no matter how trivial) can drop the score below the pay line.

I do not have the answer. I beg any reviewers reading this to play fair. Judge what is proposed; no study can examine every biological process! If your favorite pathway is omitted, fear not! Someone else may be asking that question; let this PI examine his/her idea! Occasionally I have seen someone who proposes to examine the role of process A in disease X, but is using a model of disease Z. This is a major flaw, and a change of model or disease is in order. It is NOT in order if you, the reviewer, find disease Z more interesting than disease X. Would you want someone to tell you to study disease X? I thought not…

NIH has shortened its application process and changed the scoring system. Somehow I don’t believe this will change reviewer behavior, but I will find out early in 2010.

In the meantime, I will just keep imagining worse worlds of review. I finally appreciate Candide.

Art courtesy of PhotoXpress.

The Truth About Scientific Research

I remember attending my 10th high school reunion. I was in my fellowship in pediatric nephrology, and my former classmates were amazed that I was “a real MD.” When I confided that I really liked doing research, they asked what discoveries I had made.

Science doesn’t work like that.

Yes, I know, people win Nobel prizes for “discovering” something or other. It may not be exactly what they set out to find, and it is very likely that their “discovery” is different from their original hypothesis. Novel data also get dissed. The first descriptions of many phenomena are relegated to “low-impact-factor” journals because reviewers refuse to believe something that is truly novel.

Science proceeds nonlinearly and incrementally. Only in retrospect can the path from A to B be seen. Several experiments may suggest a direction, but the overriding truth comes from a body of work. At other times an incidental part of the study proves to be more important than what was originally the goal.

For example, I start out interested in condition X which is known to be associated with factor A. I believe that factor A produces factor B which promotes condition X. I do an experiment that shows A produces B; however, my follow-up shows that B produces D in the presence of C. Someone else has shown that D causes condition Y. We now show that blocking C can prevent condition Y. Not what I set out to prove, but perhaps an important “discovery” if Y is a major health issue.

Advances depend on one thing: money. Generous sources of funding allow us to perform experiments and learn more about how life works. At the present time only 20% of studies are getting money from NIH. Peer review is a pretty good system, but you never know where a critical observation will come from. Reviewers also tend to favor more conservative work that is expected to go as planned rather than stuff that may provide novel break-through results. The NIH has tried to get around this issue, but hasn’t solved the problem yet.

Science is like a river of facts. As it flows and eddies form, seemingly unrelated streams from different sources may come together, making a whole that is more than the sum of its parts. And you never know which creek upstream will provide the critical drop.

Photos courtesy of PhotoXpress.com

The Toughest Loss

Yesterday the first word in my daily email from The Scientist was “renal” so I had to read on; unfortunately, the next words were “researchers faked data.”

These researchers violated their experimental protocol, resulting in 16 papers with results that may be better than they should be (according to the post, 14 have been withdrawn and the other 2 are being withdrawn). The really scary part buzzed through the comments, namely that this fraud came to light because the investigators had a falling-out. How many fake papers out there will never come to light because those involved will not “kiss and tell?” I wonder about this every time I get different results from the established paradigm. Did I do something wrong? Were their published methods incomplete? Or did they present “edited” data? Let’s face it; you can Photoshop damn near anything!

What were these “scientists” thinking?

The sanctions officially leveled in this case seem mild, at least to me; 10 years of exclusion from federal research activities for the PI and 3 years for the post-doc/junior person. They have also lost their academic positions. Perhaps the most damning punishment is that they have lost their reputations.

This group was researching novel immunosuppressant drugs for kidney transplantation. This is not my research area (and rumors about this team have apparently been circulating in the transplant immunology community for some time), but I suspect many in this world will have the same reaction I do whenever a paper is retracted: never to trust any data from these people again. Once lost, a reputation is the most difficult thing to regain. There will always be a question mark in the back of peoples’ minds, even if it is never spoken.

While various levels of supervision and the peer review process can help weed out fraud, ultimately editors (and readers) have to trust what the researcher says. If you can’t be trusted, then your research career is over.

Art courtesy of PhotoXpress.

Thoughts from a Study Section

It's my lunch break at study section, and I felt compelled to share a few thoughts.

What is the MOST IMPORTANT part of your grant application? Shiny colored figures? Cover story on the JCI? Novel, patented assay?

Silly people: it's the title and abstract! Why is this so? Because your study section assignment and reviewers will be based primarily on these parts of the grant!

My current assignment is with a VA Merit Review section. We have a wonderful system where the reviewers can see titles and abstracts to pick ones they would love to review and those they absolutely should not review. Over time, I have discovered that prostaglandins confuse me. It is unfair to the PI to have me review a grant dealing primarily with prostaglandins. Thus, I exclude myself from those applications.

Every once in a while, I get assigned an application with significant prostagladin work that wasn't described in the title or abstract. I have finally decided that the PI, in these cases, deserves what they get. Which is generally me dozing off somewhere in the experimental plan. Falling asleep during a review NEVER helps.

We also see grants with titles so vague or misleading to be of no help whatsoever. I know we have limits on the length of the title, but please! If we can communicate on Twitter in 140 characters, we can say something coherent in a grant application!

So make sure your titles and abstracts communicate your topics well. Otherwise you may get me reviewing your prostaglandin grant, and neither one of us wants that to happen.