Government Granting Agencies and Wastin’ Time

NPR some time back had an article on scientists leaving science, mostly because government funding of research has become more like a sweepstakes than a well. I had posted a couple of comments in the following thread, and I’d rather those didn’t get lost, as Google seems not to take note of or index comments in Disqus for search.

Tom, a self-described retired NIH grant reviewer, had weighed in, generally saying that the system worked fine, the article was describing a non-issue. And in one comment, Tom chided another commenter for not focusing on improving “the system”. I responded, then Tom wrote back.

Tomonthebeach -> John A • 20 days ago

You do not sound like you want to improve the system, just benefit from it. If review is a game, play it well. Once you have mastered it, convince “the system” to change.

Wesley Elsberry -> Tomonthebeach • 19 days ago

Do we want to talk about improvement of the system? OK.

Let’s have a look at one aspect of the NIH and NSF sweepstakes, the “public burden”. I’ve been involved in the production of several NSF grant applications, and recent information says that NIH is, like NSF, funding a small fraction of grant applications. The NSF proposal guide I used indicated a public burden of 120 hours, that is, they *expected* applicants to invest 120 man-hours of effort in putting together an application. I’m assuming NIH is similar. If we are talking about a single researcher tackling one of those, then we are talking about that person spending 6% of a full-time equivalent on a single application. If each application had a uniform probability of acceptance, and the acceptance rate were 10% of applications (a figure that I think is much higher than the actual odds now), an applicant would have to submit 7 applications to have just better than even odds of actually getting one of them funded, chewing up 42% of a full-time equivalent in public burden. To get to a 90% chance of acceptance, our hypothetical researcher would have to submit 22 applications with an associated 1.32 full-time equivalents in public burden (which is more than a bit onerous for the lone researcher). The fact that the application process is competitive, though, means that serious applicants are in all likelihood investing more than the stated public burden. So here’s the thing… one can readily see that a low acceptance rate induces a high wastage of effort on the part of researchers. At the 10% acceptance rate, for every hundred applications considered, 5.4 full-time equivalents (man-years) are wasted. This seems to me to be inefficient. We do need to balance responsible use of public funds with the effort we exact from researchers in order to get those funds, but it seems to me that the NIH/NSF application process model has things way, way out of balance. The obvious place to start is with the public burden. Cut that down to 2 to 10 hours of public burden for an initial concept round, whose winners would be invited to submit full applications where the expected acceptance rate is 50% or better, and we could stop wasting so much researcher time and effort, while still being able to assure ourselves that we are funding projects responsibly.

Here’s a few questions for Tom: what’s the average public burden and average acceptance rate for NIH applications now, and what’s the total number of applications considered by NIH in the past year? With that information, I can estimate the amount of wasted effort researchers have put into trying to get that relatively small pot of NIH money. Who knows, maybe it isn’t as dire as what the situation appears to be at NSF.

If there is also a figure for average man-hours or FTE per NIH grant, I could even work up a net man-hours funded, taking into account the wastage that happens.

Tomonthebeach -> Wesley Elsberry • 19 days ago

The data you seek is available on the NIH.gov webpages. It doubtless will confirm your general hypothesis, so I will grant your points in general as reasonable. Keep in mind that government money is spent at the direction of the political system – Congress. Each district wants a piece of the NIH (and other federal research grantor) funding pie in their universities, and they do fuss when their schools are not getting their “fair share.” The slow application and blind peer review process is a significant, albeit costly (and yes sometimes wasteful) way to create an even playing field for national competition. Until a better system is accepted, the process will endure.

Senior researchers fare better in competition because over the years they have proven their talent in concrete terms, and have learned from unsuccessful applications how best to articulate to peer reviewers the rationale for their new projects. Successful grantees also impress me as being very humble (accommodate criticism from peers) and very persistent – they don’t “give up.”

Well, Tom might be right that the information I requested is somewhere in the elephantine expanse of the NIH website. I don’t feel like spending hours of my time trying to narrow things down to locate them, especially when a person claiming inside knowledge might be a guide. But apparently not. Sure, government money is spent at the direction of Congress. But the particulars of policy on exactly how the award of a grant happens isn’t all handed down from above, NIH and NSF have developed those policies to some degree themselves, and have the ability to modify them themselves. But it would require a recognition that the system is busted and wasteful of research effort, which does not appear to be close to happening. I attended a workshop at an Animal Behavior Conference meeting (2006, I think) where the NSF personnel attending offered the advice to spend much more time on the application than the “public burden” specified, so the various estimates I made are likely very conservative in estimating wasted researcher time.

Tom does continue to advocate the notion that things are just peachy with the NIH process, citing the observation that those who succeeded in obtaining NIH grants are “persistent” and didn’t “give up”. Well, yeah, that is an archetypal example of a a self-selected sample. It isn’t a point in favor of complacency.

There was another part of the thread where I put in my $0.02.

truth seeker -> James • 20 days ago

Student loan debt is the debt owed by the students, not government funding. Students can pay it back if they find jobs.
Research funding would help some of these students to find those jobs and also new research can increase jobs in other sectors too. Remember USA is what it now because of research.

Wesley Elsberry -> truth seeker • 20 days ago

Agreed. Further, what the USA will become is going to be what it can manage to find out via research. And that will be done competitively. I get the feeling that many people think that we managed to get to the head of the global economy and that will simply continue as a matter of course. I see that as an extremely dangerous point of view. If we want our country to continue to be an economic leader, investing in education and basic research is an essential policy stance. Contrariwise, if we want to settle for handing off economic pre-eminence to other nations, all we have to do is continue to whittle away at education and basic research.