The Art and Science of Successful Scientific Grant Writing - Part 5

This post will be my final one in the series on "The Art and Science of Successful Scientific Grant Writing". Here I would like to discuss the grant review process and convey to any New Investigator the importance of putting yourself in the reviewer's shoes as you develop and polish your proposal.

Here are links to the previous posts in this series:

The Art and Science of Successful Scientific Grant Writing - Part 1
https://steemit.com/steemstem/@davidrhodes124/the-art-and-science-of-successful-scientific-grant-writing-part-1

The Art and Science of Successful Scientific Grant Writing - Part 2
https://steemit.com/steemstem/@davidrhodes124/the-art-and-science-of-successful-scientific-grant-writing-part-2

The Art and Science of Successful Scientific Grant Writing - Part 3
https://steemit.com/steemstem/@davidrhodes124/the-art-and-science-of-successful-scientific-grant-writing-part-3

The Art and Science of Successful Scientific Grant Writing - Part 4
https://steemit.com/steemstem/@davidrhodes124/the-art-and-science-of-successful-scientific-grant-writing-part-4

Hopefully after about 9 weeks, you should have now completed your Project Summary, Project Description and References, have had them reviewed and edited by your mentor, colleagues and friends, and generated a 3rd and near final draft of these documents. You should have also made significant progress on developing the Cover Page, Curriculum Vitae, Budget, Budget Justification, Facilities and Equipment, Current and Pending Support, Certifications, and have acquired letter(s) from Collaborators. During the last 3 weeks before the deadline for submission of your proposal, double-check that your documents meet all of the agency guidelines in terms of length, font size, type density, margin widths and format. Each agency usually provides a check-list that you can tick off as you prepare your proposal components. You should have been in close contact with your assigned Pre-award Specialist at your institution throughout development of these proposal components.

Note that some agencies require you to submit a letter of intent to alert the agency that you will be submitting a proposal. This is usually clearly indicated in the guidelines.

During the last 3 weeks before submission of your proposal, take time to read some of the reference materials provided in Part 4 of this series. I particulary like Jacob Kraicer's article on "The Art of Grantsmanship" (https://cdn.dal.ca/content/dam/dalhousie/pdf/research-services/The_Art_of_Grantsmanship%20-Kraicer.pdf). While this article gives a somewhat different strategy than I have articulated (e.g. it suggests a much longer time-line for proposal preparation, suggests writing the Project Summary/Abstract last rather than first, and assumes submission of the grant by mail rather than electronically ... Kraicer's article was written before electronic submission was available via GRANTS.GOV), nevertheless it gives extremely valuable insight about the grant review process and many other key tips. Bear these in mind as you re-read your proposal before final submission. Double-check that you have put everything in your own words and have not plagiarized large sections. Are your figures and legends legible when the document is printed? Many reviewers will print a copy of your proposal before reviewing it! Please consider also adding a timetable to the end of your "Project Description" outlining when you anticipate that each objective/aim will be completed.

Consider also these "Ten Fatal Mistakes in Grant Writing". W. A. Tacker Jr. in "Research Review" (May 1991, Vol 4. No. 5) has noted 10 "fatal" mistakes in grant writing which can be avoided by answering the questions (Q) accompanying each "fatal" mistake:

1. Lack of new or original ideas. Q. Exactly what are the new and original ideas in my proposal? Where have I really pointed out clearly which ones are new?

2. Diffuse, superficial, or unfocused research plan. Q. Where do I clearly point out the central aspects of my research plan and make them distinct from the secondary aspects?

3. Lack of knowledge of published relevant work. Q. Have I done a complete literature search for this research area and included the important findings of scientists other than myself and co-workers?

4. Lack of experience in the essential methodology. Q. Does the description of my feasibility studies demonstrate that I can carry out all of the proposed scope of work?

5. Uncertainty concerning future directions. Q. Where do I describe what I will do in the later experiments if early experiments don't work? Do I describe what applications will stem from the results? Do I describe how other scientists can use my findings?

6. Questionable reasoning in experimental approach. Q. Where have I explicitly stated the hypothesis to be tested? Is there any circular logic in the proposal? Is every step of reasoning actually written down in the text, and have I left out steps in the approach which I hope the reviewer can figure out?

7. Absence of an acceptable rationale. Q. Where have I stated the fundamental reasons why this research will provide an answer to the hypothesis?

8. Unrealistically large amount of work. Q. Can this work actually be carried out in the stated time and with the resources listed in the proposal? Have I made projections for: inflationary costs, sick leave and vacation time for personnel, time to present results at meetings and to write papers for publications?

9. Lack of sufficient experimental detail. Q. Could another competent scientist successfully carry out this work by using the experimental detail I have given in the proposal?

10. Uncritical approach. Q. Have I included a discussion of anticipated problems and my plans for dealing with them? Have I fairly presented ideas and studies of scientists in my field with whom I disagree?

Take the time to view the following video: "NIH Peer Review Revealed provides a front-row seat to a NIH peer review meeting. Real scientists from the scientific community review fictional but realistic grant applications for scientific merit."

For NIH R01 grants please also read: "NIH R01 Grant Application Mentor: An Educational How-to Manual — 5th Edition" (https://cvm.ncsu.edu/wp-content/uploads/2017/02/NIH_R01_Grant_Application_Mentor_5th_edition_ONLINE.pdf), giving particular attention to the section entitled "11 SIMPLE MISTAKES THAT CAN DERAIL YOUR GRANT APPLICATION" on pages 270-271.

It is essential, before submission of your proposal, that you are aware of the review criteria that will be used to evaluate your proposal. You should spend the last few weeks before submission fine-tuning your proposal so that it satisfies these review criteria.

The Grant Review Process at USDA

Because the above video focuses on the NIH grant review process, I would here like to give my insight and experience serving as a panel member and panel manager for a different agency, USDA, and then finally give brief commentary on the processes that occur at NSF and DOE.

The grant review process at USDA begins about 1 year before proposals are due. The Program Director (a permanent member of staff at the agency) identifies an experienced reviewer (and prior panel member from previous grant review panels) who is willing to serve as a Panel Manager for the upcoming proposal submissions. The Panel Manager works with the Program Director to identify possible candidates as panelists for the next grant review panel. These panelists might include prior panel members, and investigators who were successful in their grant application in the previous panel. The panel must consist of an equal number of males and females, an equal number of assistant, associate and full professors, a representative from industry, at least one minority, and no more than one person from any one state. The panel should exclude anyone who is submitting a proposal for review. Thus, the panel cannot be chosen until all proposals are received. Once proposals to the panel are received after the deadline for submission, these are catalogued by the Program Director and Panel Manager and these individuals then proceed to invite panelists to serve on the panel. Suppose that 100 proposals were received, the Program Director and Panel Manager would seek 10 panelists who had appropriate expertise in the program's research mission and expertise covering the spectrum of topics of the proposals submitted, and satisfying the diversity criteria noted above. Ideally the panel should be fully established 3 months before the panel meeting is planned. Each panelist must agree to review 20 proposals assigned to them, read a further 10 assigned to them, submit written reviews for their assigned proposals before the panel meeting, and be prepared to attend a panel meeting held at the agency headquarters in Washington D.C. over a period of 3 days. The agency reimburses the panelists for their travel, accommodation and meal expenses. Each panelist would be urged to maintain strict confidentiality about the proposals that they receive for review and ultimately discuss at the panel meeting.

Agreeing to serve on a panel involves a major commitment of time and effort, but it provides extremely valuable insight concerning the review process!

Typically each panelist is asked to be a Primary Reviewer for 10 proposals, a Secondary Reviewer for 10 proposals and a Reader for 10 proposals. The roles of the Primary, Secondary Reviewer and Reader at the panel meeting will be explained in more depth below.

Prior to proposal assignment to the panelists, the Program Director and Panel Manager send to the panelists copies of all the proposal Project Summary pages and ask the panelists to identify any conflicts-of-interests. A panelist should not be assigned a proposal to review if the panelist has collaborated with, or co-authored a paper or proposal with the investigator(s) during the last 5 years. A panelist should not be assigned a proposal from an investigator at the same institution as the panelist. Ideally the Primary review assignments go to individual panelists who have considerable familiarity with the subject matter of the proposal. Secondary review assignments go to panel members who have moderate familiarity with the research area, while Reader assignments may go to panel members with little familiarity with the exact subject area.

When the review assignments have been made, each panelist receives electronic copies of the proposals that they have been assigned, usually at least 2 months before the scheduled panel meeting. The Program Director and Panel Manager then proceed to request "ad-hoc" reviews from experts in the field from around the world. When I served as Panel Manager, the Program Director and I sought "ad hoc" reviews from 5 individuals for each proposal. The "ad hoc" reviewers would be asked to submit written reviews before the panel meeting. Each written review (including those from the Primary and Secondary reviewer) provides an overall rating assessment chosen from the following list ... "Excellent, Very Good, Good, Poor, Very Poor".

Panelists would be expected to arrive the day before the panel meeting and attend an orientation meeting with the Program Director and Panel Manager on the evening before the panel meeting, stressing the importance of maintaining confidentiality and providing guidance for any novice panelist(s) (i.e. any panel member who had not served on the panel before).

The panel meeting would typically begin the next morning at 8:00 am over coffee and breakfast. The roles of the Program Director and Panel Manager at the panel meeting are essentially to keep the meeting on track, answer questions from the panelists, provide all available "ad-hoc" reviews to the panelist, and serve as scribes to record the rankings assigned to each proposal that is reviewed. The Program Director and Panel Manager are not permitted to comment on any proposal that is discussed, except to introduce the proposal title, PI's and institutions, and identify the Primary and Secondary and Reader at the panel meeting. The Program Director and Panel Manager must ask any panel member who has a conflict-of-interest with the proposal to be discussed, to step out of the meeting room before discussion occurs.

The first proposals to be discussed are those that have been assigned to experienced panelists, so that the novice panelists can observe the process. The Reader, who should have already read the proposal being discussed, takes copious notes on the discussion that follows. The Primary Reviewer first gives an overview of the rationale, significance and objectives of the proposal to the panel, and then explains his/her opinion of the strengths and weaknesses and scientific merit of the proposal, feasibility, probability of success, appropriateness of the budget, and qualifications of the investigator. The Primary Reviewer will then briefly summarize the opinions of the "ad-hoc" reviewers who submitted written reviews and who are not in attendance at the panel meeting. The Secondary Reviewer will then give his/her opinion of the strengths and weaknesses, scientific merit, feasibility, probability of success, appropriateness of the budget, and qualifications of the investigator. The Reader may then comment, and the proposal is then open to discussion/comments/questions by other panelists (who may have only seen the Project Summary). This entire discussion may take only 10 - 20 minutes! If there is uniform opinion from the Primary, Secondary, Reader and ad-hoc reviewers that the proposal is excellent, this may take only 10 minutes (likewise if there is uniform opinion from the Primary, Secondary, Reader and ad-hoc reviewers that the proposal is poor or very poor). Where there are large discrepancies between individual reviewers there may be longer discussion (20 minutes or more) in order to come to consensus. If parts of the budget are deemed inappropriate, the panel may recommend cutting the budget if the proposal is eventually recommended for funding. The panel is alerted whether or not the investigator is a "New Investigator" (an individual who may be just starting their career and who has not received grant funding before), and allowances are made by the panel in the evaluation and ranking of the proposal. New investigators may have fewer publications, and little or no preliminary data!

After discussion is complete, the Primary Reviewer makes a suggestion/recommendation of where the proposal should be ranked among the following categories ... Must Fund, High Priority for Funding, Medium Priority for Funding, Low Priority For Funding, Some Merit, and Do Not Fund. The Secondary Reviewer agrees or disagrees with this proposed ranking. The Reader and other panelists comment until consensus is reached, and the Panel Manager then records the consensus ranking on a Blackboard or Whiteboard. After ranking the Reader prepares a summary of the afore-mentioned discussion and the ranking outcome. This summary will eventually become the "Panel Summary" after approval by the Primary Reviewer, Secondary Reviewer, Panel Manager and Program Director. All Panel Summaries are read aloud to the panel by the Readers at the end of the day and appropriate edits are made.

If the panel has 100 proposals to consider, 50 on day 1, and 50 on day 2, and if each proposal discussion and Panel Summary reading takes 15 minutes on average, that means each day of the panel will take 50 x 15 minutes = 12.5 hours. The panel typically meets from 8:00 am to 9:00 pm with brief breaks for lunch and dinner. It can be an intense, tiring yet very educational experience. One soon begins to recognize what it takes to be successful in developing a strong proposal that is highly ranked by reviewers!

The third half-day of the panel meeting consists of the Panel Manager going through the titles and brief explanation of the thrust of each proposal and asking the panel to assess whether each proposal is correctly ranked relative to others. Particular attention is given to those in the Must Fund, High Priority for Funding, and Medium Priority for Funding categories. Is proposal X superior to Y and Z, or vice versa?

At the end of the panel meeting, the final list of proposals in each category is recorded, and this list becomes the final "Panel Recommendation". The panel is thanked and then dismissed.

The Panel Manager and Program Director use the "Panel Recommendation" and the budget allocated to the program to identify the proposals that can be funded. Some Panel Managers and Program Directors work down the list of top-ranked proposals, funding proposals at the requested level (or level recommended by the panel) until the program's budget is fully allocated. This may mean that fewer than 10% of the proposals are funded. Some Panel Managers and Program Directors try to achieve a goal of funding as many proposals in the "Must Fund" and "High Priority for Funding" categories as possible. To achieve this it may be necessary to fund those in The "Must Fund" category at the requested level (or level recommended by the panel), but then have to make strategic cuts in the budgets of the proposals in the "High Priority for Funding" category in order not to exceed the program's budget allocation. This may permit funding of up to 20% of the proposals received by the panel. While most proposals in the "Medium Priority for Funding" do not receive funding, it is important to note that any "New Investigator" whose proposal was ranked in the "Medium Priority for Funding" category (or above) would be recommended for funding using agency funds that are specifically assigned to support and encourage young scientists who are just starting their careers. Thus, "New Investigators" have a significant funding advantage over established investigators!

The Panel Manager and Program Director then proceed to contact each of the investigators to let them know whether or not their proposal was funded, whether budget cuts were made, and to provide all investigators with copies of all of the written reviews (with names of the reviewers redacted) and Panel Summary explaining why their proposal was, or was not, funded. If a budget was cut, the investigator would have the opportunity to submit a revised budget and a revised list of objectives.

If you are funded, then congratulations are in order. Be prepared to be asked to serve on a grant review panel in the near future!

If you are not funded, try not to be discouraged. Read the written reviews and Panel Summary carefully, and talk with the Program Director by telephone to ask if it would be appropriate to resubmit your proposal (making adjustments to the proposal that address the reviewer's critiques) at the next panel deadline.

The Review Process at NSF and DOE

The grant review processes and panel meetings at the NSF and DOE are quite similar to those described above for USDA in my experience. At NSF there may often be two panels per year for each program area. Thus, the entire review process described above may be compressed into 5-6 months rather than approximately 7 months for USDA. On a tight schedule of 5-6 months turn-around, fewer "ad-hoc" reviews may be requested for NSF proposals. NSF meets at their headquarters in Arlington, VA. DOE panels may meet at different locations depending upon time of year. For example, they may hold a panel meeting in Florida during the winter to encourage panelists from colder climates to participate!

As before, please feel free to post comments or questions in the comments section below. I will attempt to answer them as soon as possible. If you enjoyed this series please consider an up-vote. Here's wishing you the best of luck in your grant writing endeavors!