Making Sense of Grantee Reports
Your program officer's toolbox contains four implements that will help you make the right decision about whether to continue funding or to cut the foundation's losses. Although all are not equally valuable, each has its merits, and each should be used as appropriate.
Interim Reports and Site Visits
Most foundations require at least one interim narrative and financial report during the course of a funded project. Typically, these are required on an annual basis. One such report provides a valuable insight into how the project is developing, but multiple interim reports allow for a better sense of how the project is improving (or deteriorating) over time. The site visit to the funded project, of course, is an invaluable firsthand opportunity for you to assess how the grantee's efforts are faring. If you study the interim reports assiduously, paying particular attention to the progress of the project in answering the important evaluation questions, and if you make the site visit, chances are that you will form a very good notion of whether renewal of funding is merited even before the term of the grant has expired.
Final Financial Report
This report offers a wealth of precise data, but the meaning of that data is notoriously difficult to discern. For example, let us suppose that during the course of the project, the grantee requested permission to make numerous shifts of grant funds from one line item to another, and even secured permission to create new line items. The final financial report, therefore, bears little resemblance to the originally submitted budget. What does this mean? Does it signify that the grantee is a poor planner and an indifferent financial manager? Or might it be that the grantee is wonderfully flexible and adept at seizing unforeseeable opportunities? Another example: the final financial report reveals that the grantee still has a balance of grant funds left unspent. What does this mean? Does it signify that the grantee lacks energy and is not competent to complete the project according to the agreed-on timeline? Or might it be that the grantee is a careful financial manager, has found ways to economize and leverage funds, and is getting maximum bang for the foundation's buck?
The answers to these questions, of course, are a matter of interpretation, and the basis for making the judgment must come from the interim and final narrative reports, from site visits, and from the project evaluator's assessment. There are, however, a few things you can learn unambiguously from final financial reports. If the numbers do not add up, especially if there are big discrepancies, it means trouble. If the numbers reveal a series of unrequested and unauthorized line-item changes, it means trouble. If a huge amount of grant funds remain unspent, it raises many concerns. Is the grantee using the award as a de facto endowment? Why is the grantee asking for a second grant if it is nowhere near completing the first? Would not the more appropriate course be an extension of time without any additional funds at present? Finally, it is a good precaution to check that the financial report has been signed by two people, preferably the CEO and the chief financial officer of the grantee organization. This reduces (but does not wholly eliminate) the likelihood that one dishonest person is “cooking the books.”
Final Narrative Report
This is an important tool for making the decision to renew or not to renew, but it is a flawed tool. Simply put, the project director, who is writing the report, has a vested interest in putting the best face on things. As a program officer notes, “Final narrative reports are a lot like corporate annual reports. Every year is always a ‘very good year.’” The final narrative report, unlike the final financial report, does not need to add up, so it is highly likely that the project director will make every effort (borrowing from Johnny Mercer) to “ac-cent-tchu-ate the positive, eliminate the negative.”
It should be acknowledged that there are many project directors who are very conscientious about writing balanced final reports. Even in the case of the majority who accentuate the positive in their reporting, not all is lost; in fact, there are several useful items of information to be gleaned from such reports. First, what outcomes did the project director decide to highlight? How do these compare to the original goals of the project? What might explain any discrepancies? Second, disappointing outcomes may sometimes be simply omitted from the report. What was left out and why? Third, does the narrative report agree with the financial report? If the narrative touts the outcomes achieved by intensive work in one area and the line item for that area is badly underspent, what explains the apparent contradiction? Fourth, is the narrative report generally consistent with the interim reports and your own observations from the site visit? If not, what explains the inconsistency? Fifth, how well does the project director's assessment of outcomes match that of the project evaluator? Presumably more objective than the project director, the evaluator should be able to render a more dispassionate judgment on the outcomes. If the two reports do not agree, it certainly suggests that you should investigate the discrepancy before making a decision on renewal of funding.
Final Evaluation Report
This is the most useful tool in the box. The evaluator is being paid explicitly to observe the project and assess its outcomes. As a third party formally employed by neither the foundation nor the grantee, the evaluator should not have any particular ax to grind. Of all the reports, therefore, the evaluator's is likely to be the most objective and the most useful for determining next steps. This is not to say, however, that the evaluator is a completely disinterested party. Two factors, one human and the other financial, make pure neutrality impossible.
The human factor springs from the fact that the evaluator has worked with the grantee during most or all of the project. This means that the evaluator has forged either good or poor working relationships with the project's leaders and thus has become either attached to or disaffected from the project. Obviously, attachment is likely to bias the evaluator in favor of positive outcomes (he or she likes the people or believes in the project); disaffection is likely to bias the evaluator in favor of negative outcomes (he or she dislikes the people or has lost faith in the project).
The financial factor enters the picture because the evaluator, whether paid directly by the foundation or indirectly by the foundation through the grantee, probably wants to continue the relationship should the grant be renewed and also probably wants to be the evaluator for other foundation-funded projects in the future. Some evaluators fear that a negative evaluation report would not be appreciated by the grantee or the foundation and might result in their getting replaced as the project goes into second-stage funding. Similarly, the evaluator may fear that a negative report could land him or her on a foundation “list” of proscribed evaluation consultants. Such fears are usually groundless, for most grantees and foundations recognize the value of constructive criticism, but every evaluator of any experience has a horror story to tell about the dangers of being too frank in his or her evaluation report.
Usually the human and financial biases, if evident at all in a report, are very subtle in nature. Only rarely do they get as blatant as those related by a program officer for a large midwestern foundation: “I had a project about which everyone in the state, from the governor's office to neighborhood groups, was bitterly complaining. Everyone was very vocal and very public about their dissatisfaction with the project and its director. Yet the evaluator, in his summative report, raved about how wonderful the project was and about how well respected its director was. I was stunned—it was as if the evaluator was the only one in the state who didn't know—but of course, he had gotten an earful from some of the same people who had complained to me.”
Fortunately, that sort of outright prevarication is an extreme rarity. Nonetheless, it is important to recall that evaluators, third parties though they may be, cannot be completely impartial. Yet even after you have taken this caveat into account, the evaluator's report truly provides the sharpest tool in the box. The evaluator may have biases, but they are generally milder than yours or those of the project director. Although it is true that a summative report done by an evaluator brought in exclusively for that purpose may be more objective than one done by the project's formative evaluator (the purely summative evaluator would not have had the chance to become either attached to or disaffected from the project), it is still acceptable to have one evaluator do both. After all, the formative evaluator knows more about the project after having worked on it for so long.
Summative evaluations are of more value to you if they are clear-cut as opposed to equivocal. It is difficult to decide on a course of action if the report offers mostly ambiguous conclusions. Summative evaluators are most valuable when they offer lessons that can be learned by both grantee and foundation alike, so that the grantees may improve their future performance and the program officers may sharpen their grantmaking skills. You are wise to cultivate a candid and open relationship with evaluators, the better to encourage them to tell the whole story in summative evaluation, whether that story be good, bad, or indifferent.
If there is no external evaluator, and the evaluation is being done by the project management team themselves, there is an obvious danger of the report being tainted by self-interest. In such cases, it is important for you and the grantee to clearly agree on the key evaluation questions, milestones, and targets in advance, before the grant commences. If you have this agreement, both interim and final reports can be used to measure progress or to explain why expected goals are not being met.