Sunday, December 15, 2013

Conducting an evaluation with Outcome Harvesting

Over the last few months, I have been working together with a colleague on the final evaluation of a medium-size program, implemented in three different continents by four implementing agency. The lack of a baseline, the complex context and my dissatisfaction with a straight results-based approach convinced us to integrate an outcome harvesting component to the methodology been used. It was not my first time dabbling with elements of OH, but it was the first time I experimented with the entire process. To be clear, OH was used in parallel with a traditional results-based approach both to reassure the client and as an insurance policy for the evaluation team. I cannot name the agencies involved or the name of the program because of the confidentiality agreement. The program ran for six years, in four widely different regions, each region having a budget of roughly 2.5 million USD to promote regional integration in the disaster management sector.

Outcome harvesting allowed us to produce a meaningful analysis of the (scarce) outcomes of the program. I am convinced that without OH we would have had a really hard time identifying and making sense of the modest changes introduced by the program. However, adopting OH was not a smooth process for a number of reasons. The whole exercise was at times very frustrating but I believe that it could be interesting to share my experience with other practitioners.

Here are some of my observations.

1. Insufficient response rates to initial survey. After analyzing the available literature and identifying a first list of probable outcomes, the evaluation team prepared a simple questionnaire as a word document, in both the official languages of the program and sent it to the four regional implementing agencies. The questionnaire consisted of 8 questions (some of them multiple choice, others open ended) and it was estimated that it would take roughly twenty minutes to fill it out. After 4 follow-up emails reminding the agencies of the importance of returning the questionnaire with their inputs, the evaluation team manager to receive replies from two regions. However one of the two completely ignored the questions and format, opting to send a different document largely based on descriptions of the outputs of the intervention.

2. Low profile of the project. Identifying boundary partners with enough knowledge of the intervention or ability to differentiate between this and other actions implemented by the regional agency proved to be extremely challenging. By and large, boundary partners belonged to two categories: national disaster management agencies and international organizations. Both saw the program as part of the institutional mission of the implementing agency and could not provide feedback on the action that was being evaluated.

3. The contribution of the project was hard to separate from other projects run by the same implementing agency. Our own evaluation team struggled to attribute outcomes to this project rather than other funding streams managed by the same implementing agency. Often agencies worked on long-term strategies of which this program covered just one or more steps. Other projects executed in the past or being implemented simultaneously had also contributed to advance the goals of the project. Separating who did what using whose money proved to be almost impossible. Knowing that OH focuses on contribution rather than attribution offered little consolation. At times the evaluation team felt that it was evaluating the last decade of work of an entire sector rather than just one project.

4. Positive bias. As a consequence of the difficulty to separate the outcomes of this project from the other programs of the same agency, the number of outcomes was inflated. The contribution of the project to some of the outcomes was perceived to be minimal by the evaluation team, but there was no way to definitely prove it since the implementing agencies had a clear incentive to exaggerate their achievements and other parties had not enough insights to disprove those claims.

5. Confusion in understanding the difference between outputs and outcomes.  Few if any of the interlocutors within the implementing agency had a clear idea of the difference between outcomes and outputs. Even after the evaluation team conducted briefing sessions (necessary short due the limited time available for the interviews) the subtleties of the difference between the two concepts remained largely obscure to the project management units. Again, outcome harvesting had nothing or little to do with it. However, the introduction of a new definition of “outcome” proved to be far too much even for those who had manage to master the difference between outputs and outcomes.

 6. Changes take time. Not enough time had passed to test the depth of the adoption of the changes in behavior. Observing a change in the institutional framework (adoption of policies, etc) can take a mighty long time in certain countries. While changes in operating procedures in the disaster management sector become only evident during an emergency.

Notwithstanding all the difficulties encountered, outcome harvesting proved to be an invaluable resource when the evaluation team started the task of analyzing the information collected and preparing the final report. The intervention had been mired in serious administrative issues that delayed implementation, virtually precluding the possibility of achieving a decent adoption rate of the outputs. The analysis of the effectiveness and potential impact would have been very limited if done purely on the basis of a classic results-based approach. Through outcome harvesting the evaluation team was able to conduct a much more meaningful analysis that contributed to the understanding of the dynamics of the project both at global and regional level.