Over the last few months, I have been working together with
a colleague on the final evaluation of a medium-size program, implemented in
three different continents by four implementing agency. The lack of a baseline,
the complex context and my dissatisfaction with a straight results-based
approach convinced us to integrate an outcome harvesting component to the
methodology been used. It was not my first time dabbling with elements of OH,
but it was the first time I experimented with the entire process. To be clear,
OH was used in parallel with a traditional results-based approach both to
reassure the client and as an insurance policy for the evaluation team. I
cannot name the agencies involved or the name of the program because of the
confidentiality agreement. The program ran for six years, in four widely
different regions, each region having a budget of roughly 2.5 million USD to
promote regional integration in the disaster management sector.
Outcome harvesting allowed us to produce a meaningful
analysis of the (scarce) outcomes of the program. I am convinced that without
OH we would have had a really hard time identifying and making sense of the
modest changes introduced by the program. However, adopting OH was not a smooth
process for a number of reasons. The whole exercise was at times very
frustrating but I believe that it could be interesting to share my experience
with other practitioners.
Here are some of my observations.
1. Insufficient
response rates to initial survey. After analyzing the available literature
and identifying a first list of probable outcomes, the evaluation team prepared
a simple questionnaire as a word document, in both the official languages of
the program and sent it to the four regional implementing agencies. The
questionnaire consisted of 8 questions (some of them multiple choice, others
open ended) and it was estimated that it would take roughly twenty minutes to
fill it out. After 4 follow-up emails reminding the agencies of the importance
of returning the questionnaire with their inputs, the evaluation team manager
to receive replies from two regions. However one of the two completely ignored
the questions and format, opting to send a different document largely based on
descriptions of the outputs of the intervention.
2. Low profile of the
project. Identifying boundary partners with enough knowledge of the
intervention or ability to differentiate between this and other actions
implemented by the regional agency proved to be extremely challenging. By and
large, boundary partners belonged to two categories: national disaster
management agencies and international organizations. Both saw the program as
part of the institutional mission of the implementing agency and could not
provide feedback on the action that was being evaluated.
3. The contribution
of the project was hard to separate from other projects run by the same implementing
agency. Our own evaluation team struggled to attribute outcomes to this
project rather than other funding streams managed by the same implementing
agency. Often agencies worked on long-term strategies of which this program
covered just one or more steps. Other projects executed in the past or being
implemented simultaneously had also contributed to advance the goals of the
project. Separating who did what using whose money proved to be almost
impossible. Knowing that OH focuses on contribution rather than attribution offered
little consolation. At times the evaluation team felt that it was evaluating
the last decade of work of an entire sector rather than just one project.
4. Positive bias.
As a consequence of the difficulty to separate the outcomes of this project
from the other programs of the same agency, the number of outcomes was
inflated. The contribution of the project to some of the outcomes was perceived
to be minimal by the evaluation team, but there was no way to definitely prove
it since the implementing agencies had a clear incentive to exaggerate their
achievements and other parties had not enough insights to disprove those
claims.
5. Confusion in
understanding the difference between outputs and outcomes. Few if any of the interlocutors within
the implementing agency had a clear idea of the difference between outcomes and
outputs. Even after the evaluation team conducted briefing sessions (necessary
short due the limited time available for the interviews) the subtleties of the
difference between the two concepts remained largely obscure to the project
management units. Again, outcome harvesting had nothing or little to do with
it. However, the introduction of a new definition of “outcome” proved to be far
too much even for those who had manage to master the difference between outputs
and outcomes.
6. Changes take time. Not enough time had
passed to test the depth of the adoption of the changes in behavior. Observing
a change in the institutional framework (adoption of policies, etc) can take a
mighty long time in certain countries. While changes in operating procedures in
the disaster management sector become only evident during an emergency.
Notwithstanding all the difficulties encountered, outcome
harvesting proved to be an invaluable resource when the evaluation team started
the task of analyzing the information collected and preparing the final report.
The intervention had been mired in serious administrative issues that delayed
implementation, virtually precluding the possibility of achieving a decent
adoption rate of the outputs. The analysis of the effectiveness and potential
impact would have been very limited if done purely on the basis of a classic
results-based approach. Through outcome harvesting the evaluation team was able
to conduct a much more meaningful analysis that contributed to the
understanding of the dynamics of the project both at global and regional level.
