We talked about the importance of evaluation, methods of evaluation, challenges of evaluation and what should be evaluated a lot. During the last class we also raised the question of competence of individuals and organizations who are conducting evaluation.
I remember once we had an independent evaluation of the project funded by Spanish agency. The task of the expert from Spain was to evaluate 4 year project in four rural communities of Armenia having interviews with beneficiaries and project team within three days. Her evaluation showed that the project was not effective. When the report arrived it was clear why it was not successful. The expert was really good in evaluation methods but she did not know anything about the cultural specifics of the region, anything about the governance system, and the time was not sufficient to discover all these aspects, analyze and then prepare the report. The interpreter with whom she worked used different terms while interviewing the beneficiaries and beneficiaries were answering “no” to very obvious questions. One of them was “Did the project team organized training and other activities to encourage community mobilization? “. The interviewees answered no because they have never heard term mobilization. The project team used other translation for the word mobilization while working with the community.This is a small example of the independent evaluation that could destroy the collaboration of the donor agency and implementing agency. Luckily the donor agency was willing to hear the implementing agency's feedback and justification regarding each point of the evaluation. There was another 4 year project approved after that evaluation and the evaluation of that project was also independent but was done through a professional agency located in Armenia.
So who can or should evaluate? Organizations themselves ? Audit or other independent agencies who are professionals in the field of evaluation ? Beneficiaries ? Volunteers/ community members? Evaluation of each of these groups has its advantages and disadvantages. Any ideas ?
Your post points out that cultural competence and understanding of the work of organizations is critical to effective evaluation. I'd argue as well that outcomes and impact may be more important than outputs (though as the Ebrahim article points out, for some services, outputs are all that you can get).
ReplyDelete