Valuation of the Effectiveness of AI for the Zambia Police Services Victim Support Unit

By Mette Jacobsgaard

The following describes a valuation of the work of the Victims Support Unit (VSU) within the Zambia Police Services. VSU was supported as part of a foreign aid program to support access to justice in Zambia.

Introduction

A number of journals, associations, and trusts are dedicated to evaluation. Companies, aid donors, and many others have specific guidelines for evaluation. Evaluation is important. Looking at the literature, there does not appear to be a unanimous definition of evaluation. However, those who do try to define what evaluation is suggest some common traits. According to Wikipedia, “Evaluation is systematic determination of merit, worth, and significance of something or someone using criteria against a set of standards.” The Evaluation Trust says that “Evaluation is assessing and judging the value of a piece of work, an organization or a service.” The Danish International Development Agency (DANIDA) explains, “An evaluation is an examination, as systematic and objective as possible, of the design, implementation and results of projects and their objectives.” Value judgments seem to be implied in all cases.

Whereas I do not have a problem with judgments per se, I do question situations where people depend on so-called expert evaluations.

It is not difficult for us to develop criteria for our judgments, but our judgments also involve our values. We have different ways of valuing based on culture, background, access, experience, and education. Evaluation is comparative by nature. Given that evaluations are often carried out by external evaluators, those value judgments are essentially made by the evaluator. They become the judgments of the expert evaluator. At the same time, the whole point of evaluating is to look back to learn in order to move forward and possibly improve on the past. How is it possible that recommendations of an evaluator living one reality can be relevant and even understood by people and projects living another reality? And by the way, by the time the information is filtered it is outdated. This is where Appreciative Inquiry comes in.

Appreciative Inquiry is (e)valuation. Valuation is the determination of the value of something or the act of looking back to see what worked best—what was most valued in the past with a view to increase such value in the future. Appreciative Inquiry not only facilitates self-valuation—it also embeds the valuative process within the organization or project.

Valuation of Development Aid Projects and Programs

Third-World development is based on assumptions of need. If no need is portrayed, no funds will come forward. This is the basic premise and reality of the world of development aid. “Third World Aid” is in itself a social construct, a reality in which one part of the world has great needs and wants which can be fulfilled by those living in another part of the world, providing the right planning and actions are in place. However, as many evaluations show, efforts generally fall short of the plans, perhaps precisely because we evaluate against the plans and not against the objectives of the project or program—or indeed against the wants and needs of those who should be assisted.

There are several reasons for using Appreciative Inquiry for evaluation. These are:

• The assumption that everyone has something to contribute
• The active engagement of the project and those involved with it
• The “empowerment” that comes from respectful self-assessment
• The ownership of the decisions made through the self-assessment.
• AI goes beyond data collection and analysis
• AI goes beyond judgment and taking stock
• AI is participatory
• AI creates ownership of outcome
• AI involves the whole system
• AI embeds results
• AI is a process and perspective that can be applied to existing evaluation methods

In most project and program evaluation, there is a need and even a requirement to describe or document what happens in particular with respect to output and impact (written reports are a big part hereof). Using Appreciative Inquiry does not negate this requirement. In fact, in order to secure continued funding, progress has to be documented or related in such a way that the reader feels convinced that the activities/project meets its objectives. Telling, sharing, and documenting stories is the most powerful way of documenting impact and change.

VSU Zambia

The Victims Support Unit (VSU) was established as a unit within the Zambia Police Service in 1996. VSU was created specifically to address problems of violence affecting women, children, and the elderly. The unit has two main functions: (1) creating awareness amongst the general public through sensitisation and information workshops and (2) counseling victims of crime, crisis negotiation, and intervention to address trauma resulting from victimization.

There are a total of about 350 police stations country-wide in Zambia and about one thousand police posts. The VSU has officers in three hundred of the police stations. Each police station has two or three trained officers, at least one of each gender. The number of cases that the VSU deals with has increased steadily since it opened, from 657 cases in 1997 to 9,282 cases in 2002. The VSU is staffed with specially trained police officers and is a very popular unit within the Police Service, with many officers applying to be part of the unit. The VSU had quickly become widely known and very popular in Zambia. Their success was reported in the local papers, and they saw themselves as “the human face” sympathizing with the public, especially victims of crime.

The VSU has received aid funding from the foreign donor in Lusaka in 2000, 2001, and 2002. The money was spent on training of officers, community workshops, information dissemination, and office equipment. In 2003 I was asked by the foreign donor to review the achievements of VSU with a view to recommend whether future assistance should be given. Clearly, Zambia Police Service, and the VSU especially, were apprehensive about being evaluated, as the outcome of the review would be decisive in terms of whether future funding would be granted by the donor, which at the time was a major donor in Zambia.

Although there was every reason to believe that funding would continue given the success of the VSU, the fact that an external consultant arrived to evaluate the work put the VSU on the spot and possibly having to defend themselves and their actions. This is a normal reaction in all evaluation cases, and in many cases the client has reason to be worried. Different people, with different values, evaluate change in different ways, and you do not know what you will get when external consultants arrive.

In my initial meetings with the VSU I made it clear that this evaluation would be in their own hands, that they were to evaluate themselves, and that we would focus on their strengths and why it was so important to discover their very best practices in order that they could be learned from and built upon. In the first meeting, we discussed establishing a core group of especially interested officers from the VSU. In the next meeting with those who had volunteered and been selected from the VSU and the Police Service, I asked: “If you were to ask the question; tell me a story about the best experience you have had with the VSU—whose story would you be curious to hear?” It became clear to the officers that they would appreciate the views of stakeholders such as their clients, representatives from social services, the hospital (which dealt with forensic evidence and rape cases), legal aid providers, NGOs, and more. It was agreed that representatives from these stakeholder groups should be included in the evaluation. We agreed to divide the work into three phases:

In Phase I we started the workshop with the core group by having them interview each other about their best experience of being a member of the VSU and also what their aspirations were for the unit. Based on the themes from the interviews, we discussed the sort of questions they had for further elaboration from the stakeholder group and their clients. We agreed on a protocol of questions for the stakeholder workshop and the clients. We discussed and practiced interviewing and the importance of staying with the generative interview protocol.

The following themes arose from the interviews:

A discussion was held with respect to what the VSU would like to know from the stakeholders and from their clients. The following were listed:

Stakeholders

• How VSU and the stakeholders (NGOs) complement each other
• The wishes of the stakeholders
• Their attitude (toward VSU)
• To know more about working relations (with Social Welfare)
• How they view VSU
• Suggestions (for improvement)
• How much they know about VSU
• Weaknesses (of VSU)
• How VSU can help

It was agreed that the same questionnaire as the one used with the core group could be used with the wider stakeholder group with a few modifications, as follows:

We had a long discussion about how to include and interview the clients who were, after all, victims of crime and violence. Not only were they vulnerable but it was claimed that they may also be suspicious if interviewed by the VSU. At the same time, there was an argument for having the VSU officers conduct the interviews—not of their own clients but VSU officers from other parts of the country interviewing clients of their colleagues. I felt it was important for the VSU officers to hear from the clients directly, to feel the impact of their work. I also reminded the VSU officers that the questions we asked were about best experiences and not about complaints, and the clients would therefore be more likely to feel free to come forward with their experiences.

There was great excitement in the group after the interviews in Lusaka and great expectations and energy to carry the exercise on into two rural provinces, one in the Western and one in the Southern province. As part of the planning, we discussed what aspects of the interview protocol had worked well and whether there was a wish for changes, especially whether there was anything in particular arising from the interviews that the VSU officers were curious to know more about.

At this point we had defined by planning and designing the protocols—discovered in the first workshop and redefined following the first workshop. We had created a new protocol between the two workshops, recorded and described the themes, and used these to redefine and created the protocol to be used in the two up-country workshops. I would like to point to the fact that we did not go through the 4 Ds in the first two workshops. Instead, we circled back around definition and discovery—gathering data.

Only in the two following workshops up-country did we go through all of the 4 Ds.

The Up-Country Workshops

In the two workshops up-country the VSU had gathered officers from the entire province. Some had to come by boat up rivers and some had long journeys on foot and bicycles. The workshops were held at a central place that had both meeting room and lodging facilities. The logistics involved were substantial.

Apart from the VSU officers, representatives from local NGOs and local government participated. The first day of the workshop was another discovery and recording and discussing the themes. The following day the participants went to the field to interview the clients of VSU, and the day after that they shared the stories they had heard in the field and pulled out even more themes. This way we now had data about VSU and their work from the VSU officers at the provincial level, other stakeholders, as well as the clients of VSU. A substantial amount of data and themes had been collected, which in turn informed the Dream phase and the remainder of the exercise.

What Stands Out

What stands out for me from this valuation are a number of things best expressed by examples:

VSU, like the rest of the Zambia Police Service, lacks transport. Police officers, including the VSU, often have great difficulty getting transport to investigate crime and make arrests, let alone just showing their faces in the communities they serve. There are few resources allocated for cars and motor bikes and, even if the equipment is there, the funds may not be there to buy the fuel to run it. This is a big problem and a complaint that would possibly override everything else had we chosen a problem-focused approach to this review. The resulting recommendation would have been: “Zambia Police Service must allocate funds for fuel.” Through our AI process, we heard stories of how successful the VSU had been in reaching their clients—often without delay. This was made possible in those cases where the VSU had worked closely with NGOs operating in their area who were able to give them lifts. Part of the design process therefore focused on how they could collaborate closer with NGOs in general to mutual benefit. The fact that transport is a big problem due to lack of equipment and funds remains. However, the objective of reaching clients can be dealt with without being stuck due to lack of funds.

The power of the stories is not just the story itself—it is the solutions to the problems that almost automatically present themselves.

The 6th D

There is usually a big requirement for reporting in aid projects. The project needs to report on a regular basis. Using the AI process is an ideal way of gathering data to continuously monitor progress, which in turn will make valuations easier. This can be done by making an inquiry within each of the five Ds generating stories and valuable information about what has worked and how.

• Discovery: Describe what important data the interviews generated—What did we hear?
• Dream: Describe the dreams and how it relates to the objectives of the project.
• Design: Describe the design and explain how it will benefit the project.
• Delivery: Describe how the design will be implemented, and finally
• Describe: How you plan to move ahead with the next discovery—Definition.

I have encouraged projects that I have worked with to take time to document and describe each step as they move along with implementation of the project.