End-of-term evaluation

Background

This evaluation was specifically designed to incorporate the participation of in-country stakeholders in the form of a ‘Local Evaluation Team’ or LET. Through participation in the design, implementation, review and dissemination processes of the evaluation, the LET members enhanced their appreciation for evaluation, strengthened their evaluation capacity and assisted in focusing the evaluation on areas of particular interest for the national response.

Additionally, an ‘External Evaluation Team’ or EET was deployed to bring an independent point-of-view and relevant expertise to the evaluation. The EET was made up of regional HIV experts and evaluation specialists, who worked to draw out conclusions and articulate lessons learned for the evaluation. In the process, EET members strengthen their skills in undertaking evaluations of national programmes, thus further expanding the pool of experienced evaluators throughout the region.

This was the sixth ‘learning by doing’ evaluation undertaken by CARPHA (and formerly the Caribbean Health Research Council) under the existing PANCAP/GFATM Agreement and within the context of a national response. These evaluations are intended to complement the theory-based training courses implemented by CARPHA, as well as to respond to a demand throughout the region for more exposure to evaluation.

The LET for this evaluation included:

• 
  Ms. Nicole Drakes - Assistant Director, NHAC;
  
• 
  Ms. Alexis Nurse - BCC Specialist, NHAC;
  
• 
  Dr. Anton Best - Senior Medical Officer Health (Communicable Diseases), MOH;
  
• 
  Dr. Dale Babb - Project Director, MOH;
  
• 
  Ms. Chisa Cumberbatch - Health Planner, MOH;
  
• 
  Ms. Shawna Crichlow - Data Analyst, MOH;
  
• 
  Ms. Madge Dalrymple - HIV Coordinator, Ministry of Transport;
  
• 
  Mr. Teddy Leon - Senior Programme Officer, CHAA;
  
• 
  Ms. Patsy Grannum - Director, Movement against Discrimination Coalition (MOVADAC);
  
• 
  Ms. Laura-Lee Foster - Research Assistant, National Council for Substance Abuse (NCSA); and,
  
• 
  Ms. Nia Salankey - Safety and Health Officer, Labour Department, Ministry of Labour.
  

The Regional Partners/Counterparts on the EET included:

• 
  Dr. Ayanna Sebro - Director HIV & AIDS Coordinating Unit, Ministry of Health, Trinidad & Tobago;
  
• 
  Mrs. Sannia Sutherland - Director of Prevention, National HIV Programme Ministry of Health Jamaica; and,
  
• 
  Ms. Audrey Brown - Focus Group Consultant and Behaviour Change Communication Expert.
  

• 
  Mr. Erin Blake - M&E Specialist;
  
• 
  Ms. Patricia Smith-Cummings - M&E Officer; and,
  
• 
  Mr. Garth Watson - M&E Officer.
  

Methodology

This evaluation was undertaken using a ‘learning by doing’ methodology grounded in Utilization-Focused approach (Patton, 2008) and an RBM&E framework (Kusek and Rist, 2004). This methodology was used in order to produce practical recommendations that can be incorporated in the implementation of the next NSP for HIV (2014-2018) in Barbados.

Utilization-Focused Evaluation

Utilization-Focused Evaluation has been described as one of the most promising approaches for evaluations in the 21st century (Stufflebeam, 2001). As the name implies, Utilization-Focused Evaluation emphasizes the utility of evaluation findings and recommendations for an evaluations intended audience.

‘Utilization-focused evaluation begins with the premise that evaluation should be judged by their utility and actual use; therefore, evaluators should facilitate the evaluation process and design any evaluation with careful consideration for how everything that is done, from beginning to end, will affect use.’ (Patton, 2008, p.37)

Thus, Utilization-Focused Evaluation concentrates on evaluation questions that are of interest to stakeholders and promotes the use of findings through stakeholder involvement throughout the evaluation process (Patton, 2008). Utilization-Focused Evaluation does not depend on or advocate any particular evaluation content, model, method, theory, or even use. Rather, it is a process of helping primary intended users select the most appropriate content, model, methods, theory and uses for their particular situation. Thus, the strength of the Utilization-Focused Evaluation approach lies in the process by which stakeholders are engaged to determine the purpose of the evaluation, the procedures employed during the evaluation, and the subsequent findings of the evaluation.

Results-Based Monitoring and Evaluation

RBM&E can be broken up into two parts; Results-Based Monitoring and Results-Based Evaluation (Morra Imas and Rist, 2009). Results-Based Monitoring is the continuous and routine process of collecting and analysing data to assess an intervention's implementation through pre-determined indicators and targets. Progress is measured by comparing the current situation with the desired results identified for key performance indicators. Results-Based Evaluation takes a broader perspective on an intervention and seeks to determine its relevance, efficiency, effectiveness, impact, and sustainability. The intention of Results-Based Evaluation is to provide information that credible and useful; that facilitates the development of lessons learned which can be used to tailor an interventions design and implementation. Results-Based Evaluation also seeks to account for any confounding factors that could have influenced an intervention’s outcomes.

Appropriateness of Approach

These approaches were appropriate for this evaluation for the following reasons:

1. 
   The NSP being evaluated was developed using an RBM&E framework, and is linked to an M&E Framework that incorporates clearly states objectives and indicators for which the results and progress can be measured;
   
2. 
   The purpose of the NSP is to guide the national response to achieving tangible results for target populations;
   
3. 
   This evaluation was designed expressly to draw out lessons learned that can inform implementation of activities under a new NSP;
   
4. 
   RBM&E has growing currency in the Caribbean and is viewed favourably by health professionals and decision makers;
   
5. 
   The recommendations from this evaluation will need to be practical and actionable in the Barbados context; and,
   
6. 
   These approaches lend themselves well to CARPHA’s capacity building mandate and ‘learn by doing’ approach for evaluations.
   

Learning by Doing Approach

The ‘learning by doing’ aspect of this evaluation reflects CARPHA’s desire to enhance the capacity for evaluation throughout the region. The approach seeks to build the capacity of local personnel and regional counterparts to conduct evaluations of health interventions by involving them in ‘real life’ evaluations. This approach builds on training sessions and other capacity building efforts implemented by CARPHA as part of a broader M&E capacity building strategy and health system strengthening in the region. Project personnel participates in the evaluation planning, execution, and dissemination through appointment to the LET while regional peers participate as part of the EET. This lends greater credibility to the evaluation by presenting an independent perspective of project activities while developing their capacity through greater exposure to evaluation activities.

Data Collection Methods

The answers to the evaluation questions were developed based on a mix of quantitative and qualitative methods. Data collection methods included a combination of primary and secondary data collection. Primary data collection took the form of key informant interviews and focus group discussions with key populations. Secondary data collection consisted of document and data reviews of reports, indicator data, and research.

Key informant interviews are the process of interviewing persons who have insight or knowledge on a specific issue. In this case, the key informants represented people with in-depth knowledge of different aspects of the HIV and AIDS response in Barbados, and associated key populations. This data collection method is time efficient and inexpensive, and is relatively low burden on respondents. To cover all the interviews in a timely manner and to lessen the burden on the key informants, the EET split up into smaller teams to conduct most of the interviews, sharing questions ahead of time to ensure all pertinent questions were asked of each key informant. In a few cases, when the key informant had a broad understanding of the national response, the entire EET were present for interviews. Interviews were conducted with one key informant at a time and in small groups of people representing similar positions, in rooms that facilitated privacy. It should be kept in mind that a weakness of the key informant data collection process is its susceptibility to interview and question bias. Additionally, the views of key informants are not generalizable.

Focus group discussions are interviews of a small group of persons conducted by a skilled moderator. For the purposes of this evaluation, focus groups were conducted with groups of people representing different key populations identified as especially vulnerable to contract HIV and AIDS in Barbados based on the available data. The aim of these focus groups was to gather an in-depth understanding of key population perceptions of the national response and associated services. This data collection method is also relatively low cost, flexible and a useful tool for gathering a broad range of information on a specific topic. However, the findings from focus groups are not generalizable and cannot be said to represent the ‘private’ views of participants, especially on sensitive topics. Focus groups also rely heavily on the skills of the moderator and are vulnerable to moderator and question bias.

Document review is the process of systematically analysing documents to develop a case. This method of data collection was employed in this evaluation because it is time efficient, inexpensive and can yield insight into the thinking behind a project at different junctures. This data collection method is considered free from interview bias, although writer bias may be present. The method is also limited in that documents are static and are not always well aligned to specific evaluation questions.

In order to balance the shortcomings of each of these data collection methods, the evaluation team sought to balance and triangulate the findings from the different data sources. The evaluation team was also chosen purposefully to ensure representation of professionals from differing backgrounds, equipped with varying perspectives on national responses.

Download 0.92 Mb.

Share with your friends: