End-of-term evaluation

Appendices

Appendix 1 – Evaluation Protocol and Matrix

Contents

ACRONYMS

BCC Behaviour Change Communication

CARPHA Caribbean Public Health Agency

CSW Commercial Sex Worker

EET External Evaluation Team

HIV Human Immunodeficiency Virus

LET Local Evaluation Team

M&E Monitoring and Evaluation

MOH Ministry of Health

NHAC National HIV/AIDS Commission

NSP National Strategic Plan

PANCAP Pan-Caribbean Partnership Against HIV and AIDS

RBM&E Results Based Monitoring and Evaluation

RePDU Research, Training and Policy Development Unit

STI Sexually Transmitted Infection

TB Tuberculosis

U-F Utilisation Focused Evaluation

UNDP United Nations Development Programme

INTRODUCTION

The Research, Training and Policy Development Unit (RePDU) of the Caribbean Public Health Agency (CARPHA) – formerly the Caribbean Health Research Council (CHRC) – is the principal regional health institution charged with providing support to countries in the monitoring and evaluation (M&E) of national HIV and AIDS responses. RePDU recognizes the importance of developing country capacity to conduct periodic evaluations in order to assess their progress toward achieving stated outcomes and goals. The focus of RePDU’s work is on helping countries to generate information which can be used by programme implementers to strengthen their programming to attain stated results. RePDU utilizes a ‘learn by doing’ approach, working alongside country staff and utilizing the skills and experience of well-trained M&E professionals and subject experts from fellow Caribbean countries to conduct these evaluations.

RePDU in partnership with the National HIV/AIDS Commission (NHAC) in Barbados proposes to undertake an end-of-term outcome evaluation of the Barbados National Strategic Plan (NSP) for HIV Prevention and Control 2008-2013. It is intended for this evaluation to inform the implementation of activities under the NSP 2014-2018 which is currently being finalised. As such, the evaluation will focus on the development, implementation and performance of the programme strategic response during the period 2008-2013.

The goal of the 2008-2013 NSP was the ‘Mitigation of the social and economic impact of HIV and AIDS on the population thereby reducing new cases (incidence) and ensuring the sustainable development of our nation’. This goal is in keeping with the objectives of Goal 3 of the National Strategic Development Plan of Barbados 2005-2025.

The 2008-2013 NSP included the following five Priority Programme Areas for Action:

1. 
   Prevention and Control of HIV transmission;
   
2. 
   Diagnosis, Treatment and Care of PLHIV;
   
3. 
   Support for PLHIV;
   
4. 
   Programme Management and Institutional Performance; and,
   
5. 
   Surveillance, Monitoring and Evaluation, and Research.
   

1. 
   Gender power relations and dynamics and HIV/AIDS;
   
2. 
   Human resource management; and,
   
3. 
   Human rights policy and legislation.
   

EVALUATION PURPOSE

The aim of this evaluation is to explore the strengths and weaknesses of the National response to HIV, with the view to strengthening programme development and implementation under the new NSP. To ensure that the evaluation remained closely aligned to the NSP, the Priority Programme Areas for Action and the Strategic Objectives were used as the basis for developing this evaluation protocol and the corresponding evaluation matrix.

EVALUATION OBJECTIVES

• 
  Determine the relevance of the programmatic response to the NSP;
  
• 
  Assess the extent to which the strategic objectives of the NSP were achieved;
  
• 
  Determine whether the programme objectives of the national response were achieved; and,
  
• 
  Assess the extent to which programme resources were utilised to achieve programme objectives.
  

EVALUATION QUESTION

1. 
   Do the goals and strategic objectives of the NSP address the needs of key populations?
   
2. 
   To what extent have the strategic objectives been achieved through programme implementation?
   
3. 
   Have the programme objectives of the national response been achieved, if not why?
   
4. 
   How have resources been utilized in the implementation of the NSP?
   

EVALUATION DESIGN

Theoretical Framework

This evaluation will employ aspects of the utilization focused (U-F) and results-based monitoring and evaluation (RBM&E) approaches. U-F evaluation concentrates on the evaluation questions which are of interest to stakeholders and promotes the use of findings through stakeholder involvement in the evaluation process (Patton, 2008). It has been described as one of the most promising approaches for evaluations in the 21st century (Stufflebeam, 2001). The strength of the U-F approach lies in the process by which stakeholders are engaged to determine the purpose of the evaluation the procedures which are to be employed, and by the close attention which is paid to the contextual dynamics. The RBM&E approach complements the U-F evaluation approach because it focuses on outcomes and uses results to improve performance, which is different from traditional M&E which measures and reports only the status of results. RBM&E is well suited to evaluations of national strategic plans and national programmes which have clearly articulated goals and objectives.

Learn by Doing Approach

By utilizing a learning by doing approach to conducting this end-of-term evaluation, RePDU proposes to build the capacity of local personnel and regional counterparts to conduct evaluations of their public health system responses. Critical steps to achieving this for the evaluation include:

1. 
   Identifying a local evaluation team (LET) comprised of key persons in the national response who can identify and gain access to the required data sources;
   
2. 
   Developing an evaluation protocol;
   
3. 
   Developing an evaluation matrix which illustrates the alignment of NSP Strategic Objectives, evaluation questions, data sources and key informants;
   
4. 
   Choosing an external team of regional peers with appropriate skills and experience to conduct an evaluation as part of an External Evaluation Team (EET);
   
5. 
   Assigning roles and responsibilities for data collection;
   
6. 
   Conducting key informant interviews/focus groups;
   
7. 
   Undertaking data analysis and synthesis;
   
8. 
   Developing a preliminary report of findings and recommendations;
   
9. 
   Developing a final report; and,
   
10. 
    Disseminating the findings to national stakeholders.
    

The national response to HIV/AIDS in Barbados is multi-sectoral effort. Correspondingly, the Local Evaluation Team (LET) sought to include wide representation from key stakeholders and includes representatives from the NHAC, Ministry of Health (MOH), Civil Society and Government Ministries. The LET members of the evaluation will include:

• 
  Ms. Nicole Drakes – Assistant Director, NHAC
  
• 
  Ms. Alexis Nurse – BCC Specialist, NHAC
  
• 
  Dr. Anton Best – Senior Medical Officer Health (Communicable Diseases), MOH
  
• 
  Dr. Dale Babb – Project Director, MOH
  
• 
  Ms. Chisa Cumberbatch – Health Planner, MOH
  
• 
  Ms. Shawna Crichlow – Data Analyst, MOH
  
• 
  Ms. Madge Dalrymple – HIV Coordinator, Ministry of Transport
  
• 
  Mr. Teddy Leon – Senior Programme Officer, Caribbean HIV AIDS Alliance
  
• 
  Ms. Patsy Grannum – Director, MOVADAC
  
• 
  Ms. Laura-Lee Foster – Research Assistant, NCSA
  
• 
  Ms. Nia Salankey – Safety and Health Officer Labour Dept.
  

External Evaluation Team

The members of the External Evaluation Team (EET) will be selected from CARPHA staff and regional peers who add value to the evaluation team with their expertise in the specific areas being explored by the evaluation. The EET for this evaluation are:

• 
  Mrs. Sannia Sutherland - Director of Prevention, National HIV Programme Ministry of Health Jamaica (Strategic Objectives 1 & 2)
  
• 
  Dr. Ayanna Sebro - Director HIV & AIDS Coordinating Unit, Ministry of Health, Trinidad & Tobago (Strategic Objective 3)
  
• 
  Mr. Garth Watson – M&E Officer, CARPHA (Strategic Objective 4)
  
• 
  Mr. Erin Blake – M&E Specialist, CARPHA (Strategic Objective 5)
  
• 
  Mrs. Patricia Smith-Cummings – M&E Officer, CARPHA (Strategic Objective 6)
  

An evaluation matrix was developed by the CARPHA team in partnership with the LET and other national stakeholders in May 2014. This matrix sought to link ‘sub-evaluation questions’, indicators, data sources and key informants to the NSP Strategic Objectives and Programme Priority Areas. This matrix will serve as a guide to the evaluation team as the shape their evaluation findings and recommendations and is included in Appendix 1.

Roles and Responsibilities

The LET will be responsible for the following aspects of the evaluation:

• 
  The provision of input on the evaluation design, including evaluation objectives and questions, and reviewing the evaluation protocol and matrix;
  
• 
  Identifying key informants to be interviewed by the EET during their in-country mission;
  
• 
  Sourcing relevant documents and data for review by the EET (and uploading these documents into a shared ‘drop box’);
  
• 
  Assisting with logistics for the in-country mission;
  
• 
  Acting as key informants where required;
  
• 
  Working alongside the EET to conduct key informant interviews;
  
• 
  Working with the EET team to develop recommendations and findings;
  
• 
  Reviewing and approving the draft report; and,
  
• 
  Assisting with the dissemination of the final report.
  

• 
  Reviewing documents (including focus group report) and data uploaded by the LET prior to the in-country mission;
  
• 
  Conducting a 5 day in-country mission to conduct interviews with key informants in partnership with the LET to formulate findings and recommendations to be included in the final report;
  
• 
  Presenting preliminary findings and recommendations at the end of the mission to local stakeholders; and,
  
• 
  Writing up findings and recommendations for the final report.
  

• 
  Draft focus group protocol for review by CARPHA M&E Team;
  
• 
  Acquire ethics approval from the IRB with assistance from CARPHA and LET (NHAC);
  
• 
  Conduct focus groups with key populations;
  
• 
  Compile transcripts and conduct analysis;
  
• 
  Prepare a focus group report based on findings from focus groups sessions; and,
  
• 
  Brief EET and LET on focus group findings.
  

This evaluation protocol and the corresponding evaluation matrix was developed by RePDU staff from the 21-23 May 2014 in partnership with key stakeholders. This Protocol provides an overview of the focus of the evaluation and how the evaluation will be undertaken.

Data Collection & Analysis

For this end-of-term outcome evaluation, answers to the evaluation questions will be arrived at based on a mix of quantitative and qualitative methods. There will be four significant data collection strategies:

1. 
   Document Review
   

2. 
   Review of Indicator Data
   

3. 
   Focus Groups
   

Key populations identified in the NSP with which focus groups will be undertaken include: MSMs, Sex Workers (Male & Female), Ex-offenders, Youth (Male & Female), PLHIV (Male & Female), Persons with Disabilities, Drug Users, Migrants, Single Unemployed and Dependent Women.

4. 
   Key Informant Interviews
   

Proposed Timetable

In-Country Mission

Final Report

A draft report will be completed by RePDU and submitted to the LET for review by October 3, 2014. The LET will send their comments to RePDU by October 10, 2014 and a final report will be submitted on October 24, 2014.

Dissemination of Findings

The final report will be shared with all stakeholders by the LET and submitted to PANCAP as a deliverable for the Global Fund Round 9 Grant upon approval from the NAC. Further dissemination of the final report will be undertaken by the LET.

CARPHA will provide follow-up support to implement recommendations adopted by the NHAC.

Barbados HIV/AIDS NSP EVALUATION MATRIX

EVALUATION QUESTIONS:

1. Do the goals and strategic objectives of the NSP address the needs of key populations?

2. To what extent have the strategic objectives been achieved through programme implementation?

3. Have the programme objectives of the national response been achieved, if not why?

4. How have resources been utilized in the implementation of the NSP?