Programme Area 5: Surveillance, Monitoring, Evaluation and Research

Programme Area 5: Surveillance, Monitoring, Evaluation and Research

While some progress has been made over the NSP period to strengthen the collation and use of strategic information, more effort is required to ensure that M&E is fully operationalised and institutionalised, to ensure evidence-based decision making at all levels. Key areas for strengthening include the development of a mechanism for the periodic dissemination of data and information to stakeholders based on their programme needs, capacity-building of the M&E sub-committee to provide technical support to the M&E Unit, improvements in the standardized reporting and monitoring templates, and the conduct of routine reviews of HIV programme interventions using standardized guidelines.

1. 
   Is data being used to inform decision-making?
   

In pursuit of its M&E function, the NHAC developed an M&E Framework in parallel to the NSP with the intent of ensuring that information collected was relevant to local context and persons working on the ground, and thus provide a framework for evidence-based planning. The M&E Framework provides direction for the collection, analysis and use of data for the indicators identified in the Framework. For each indicator, a data collection strategy and the strengths and limitations were stated. The M&E Framework also provides guidance for national, regional and international reporting, inclusive of performance indicators for the M&E System. This M&E Framework was also revisited in 2011 and updated. Over the NSP period, the majority of the performance indicators were being reported with the exception of stigma and discrimination indicators, which remains a challenge. The implementation of the National Research Agenda will help to strengthen the collection of targeted data on key populations that will also facilitate indicator reporting.

However, despite considerable effort going into the design and review of the M&E Framework, the NHAC has been challenged in implementing the M&E system envisaged in the NSP. Key challenges have included a lack of staff and capacity and weak collaborations with the MOH.

In exploring the use of M&E data to inform decision making, the evaluation team noted that the ‘Report on 2008 Monitoring and Evaluation Needs Assessment’ undertaken by a consultant on behalf of UNAIDS, reported that 22% of stakeholders reported that they used monitoring data for annual planning and programme improvement. While a 12 Component Assessment of the Barbados National M&E System, completed by an M&E Consultant in 2014, showed some small improvements from the 2008 Needs Assessment. Approximately 35% of stakeholders reported that they use monitoring data (see Figure 9) suggesting a 59% increase over the NSP period.

An example of data being used to inform decision-making, was the development of a ‘Crop Over Intervention’ based on thematic HIV messaging. Using the available data, this intervention was designed to focus on alcohol use and exposure to HIV among festival participants. Another partner used data to inform decision-making by conducting an evaluation to determine the best month and time to conduct AIDS awareness week activities and focus groups. This information was later used to inform the development of a grant proposal.
Figure 9: 12 Component Assessment Dashboard

Source: 12 Component Assessment Report 2014

2. 
   Is data available to inform programme development and implementation?
   

• 
  % of institutions that have incorporated M&E components into their work plans (public, private, civil society)
  
• 
  % of line ministries submitting HIV work plans in accordance with national guidelines
  
• 
  # of implementing partners who report annually on program results
  

The NHAC has made notable effort to design and standardize data collection forms to facilitate the routine collection of data over the NSP period. Stakeholders utilize the HIV/AIDS Progress Reporting Forms designed by the NHAC to report on activities conducted on a quarterly basis. However, the consistency of reporting remains a challenge. The current reporting template allows for the collection of data on target group demographics, activity goals, objectives, number of persons reached, outcome achieved, and challenges encountered. While this data is important for reporting requirements, more could be done to leverage this system to provide data against indicators in the M&E Framework to generate more readily available information that could be used to inform decision making by partners and the NHAC.

Data for national, regional and international reporting is collected and reported based on donor reporting requirements and timelines. The 2014 GARPR is one such report. A review of the report shows that data was collected for the majority of the targets, however, stigma and discrimination, sero-prevalence, and behavioural data on key populations (MSM & CSWs) was not readily available.

The MOH is a key partner in the generation and use of strategic information. The MOH collects and reports surveillance data systematically using a case-based surveillance system to monitor changes and track trends in the epidemic. This system and data appear to be quite robust. However, the evaluation team found that that there was a clear disconnect between the NHAC and the MOH in the sharing and use of data. Although data collection tools for reporting were standardized, and the NHAC collected data, this was done largely in an ad-hoc manner. There is need for a more systematic and structured mechanism to monitor and evaluate the implementation of the NSP and to better understand the driving factors of the epidemic, including the factors driving the prevalence among males. This can only occur if the MOH and NHAC are collaborating effectively.

Data dissemination and use for programme planning, implementation, improvement, and policy formulation needs further strengthening the national response. Data is being collected, and reports are being generated at the NHAC and by individual agencies, but there is a little communication of the information or systematic process to promote the sharing of data. The National HIV & AIDS Draft Communication Plan 2009 was developed through a national initiative to ensure the dissemination of timely and accurate information for persons to make informed decisions, however, the plan has not been approved for implementation.

Some partners shared that data use is still an area that needs significant improvement and advocacy. However, there were a number of challenges to achieving this. One key informant noted ‘reports are not being read, it may be the presentation of the data… some people may not want to read an entire report’. Another stakeholder said ‘more research is needed to inform programmatic decisions’. While another stated ‘the real problem is that there is no funding to direct the research although it has been prioritized’.

In terms of research, the evaluation team found that there is a functional Research Sub-Committee led by the Assistant Director of the NHAC. It met on a regular basis. However, there are varying perceptions of its effectiveness. Of particular concern for the evaluation team was the fact that the MOH was not represented on the Research Sub-Committee, given their role in generating research and managing the health information system.

The role of the Research Sub-Committee is to coordinate HIV Research, and in doing so, the Sub-Committee developed an HIV Research Agenda. The Research Agenda aims to identify national research priorities, increase knowledge on the drivers of the epidemic, to support the conduct of research and information sharing of evidenced based research. However, several stakeholders were unaware of the existence of the Research Agenda.

The Research Sub-Committee also coordinates a Research Symposium every two years, as well as a Virtual HIV Research Unit. The objective of the research symposium is for all stakeholders to, present the most recent research findings on HIV-related research, provide a platform for strengthening the response to HIV and bring renewed awareness of the impact of HIV/AIDS. The Virtual HIV Research Unit is an online database of HIV-related research for Barbados and the Caribbean. It is designed to ensure access to researchers, students, officials and civil society groups’ up-to-date research and the work of the Research Sub-Committee. However, uptake and use of the system appears limited, with several stakeholders indicating that they were not aware of it.

3. 
   Is there capacity to implement a strategic information system?
   

In the face of these challenges, the evaluation team commends the commitment and dedication shown by the NHAC in attempting to further M&E. Several capacity building initiatives aimed at equipping stakeholders with the skills and competencies needed to support the implementation of a strategic information system were facilitated by NHAC staff over the period of review. All stakeholders, including HIV/AIDS Coordinators, indicated that they had received this training, with some stakeholders mentioning that they had benefited from multiple trainings over the duration of the NSP period. A mentorship programme was another initiative in the pipeline, but this was not implemented. However, it is the view of the evaluation team that much more is required to strengthen the capacity of key stakeholders. This work will need to start at the NHAC. The Final M&E System Assessment 2014 supports this position, indicating that there are insufficient M&E personnel and no structure for building M&E human capacity.

4. 
   What can be done to strengthen strategic information across the national response?
   

1. 
   Determine the barriers to implementing the recommendations of the previous assessments (MTR, M&E Assessment 2014) and develop an action plan to address the strategic information components.
   
2. 
   Ensure that the information products developed are shared with the wider stakeholders, including facility level and not only with stakeholders who provided data.
   
3. 
   Review the information products to ensure that the information is communicated and distributed based on targeted stakeholders. This will improve the chances that the information will be reviewed and used for decision-making.
   
4. 
   Review, analyze and summarize the results of surveys and studies in a reader friendly way and conduct a discussion forum in which stakeholders can discuss the results and its possible use to inform programmatic decision-making.
   
5. 
   While data is collected routinely, the data collection form needs to be reviewed and updated to capture additional data that will help the NHAC to quickly assess the outcomes of the interventions and the various programmes being implemented to assure effectiveness. Develop an electronic reporting system that will allow for the easy collation of the data being submitted. This will improve the monitoring process and allow for more timely feedback to stakeholders on both agency and national progress.
   
6. 
   A forum needs to be created to facilitate the review and analysis of existing data (surveillance, programmatic, seroprevalence and behavioural) to give a holistic view of the epidemic and create a better understanding of the changes. The re-establishment of the M&E Technical Working Group will facilitate this much-needed forum, promote accountability, provided leadership, guidance, and support.
   
7. 
   Review and update the M&E TWG terms of reference prior to re-establishment to ensure it comprises of key representatives skilled in M&E, including a representative from the MOH and the University.
   
8. 
   Review the organizational structure of the NHAC to ensure there is adequate staff within the NHAC to support the implementation of a fully functional M&E system. This includes a review of the current staff complement and the conduct of a job analysis assessment to enhance the validity and utility of its human resources.
   
9. 
   Support stakeholders M&E initiatives through regular supportive supervision, on the job training and the implementation of the mentorship programme.
   
10. 
    Resource and implement the training plan developed based on the stakeholder needs assessment.