Chapter 4: A New Holistic Evaluation Framework

Chapter 4: A New Holistic Evaluation Framework

12Towards the Formation of a Holistic Framework for the Evaluation of an Investment on the Medical Web Interventions

There are very few Cost-Benefit Analysis studies from the healthcare system’s provider’s perspective. There are only studies assessing the costs of these services (Munro et al, 2001). In order to conduct an economic evaluation, a framework that can be used as a guide throughout the evaluation is needed. Past studies on economic evaluation frameworks, frameworks on Health Information System (HIS) evaluation, Frameworks on general medical technological interventions as well as frameworks regarding the evaluation of medical web-based services are presented and analysed below in order to end up to the formation of a holistic, comprehensive and suitable framework for the evaluation of medical web-based services.

Estrem (2003), pinpoints that evaluation is a crucial aspect of web-based services/applications and is needed to determine the impact of web-based service innovation on the organization, even before this innovation is to be applied. The difficult part according to him, is to identify and quantify the costs of developing and applying the medical web-based services, since, expect of the cost to acquire the service, an organization must also form internal processes and applications to support the web-based service and also help their integration in the organizations culture processes and functionality.

Moreover, Daghfous and White (1994), suggest that technological innovation should be evaluated by taking into consideration 4 special characteristics/ dimensions:

• 
  Inventive concept. This refers to the constraints that can be solved by the implementation of the web-based service innovation as well as the new constraints that may appear due to this implementation.
  
• 
  Embodiment merit. This examines the relationship to the resources needed to deploy and fully integrate the new innovation.
  
• 
  Operational practice. This term emphasises the focus on the improvements of the new innovation in the operational practice of the organization.
  
• 
  Market dynamics. This term refers to the relation between how a new web-based service improves effectiveness and user satisfaction and how it changes the sector structure.
  

Figure 9 presents the steps of an economic evaluation according to Stone (2005). Stone (2005) takes the view that in order to frame an economic evaluation, researchers, analysts and decision makers should take into consideration the perspective of the analysis and its main aims, the boundaries and the time limitations. The boundaries refer to the scope and the aims for which the economic evaluation process is undertaken. Moreover Stone (2005) proposes that sensitivity analysis and cost-effectiveness acceptability curves can be used to determine to what degree uncertainty will influence the results of the economic evaluation process.

Figure : The steps of an economic evaluation according to Stone (2005)

• 
  The evaluation purpose must be clear, focusing on the purpose of the evaluation and formed before the developments of an evaluation plan
  
• 
  The feasibility of the examination of the impacts and consequences of the evaluation programme as well as the feasibility of the application of the evaluation method must be examined.
  
• 
  Programme objectives must be also formed to drive the evaluation process towards the correct direction.
  
• 
  Researchers and analysts must also take in to consideration the various sources for the data they need to apply an evaluation process and the timing of data collection.
  
• 
  An issue that is overlooked in most evaluation processes is to find ways to isolate the effects of the target under evaluation
  

In addition to the above, Kazanjian et.al (2002) suggests that decisions about healthcare IT should be taken after thorough consideration from the decision maker of all the available information about effectiveness, costs and ethical, political and legal effects that these technologies will have on healthcare. According to him:

• 
  Patients want to increase their access in healthcare information and in general in healthcare but without worsening healthcare standards
  
• 
  Providers have an interest in how new medical technologies as tele-health and new HIS will improve efficiency
  
• 
  Third party payers want to know the resource allocation for the healthcare technology implementation and improve accountability
  
• 
  Finally, Technology producers want to improve their market position and overcome competition.
  

Dixon (1999) presented the Information Technology Adoption Model (ITAM) (Figure 10), which is the product of a combination of various theoretical models. This framework, focuses more on the human factor and more specifically on the users of the new technological innovations implemented in the IT field, attempting to predict the final decision that will lead to the adoption of the innovation, and its implementation or not. Except from the prediction feature, ITAM can highlight areas where further evaluation should be implemented and new strategies can be applied. The most important aspect of this framework is that it incorporates the users’ perspectives in the evaluation process and decision making, showing clearly that the development of IT innovations in general and in the case of this research of web applications/services should be user-oriented and user-driven.

Table : Framework for health technolody decisions (Kazanjian et al., 2002)

Phase 1 refers to the design of a prototype “test” system in order to assess its functionality and what other components and resources may be needed for its better application. Phase 2 refers to the evaluation of the prototype, while Phase 3 refers to the evaluation of the system after it is fully installed or during its mature period. Finally, Phase 4 refers to the need to re-evaluate the system in reasonable time intervals with a periodic character.

Pulliam and Phillips, present the building of the ROI process as a puzzle where the main pieces of the puzzle are the evaluation framework, the model of the process, the operating standards and philosophy of the organization that applies the ROI process, and the practice of ROI process all connected around the centre piece which is the implementation. Every each one of these pieces is needed for so the puzzle to be complete and the application of ROI methodology to be successful and produce useful results.

Figure 13 represents the ROI methodology (Pulliam and Phillips, 2000) with its various levels, in which in order to evaluate complicated processes, these processes are broken down into simpler steps. Level 0 of the evaluation methodology is the development of the objectives of the evaluation process. The main purpose of this level is to answer the questions below:

• 
  What steps should be taken toward the implementation of the evaluation process and how many people are involved in the process?
  
• 
  How much time has been spent on the programme and what has the programme cost to the organization.
  

Figure : The ROI Process Model (according to Patricia and Jack Phillips, 2000)

Figure : Evaluation Aspects

In Table 5, the HOT-fit evaluation framework (Mohd. Yusof’s et al, 2008) is presented for evaluation of Healthcare Information Systems. HOT- fit incorporates human technological and organizational characteristics. For the evaluation of medical web-based services, a similar framework can be used that could incorporate the same evaluation aspects and characteristics.

Table : HOT-fit framework evaluation aspects and targets Cost-Benefit Analysis

Schaupp et al (2009) used Delone’s and McLean’s (2003) well-known framework for IS interventions success (Figure 14) and transform edit into a framework to assess the website’s user satisfaction based again on qualitative aspects and not providing website comparisons (Figure 15). In their work, the website satisfaction is considered the most crucial factor for the success of a website.

Figure : Framework for IS interventions success (Schaupp et al., 2009)

On the other hand according to NetGenesis/SPSS Inc. (2003), the websites require non-traditional metrics, or e-metrics to be used for their evaluation and success assessment. Patton (2002) identified various Web metrics for evaluation of business-to-consumer (B2C) website activity (net dollar per visitor, clickstream, customer drop-off rates), content sites (loyalty index, customer satisfaction), and business-to business (B2B) websites/ website activities (site performance, user efficiency, average time spent on system). Finally, Plaza (2011) presents a quantitative methodology for measuring websites performance from the owner’s point of view using regression models to assess the impact of various factors like the number of unique visits and the number of return visits. Although Plaza (2011) work utilised quantitative metrics to access the success of a medical website, it was focused only on one website due to the limitations of the Google Analytics tool that was used for data harvesting.

4. The Developed Holistic Evaluation Framework

The aspects that each of the aforementioned frameworks examined where categorised and the most relevant aspects according to the targets of the current research were selected and added in the framework

The first level of the evaluation is to plan the evaluation process by identifying and focussing its main objectives, which in this case are the economic evaluation and efficiency assessment of a medical web intervention. At this level, the scope of the evaluation must also be decided. The actors involved in the evaluation of medical web-based services have different points of view and expect different results from the evaluation process. This helps the analysts/ decision makers to form a comprehensive stakeholder analysis, by identifying the key stakeholders involved in the medical web intervention, by analysing their interests and motivation, and the limitations that the developers of the medical web-based services have to address and overcome in order to satisfy the key stakeholders. Moreover, “Influence vs. Interest” diagrams can help the decision makers to understand more clearly which actors are of high influence to and interest in the medical web intervention, so that they can focus more on them. Furthermore, “Influence vs. Interest” diagrams among the key stakeholders are useful in order for the decision makers to identify the interrelationships among them.

The above-described evaluation process aims to enhance the efficiency of web medical services/ applications, and assess the return on investment and the net profits that these services/ applications can offer to their provider. In addition to this, the users of such applications demand enhanced reliability, accurate and comprehensive information, and ease of use.

The second level of the proposed evaluation framework refers to the focus that is needed on the main aspects of the evaluation target under examination and its specific characteristics. In order to evaluate medical web-based services, the technological, human and organizational aspects must all be taken into consideration in order to produce accurate and valid evaluation results with high level of confidence as well as generalizability (if and when needed).

The Technological aspects refer mainly to the system quality (data accuracy, database depth of content, reliability of the functionality of the service, security and response time), information quality (importance, relevance, usefulness, accuracy, completeness, reliability of information) and service quality (responsiveness, support, effectiveness). Human factors refer to system usage by its users (purpose of use, expectations, attitude towards the information or service received, level of acceptance of the information) and user satisfaction (usefulness of the information received and overall satisfaction). Finally, the Organizational aspects that should be taken into consideration are the structure of the organization (nature of the organization, medical sponsorship and support) and the environment around the organization (population served, user base/community).

Report Data and assess the impact

Reporting
Estimate present value

Separate Costs and Benefits

Identify Intangibles

Data analysis

Convert Data to monetary units (QALYS, WTP approaches)

Use intangibles that cannot be quantified to assess broader value, survivability/ sustainability

Structure

Environment

Technology

System Quality

Information Quality

Service Quality

Form the objective

Decide the perspective and identify and analyse the key stakeholders involved

Healthcare System

Service providers

Plan Evaluation

Human

System use

Data collection

Use tools to get data from user’s comments, use databases, interview/questionnaires, annual reports, NHS evaluations

Data Collection

Calculate Benefit to Costs ratio and Break Even point

Economic Evaluation

Discuss the aspects and targets of evaluation


Figure : Evaluation Framework for Medical Web-based service

In addition to the above, intangible data should be also identified and attempts should be made to express them in monetary units. Various methods will be used in order to monetize intangible data depending on the specific characteristics of its intangibles. For example, in case of web medical applications, if a value is needed to represent the value of human life, then the Sugden and Williams (1978) approach can be used. They propose that a more straightforward approach to value human life is to identify situations where individuals have a choice between ‘safety’ and ‘extra monetary gain’ and the value they attribute to ‘safety’, and in this way an analyst can derive the minimum value of life, as equal to the monetary gain an individual is asking for, in order to fully compensate against living/working/interacting in a completely unsafe environment. Alternative methods like the Willingness to Pay approach can be used to monetise the intangible effects of an intervention.

The intangibles that cannot be transformed into monetary units can be used to assess the broader value that the medical web-based services may have and also to assess the future potential and survivability of these services using qualitative methods.

The estimation of the present value and the discounting of the future value of the costs and benefits involved in the development of IS interventions are vital, if the data are referring to past economic periods/years or a future projection of the evaluation results is required. Estimation of the future value is important for a complete and useful interpretation of the results. Robert Sugden and Alan Williams (1978) propose a solution to the problem of which rate is more appropriate to be used in the discounting process. One approach they propose is to seek for consistency in the criteria used to evaluate projects in both the public and the private sector of the economy. Another approach is the time preference approach which refers to the existing difference of relative valuation placed on a good at an earlier period of time compared with its valuation at a later period of time and the discounting that is required in order to make the evaluation results at the two periods easily comparable to each other (Moseley, 2001). Finally, they suggest that the best solution would be a synthesis of the two methods.