Economic Evaluation of an Investment in Medical Websites and Medical Web-Based Services

Research Methodology

• 
  The investigation of the emerging market of the medical websites for factors that can influence their success, survivability and sustainability.
  
• 
  The development of a holistic framework for the evaluation of an investment on the medical websites, based on previous well-established frameworks in the field of medical wed applications, healthcare technological interventions and healthcare information systems,
  
• 
  The application of the evaluation methods (Cost-Benefit Analysis, Willingness to Pay and Return on Investment)
  
• 
  The assessment of the final results.
  

1. Investigation in Success and Failure Factors for the Market Of Medical Web-Based Services

The first step to achieve the aims of this research was the formation of a sample of the population of the Anglophone medical websites, belonging into different categories, according to their content, and having as targeted audience people with different needs and/or conditions.

An initial sample of about 500 medical websites was obtaining using:

• 
  Keywords like “medical websites”, “symptoms and conditions”, “online doctor”, “medical resources”, “symptom checker”, “online medical appointment”, “ask a doctor online” and “drugs and interactions” in various search engines (Google.com, Yahoo.com, Ask.com).
  
• 
  Past qualitative and quantitative studies that used popular medical websites.
  
• 
  Lists of medical websites that were sourced from the recommendation or affiliation sections of other medical websites.
  

Each medical website in the sample was visited by the researcher and data were collected, including variables such as the traffic rank of the websites, the interactive services they offer to user/patients or medical practitioners, the nature of the organization that supports them, how long they have been/were online, their activity in social networks, the web presence of any kind of certification awarded from well-known organizations, the number of unique visits that they have received as well as the number of the linked websites to them. Following the sample formation, statistical methods were implemented to examine the basic characteristic of the sample in order to extract useful knowledge and concluding remarks regarding the overall population of the Anglophone medical websites. The various variables were then coded and analysed with the use of statistical methods and more specifically linear regression models.

In order to identify the factors of success and failure of the medical websites, a feature selection process was implemented. The sample’s medical websites were firstly ranked according to their global rank, and the characteristics of the higher and the lower ranked medical websites were compared in order to identify the characteristic/features that present high variance among them and can be considered as factors that affect the grade of excellence and the sustainability/survivability of the medical web websites. The impact of these factors were assessed then, using statistical methods and more specifically regression models. The websites were ranked then once again, according to the factors that will have greater influence on their success. Two taxonomies were formed, one with the “highest ranked” and the other with the “lowest ranked websites” (Figure 3). The same feature selection process and the same methodological steps followed to form a regression model were used also in order to select and examine the impact that some variables can have on the frequency with which the websites appear in a search engine, on the time that the users tend to spend on the medical website and on the offering of interactive medical web-based services, as other factors of excellence. Finally, after the factors of excellence were assessed, research was conducted to assess also the factors that enhance the survivability of the medical websites based on the above methodological structure.

Global Rank

Lower ranked websites

Global Rank

Lower ranked websites

Higher ranked websites

Comparison

Step 3: Choose the factor that present greater difference and run a linear regression model to measure their impact

Step 4: Take the significant variables from the regression outcome, rank the websites according to them and choose the 100 “highest ranked” and 100 “lowest ranked” websites from its rank. From its new lists choose those that appear in higher frequency among the lists.

Significant variable 1

Significant variable 2

Significant variable 3

100 “lowest ranked” websites

100 “Highest ranked”

Websites
100 “lowest ranked” websites

100 “Highest ranked”

Websites

100 “Highest ranked”

Websites


LRM

• 
  Website status
  
• 
  Percentage of visits from search engines
  
• 
  Number of linked websites
  
• 
  Nature of the organization that supports the website
  
• 
  Category of the website
  
• 
  Average Reputation
  
• 
  Source of income
  
• 
  Time on website
  
• 
  Other factors
  

Comparison

“Best of the best”

“Worst of the worst”

The variables that formed the final dataset were chosen according to the rating/evaluation variables that were used in the extant literature for the evaluation of the medical websites and according to the scope of the current research. By investigating the well-established body of literature, we identified the need to include variables concerning the certification of the websites, their loading time, the offering of interactive services and their ranking estimated based on various measurements like their traffic flow on specific time interval (daily, monthly, weekly) and their reputation among the Internet users. These variables were considered core to the success and survivability of medical websites, since they can affect the user experience, the traffic, the popularity and in general the excellence of the medical websites in order to produce valid and comprehensive research results. In contrast to the aims of the Aforementioned past researches, the purpose of this research is to examine the user experience or the accuracy of the medical information provided on the World Wide Web. Instead this research aims to examine the factors of success and the factors that lead to or enhance the survivability/and sustainability of medical websites from a business-oriented point of view, rather than focusing on subjective assessments of the quality of the information they provide.

Moreover, variables were formed to represent the current status of the medical websites (online/offline), the offering of interactive medical applications from the medical websites, the category in which the website belongs to according to its content, and the nature of the organization that provides and/or supports the website. Furthermore, variables were formed to reflect also the activity of the medical websites in social networks, the offering of blogs/forums and other forms of online communities, the disclosure of any kind of financial information about the organization supporting them, evidence of any kind of certification awarded to the medical website from well-known organizations, and finally the actors (patient/user, stakeholders, providers) for whom the offered information and services by the medical websites have value.

Furthermore, by implementing a search strategy using syntax commands on the Google search engine, it was possible to retrieve the number of websites of any kind that are/were linked to a given medical website in the sample (target website), or include links of the target website in their content or list of affiliations, or even cite content of the target website/websites. In addition to the above, using information from Alexa the Web Information Company (http://www.alexa.com/, accessed on 10/1/2012), it was possible to collect data about the average three-month global traffic rank of the medical websites, their reputation among the Alexa-tools’ users, the country from which the website has the more visits, the time that the users spent on the website and the loading time of the website and the percentage of the websites that are faster or have the loading same speed with the target website, the websites that the users visit before and after the target website, as well as the fraction of the total Internet users that have visited the website (three-month average) and the fraction of the visits to the medical website referred to from search engines. In addition to this, by using the Web Rank SEO (Search Engine Optimization) tool provided as add-ons by Google for the Google Chrome Web Browser, it was possible to enhance the accuracy of the estimation of the lifespan that the medical websites were online, previously estimated using WayBack machine, and also estimate the Google Rank and the “Facebook Likes” for the medical websites in the sample.

Additionally, by using also Compete.com, the website analytics, the researcher was able to estimate the number of unique visits to each medical website in the form of one– month average visits and only for the USA region. Comparing the data retrieved from Compete.com to those from, similar tools like the Google Ad Planner’s unique visit data, the data from Compete.com were not rounded up, and in many cases were far more accurate, so were considered more reliable to be used as a variable’s data in the statistical analysis. It is important to mention that it was not possible to find the unique visits from the global Internet users, but only for the users based in the USA. Since the majority of the users of the medical websites in the sample were from the USA, we consider that this will not cause any significant loss of validity of the models that are applied in this research.

• 
  There are sufficient events per variable. This refers to the fact that there should be enough variables, so the model to fits well to the data. Moreover, a rule of thumb is proposed for the Logistic regression. This rules expression is the total of the least possible outcomes divided by the number of the predictors to have a result equal or greater to ten.
  
• 
  Conformity with linear gradient for continuous variables. Extant literature suggests that any change in a continuous independent variable used in the model “should have an effect on the log-odds of a positive outcome that is of the same magnitude” regardless the value of the predictor variable.
  
• 
  Test for interactions. Knowledge of the domain of the study can help analysts/researchers to identify interactions between two variables that, when included, can influence the model, and their significance should be measured and reported.
  
• 
  Test for collinearity. If there are variables that are highly correlated, this can affect the precision of the estimation of their contributors in the model and cause the variance of these variables to be inflated.
  
• 
  Finally the validation of the model.
  

• 
  Normality of the dependent variable. The problem of the non-normality of the independent variables can be corrected by applying the linearity assumption.
  
• 
  Linear relationship between the dependent variable and the independent. If there is no linear relationship between the dependent and the independent variables, then the results of the model will have very large errors the will make the model invalid.
  
• 
  There must be no serial correlation (independence) concerning the errors. Large serial correlation of the errors means that the model needs lot of improvement and is mis-specified.
  
• 
  Homoscedasticity of the errors. If homoscedasticity checks are not taken into consideration, this can make the estimation of the standard deviation of the errors resulting in very narrow or very wide confidence intervals.
  

2. Cost-Benefit Analysis and ROI

Before the final comparison of the costs and the benefits that arise from the operation activities of the medical websites, a stakeholder analysis was performed in order to obtain a deeper understanding on the main/key actors that affect and are affected from the medical website’s operation.

The first step included the identification of the key stakeholders and the comprehensive analysis of their profiles. The key stakeholders that were identified were:

• 
  The Government/ Ministries of Health
  
• 
  Private Hospitals and Hospital networks
  
• 
  Individual medical practitioners and Medical Practitioners Associations
  
• 
  Non-for-profit Organizations
  
• 
  Patients’ Associations and Individual patients/users
  
• 
  Web-content providers and other relevant stakeholders
  
• 
  E-publishers
  

Identification of the “benefits” using a WTP approach

In order to identify the benefit (value) of an investment in medical websites to compare it with its cost for conducting the Cost-benefit Analysis, a Willingness to Pay approach in an online survey setting was employed. This survey had two aims: a) to identify the value (benefit) that the users perceive having from the use of services offered by a medical website and b) to identify their willingness to pay towards the services offered from medical websites in order to identify the value (profit) that an investment on a medical website might have.

Survey Formation

An online survey setting was used in order to conduct the study. Various studies have shown that this kind of survey for accessing the WTP of the participants can produce results equivalent to those of face- to-face interviews and has many advantages compared to more traditional survey methods (Nielsen, 2011; Dillman, 2007; Iraguen et al., 2004). It can access only the sample of the population that use the Internet, which is actually the group of participants that this study seeks to address, in order to produce valid results (Nielsen, 2011; Dillman, 2007; Iraguen et al., 2004).

Prospective participants were selected purposively and on voluntary basis. The online survey was circulated via email to various mailing lists of organizations, such as universities, non-profit health organizations, and via social networks such as Facebook and Twitter using novel crowd-sourcing methodology/technique and research tools. The participants were informed about the voluntary character of the survey and their anonymity was ensured. In the first pages of the survey, the purpose of the study was explained to the participants as well as a statement that will informed them that their anonymity is ensured. They were informed also that if they continue to complete the survey, then they give their agreement to take part to it. (Appendix B).

The first section of the survey included questions related to age, sex, and occupation, level of education and level of income of the participants.

Initial survey text for the participants

Welcome to this survey conducted by Department of Information Systems and Computing of Brunel University. This survey aims to investigate the Willingness to Pay of current and potential users of medical web sites toward the use of various medical websites and online web applications (e.g. ask a doctor online, use a symptom checker, use a drug identifier etc.) that the websites offer to them.

This survey is part of a research project that is conducted in Brunel University London and is related to the economic evaluation of technological initiatives in healthcare. The survey is anonymous and your participation is entirely voluntary. Cookies and personal data stored by your Web browser, are not used in this survey. It takes around 10 minutes to complete. The data of this survey will be used only for academic/scientific and not for commercial purposes.

By completing the survey you can take part in a competition giving away an iPad Mini to a lucky winner. The participation to this competition is voluntary and the personal data required (full name & email address) will be used only for the purposes of the competition, not for commercial purposes and will be deleted immediately after the competition ends (30/4/2013).

By clicking the Continue button you agree in taking part to this survey under the aforementioned terms and conditions.

If you have any concerns or complaints regarding the ethical elements of this project, please contact siscm-srec@brunel.ac.uk or Professor Zidong Wang on Tel. No. 01895 266021.

The second part of the survey included questions related to whether the participants have ever used a medical website to seek medical advice and if they had used any web medical application (e.g. ask a doctor online, find a doctor, an online symptom or drug identifier etc.). Moreover, this part included questions related to whether they are willing to use medical web-based services such as these in the future, and how much money they might be willing to pay in case these applications are not offered without any monetary cost. The participants were asked also to judge their decision thinking carefully about the time they save from not having to visit consultancy rooms, the cost to go to the GP for minor issues, the quick search for alternative diagnoses, the better results that they might have from the online communication, instead of the traditional telephone communication with their doctor, by using these online applications.
Finally in the third part of the survey, questions are aimed to assess how confident the participants are about their decision to spend the amount of money they have stated for the use of a web medical application and a question about the difficulty they had met in answering the questions of the survey.

The questions of the second part regarding the willingness to pay of the participants were offered in a combination of (closed form) discrete-choice and open-end form of questions, since the participants were asked to choose one of a set of predefined monetary values, but also there was an option asking if they want to state another value. By using this structure of questions, we can overcome the difficulty of the participants to answer open-ended questions regarding their willingness to pay and also eliminate the bound bias of discrete-choice form (Frew et al., 2004, Ryan et al., 2004).

In the last part of the survey, the question about the participant’s certainty/confidence about the use of the services if provided in the prices that as they stated before that they would prefer, is a method to solve the hypothetical bias issue in WTP surveys by distinguishing between the True and False Yes answers and taking the definitely sure Yes answers as the true Yes. (Johannesson et al., 1998; Bluumeschein, 1998).

After the completion of the survey, statistical methods will be used to analyse the data and derive the maximum average WTP of the participants in order to use it to complete the overall research purposes.

Formation of WTP questions

The questions of section 2 regarding the Willingness to Pay of the participants were a combination of (closed form) discrete-choice and open-ended form of questions, since the participants were be asked to choose one of a set of predefined monetary values the option that represents better their maximum WTP but also there was an option if they want to state any other value in case they don’t find the discreet choices representative enough. By using this structure of questions, we can overcome the difficulty of the participants to answer open-ended questions regarding their Willingness to Pay and also the bound bias of discrete-choice form can be eliminated (Frew et al., 2004, Ryan et al., 2004).

According to Ryan et al. (1997), there are four main methods used in a contingent valuation. The first is the bidding method, which is the first method developed and its main use was in Internet services. The participants in this setting are asked if they accept a specific initial value for a commodity/service and depending to their answer the price is increasing or decreasing in order for the actor of the evaluation to determine their maximum WTP (Ryan et al., 1997). This method is prone to the starting-point bias, meaning that the valuation is heavily influenced by the first/initial value that is proposed to the participants (Frew et al., 2004). A variation of this, is the payment card method in which the participants are asked to choose from a range of bids which one presents better their own preference with the actual amount to be between their choice and the next bigger bid (Ryan et al., 1997; Mitchell and Carson, 1989).

The third method is the close-ended method, which presents some similarities to the bidding method but here the participants sample is separated into subsamples and in its one is presented a different value for the commodity/service under evaluation (Ryan et al., 1997; Mitchell and Carson, 1989; Bala et.al, 1998). Then the participants are asked if they accept that value or not. In a more recent variation of this method, two dichotomous questions are asked to the participants in order to define also the level of certainty of their answer. This technique has the advantage that the everyday preferences of the public are reflected better, but the estimation of the maximum WTP can be proved particularly difficult or tricky and require the plotting of the data as a demand curve with axis the bid level and the probability of saying yes or using a Radom Utility Theory model (RUT) (Ryan et al., 1997).

The fourth method is the open-ended question setting, in which the researcher asks directly the participants to state their WTP without providing them any intervals or discreet choices. The actual WTP is estimated by the sample mean and, in general, this method does not require a large sample (Ryan et al., 1997; Bala et.al, 1998). The disadvantage of this technique is that participants in such surveys often have a difficulty to answer and avoid answering since there are no clues to guide them to plausible values (Frew et al., 2004). Comparing the bidding game method to the open-ended question format, Frew et al. (2004) concluded that the former produces higher average WTP but this might be a result of starting-point bias. Finally, payment scale or card-based method can be considered as a variation of open-ended method but the scale makes answering easier (Frew et al., 2004). The disadvantage of this method the bounds of the scale might influence the valuation judgement of the participants (Frew et al., 2004, Ryan et al., 2004).

Finally, a question was used to investigate the level of certainty of the participants for his value and taking the “very sure/ sure” Yes answers as the true Yes. This aimed to mirror better the real behaviour of the participants/users, solving the hypothetical bias by distinguishing between the true and false yes answers (Johannesson et al., 1998; Bluumeschein, 1998).

Survey Spreading and Crowd-sourcing methodology

BOS platform is an easy-to-use Web-based application that allows you to create and run online surveys, analyse the results and transfer the data to other software packages. The following national surveys have been run using BOS (BOS website, 20/5/2013):

• 
  Careers in Research Online Survey (CROS) is running on annual basis since 2001. CROS’s audience is researchers in Higher Education and over 60 UK universities have participated in CROS.
  
• 
  Athena Survey of Science Engineering and Technology (ASSET) is running on annual basis since 2002. It seeks to understand the career experiences, perceptions and ambitions of all career scientists.
  
• 
  The Postgraduate Research Experience Survey (PRES) is run by the Higher Education Academy to allow participating institutions to collect feedback from their postgraduate research students on their experiences of their research degree programmes. In 2009 more than 18,644 students completed PRES through the BOS platform.
  
• 
  The Postgraduate Taught Experience Survey (PTES) is run also by the Higher Education Academy to allow participating institutions to collect feedback from their taught postgraduate students on their experiences of their degree programmes. In 2010 32,638 students completed the survey using the BOS platform.
  

As it was mentioned above every possible online spreading route was utilised in order to spread the survey and attract as more as possible participants. The survey itself was created on the Bristol Online Survey platform (BOS).
The web link of the survey was posted on various social networks (Twitter, Facebook, LinkedIn) and spread also via emails using staffs’ and students’ mailing lists of four UK universities (Brunel, Swansea, Cardif, Glascow) as well as mailing lists of non-profit health organizations. The web link of the survey was spread via social networks and targeted mails first to “friends”, “followers”, colleagues and targeted Profit and non-Profit organizations that have activity and supporters in UK (Figure 4). These initial “targets” became the means to spread the survey to a large number of people belonging to the “general public”, because they are not patients or members of healthcare organizations. The main aim of these steps was to involve the participants in a “crowd engagement” activity creating a self-sustainable spread chain for the survey link.



Figure 4: General crowd-sourcing strategy
The survey spreading activities followed two successive phases:

• 
  First Phase: First of all the survey was spread to the email lists of the staff and the students of almost all the faculties of four UK universities, aiming to have a first sample of participants having diverse ethnical backgrounds living in UK as well as of diverse ages, occupation state and income level. This sample can be considered biased in terms of age groups and educational level. Moreover, the link of the survey as well as an explanatory text was sent together with the emails of organizations related to healthcare (NHS, Doctors without Borders, Patients.co.uk) with a request to forward the survey to their staff and members (Figure 5).
  

Figure : First phase of crowd-sourcing activities

• 
  Second Phase: The Second phase of the spreading campaign of the survey involved the use of social networks and, more specifically the use of LinkedIn, Facebook and Twitter. The methodology followed to spread the survey through Facebook was rather simple and based on the same assumptions and principles as the methodology followed to spread the survey via the email lists. The web link of the survey was posted on a personal LinkedIn/Facebook profile, which offers access to friends and colleagues, who in turn spread the survey to their friends and from them to others. In addition to this, the survey link was posted on the LinkedIn/Facebook profile of Organization related or not to healthcare that are active in UK (NHS Blood Donation, NHS Direct and Choices, BBC, UNICEF UK, UNESCO UK, Doctors without Borders UK, United Nations UK etc.). Especially NHS choices Facebook profile administrators were particularly helpful showing great interest in the outcomes of this study (Figure 6).
  

Figure : Second phase of crowd-sourcing activities

While the methodology followed in order to spread the survey through LinkedIn/Facebook was rather simple, that was not the case while using the Twitter for this purpose. In order to spread the survey through Twitter a novel crowd-sourcing technique/methodology was developed incorporating novel and sophisticated tools that will be described in the following paragraphs. The key tool that was used was the MATCH Tweet-catcher, a novel tool developed by the Multidisciplinary Assessment of Technology Centre for Healthcare Project (MATCH)¹.


Tweet-catcher is a unique Twitter ™ crowdsourcing application. The Tweet-catcher crowdsourcing tool allows users to administer complex Meta searches using relational algebra and predicate logic. Tweet-catcher supports Structured Query Language (SQL) style syntax including the use of AND, OR, NOT (exclusion) and parenthesis; which provisions extensive and optimal filtering techniques respective to the searching carried out. Tweet-catcher enables its users to get instant preview of the Twitter content based on the search terms which they are providing. In addition to the searching, its users can also download the searched content in excel format. Tweet-catcher begins harvesting Tweets, timelines associated with the search criteria, user handles of the Twitter user (and their given display name), friends and followers, user Tweet count, geo-coordinates, and resolved links using the Tweet itself. Tweet-catcher continues to harvest data as long as the user requires, and can be accessed throughout the harvesting process in order to check the relevancy and worth of the search being conducted¹ (Match website, accessed 20/5/2013).

For the purposes of this research, Tweet-catcher was used to search the tweets that were posted on the Twitter profiles of specific healthcare related organizations and websites (NHS choices, Patients.co.uk, Medhelp.co.uk, NHS Direct, Net Doctor, Altzheimer Society) by using their “tweet handles” (Twitter IDs) as search terms/query (e.g. “@nhsdirect OR @nhschoices OR @netdoctor OR @patientuk OR @alzheimerssoc”). This search query produced a list of the Tweets, the user Twitter names/IDs that made these tweets, their sentiment (positive or negative) towards the specific topic referred in the tweet, their geographic location as well as the number of their followers and the number of the users that they follow.
The aim of this search query was to identify users that are i.e. actively using medical websites and to shortlist those which are more actively engaged in Twitter, those with the larger number of followers and tweets in order to spread to them the survey link. Finally, the geo-coordinates accompanying the harvested data i.e. the output of the MATCH Tweet-catcher tool, helped in the identification of users that are living in UK.

Figure : Second phase of crowd-sourcing activities

As it is presented in Figure 8, the search terms were entered into the MATCH Tweet-catcher tool which interacted with Twitter social network and harvested the tweets, the user IDs and the other aforementioned variables in a file. After identifying the users with significant activity that is considered promising for the further spreading of the survey, the researcher tried to contact these users in a crowd engagement process, using personal messages using their Twitter IDs (@) and also a thread about the survey marked with # (in this case #wtpsurvey). These users engaged then in conversations on the Twitter social network and were offered guidance, in case they required further information about the purpose of the survey, and in turn with their own friends/followers spreading the survey to them either by personal messages or by posting them on their profile. The “general public” completed the survey and the results were available for processing through the BOS platform.

Figure : Analytical survey spreading activities through Twitter

Assessing the costs

The task to assess the costs related to the operations of the medical websites and the medical web that they offer to the public is a very complicated task that cat can lead to inappropriate and misleading estimations during the economic evaluation of the medical web-based services . Thus instead of estimating the costs related to the medical websites’ operations we will evaluate the whole concept of using a medical website offering medical web-based services by estimating the cost of acquiring and managing a website like this.

SEO tools add-on for Google Chrome Browser uses information from various valid sources (Ask.com, Archive.org, Alexa Rank, Baidu, Bing, Compete Rank, CoralCDN, Delicious, Digg, Dmoz, Google Page Rank, Majestic SEO, OpenSiteExplorer, Quantcast, SEMRush, SEOmoz, Linkscape, StumbleUpon, Technorati, WebCite, Yandex Quotation Index, Yahoo) and combines them in a Dashboard setting providing information about the popularity the traffic and the traffic cost of a target URL.

Using WebRank SEO add on for Google Chrome Browser we were able to calculate the cost of acquiring and managing a medical website. From the sample of the 317 websites that we had formed for the afore-presented purpose of the success and survival factors analysis, and by using the WebRank SEO tool it was possible to estimate the cost that is required for each one of the medical websites to pay in USA dollars $ in order to “buy” traffic flow equal to that they already have. This cost was considered the minimum cost that somebody should pay in monetary units in order to acquire the medical website under his own management and was the cost that will be compared with the Benefit in the following Cost Benefit analysis and Return on Investment estimation.

3. Applying Evaluation Methods

The next step in the evaluation process is the application of economic evaluation methods. Our suggestion is that Cost-Benefit Analysis (CBA) can be used in the context described by Robert Brent (2003), as a general method that will include also Cost-Efficiency Analysis, Cost-Minimisation and Cost-Utility Analysis as its special cases. Breakeven Point is the point where the costs associated with the development and implementation of the medical web-based services become equal to their benefits.

Benefits to Costs ratio	Ratio of benefits to costs
ROI/ CBA	Ratio of adjusted benefits to costs	ROI
Breakeven point	Point where benefits meet or exceed costs

Applying Cost-Benefit Analysis

National Institute for Clinical Excellence (2004) also suggests that cost effectiveness analysis is the most appropriate method to be used for healthcare evaluations because its main purpose is to compare cost differences of various options and justify them in terms of changes in healthcare intervention’s effects. The opinion that Cost-Benefit Analysis is more appropriate to be applied in healthcare field is also supported by Brent (2003).

The CBA for a program is calculated as:

Measuring ROI

According to Patricia and Jack Phillips (2008), measuring ROI is a much-debated topic. Many characterize the measurement of ROI as inappropriate while others believe that “is the only answer to accountability concerns”.

The ROI for a program is calculated as:

Stone (2005) presents a comprehensive table (Table 3) of the steps of ROI calculation methodology by setting the formulas that should be used in order. First of all estimates costs and then benefits. After the cost and benefits estimation the estimation of the net present value of past costs and future benefits in is necessary although benefits can be brought to the period the costs occurred rather than the present. Then the Benefits to Costs ratios is estimated followed by the ROI calculation. Finally, a Breakeven point is estimated which is the measure that the benefits are even or exceed costs.
Table : ROI calculation process (Stone, 2005)

Metric	Definition	Formula
Costs	Total amount of money spent on new investment
Benefits	Total amount of money gained from new investment
Net present value	Discounted benefits based on inflation
Benefits to Costs ratio	Ratio of benefits to costs
ROI	Ratio of adjusted benefits to costs
Breakeven point	Point where benefits meet or exceed costs

Patricia and Jack Phillips (2008), state that during the calculation of ROI, they state that it is important to isolate the effects of the program/investment outcome from exogenous effects that can influence the evaluation results. They propose that this can be achieved through techniques such as:

• 
  The use of a control group arrangement with a control group and a group that participates in the program.
  
• 
  The use of trend lines that project an estimation of values of some characteristics and then after the evaluation process to compare the findings
  
• 
  Participants’, supervisors’ and panel of experts’ opinions of the estimated value of impacts.
  
• 
  Finally the identification of influence factors can lead in the conclusion that the improvement that remains unexplained by these factors can be directly attributed to the program.
  

Data conversion also is an important issue. Impact data must accurately converted into monetary units in order to calculate ROI. Furthermore except of the tangible effects there are also intangible effects of the program that must be converted into monetary values. Patricia and Jack Phillips (2008) propose that this can be achieved through:

• 
  The conversion of output data to profit contribution or savings.
  
• 
  Improvements in quality usually are treated as cast savings.
  
• 
  Historical costs can be used from cost statements.
  
• 
  Participants’, supervisors’ and panel of experts’ opinions of the monetary value of data and outputs.
  

According to Jack Philips (1997), “if an organization needs ROI calculation for some courses it must to find the desired level of ROI calculations”. Some organizations using statistical sampling methods to select the number of programs they want for ROI calculation while others just evaluate the two more popular programs they have. There is not a standard benchmark for the number of programs and ROI calculations and they depend on:

• 
  The expertise, experience, discipline and commitment of the personnel involved in the evaluation process.
  
• 
  The resources that the process is allowed to consume in order to be completed.
  
• 
  The nature of the programs.
  
• 
  The nature of the organization.
  

Download 1.95 Mb.

Share with your friends: