Survey Completion Rates and Resource Use at Each Step of a Dillman-Style Multi-Modal Survey
Download 116.31 Kb.
|
- Navigate this page:
- Abstract
- Methodology
- Table 2: Timeline of Data Collection Steps
- Final Case Disposition
- Total Non-Completes
- Discussion
| Survey Completion Rates and Resource Use
at Each Step of a Dillman-Style Multi-Modal Survey Submission for Public Opinion Quarterly Authors’ Names and contact information: Andrea Hassol, Abt Associates Incorporated, e-mail: andrea_hassol@abtassoc.com Holly Harrison, Abt Associates Incorporated, e-mail: holly_harrison@abtassoc.com Brenda Rodriguez, Abt Associates Incorporated, e-mail: brenda_rodriguez@abtassoc.com Ricki Jarmon, Abt Associates Incorporated, e-mail: ricki_jarmon@abtassoc.com Austin Frakt, Abt Associates Incorporated, e-mail: frakt@bu.edu
Phone: 617-492-7100 Fax: 617-492-5219 Supported by contact #CMS-500-00-0032/TO2 AbstractConsiderable research has been conducted on methods for improving response rates to surveys. Several key factors are known to affect response rates, including: salience of the survey, survey mode, monetary incentives, and multiple contacts. A response-maximizing approach to multi-modal surveys, as best articulated by Dillman (2000), includes up to five contacts with survey recipients, stamped return envelopes, personalized correspondence and a token financial incentive. In designing data collection strategies, survey researchers must weigh available resources against expected returns. Before beginning our survey of Medicare beneficiaries, we searched the literature for information on large surveys where researchers measured the additional increment in completion rates achieved by each step in the process, as well as the costs of each sequential step. We found minimal research that tangibly demonstrated the effects on completion rates of reminder postcards, additional mailings, and the use of first class versus priority mail. This article describes the results of a Dillman-style multi-modal approach, questionnaire completion rates, and costs associated with each step in the approach. The survey was conducted as part of an evaluation of beneficiaries’ experiences with a new Medicare insurance option. This article does not describe the effects of the option; it focuses on survey methodology and completion rates rather than on the substance of the survey itself. This work validates many of the axioms of survey research, in a much-studied population, and offers survey researchers benchmarks for at each additional step in the process. Considerable research has been conducted on methods for improving response rates to surveys. Several key factors are known to affect response rates, including salience of the survey (Herberlein and Baumgartner, 1978), form of mailing and monetary incentives (Dillman, 1991), and multiple contacts (Linsky, 1975, Dillman, 1991). A response-maximizing approach to multi-modal surveys, as best articulated by Dillman (1978), includes: A respondent-friendly questionnaire; Up to five contacts with recipients of the survey (a brief pre-notice letter sent a few days prior to the arrival of the questionnaire; a questionnaire mailing that includes a detailed cover letter; a thank you/reminder postcard; a replacement questionnaire; a final contact, possibly by telephone.); Inclusion of stamped return envelopes; Personalized correspondence; and A token financial incentive. When designing cost-efficient data collection efforts, researchers must usually balance available resources against expected returns. For example, telephone follow-up can be extremely costly and might yield fewer completes than would an additional mailing of a questionnaire. To maximize response rates, we sought to utilize as many of Dillman’s recommendations as were feasible in the context of the study. The findings in this paper are from a survey funded by the Centers for Medicare and Medicaid Services (CMS). CMS specified a response rate of over 70% for the survey, using a mail survey with telephone follow-up. A literature review was conduced to identify efficient strategies for maximizing response in this type of multi-modal survey. We found no published studies that specifically demonstrated the results of the “Total Design Method” (TDM) approach made famous by Dillman (1978). Other researchers have focused on the effects of questionnaire length on response rates (Dillman, 1993; Axford, 1997); ways to increase mail response rates (Finke, 1985; House, 1977; Nederhof, 1988); and the effects of multiple mailings on external validity in mail surveys (Dalecki, 1983). A common methodology in survey research is repeated mailings with telephone follow-up, but there is a dearth of information about expected results for each step of the process. Such information could guide researchers in tailoring their survey strategies to maximize efficient use of resources for a given field period. This article seeks to fill a gap in the literature by describing the questionnaire completion rates yielded from a TDM approach over a 21-week field period. These data serve to validate many of the axioms of survey research, in a much-studied population, and offer survey researchers benchmarks for each additional step in a Dillman survey process. Although this survey was implemented in a manner consistent with Dillman’s strategy of repeated contacts, advance letters, and so on, we were unfortunately forced to violate some of Dillman’s suggestions. For example, the questionnaire, although attractive, was not as respondent-friendly as we would choose, and it had many ‘skip patterns’ that respondents were required to follow. This complexity was necessary because certain sections of the questionnaire were appropriate only for respondents with particular insurance coverage. In addition, the topic of this survey was more salient to some groups of respondents than to others, and we hypothesized that completion rates would differ accordingly. The survey was entirely voluntary (as explained in the advance letter and all cover letters), and no token financial incentives were allowed for this Federally sponsored survey. Because this survey was implemented without many of the advantages other surveys employ to enhance response rates, implementation methods were the primary tools to maximize response rates and reduce non-response bias.
The eligible sample was defined as community-dwelling Medicare beneficiaries, with valid addresses, who were either aged or disabled (but not individuals with End Stage Renal Disease). The initial sample of 7,730 was drawn from Medicare’s Enrollment DataBase (EDB), which contains: name and address, date of birth, race, sex, language preference (either English or Spanish),1 and disability status of all Medicare beneficiaries. The total sample, spread across 25 states, included beneficiaries who had enrolled in a new Medicare-approved insurance plan (Private Fee-For-Service), as well as other, similar Medicare beneficiaries who either remained in traditional Medicare with an indemnity-style Medigap supplemental policy or who were in managed care plans approved by Medicare. The demographics of the sample are shown in Table 1. We hypothesized the questionnaire would be more salient to respondents with PFFS experience, and that completion rates would be highest in this group.
* EDB data do not distinguish white Hispanics from black Hispanics; many black Hispanics may be listed as ‘black’ but many other black Hispanics could be listed as ‘other’. Individuals living in nursing homes were ineligible, as were persons who were impaired in ways that made response impossible. Proxies were permitted for this latter group including spouses, life-partners, or adult children. We also learned of some deaths of respondents that occurred after the sample was drawn. Some of the sampled beneficiaries had only guardians or conservators/trustees listed as their addresses; these individuals were assumed to be incompetent or institutionalized and therefore ineligible. The final eligible sample was 7,4572. Despite repeated attempts by mail and phone, no contact was ever made with 1,252 sample members to verify eligibility. We do not know the portion of this group who were ineligible (deceased, institutionalized, and so on.). We followed standard survey procedures and estimated which members of this ‘non-locatable’ group would be eligible based on known characteristics. This estimation process subtracted 221 cases, or 17 percent of the 1,252 cases where eligibility could not be verified. Our final sample was 7,236 eligible and ‘assumed eligible’ Medicare beneficiaries.
The cover of the questionnaire contained a text box in which a Spanish instruction appeared, advising any individual who wanted a Spanish version of the questionnaire to call a toll-free number to request the Spanish version. Respondents who preferred to conduct the interviews by phone, in English or Spanish, could call the same toll-free number, which was staffed by a bilingual interviewer. Throughout the course of the 21- week field period, there were only 15 inquiries to this toll free line. Most of these requests were for Spanish versions of the survey or to complete telephone interviews in Spanish. Survey Implementation: The field period was scheduled to last 21 weeks, between July and December 2002. It began with an advance letter (sent to all sampled beneficiaries), and included three mailings of the survey to all remaining eligible non-responders before beginning phone follow-up. All eligible non-respondents were also sent two reminder postcards (one between each mailing of the questionnaire). Each time a questionnaire was mailed, a business reply envelope was enclosed to minimize burden on the sampled beneficiary and to help increase completion rates. All response envelopes were returned via postage-paid, first-class mail. When the first mailing of the survey was sent, telephone interviewers were trained and ready to accommodate any incoming calls from respondents who chose to complete the survey over the telephone. In week 12 of data collection, we used an automated directory assistance (tele-matching) service to locate telephone numbers for individuals who did not respond by mail. The search used surname and street address as listed in Medicare’s EDB. Less than half of the cases produced a matching telephone number. For the third mailing in week 13, we used Priority mail to ship the survey packages, as opposed to the First Class mail (used in the previous two mailings), because the outer encasement of the priority envelope conveys the special importance of the contents and might attract greater recipient attention. In addition, the cost of Priority mail is much less than that of other attention-getting envelopes such as Federal Express. In weeks 14-21, we conducted telephone follow-up using computer-assisted telephone interviewing (CATI), in English and Spanish, to all eligible non-responders from whom we could match a telephone number to their address. Table 2 summarizes the steps and timeline in survey implementation.
Throughout the 21week field period, it was imperative to be able to monitor not only the overall sample but also completed cases by sample types. Sample Management: Our monitoring system enabled us to mail questionnaires only to eligible non-responders and to pull any cases from the mail file that were completed in the time between drawing the initial file for the mail house and the mailing date. Once phone follow-up began in week 14, this database was updated daily, downloading case dispositions from the computer assisted telephone interviewing (CATI) system, as well as providing reports to the telephone center on which cases needed to be updated, with mail data, in CATI as either complete or ineligible. This database was an essential component of our sample management system and minimized burden on the respondent population by avoiding re-mailings to ineligible sample members or to individuals who completed the survey by mail or phone. In addition, we carefully assigned interviewer labor only to non-responding cases. The success of this management system was demonstrated most clearly in the fact that only 21 cases completed the survey by both phone and mail over the course of the entire field period. Results Mailed questionnaires were considered complete if they contained responses to at least 40% of the questions (which all did). In addition, there were 21 sampled beneficiaries who firmly stated they are “not on Medicare,” although the EDB indicates otherwise3. Table 3 describes the final disposition of cases in the eligible sample.
Figure 1 shows the weeks of data collection, as well as the completion rate during the field period. This figure shows that approximately 1 to 2 weeks after each questionnaire mailing, numbers of returned questionnaires increased dramatically, leveling out again until the next questionnaire mailing. The reminder postcards, particularly the second one, generated little increase in returned questionnaires. 
For this survey, 87% of completes were mailed questionnaires, and 13% were computer- assisted telephone interviews. We anticipated that the questionnaire would be more salient to respondents in PFFS, and, indeed, completion rates were highest for this group. In the PFFS group, 79% of eligibles completed the survey; in the Traditional Medicare group, with or without a Medigap Supplement, 69% completed the survey; and in the Medicare Managed Care group, 67% completed the survey. Salience of the questionnaire does appear to have influenced completion rates. As noted earlier, the group who probably found this survey most relevant (PFFS group) had the highest completion rate. Costs: Fixed costs of the survey included: the design of the questionnaire, OMB processing, programming the questionnaire into CATI, translating the questionnaire into Spanish, coordinating with mail house and print vendors, managing the data collection process, and training interviewers. These costs are fixed, regardless of the number of completed questionnaires. Incremental costs (postage, printing, interviewer time, coding) varied depending upon the number of completed surveys. Figure 2 shows total costs, as they accrued over time, through each successive month during the 21 week filed period. The remaining 13% of costs were for data file preparation. 
In terms of response mode, CATI accounted for 23% of total spending, from programming the instrument through interviewing and coding and the mail/paper effort comprised for 64% of spending (data not shown). DiscussionBased on these data, we find that it is possible to attain a response rate of 73% with an aged and disabled Medicare population, using an enhanced Dillman multi-modal approach, in approximately 21 weeks of data collection. Each re-mailing of the questionnaire resulted in a noticeable increase in response during the field period. There was less indication of a similar ‘spike’ following the thank you/reminder postcards. It is not clear that this funding means that the reminder postcards could be eliminated without damaging response; the reminders are part of a series of contacts which may – in their entirety – have generated a sense of obligation on the part of potential respondents. The Dillman approach includes two mailings of the questionnaire itself; we added a third (via Priority Mail) to determine whether or not it would yield substantial additional response, which it did. Because CATI follow-up is more costly than a Priority Mailing, we can recommend this third ‘special’ mailing as being a cost-effective addition to the standard Dillman approach. Telephone follow-up yielded only 690 additional completed responses. This result could be due to many factors including: a general and widespread disinclination to respond to telephone surveys (Kessler, 1995; Allen, 2002; Massey, 1997; Cox, 2002); active scam-avoidance campaigns aimed at protecting seniors in particular (AARP Bulletin, 2002; NYC Department of Consumer Affairs, U.S. Department of Justice); and the fact that the telephone effort targeted the hardest-to-reach remainder of the sample population – the easy-to-reach responds replied by mail. As the telephone follow-up yielded only a small additional response (13% of all completed surveys) but consumed 23% of total costs, it is a very costly method for boosting survey completion rates a relatively small and final increment. These findings may be helpful to researchers planning large-scale surveys of the Medicare population as well as for other large surveys.
Dillman, DA. Mail and Internet Surveys: the tailored design method. (2000). Wiley & Sons, NY. Dillman, Don A.; Sinclair, Michael D.; Clark, Jon R. “Effects of Questionnaire Length, Respondent Friendly Design, and a Difficult Question on Response Rates for Occupant Addressed Census Mail Surveys.” Public Opinion Quarterly; 1993, 57, 3, fall, 289 304. Dillman, Don A. “The Design and Administration of Mail Surveys.” Annual Review of Sociology; 1991, 17, 225 249. Dillman, Don A. Mail and Telephone Surveys: The Total Survey Design Method. NY: Wile, 1978. Fink, A. and Kosecoff, J. How to Conduct Surveys: A Step-by-Step Guide. Beverly Hills, CA: Sage Publications, 1985. HealthCare Financing Administration. Medicare Pamphlet; Protect Yourself Against Health Care Fraud; Publication No. HCFA-10111. Herberlein, TA and Baumgartner, R (1978). “Factors affecting response rates to mailed questionnaires: a quantitative analysis of the published literature.” American Sociological Review, 43, 447-462 House, James S.; Gerber, Wayne; McMichael, Anthony J. “Increasing Mail Questionnaire Response: A Controlled Replication and Extension.” Public Opinion Quarterly. 1977, 41, 1, spring, 95 99. Kessler, R.C., Little, R.J.A., Groves, R.M. (1995). Advances in strategies for minimizing and adjusting for survey non-response. Epidemiological Reviews, 17:192-204. Linsky, AS (1975). “Stimulating responses to mailed questionnaires: a review.” Public Opinion Quarterly, 39, 82-101. Marillene Allen, Chair, Response Rate Committee, Professional Marketing Research Society, 2002; http://www.pmrs-aprm.com/SpecialResponse Massey JT, O'Connor D, Krotki K, RESPONSE RATES IN RANDOM DIGIT DIALING (RDD) TELEPHONE SURVEYS; Education Statistics Services Institute; presented at American Statitistical Association meetings, 1997. National Sheriffs Association, Operation Fraud stop: A Partnership to Reduce Telemarketing Fraud and Assist Victims. Nederhof, Anton J. “Effects of a Final Telephone Reminder and Questionnaire Cover Design in Mail Surveys.” Social Science Research; 1988, 17, 4, Dec, 353 361. Nederhof, Anton J., “A Survey on Suicide: Using a Mail Survey to Study a Highly Threatening Topic.” Quality and Quantity; 1985, 19, 3, June, 293 302. NYC Dept. of Consumer Affairs. Telephone Scams & Senior Consumers: A Word to Seniors from NYC Dept. of Consumer Affairs; http://www.nyc.gov/html/dca/pdf/telscam.pdf Schaefer, David R.; Dillman, Don A., “Development of a Standard E Mail Methodology: Results of an Experiment.” Public Opinion Quarterly; 1998, 62, 3, fall, 378 397. Telephone Scams And Older Consumers; Protect Yourself Against TELEMARKETING FRAUD; http://www.discoveringmontana.com/doa/consumerProtection/Downloads/TelemarkPrevention.pdf U.S. Dept. of Justice, Elder Financial Exploitation Prevention Program 1 Few of the 7,730 were listed in the sample database as preferring Spanish rather than English (64 of 7,730), and indeed only 11 respondents completed the survey in Spanish. 2 Eliminated from the sample were 56 deceased individuals, 143 individuals dwelling in nursing homes or other institutions, and 74 who were too impaired to respond (and had no proxies). 3 These 21 individuals could arguably have been removed from the denominator as ‘ineligible,’ but we know from our considerable experience with this population that a small percentage of sample members state – in error – that they are not on Medicare. For some surveys, including this one, analysts want to know which respondents believe they are not on Medicare, because those in managed care plans tend to be more likely to hold this erroneous view. Because this survey concerned experiences with different forms of Medicare insurance, a misperception about not being on Medicare is valuable information, and these responses were treated as completes. Rates of Survey Completion and Resource Use at Each Step of a Dillman-Style Multi-Modal Survey Directory: presentations presentations -> Enterprise Network Management iPost: Implementing Continuous Risk Monitoring at the Department of State presentations -> O. P. Singh saarc meteorological Research Centre (smrc) presentations -> Plotting learning presentations -> Friends/Partners in Aviation Weather presentations -> The potential of zakat scheme as an alternative of microcredit to alleviate poverty in Bangladesh presentations -> Managing Millennials presentations -> The Value of Modeling and Simulation Standards presentations -> Assessing the stability and resilience of islamic banks through stress testing under standardized approach of the ifsb capital adequacy framework presentations -> Background of vietnam ict data collection and dissemination number of telecoms, internet service providers presentations -> Guide for applicants Download 116.31 Kb. Share with your friends:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||