Reliability in Unobtrusive Research

LEARNING OBJECTIVES

1. 
   Define stability and describe strategies for overcoming problems of stability.
   
2. 
   Define reproducibility and describe strategies for overcoming problems of reproducibility.
   
3. 
   Define accuracy and describe strategies for overcoming problems of accuracy.
   

This final section of the chapter investigates a few particularities related to reliability in unobtrusive research projects (Krippendorff, 2009) [1] that warrant our attention. These particularities have to do with how and by whom the coding of data occurs. Issues of stability, reproducibility, and accuracy all speak to the unique problems—and opportunities—with establishing reliability in unobtrusive research projects.

Stability refers to the extent to which the results of coding vary across different time periods. If stability is a problem, it will reveal itself when the same person codes the same content at different times and comes up with different results. Coding is said to be stable when the same content has been coded multiple times by the same person with the same result each time. If you discover problems of instability in your coding procedures, it is possible that your coding rules are ambiguous and need to clarified. Ambiguities in the text itself might also contribute to problems of stability. While you cannot alter your original textual data sources, simply being aware of possible ambiguities in the data as you code may help reduce the likelihood of problems with stability. It is also possible that problems with stability may result from a simple coding error, such as inadvertently jotting a 1 instead of a 10 on your code sheet.
Reproducibility, sometimes referred to as intercoder reliability (Lombard, Snyder-Duch, & Campanella Bracken, 2010), [2] is the extent to which one’s coding procedures will result in the same results when the same text is coded by different people. Cognitive differences among the individuals coding data may result in problems with reproducibility, as could ambiguous coding instructions. Random coding errors might also cause problems. One way of overcoming problems of reproducibility is to have coders code together. While working as a graduate research assistant, I participated in a content analysis project in which four individuals shared the responsibility for coding data. To reduce the potential for reproducibility problems with our coding, we conducted our coding at the same time in the same room, while sitting around a large, round table. We coded at the same time in the same room so that we could consult one another when we ran into problems or had questions about what we were coding. Resolving those ambiguities together meant that we grew to have a shared understanding of how to code various bits of data.
Finally, accuracy refers to the extent to which one’s coding procedures correspond to some preexisting standard. This presumes that a standard coding strategy has already been established for whatever text you’re analyzing. It may not be the case that official standards have been set, but perusing the prior literature for the collective wisdom on coding on your particular area is time well spent. Scholarship focused on similar data or coding procedures will no doubt help you to clarify and improve your own coding procedures.

KEY TAKEAWAYS

• 
  Stability can become an issue in unobtrusive research project when the results of coding by the same person vary across different time periods.
  
• 
  Reproducibility has to do with multiple coders’ results being the same for the same text.
  
• 
  Accuracy refers to the extent to which one’s coding procedures correspond to some preexisting standard.
  

EXERCISE

1. 
   With a peer, create a code sheet that you can use to record instances of violence on the television program of your choice. Now, on your own, watch two or three episodes of that program, coding for instances of violence as you go along. Your peer should do the same. Finally, compare your code sheet to that of your peer. How similar were you in your coding? Where do your coding results differ, and why? Which issues of reliability may be relevant?
   

---

[1] Krippendorff, K. (2009). Testing the reliability of content analysis data: What is involved and why. In K. Krippendorff & M. A. Bock (Eds.), The content analysis reader (pp. 350–357). Thousand Oaks, CA: Sage.
[2] Lombard, M., Snyder-Duch, J., & Campanella Bracken, C. (2004). Practical resources for assessing and reporting intercoder reliability in content analysis research projects. Retrieved from http://astro.temple.edu/~lombard/reliability

Chapter 12

Other Methods of Data Collection and Analysis

Why Additional Methods?

While the data collection methods described thus far in the text may be among the most commonly used in sociology, they certainly are not the only methods that social scientists use. Here we’ll describe some of the other methods used in social science, including focus groups, experiments, and ethnomethodology and conversation analysis.

12.1 Focus Groups

LEARNING OBJECTIVES

1. 
   Define focus groups and outline how they differ from one-on-one interviews.
   
2. 
   Discuss how different groups have used focus groups for different purposes.
   
3. 
   Identify the strengths and weaknesses of focus group methodology.
   
4. 
   Describe how to determine the best size for focus groups.
   
5. 
   Identify the major considerations in focus group composition.
   
6. 
   Discuss how to moderate focus groups.
   

Focus groups resemble qualitative interviews in that a researcher may prepare an interview guide in advance and interact with participants by asking them questions. But anyone who has conducted both one-on-one interviews and focus groups knows that each is unique. In an interview, usually one member (the research participant) is most active while the other (the researcher) plays the role of listener, conversation guider, and question asker. Focus groups, on the other hand, are planned discussions designed to elicit group interaction and “obtain perceptions on a defined area of interest in a permissive, nonthreatening environment” (Krueger & Casey, 2000, p. 5). [1] In this case, the researcher may play a less active role than in a one-on-one interview. The researcher’s aim is to get participants talking to each other and to observe interactions among participants.

Focus groups are typically more dynamic than interviews. The researcher takes the role of moderator, posing questions or topics for discussion, but then lets the group members discuss the question or topic among themselves. Participants may ask each other follow-up questions, agree or disagree with one another, display body language that tells us something about their feelings about the conversation, or even come up with questions not previously conceived of by the researcher. It is just these sorts of interactions and displays that are of interest to the researcher. A researcher conducting focus groups collects data on more than people’s direct responses to her or his questions; the group interaction is a key focal point. Due to the nature and unpredictability of group interaction, and the fact that focus group researchers generally want to draw out group interaction, focus groups tend to be qualitative rather than quantitative.
There are numerous examples of sociological research using focus group methodology. In their 2008 study, for example, Amy Slater and Marika Tiggemann (2010) [2] conducted six focus groups with 49 adolescent girls between the ages of 13 and 15 to learn more about girls’ attitudes toward their own and other girls’ participation in sports. In order to get focus group participants to speak with one another rather than with the group facilitator, the study’s interview guide contained just two questions: “Can you tell me some of the reasons that girls stop playing sports or other physical activities?” and “Why do you think girls don’t play as much sport/physical activity as boys?” In another focus group study, Virpi Ylanne and Angie Williams (2009) [3] held nine focus group sessions with adults of different ages to gauge their perceptions of how older characters are represented in television commercials. Among other considerations, the researchers were interested in discovering how focus group participants position themselves and others in terms of age stereotypes and identities during the group discussion. In both examples, the researchers’ core interest in group interaction could not have been assessed had interviews been conducted on a one-on-one basis; thus the focus group method was the ideal choice in each instance.
The preceding examples come from the work of academics who have used focus groups as their method of data collection. But focus groups have proven quite useful for those outside of academia as well. In fact, this method is especially popular among applied researchers. Market researchers use focus groups to gather information about the products or services they aim to sell. Government officials and political campaign workers use them to learn how members of the public feel about a particular issue or candidate. One of the earliest documented uses of focus groups comes from World War II when researchers used them to assess the effectiveness of troop training materials and of various propaganda efforts (Merton & Kendall, 1946; Morgan, 1997). [4] Market researchers quickly adopted this method of collecting data to learn about human beliefs and behaviors. Within social science, the use of focus groups did not really take off until the 1980s, when demographers and communication researchers began to appreciate their use in understanding knowledge, attitudes, and communication (Morgan, 1997). [5]
Focus groups share many of the strengths and weaknesses of one-on-one qualitative interviews. Both methods can yield very detailed, in-depth information; are excellent for studying social processes; and provide researchers with an opportunity not only to hear what participants say but also to observe what they do in terms of their body language. Focus groups offer the added benefit of giving researchers a chance to collect data on human interaction by observing how group participants respond and react to one another. Like one-on-one qualitative interviews, focus groups can also be quite expensive and time-consuming. However, there may be some time savings with focus groups as it takes fewer group events than one-on-one interviews to gather data from the same number of people. Another potential drawback of focus groups, which is not a concern for one-on-one interviews, is that one or two participants might dominate the group, silencing other participants. Careful planning and skillful moderation on the part of the researcher are crucial for avoiding, or at least dealing with, such possibilities. The various strengths and weaknesses of focus group research are summarized in Table 12.1 "Strengths and Weaknesses of Focus Group Research".

Table 12.1 Strengths and Weaknesses of Focus Group Research

Strengths	Weaknesses
Yield detailed, in-depth data	Expensive
Less time-consuming than one-on-one interviews	May be more time-consuming than survey research
Useful for studying social processes	Minority of participants may dominate entire group
Allow researchers to observe body language in addition to self-reports
Allow researchers to observe interaction between multiple participants

As mentioned, careful planning and skillful moderating are two crucial considerations in the effective use of focus groups as a method of data collection. In some ways, focus groups require more advance planning than other qualitative methods of data collection such as one-on-one interviews, where a researcher may be better able to control the setting and the dialogue, or field research, where “going with the flow” and observing events as they happen in their natural setting is the primary aim and time is less limited. Researchers must take care to form focus groups whose members will want to interact with one another and to control the timing of the event so that participants are not asked nor expected to stay for a longer time than they’ve agreed to participate. The researcher should also be prepared to inform focus group participants of their responsibility to maintain the confidentiality of what is said in the group. But while the researcher can and should encourage all focus group members to maintain confidentiality, she should also clarify to participants that the unique nature of the group setting prevents her from being able to promise that confidentiality will be maintained.

Group size should be determined in part by the topic of the interview and your sense of the likelihood that participants will have much to say without much prompting. If the topic is one about which you think participants feel passionately and will have much to say, I think a group of 3–5 is ideal. Groups larger than that, especially for heated topics, can easily become unmanageable. Some recommend that a group of about 6–10 participants is the ideal size for focus group research (Morgan, 1997); [6] others recommend that groups should include 3–12 participants (Adler & Clark, 2008). [7] The size of the focus group is ultimately your decision as the researcher. When forming groups and deciding how large or small to make them, take into consideration what you know about the topic and participants’ potential interest in, passion for, and feelings about the topic. Also consider your comfort level and experience in conducting focus groups. These factors will help you decide which size is right in your particular case.
It may seem counterintuitive, but in general, it is better to form focus groups consisting of participants who do not know one another than to create groups consisting of friends, relatives, or acquaintances (Agar & MacDonald, 1995). [8] The reason for this is that groups who know each other may share some take-for-granted knowledge or assumptions. In sociological research, it is precisely the taken-for-granted that is often of interest; thus the focus group researcher should avoid setting up interactions where participants may be discouraged to question or raise issues that they take for granted. However, groups should not be so heterogeneous that participants will be unlikely to feel comfortable talking with one another.
Focus group researchers must carefully consider the composition of the groups they put together. In his text on conducting focus groups, Morgan suggests that “homogeneity in background and not homogeneity in attitudes” (p. 36) should be the goal, since participants must feel comfortable speaking up but must also have enough differences to facilitate a productive discussion (1997). [9] Whatever composition a researcher designs for her or his focus groups, the important point to keep in mind is that focus group dynamics are shaped by multiple social contexts (Hollander, 2004). [10] Participants’ silences as well as their speech may be shaped by gender, race, class, sexuality, age, or other background characteristics or social dynamics, all of which might be suppressed or exacerbated depending on the composition of the group. Hollander suggests that researchers must pay careful attention to group composition, must be attentive to group dynamics during the focus group discussion, and should triangulate multiple methods of data collection in order to “untangle participants’ responses and their relationship to the social contexts of the focus group” (p. 632).
In addition to the importance of advance planning, focus groups also require skillful moderation. While a researcher certainly doesn’t want to be viewed as a stick-in-the-mud or as overly domineering, it is important to set ground rules for focus groups at the outset of the discussion. Remind participants that you’ve invited them to participate because you want to hear from all of them. Therefore the group should aim to let just one person speak at a time and avoid letting just a couple of participants dominate the conversation. One way to do this is to begin the discussion by asking participants to briefly introduce themselves or to provide a brief response to an opening question. This will help set the tone of having all group members participate. Also ask participants to avoid having side conversations; sharing thoughts about or reactions to what is said in the group is important and should not be limited to only a few group members.
As the focus group gets rolling, the moderator will play a less active role than he does in a one-on-one interview. There may be times when the conversation stagnates or when you, as moderator, wish to guide the conversation in another direction. In these instances, it is important to demonstrate that you’ve been paying attention to what participants have said. Being prepared to interject statements or questions such as “I’d really like to hear more about what Sally and Joe think about what Dominick and Ashley have been saying” or “Several of you have mentioned ____. What do others think about this?” will be important for keeping the conversation going. It can also help redirect the conversation, shift the focus to participants who have been less active in the group, and serve as a cue to those who may be dominating the conversation that it is time to allow others to speak.
In sum, focus groups are a useful method for researchers who wish to gather in-depth information about social processes. Focus groups are similar to one-on-one qualitative interviews in many ways, but they give researchers the opportunity to observe group dynamics that cannot be observed in one-on-one interviews. Historically, focus group research was more commonly used by applied researchers than by academics, though in recent decades social scientists from all domains have discovered the usefulness of focus groups for gaining understanding of social processes and have begun using this method of data collection in their studies.

KEY TAKEAWAYS

• 
  Focus groups are designed to elicit group interaction.
  
• 
  Focus groups are used in a variety of professions, from market research to academia to government and political research.
  
• 
  Like one-on-one qualitative interviews, focus groups can yield very detailed information, are excellent for studying social processes, and provide researchers with an opportunity to observe participants’ body language; they also allow researchers to observe human interaction.
  
• 
  Focus groups can be expensive and time-consuming, as are one-on-one interviews; there is also the possibility that a few participants will dominate the group and silence others in the group.
  
• 
  In terms of focus group composition, homogeneity of background among participants is recommended while diverse attitudes within the group are ideal.
  

EXERCISES

1. 
   Musician John Mayer held a “focus group” to get fan feedback on his career. Watch and critique his focus group facilitation style in this clip:
   

http://www.youtube.com/watch?feature=player_embedded&v=ES3wFlpwjiM

How well does Mayer play the role of a “behind-the-scenes” focus group moderator? How well does he get focus group participants to talk with each other? Knowing what you now know about interviews and focus group research, what advice would you give Mayer for improving his focus group facilitation skills?

2. 
   To see what a real marketing focus group looks like, watch the following video:
   

http://www.youtube.com/watch?feature=player_embedded&v=MuiI7BFhQl4

You’ll see several of the tips mentioned in this section applied. As you watch, what elements of the major strengths and weaknesses of focus group research seem to be in play?

---

[1] Krueger, R. A., & Casey, M. A. (2000). Focus groups: A practical guide for applied research(3rd ed.). Thousand Oaks, CA: Sage.
[2] Slater, A., & Tiggemann, M. (2010). “Uncool to do sport”: A focus group study of adolescent girls’ reasons for withdrawing from physical activity. Psychology of Sport and Exercise, 11, 619–626.
[3] Ylanne, V., & Williams, A. (2009). Positioning age: Focus group discussions about older people in TV advertising. International Journal of the Sociology of Language, 200, 171–187.
[4] Morgan, D. L. (1997). Focus groups as qualitative research (2nd ed.). Thousand Oaks, CA: Sage.
[5] Morgan, D. L. (1997). Focus groups as qualitative research (2nd ed.). Thousand Oaks, CA: Sage.
[6] Morgan, D. L. (1997). Focus groups as qualitative research (2nd ed.). Thousand Oaks, CA: Sage.
[7] Adler, E. S., & Clark, R. (2008). How it’s done: An invitation to social research (3rd ed.). Belmont, CA: Thomson Wadsworth.
[8] Agar, M., & MacDonald, J. (1995). Focus groups and ethnography. Human Organization, 54,78–86.
[9] Morgan, D. L. (1997). Focus groups as qualitative research (2nd ed.). Thousand Oaks, CA: Sage.
[10] Hollander, J. A. (2004). The social context of focus groups. Journal of Contemporary Ethnography, 33, 602–637.

Download 2.09 Mb.

Share with your friends: