Beyond Theory: the value of collecting data with psychological methods

As I said at the beginning of this paper, the central question for a designer who wants to make popular games is "how do I make my game more fun for more gamers?" and that a glib response is to "design the games better." Taking the glib answer seriously for a moment, how do you go about doing that? Presumably, designers are doing the best they can already. The Dilbertian "work smarter, not harder" is funny, but not helpful. The way to help designers is the same way you help people improve their work in all other disciplines--you provide them feedback that helps them learn what is good and not so good about their work, so that they can improve it.

1. 
   The feedback should accurately represent the opinions of the target gamers. By "target gamers," I mean the group of gamers that the game is trying to appeal to (e.g., driving gamers, RTS gamers, etc.) If your feedback doesn't represent the opinion of the right group of users, then it may be misleading. This is absolutely critical. Misleading feedback is worse than no feedback, the same way misleading road signs are worse than no signs at all. Misleading signs can send folks a long way down the wrong road.
   
2. 
   The feedback should arrive in time for the designer to use it. If the feedback is perfect, but arrives too late (e.g., post RTM, or after that feature is locked down), the feedback isn't that helpful.
   
3. 
   The feedback should be sufficiently granular for the designer to take action on it. The information that "gamers hate dumb-sounding weapons" or that "some of the weapons sound dumb" isn't nearly as helpful as "Weapon A sounds dumb, but Weapons B, C, and D sounds great."
   
4. 
   The feedback should be relatively easy to get. This is a pragmatic issue--teams won't seek information that is too costly or too difficult to get. Teams don't want to pay more money or time than the information is worth ($100k and 20 person hours to learn that people slightly prefer the fire-orange Alpha paint job to the bright red one is hardly a good use of resources.)
   

There are many feedback systems that designers use (or, in some cases, been subjected to). Most designers, like authors, recognize that they need feedback on their work in order to improve it-- few authors have reason to believe that their work is of publishable quality without some revision based on feedback. I'm going to list the feedback systems of which I am aware, and discuss how good of a feedback delivery system it is. There are two main categories of feedback loops: feedback from professionals in the games industry, and from non-professionals (i.e., gamers). While these sources obviously affect each other, it is easier to talk about them separately

1. 
   Feedback from those on the development team. This is the primary source of feedback for the designer--people working on the game say stuff like "that character sucks" or "That weapon is way too powerful." This system is useful because it ably suits criteria two through four (the feedback is very timely, granular enough, and easy to get), but still leaves the designer with a question mark on criteria one--how many gamers will agree that that weapon is way too powerful
   
2. 
   Feedback from gaming industry experts. Game design consultants ("gurus"), management at publishers, game journalists, etc. can also provide useful feedback. While their feedback can often meet criteria three (sufficiently granular), criteria two (timely) is sometimes a problem--long periods can go between feedback, and recommendations can come after you can use it. And the designer is still left with questions about criteria one (accurately represents gamers), although some could argue that they may be more accurately representing gamers because they have greater exposure to more games in development.
   

So while feedback from industry professionals is necessary when designing the game, they may not be the best at evaluating whether gamers will like something. In the end, they can only speak for themselves.

1. 
   News group postings/Beta testing/fan mail. This is reading the message boards to see what people say about the game. The main problem with this as a feedback system is with criteria two (timely). The game has to be able to be fairly far along (at least beta, if not shipped) in order to get the games to people; typically, that feedback arrives too late to make any but the most cosmetic of changes. Also, the feedback often runs into problems of not being sufficiently granular to take action on. ("The character sucks!") But at least this kind of feedback is relatively cheap in both time and money.
   
2. 
   Acquaintance testing. This is where you try to get people (typically relatives, neighbor's kids, etc.) from outside the industry to play your game and give you feedback. This feedback is often sufficiently granular and may be relatively accurate, but it is often not that timely due to scheduling problems, and can be costly in time.
   
3. 
   Focus groups/Focus testing. This kind of feedback system is typically done by the publisher, and involves talking to small groups (usually four - eight gamers) in a room about the game. They may get to see or play demos of the game, but not always. One typical problem with focus groups is that often tend to happen very late in the process when feedback is hard to action on (not timely) and not sufficiently granular. The costs for focus groups can also be quite high.
   

Up to this point, I've mostly been criticizing what is done. Now I need to show that I have a better solution. I'm going to outline some of the key factors that have allowed Microsoft to develop a feedback system that we think meets the four criteria that I set up for a "good" feedback system. We call this process of providing designers with feedback from real users on their designs "user-testing," and the people who do this job "user-testing specialists."

The importance of using principles of psychological testing. Experimental psychology has been studying how to get meaningful, representative data from people for over 70 years, and the process we use adheres to the main principles of good research. This is not to say that all psychological research is good research any more than to say that all code is good code; researchers vary in their ability to do good research the same way that not all programmers are good. But there are accepted tenets of research methodology that have been shown to yield information worth relying on, and our processes have been designed with those in mind. (For the sake of not boring you senseless, I'm not going to attempt to summarize 70 years of research on how to do research in this paper.) What I'm going to do instead is describe the day-to-day work that the user-testing group at Microsoft does for its dev teams (both first and third party).

The actual testing methods we use. The user-testing group provides three major services: usability testing, playtesting, and reviews. These services are described in detail below.

The result of each of these services is a report is sent to the team which meticulously documents the problems along with recommendations on how to fix those problems. Our stance is that the development teams are the ones who decide if and how to fix the problems.

One noticeable absence in our services is "focus groups." Our belief (supported by research on focus groups) is that focus groups are excellent tools for generation (e.g., coming up with new ideas, processes, etc.), but are not very good for evaluations (e.g., whether the people like something or not). The group nature of the task interferes with getting individual opinions, which is essential for the ability to quantify the evaluations.

How this feedback system fares on the four criteria for a good feedback system. So, how does the way we do user-testing at Microsoft stack up to the four criteria? Pretty well (in my humble opinion). A recap of the criteria, and my evaluation of how we do on them is given below.

1. 
   The feedback should accurately represent the opinions of the target gamers. We supply reasonably accurate, trustworthy feedback to teams, because:

   a. We have a large database of gamers (~12,000) in the Seattle metro area, who play every kind of game. So we can almost always bring the right kind of gamers for each kind of game.

   b. We hire only people with strong backgrounds in experimental or applied psychology in order to minimize the biases of the user-testing specialist. We also have a rigid review process for all materials that get presented to the user.

   c. We thoroughly document our findings and recommendations, and test each product repeatedly, which allows us to check the validity of both our work and the team's fixes over multiple tests and multiple participants.

   
   
2. 
   The feedback should arrive in time for the designer to use it. We are relatively fast at supplying feedback. The entire process takes about six days to get some initial feedback, and about 11-14 days for a full report. If the tests are well planned, they can happen at key milestones to maximize the timeliness of the feedback.
   
3. 
   The feedback should be sufficiently granular for the designer to take action on it. The level of feedback in the reports is extremely granular, because the tests are designed to yield granular, actionable findings. The user-testing specialist typically comments at the level of which cars or which tracks caused problems, or what wording in the UI caused problems. The recommendations are similarly specific. Usability tests typically yield more than 40 recommendations, whereas playtest tends to have anywhere from 10-30 items to address.
   
4. 
   The feedback should be relatively easy to get. The feedback is relatively easy for the dev team to get--they have a user-testing lead on their game, and that person sets up tests for them and funnels them the results. However, the feedback is relatively inexpensive, when compared to the multi-million dollar budgets of modern games. The total cost of our operation is "substantial," but economies of scale make the cost per game relatively small.
   

Current composition of user-testing group: 15 FT user-testing specialists, 3-5 contract specialists, 3 FT support staff. Almost all user-testing specialists have either two or more years of graduate training in experimental psychology, or equivalent experience in applied psychology and are gamers. All four founding members of the user-testing group are still with the group.

Special thanks to Randy Pagulayan and Ramon Romero for their help editing this article.

The play of any computer game can be described as a series of choices. A player might choose the left or right hand tunnel, decide to skip this target and save ammunition, or play a fighter rather than a mage. The total path of a player through the game is the result of a thousand little choices, leading to success or failure in the game and to enjoyment or dislike of the game itself. The principles underlying the choices players make and the way in which a designer can shape those choices is a key component of game design.

As in my previous article, the kind of psychology discussed here is often called behavioral psychology. This sub-field of psychology focuses on experiments and observable actions, and is a descriptive rather than normative field of study. Instead of looking at what people should do, it studies and tries to explain what they actually do. By understanding how people react to different kinds of choices, we can design games that help them make the kind of choices that they'll enjoy, and understand how some game designs can unintentionally elicit bad choices.

Maximizing

The most obvious thing to do when confronted with multiple options is to pick the choice or pattern of choices that maximizes reward. This is the sort of solution sought by game theory, one that mathematically guarantees the greatest level of success. While most players don't try to work out the exact algorithms behind weapon damage, they will notice which strategies work better than others and tend to approach maximal reward.

Usually, participants maximize when the choices are simple and deterministic. The more complex the problem, the more likely they are to engage in exploratory actions and the less likely they are to be sure that they are doing the optimal thing. This is particularly true in situations where the contingency is deterministic. If the pit monster attacks every time the player gets to a certain point, they'll quickly pick this up and learn the optimal point to jump over it. If it attacks probabilistically, the player will take longer to guess what rules govern the pit monster's attack.

While maximizing is the best thing for the player, it's probably not a good thing for the designer. If the player is doing as well as it's possible to do, it implies that they've mastered the game. It also means that the game has become perfectly predictable and most likely boring. A contingency with an element of randomness will maintain the player's interest longer and be more attractive. For example, subjects will generally prefer a 30 second variable interval schedule (rewards being delivered randomly between zero and sixty seconds apart) to a 30 second fixed interval schedule (rewards being delivered exactly 30 seconds apart), even though both provide the same overall rate of reward.

There is another, subtler problem with maximizing. As discussed in the previous article, sharp declines in the rate of reward are very punishing for players and can result in quitting. If the player has learned to maximize their reward in one portion of the game, creating a high and consistent level of reward, moving to another part or level of the game will most likely result in a drop in reward. This contrasting low level of reward is extremely aversive and can cause the player to quit. It may even be an effective punishment for exploring new aspects of the game, as the transition from the well understood portion to the unknown marks an inevitable drop in rewards.

To avoid maximizing, there are two basic approaches. First, one can make sure that the contingencies are never so simple that a player could find an optimal solution. The easiest way of doing this is to make the contingencies probabilistic. Massive randomness isn't necessary, just enough to keep players guessing and engaged. Second, the more options there are within the game, the more things there are to compare, the less likely it is that there will be a clear ideal strategy. If all the guns in the game work the same but do different levels of damage, it's easy to know you have the best one. If one gun is weaker but does area damage and another has a higher rate of fire, players can explore a wider variety of strategies. Once there is a clear best way to play the game, it ceases to be interesting in its own right.

Once there are multiple options producing rewards at different rates, the most common pattern of activity observed in humans and animals is matching. Essentially, matching means that the player is allocating their time to the various options in proportion to their overall rate of reward. More formally, this is referred to as the Matching Law, and can be expressed mathematically as the following equation:

Let's say our player Lothar has two different areas in which he can hunt for monsters to kill for points. In the forest area, he finds a monster approximately every two minutes. In the swamp area, he finds a monster every four minutes. Overall, the forest is a richer hunting ground, but the longer Lothar spends in the forest the more likely it is that a new monster has popped up in the swamp. Therefore Lothar has a motive to switch back and forth, allocating his time between the two alternatives. According to the Matching Law, our player will spend two-thirds of his time in the forest and one-third in the swamp.

The key factor in matching is rate of reward. It's the average amount of reward received in a certain period of time that matters, not the size of an individual reinforcer or the interval between reinforcers. If the swamp has dragons that give Lothar 100 points, while the forest has wyverns that give him only 50 points but appear twice as often as the dragons, the overall rates of reward are the same and both areas are equally desirable.

Now that I've set up a dichotomy between matching and maximizing, let me confuse things a bit. Under many circumstances, matching is maximizing. By allocating activity according to rate, the player can receive the maximal amount of reward. In particular, when faced with multiple variable interval schedules, matching really is the best strategy. What makes matching important to our understanding of players is that matching appears to be the default strategy when faced with an ongoing choice between multiple alternatives. In many cases, experiments show subjects matching even when other strategies would produce higher rates of reward.

Matching (and switching between multiple options in general) also has the helpful property of smoothing out the overall rate of reward. If there are several concurrent sources of reinforcement, a dip in one of them becomes less punishing. As one source of points falls off, a player can smoothly transition to others. A player regularly switching back and forth between options also has a greater chance of noticing changes in one of them.