Beautification

A within-subject repeated measures design was used in this experiment to measure the effects of formality on design performance. There were five conditions, all of which were presented to each subject one after another. Latin square design, as shown in Figure 1, was used in to control order effects of the conditions presented (Heiman, 2001). In addition, between-subject comparisons were also made to test Hypothesis 3 where experts and novices may be affected by formality differently during the design process.

Subjects

There was one independent variable – levels of formality of a web interface design prototype (HTML forms) with four levels: from low formality to high formality. There were five conditions in the experiment, as shown in Table 1, with four conditions each involving one HTML form design with a different level of formality presented on the tablet PC, and for comparison, as interaction with digital and traditional media differs (Bilda & Demirkan, 2003) which in turn, may affect design-decisions (Black, 1990), one condition involved one HTML form design with low formality was presented on paper.

Table 1.

Level of formality associated with each condition, and the medium used for the presentation and review of designs.

Conditions	Formality level	Medium
1	Low formality (totally hand-drawn)	Paper (and pen)
2	Low formality (totally hand-drawn)	Tablet PC
3	Medium-low formality	Tablet PC
4	Medium-high formality	Tablet PC
5	High formality ([totally] computer-rendered)	Tablet PC

To produce different levels of formality, a taxonomy of beautification was developed (see Table 2) based on previous work on beautification techniques (e.g. Chung, Mirica, & Plimmer, 2005; Plimmer & Grundy, 2005; Pomm & Werlen, 2004; Wang, Sun & Plimmer, 2005). Smoothness and alignment (vertical and horizontal) were varied systematically as described in detail in section 2.5.2, while others were held constant. The relationship between beautification and formality can be understood as: the higher the degree of beautification, the higher the level of formality of a design (i.e. design appears more formal)

Table 2

Taxonomy of beautification showing different variables associated with beautification.

VARIABLES
Hand drawn ----------------------continuum--------------------- Computer-generated (Low formality) (High formality)
Smoothness (objects, lines or characters) Rough (hand-drawn)	Computerized i.e. Smoothed, straight, formal
Size Inconsistant	Exact, standardised
Alignment, vertical Inexact	Exact, standardised
Alignment, horizontal Inexact	Exact, standardised
Spacing, vertical (between objects) Irregular	Exact, standardised
Spacing, horizontal Irregular	Exact, standardised

2.1.2. Dependent Variables: Functional changes

Measurements were taken to explore the effects of design appearance (different levels of formality) on the design performance – in this case, decision-making – during the early stages of the design process. Participants were to improve the designs presented to them by making changes to the designs; hence design performance (particularly decision making) could be objectively measured by counting the different functional changes made. Functional changes measured, in the context of web interface design, included:

• 
  Adding:
  

• 
  an element to an item
  
• 
  an item to an item set
  
• 
  an item set to the design
  

• 
  Deleting:
  

• 
  an element in an item
  

• 
  an item
  
• 
  an item set
  

• 
  Changing:
  

• 
  a control from one type to another e.g. radio button to a textbox
  
• 
  a label e.g. spelling, words, grammar etc
  

• 
  Resizing:
  

• 
  a control to suit the scenario e.g. resizing a text box to become bigger/smaller, wider/longer to suit the scenario
  

• 
  Relocating:
  

• 
  an element / an item / an item set from one area to another area in the design to suit logical flow of information.
  

In the context of this study of HRML form design, an ‘element’ is any single component within a design. An element is part of an ‘item’ (a label and its associated control for user input e.g. a label “Male”, associated with a radio button); and an item is part of to an “item set” (which consists of one or more items e.g. Gender: Female (with a radio button) / Male (with radio button). An item set is at the highest level within the hierarchy of functional changes; hence, a functional change is counted at the highest level in the hierarchy. See Figure 2 for an illustration of this tree of change.

The number of functional changes made at each level of formality was measured and were categorized into three variables:

• 
  Total Changes: any (functional) change made by the participant on the design e.g. all changes mentioned above including adding, deleting, changing, resizing and relocating elements, items, and/or item sets.
  

• 
  Quality functional changes: the number of functional changes to elements, items and/or item sets made by the participant that improved functionality in comparison to the original version – according to Web Usability handbooks (e.g. Brinck, Gergle, & Wood, 2002; Fowler & Stanwick, 2004), HTML design guidelines (e.g. Shelly, Cashman, & Woods, 2005) and interface design principles (e.g. Watzman, 2003).
  

• 
  Expected (planned) functional changes: the number of functional changes to elements, items and/or item sets, made by the participant to ‘correct’ deliberate design ‘errors’ planned by the experimenter – based on Web Usability handbooks (e.g. Brinck, Gergle, & Wood, 2002), HTML design guidelines (e.g. Shelly, Cashman, & Woods, 2005) and interface design principles (e.g. Watzman, 2003).
  

As the number of deliberate errors was the same in each design presented to the participants, the number of corrections (expected changes) made in each design was measured to allow for controlled comparisons between conditions, hence, to explore the effects of formality on design-decisions during early stages of the design process (i.e. during early prototyping). In addition to expected changes, quality changes and total changes made were also measured as it was anticipated that along with expected changes, participants would made other changes to the design that were not deliberate errors (refer to Appendix A for the outline of design errors that was present in each design). The three measurements (total changes, quality changes, and expected changes) were important for the assessment of validity and reliability of the experimental stimuli – the five designs presented to the participants i.e. whether the number of quality changes and expected changes were similar; and whether the total number of changes made was statistically different from the number of quality changes. If the number of changes made were not statistically different, then it could be statistically reasoned, that the five designs were equivalent, and that the results was due to the experimental manipulation of the independent variable.

Participants were directed to make functional changes by the instruction to “improve the design[s] to better serve its [their] purposes”, therefore, participants did not have to make any beautification changes such as alignment of elements (see other examples in Table 2) to ‘tidy-up’ the design. Thus, beautification changes made by participants were not counted as a functional change. Furthermore, beautification changes were not measured as a dependent variable it was predicted that the process of tidying up a design was time consuming (e.g. Newma, et al., 2003) especially in an experiment with only 11 minutes for each condition.

In addition, a post-task questionnaire was used to record the following variables (see Appendix B for response options details in the questionnaire):

• 
  Post-task rankings of overall enjoyment of designs in the order from the most-liked design (1) to the least liked design (5). Reasons for rankings were also recorded.
  

• 
  Preference for design medium in the experiment. Response options were preference for pen and paper; preference for the tablet PC; or no preference.
  
• 
  Preference for design medium in the real world. Response options were open.
  
• 
  Demographics including (open response) :
  

• 
  Design experience.
  

• 
  Study specialization/major.
  
• 
  Study Level.
  

Thirty adults – sixteen male (mean age of 22.81, SD = 5.87) and fourteen female (mean age of 21.14, SD = 1.03) between 18 and 44 years of age (total mean age of 22.03, SD = 4.36) volunteered to participate in the study.

Participants (n=20 who majored/specialized in Computer Science (CS) or Software Engineering (SE) in their study, as well as participants with non-CS/SE study backgrounds (n=10) were recruited from the University of Auckland. All participants were current students/recent graduates from the University of Auckland (mean years of study at university: 3.20, SD = 1.186). Out of thirty, twenty-five participants were students (twenty-two undergraduates and three post-graduates), five participants, recruited through researcher’s personal contacts, were recent graduates from the University of Auckland in 2006, all currently working in computer/software engineering-related industries. Overall, there were twenty-two undergraduates, and eight graduates/postgraduates.

Papers taken most frequently during the course of study were the computer science stage one papers including CS101. All participants (n=22) who majored/specialized in computer science(CS)/software engineering(CS) had taken or taking CS101 and only two out of ten participants who majored in non-CS/SE subjects (including other engineering, business and information-system students) had taken CS101. Relevant papers taken by participants who majored in CS/SE were: CS101 and CS105 (stage one); CS230 (stage two); CS345 (stage three) and SE SE450 (equivalent to CS345). Others papers taken by participants with non-CS/SE majors included: information system, engineering, biomedical science, psychology and business papers. Fifteen participants had CS and/or SE design experience such as HTML design, website and interface designs and software design, whereas, the other fifteen participants’ design experience ranged from no design experience to some non-CS/SE related design experience.

All participants were exposed to Inkit (the design tool used in the experiment) for the first time. All participants had normal eyesight or corrected-to-normal by spectacles or contact lenses. For the summary of demographics, see Appendix C.

Participants were each reminded not to discuss the experiment with their peers, and were thanked with $2 worth of chocolate as a token of appreciation as well as entering the draw to win $50 cash.

Study approval was obtained from the University of Auckland Human Participants Ethics Committee (UAHPEC). Each participant took part in a single session approximately one hour long. An experimental protocol was produced to standardize experimental procedures and thus, help minimize experimental errors and variability. Participants were first instructed to make changes to improve each design presented to them. Five early designs of online forms were then presented to the participants one after another – four designs were presented on the tablet PC and one was presented on paper (refer to Table 1). Upon completion of experimental tasks, participants filled in the post-task questionnaire. The detailed procedure is described next.