Ranking according to design appearance (aesthetics)

Most participants (twenty-one out of thirty, or 70%,) ranked according to the aesthetic properties of the designs (e.g. tidiness of lines and fonts, alignment of elements, and whether it was easy to follow). See Table 25 which shows the mean rank of designs with high formality to low formality according to aesthetics factors. To determine the effect of formality level (aesthetics) on participants’ enjoyment in working on the design (indicated by rankings), Friedman’s rank test for several related samples was used as proposed by Winer, Brown, and Michels (1991) and Howell (2002). Analysis of the ranked data showed that the subjects’ rankings of the five designs according to aesthetics were significantly different, χ² (4, N = 21) = 47.43, p < .0001, and the Kendall coefficient of concordance of .57 indicated strong differences among the rankings. Post hoc analysis by Wilcoxin signed-rank test, indicated that participants enjoyed working on the high formality design (lower rankings) significantly more than the other four designs, all with p < .0001, as participants indicated that the designs appeared “tidy”, “pretty” and “[format is] easy to follow”. Results also showed that participants enjoyed working on the low formality design on Tablet PC the least as it had the highest mean rank, and was significantly higher when compared to the other four designs: low formality on paper (p = .006); medium-low formality (p = .001); medium-high formality (p < .0001) and high formality (p < .0001), as many participants had noted that the design appeared “untidy”, aesthetically “unattractive", and that the layout was “hard to follow”. Interestingly participants enjoyed working on the low formality design presented on paper significantly more than the low formality design presented on the tablet PC (p = .006). No significant differences were found between rankings of other pairs of designs (medium-high formality, medium-low formality and low formality on paper) in terms of enjoyment when a design appeared to be more or less formal (beautified).

Additionally, the overall rankings of designs according to appearance (aesthetics) of designs were negatively associated with the trend of the number of changes made across levels of formality (refer to Appendix Q for subjects’ performance – those who ranked according to the aesthetics factor – across levels of formality). The number of changes made increased linearly as formality decreased, and similarly, rankings increased as formality decreased – i.e. subjects liked designs that appeared more formal than designs that appeared less formal, but on the other hand, subjects made fewer changes to the designs that they liked (higher formality designs) and made more changes to designs that that liked less (lower formality designs). However, interesting, unlike subjects’ performance, the low formality designs presented on paper was ranked lower than the low formality design presented on the tablet PC i.e. subjects enjoyed working on the low formality design presented on paper than on the tablet PC.
3.4.2. Ranking according to perceived “effort required”

Moreover, eleven out of thirty (23%) participants ranked according to the perceived effort (input) required to improve designs. The mean ranks from high formality to low formality were 1.55, 2.55, 3.82, 3.91, 3.09 (see Table 25). Subjects’ reasons associated with rankings indicated that a higher rank (e.g. 5) meant that the design “required lots of changes” and therefore more effort is needed to improve the design; and a lower rank (e.g. 1) meant that the design was generally “good” and “required fewer changes” (i.e. least effort required).

As suggested by Howell (2002) and Winer, Brown, and Michels (1991), Friedman’s rank test for several related samples was employed to determine the effect of formality on participants’ perceived effort required when working on the design (which in turn affected the overall enjoyment rankings). Analysis of the ranked data showed that the subjects rankings of the five designs, according to perceived effort required, were a significantly different, χ² (4, N = 11) = 16.16, p < .003, with the Kendall coefficient of concordance of .37 indicating fairly strong differences among the rankings. Post hoc analysis by Wilcoxin signed-rank test, (and controlling for the Type I errors across these comparisons at the .05 level using the LSD procedure) further suggested that the participants enjoyed working on the high formality design (lowest mean ranking) significant more than the other four designs: medium-high formality (p = .039); medium-low formality (p = .004); low formality on Tablet PC (p = .009) and low formality on paper (p = .015) – some participants reasoned that the high formality design had “fewer mistakes” and “not many changes needed” compared to other designs which participants indicated that they “wanted to change more” in the design. Results also suggested that subjects enjoyed working on the medium-high formality design significantly more than the medium-low formality design (p = .046) as well as the low formality design on the Tablet PC (p = .034). No significant differences were found between rankings of other pairs of designs (medium-low formality; low formality on Tablet PC and low formality on paper) in terms perceived effort required when working on a design (i.e. the overall enjoyment rankings)

Additionally, the overall rankings of designs according to the perceived effort required was negatively related to the number of changes made across levels of formality (refer to Appendix R for subjects’ performance – those who ranked according to effort required – across levels of formality). The number of changes made increased (decreased) linearly as formality decreased (increased) (see Appendix R for subjects performance), and similarly, rankings increased as formality decreased. However, the rankings of low formality designs – one presented on paper and the other presented on the tablet PC – was not in linear order expected – refer to Table 25.

Few participants (seven out of thirty, or 23%) ranked according whether designs were fun or stimulating to work on during the task. The mean rank of the designs from high formality to low formality was in the order of 2.86, 2.71, 1.86, 3.29 and 4.29 (see Table 25). Participants’ reasons associated with rankings indicated that a rank of 5 meant that the design was the least interesting and stimulating to work on; whereas, a rank of 1 meant that the design that was the most fun and/or stimulating to work on out of the five designs given. To explore the whether there were any differences in the overall enjoyment rankings of the five designs in terms of level of fun/stimulation, Friedman’s rank test for several related samples was conducted. Results showed that the ranked data did not differ significantly, χ² (4, N = 7) = 8.80, p = .66, which suggested that participants found the five designs similar in terms of the level of fun/stimulation when working on the designs. Visual inspections suggested that subjects enjoyed working on the medium-low formality design as it was more fun/stimulating than other designs (mean rank of 1.86 – lowest compared to other designs); but found the low formality design least fun/stimulating to work on (mean rank of 4.29 – highest in relation to other designs). The fun/stimulating level of other designs did not differ very much for the subjects, which was shown in the similar rankings (2.86, 2.71, and 3.29 – refer also to Table 25 for variance). However, results were only preliminary and inconclusive as the number of subjects was very small (n= 7).

Additionally, although with varying perception of whether a design was fun and/or stimulating to work on (see Table 25), it made no difference to the effect of formality – the number of changes made as formality increased (decreased) still decreased (increased) – see Appendix S for subjects’ performance across levels of formality – those who ranked according to the fun/stimulation level of working on the designs. This may suggest that the perception of fun/stimulation level when working on the designs did not correlate strongly to the number of changes made to the designs; however, since fun/stimulation level of designs varies in different individuals, and that there were only seven subjects examined, interpretations must be made with caution.
3.5 Design Tool Preference

3.5.1. Design tool preference in the experiment

Subjects’ design tool preference in the experiment was examined to determine whether subjects preferred using pen and paper, and/or the Tablet PC or whether there was no preference between the two tools during the design tasks.

Figure 28 shows that overall, 50% of the participants preferred using the tablet PC (Inkit), while thirteen (43.3%) participants preferred using paper and pen during the design tasks; and two participants responded with no particular preference between using pen and paper, and the Tablet PC during the design tasks. See Appendix T for the detailed responses from subjects on design tool preference during the experiment.

Visual inspection of the Figure 29a revealed that in the CS/SE major group, more subjects preferred using the Tablet (55% or eleven out of twenty) than using paper and pen (35% or seven out of twenty subjects), while two subjects had no preference; and in contrast, in the non-CS/SE majors group, more subjects preferred using paper and pen (60% or six out of ten) than using the Tablet (40% or four out of ten); and hence, with a smaller number of subjects in the group, interpretation of group differences must be made with extreme caution. Similarly, such contrasting pattern was also found in terms of design tool preference between subjects with different design experience – in the CS/SE design experience group, more subjects preferred using pen and paper (46.7% or seven out of fifteen) than using the Tablet (40%, six out of fifteen); and in contrast, more subjects with none-to-some non-CS/SE design experience preferred using the Tablet (60%, nine out of fifteen) than using paper and pen ( 40% or six out of fifteen). With respect to no particular preference for either of the two design tools used in the experiment, not much could be concluded as there were only two subjects responding this way, and both with CS/SE major and design experience.

3.5.2. Design tool preference in the experiment

Subjects’ design tool preference in the “real world” was also examined for comparisons with design tool preference during the experiment – the similarities and differences. Figure 30 shows that overall, most participants expressed that he/she would prefer using pen and paper first then move on to using computer if they were in real life design situations. On the other hand, many participants preferred using computers (with other popular applications such as Photoshop, Visual Basic.Net). The number of subjects expressing other preferences was similarly low. See Appendix U for the detailed responses from each subject.

Figure 30. Bar graph showing subjects’ design tool preference in real life design situations

Factors that were likely to affect design tool preference in real life design situations were briefly examined, including study major/specialization (see Figure 31a) and design experience (see Figure 31b) – also refer to Appendix U for indication of subjects’ background along with the detailed responses.

Visual inspection of the Figure 31a revealed that subjects who majored in CS/SE had a wider variety of preferences for a single and/or a combination of design tools, compared to subjects with non-CS/SE majors. There was also a greater proportion of subjects with non-CS/SE major preferring paper and pen than subjects with a CS/SE major. Interestingly, no subjects with non-CS/SE major preferred using tablet (Inkit) if they were doing design in the real world, compared to a small number of subjects with CS/SE major who would prefer (choose) tablet as their design tool. When subjects were examined according to their design experience, a different pattern of preference was found – as shown in Figure 31b – there was a distinct preference for particular design tool(s) between subjects with CS/SE design experience and subjects with none to some non-CS/SE design experience. The majority of subjects with CS/SE design experience preferred using computer (that has other design tools such as VB.net and Photoshop) as opposed to the majority of subjects with none to some non-CS/SE design experience preferring the use pen and paper first, then computer (software). Furthermore, the use of pen and paper, then computer was the second most popular preference in the CS/SE design experience group; however, as the number of subjects in the group with none to some non-CS/SE design experience, no such statement could be made.

Figure 31a. Bar graph showing the proportion of subjects – according to study major: CS/SE (n=20) and non-CS/SE majors (n=10) – preferring different design tools in real life design situations.

Figure 31b. Bar graph showing the proportion of subjects – according to study major: CS/SE design experience (n=15) and none to some non-CS/SE design experience (n=10) – preferring different design tools in real life design situations.

In the following section, the mean total number of changes is referred to as “total changes”; the mean number of quality changes made is referred to as “quality changes”; and the mean number of expected change made is referred to as “expected changes”.

Overall, there was an effect of formality on the number of changes made (in terms of total, quality and expected changes), and more specifically, the number of changes made decreased as the designs’ level of formality increased; hence, a negative relationship between formality and design performance. This suggests that formality plays an important role in affecting the subjects’ design performance; especially on decisions on making changes to improve the designs presented (in terms of functionality and usability).

Field’s (2004) findings showed that aesthetics play an important role in problem solving, which further support the findings of this study that formality (as a kind of aesthetics, a strong emotional response) play an important role design performance (as problem solving performance). Furthermore, as design can be seen as a kind of problem solving by many (e.g. Goldschmidt, 1997; Smith and Browne 1993; Thomas & Caroll, 1979), the underlying visual mechanisms, for example, maybe visual attention in diagram-based problem-solving (e.g. Grant & Spivey, 2003) may help explain why designers make more changes to a design that is hand-drawn and appear rough and sketchy (lower levels of formality), compared to tidier designs, that appear more precise, polished and formal (higher levels of formality) – maybe more visual attention is required to first understand and then work on a design with lower levels of formality than designs with higher levels of formality. Although no research directly supported this claim (question), Grant and Spivey (2003) measured eye movements in problem solving, suggesting that visual attention was an important factor in problem solving and highly correlated to the frequency of correct solutions; in this case, visual attention may play a role in design (problem-solving) performance, in terms of improving a design by making changes. A factor that affects visual attention is perceptual grouping of elements (an aspect of Gestalt psychology – Koffa, 1935) – in the context of the present study, in the low formality designs, no elements are aligned exactly, therefore more elements are scattered, thus, more attention and eye fixations (and time) are needed to process the initial design presented; however, in high formality designs, more elements are aligned (i.e. perceptually grouped), which allows easier and fast scanning of the design, hence, less attention on details. In addition, in higher formality designs, lines are straightened and text is presented in standardized computer fonts, thus allowing even fast scanning and better readability compared to low formality designs with squiggly lines and handwriting, which maybe harder to follow, thus requires even more time and attention is to process the initial design presented; thus requires more attention on details.

Such difference in attention required to process a design presented may be useful to explain the effects of formality on design performance – as the level of formality of increases, the design may become easier to process (e.g. reading, scanning) as less attention is required on details, thus, maybe fewer errors are noticed; and as for low formality designs, as more detailed attention is required, more errors maybe noticed during the course of processing of the design. Hence, design performance – number of changes made (total, quality and expected changes) to improve the design presented – may be affected in such way.

There are other alternative explanations for design performance across levels of formality. Skeptically, it can be argued that the effect of formality on the design process, and more specifically on the number of changes made, was simply due to the experimenter’s subconscious bias during the course of designing of the five online forms to be presented to the participants. It was possible that the higher formality designs were subconsciously designed as being more “correct” than designs of lower formalities – such effect could be seen as one of the limitations of the study. However, optimistically, it is unlikely that results from the study was purely due to experimental bias as there were statistically significant results and noticeable trends shown in the graphs including significant negative linear trend as formality increases in total changes, quality changes and expected changes decreases; plus differences in terms of number of changes made between designs with different levels of formality. Furthermore, as the independent variable (formality) was carefully manipulated and controlled for, and that the number of planned deliberate “errors” to be corrected was the same in each design, the analysis of expected changes enabled a controlled and systematic way of examining the differences in subjects’ performance across levels of formality; and indeed, significant formality effect and a linear trend were found.

On top of this, it is also debatable whether the subjects noticed more errors in the low formality designs than higher formality designs (effects of formality); or whether the errors in the low formality designs were just easier to detect or easier to improve than errors in the high formality designs – in other words, varying difficulty in improving the five designs (an experimental confound). The latter of the two arguments, can be viewed as one of the uncontrollable limitations in the study even though all objective measures have been taken to make each design as similar as possible. It is inevitable that perception of task difficulty is subjective and personal for each individual, as such variable is dependent on personal experience, skills and ability – according to Spector (2006), in the context of Industrial and organizational psychology, work/task performance is dependent on such factors. It becomes the question of interaction rather than causation when it comes to debating about the effects of formality on errors being noticed in the designs presented – results could be interpreted as: 1) formality affecting subjects’ visual attention on a design, for example, more eye fixations in stimulus that appeared more complex (Grant & Spivey, 2003) which could be the case with the lower formality designs as it was scattered and less clustered and aligned as the high formality design). From aesthetics approach in art (see Levinson, 2003 for a fuller account and discussion), it may be that the more formal (or aesthetically pleasing) a design appears, the fewer errors one may notice, as beauty may have ‘blinded’ the eyes (and the mind).