Beautification

A post-task questionnaire (see Appendix B) was used for recording participants’ demographic information such as age, gender, education level, programme studied, papers taken, occupation, and design experience. Preference for design tools (pen and paper verses the tablet PC) during the design tasks and in real-world design situation, as well as participant’s “overall enjoyment” when working on each design in comparison to another (by ranking from 1 to 5, from the most-liked design to the least-liked design; and the reasons for the rankings) were also recorded in the questionnaire. Such information was used to explore whether performance – in this case, design decisions to improve a design at different levels of formality, was affected by factors such as design experience (e.g. Cross, 2004,; Kavakli & Gero, 2002), study major/specialization and study level (e.g. Atman, et al., 2005; Atman, et al., 1999) and design medium preference (e.g. Bailey & Konstan, 2003; Black, 1990; Hann & Barber, 2001; Newman, et al., 2003), during the design process.

Chapter 3. Results
For the purpose of analysis, design performance was measured in terms of number of (functional) changes made. In other words, one dependent variable with three levels (the total number of changes, quality changes and expected changes) was measured at each level of the independent variable (formality). Subjective measures included overall enjoyment rankings of designs, design tool preference. Data were analyzed using SPSS for Windows version 14.0 (SPSS Inc.). Analysis of variance (ANOVA) with repeated measures and unplanned pair-wise comparisons were conducted to analyze the effects of formality on outcome measures (number of changes made). Between-subject effects were also analyzed. Friedman’s rank test for several related samples were conducted to analyze subjective measures including rankings of designs.
3.1. Data-screening of performance data

In order to test whether the data satisfied the normality assumptions for a parametric repeated samples t-test (see Cohen, 1988), histograms, homogeneity of variance, the skewness and kurtosis statistics, and normality tests and plots for the scores, as well as assumptions of sphericity were examined. For all normality information on total changes, quality changes and expected changes, see Appendixes H, I and J respectively.

The histograms with normal curves were created for the mean scores of each of the three dependent measures: 1) total changes (see Appendix H), 2) quality changes (see Appendix I) and 3) expected changes (see Appendix J), across all five levels formality. Initial visual inspection showed roughly normal distributions and a few slightly skewed distributions. However, it was inadequate to conclude that the distribution was non-normal from the skewed data from a small sample of n = 30 (Cohen, 1988). Thus, all data were also plotted against the standardized version of the data and showed that the scores were normally distributed at each level of formality i.e. a roughly linear relationships (see normal Q-Q plots for level of formality in Appendix H, I and J).

The comparison of variance between levels of formality in each category of change made, indicated that the scores were similar enough (Coakes & Steed, 2001), therefore the homogeneity of variance assumption for each measure group (DV) was not violated. See variance across level of formality in Appendix H, I and J.

The 95% confidence interval around the skewness and kurtosis scores of neither cell included zero indicating that the scores were not normally distributed, with data skewed both ways in positive and negative directions, with negative and positive values for kurtosis indicated leptokurtic and platykurtic distribution (Heimanz, 2001). However, looking at the values for skewness and kurtosis in each level of formality, the absolute values of skewness and kurtosis statistics were almost always smaller than the standard errors, indicating that the skewness and kurtosis were comparable with the zero value in a normal distribution.

In order to examine the relationships between the skewness, kurtosis and variance, the Kolomogorov-Smirnov (KS) and Shapiro-Wilk (SW) tests of normality were both conducted (see the test of normality in Appendix H, I and J). KS tests (with Lilliefors Significance Correction) assessed the kurtosis and skewness of each data group, demonstrating that data were suitable for parametric testing (i.e. normally distributed) as KS statistics at each level of formality showed no significance (p > .50). SW, calculated if sample size is less than fifty (Coakes & Steed, 2001), also showed similar values, further indicating that the data did not violate the normality assumption of parametric tests.

Moreover, there were no missing data, and there were no outliers except for one in the low formality scores (participant 10) in the quality changes data group. However, this outlier was included in the analysis as it fell within the upper quartile range in the other levels of formality (see Figure 5 in the analysis section for quality changes for the box plot of quality changes made at each level of formality).

Overall, therefore, the data were reasonably normally distributed, hence, normality assumptions not violated and it was justifiable to use parametric analysis.

One-way ANOVA with repeated measures were conducted to examine each dependent variable (total change, quality changes and expected changes) under each independent variable (level of formality). An alpha level of 0.5 was used for all statistical tests.

Before analyzing one-way ANOVA with repeated measures for each type of change made across levels of formality, sphericity had to be examined. According to Brace, Kemp and Snelgar (2006), the assumption of sphericity is that the correlations between all of variables are roughly the same – in other words, the null hypothesis in the sphericity assumption is that the correlations among the number of changes made in each level of formality are equal. Moreover, Tabachnick and Fidell (2001) suggested when there are more than two levels of IV (in this case, formality) the test of sphericity must be conducted to decide which test should be used to interpret significance. Therefore, the Mauchly’s test of sphericity was conducted on each data group. The test was significant for total change (Approx. Chi-square = 24.644, p < .05), indicating that the sphericity assumption was violated. On the other hand, the test yielded no significance in both quality changes (Approx. Chi-square = 10.004, p = .351) and expected changes (Approx. Chi-square = 5.170, p = .820), meaning that the null hypothesis of sphericity was accepted. Hence, the assumptions of sphericity were met and the normal within-subjects ANOVA was not violated for both data groups.

Table 5 shows mean and standard deviation of total changes made at each level of formality.

	Mean	Std. Deviation
1. Low formality (on paper)	18.73	6.57
2. Low formality (on tablet PC)	15.17	4.14
3. Medium-low formality	14.00	4.07
4. Medium-high formality	13.13	3.86
5. High formality	11.27	3.51

Since the Mauchly’s test of sphericity (for the data group of total change) was significant, an alternate, multivariate approach was adopted (Brace, Kemp & Snelgar, 2006). The results for the ANOVA indicated a significant main effect of formality on the total changes made to the designs, Wilk’s Lambda = .265, F (4.26) = 17.99, p < .001, multivariate partial η² = .74. A strong significant linear trend was also found, F (1, 29) = 59.59, p < .001, partial η² = .67, over the mean value of total changes made at each level of formality (illustrated in Figure 11). A weaker but significant cubic trend was also found, F (1, 29) = 8.529, p < .01, partial η² = .23, suggesting that overall, the number of total changes made were the highest when participants were presented with the low-formality design on the paper, and decreased as formality increased. Subjects’ performance was slightly unstable, however, suggesting that order effects (discussed later) may have played a role in contributing to the combination of linear and cubic trends.

Figure 11. Multi-line graph showing mean total changes made across levels of formality which is represented by the black bold line; each participant’s performance (in terms of total changes made across levels of formality) is also illustrated – see individual lines.

Pair-wise comparisons (with Bonferroni adjustment for multiple comparisons) showed that the total number of changes made was significantly lower when participants were presented with the high formality design, compared to other designs with lower levels of formality: medium-high formality; medium-low formality; low formality on the tablet PC and low formality on paper. Differences increased as the level of formality decreased, as shown in Table 6. On the other hand, the total number of changes made in the low formality design presented on paper was significantly higher than other levels of formality presented on the Tablet PC: low formality on the Tablet PC; medium-low formality; medium-high formality and high formality. Differences increased as the level of formality increased, also shown in Table 6. Interestingly, even though there were two low formality conditions, one presented on paper and one presented on the tablet, the total number of changes made still differed significantly between these conditions – the mean difference was 3.57 as can be seen in Table 6. This was also shown in Figure 11 where the mean number of total changes made was much greater when made on paper than on the Tablet PC. Furthermore, as shown in Table 6, no significant difference was found, in terms of mean total changes, between medium-high formality and medium-low formality; and between medium-low formality and low formality on the Tablet PC. However, the total number of changes made at low formality was significantly higher than at medium-high formality.

(I) Factor 1	(J) Factor 1	Mean Difference (I-J)
Low formality (on paper)	Low formality (on Tablet PC)	3.57*
	Medium-low formality	4.73*
	Medium-high formality	5.60*
	High formality	7.47*
Low formality (on Tablet PC)	Low formality (on paper)	-3.57*
	Medium-low formality	1.17
	Medium-high formality	2.03*
	High formality	3.90*
Medium-low formality	Low formality (on paper)	-4.73*
	Low formality (on Tablet PC)	-1.17
	Medium-high formality	0.87
	High formality	2.73*
Medium-high formality	Low formality (on paper)	-5.60*
	Low formality (on Tablet PC)	-2.03*
	Medium-low formality	-0.87
	High formality	1.87*
High formality	Low formality (on paper)	-7.47*
	Low formality (on Tablet PC)	-3.90*
	Medium-low formality	-2.73*
	Medium-high formality	-1.87*