Analysis of “Quality Changes” made across levels of formality

Table 11 shows mean and standard deviation of quality changes made at each level of formality.

Table 11

Formality level	Mean	Std. Deviation
1. Low formality (paper)	15.73	5.50
2. Low formality (on tablet PC)	13.05	4.01
3. Medium-low formality	12.90	3.81
4. Medium-high formality	10.80	3.92
5. High formality	9.02	3.54

Since the Mauchy’s test of sphericity was not significant, the traditional test of within-subjects effect was conducted and the results from ANOVA showed that there was a significant main effect of formality on the number of quality changes made, F (4, 116) = 31.763, p < .001, partial η² = .48. A significant linear trend was also found, F (1, 29) = 76.91, p < .001, partial η² = .73, over the mean quality changes at each level of formality, indicating that subjects made most quality changes in the low formality design on paper, followed by low formality design on the Tablet PC, and the numbers dropped as formality increased – see Figure 15; and also refer to the bold line for the means across levels of formality. However, no significant quadratic, cubic nor order 4 trends were found.

Figure 15. Multi-line graph showing mean quality changes made across levels of formality which is represented by the black bold line. Each participant’s performance (in terms of quality changes made across levels of formality) is also illustrated – see individual lines.

As noted before, due to the lack of previous empirical research, unplanned pair-wise comparisons were conducted to examine the difference in the mean quality changes between levels of formality.

Pair-wise comparisons (with Bonferroni adjustment for multiple comparisons) showed that the number of quality changes made was significantly lower when participants were presented with the high formality design, compared to the designs with lower levels of formality: medium-high formality; medium-low formality; low formality on the tablet PC and low formality on paper. Difference increased as the level of formality decreased, as shown in Table 12. On the other hand, the mean number of quality changes made in the low formality design presented on paper was significantly higher than all other levels of formality presented on the Tablet PC: low formality on the Tablet PC; medium-low formality; medium-high formality and high formality. Difference increased as the level of formality increased, also shown in Table 6. The increasing and decreasing differences further emphasized the significant linear trend found. Furthermore, the mean number of quality changes made was different between each level of formality, except between medium-low formality and low formality on the Tablet PC (mean difference = .33) –illustrated in Figure 15. It was also interesting to note that, similar to the data for total changes, even though there were two low formality conditions, one presented on paper and one presented on the tablet, the number of quality changes made still differed significantly between these conditions – the mean difference was 2.68 as can be seen in Table 12. This was also shown in Figure 15 where the mean number of quality changes made was higher when made on paper than on the Tablet PC.

(I) Factor 1	(J) Factor 1	Mean Difference (I-J)
Low formality (on paper)	Low formality (on Tablet PC)	2.68*
	Medium-low formality	2.83*
	Medium-high formality	4.93*
	High formality	6.72*
Low formality (on Tablet PC)	Low formality (on paper)	-2.68*
	Medium-low formality	0.15
	Medium-high formality	2.25*
	High formality	4.03*
Medium-low formality	Low formality (on paper)	-2.83*
	Low formality (on Tablet PC)	-0.15
	Medium-high formality	2.10*
	High formality	3.88*
Medium-high formality	Low formality (on paper)	-4.93*
	Low formality (on Tablet PC)	-2.25*
	Medium-low formality	-2.10*
	High formality	1.78*
High formality	Low formality (on paper)	-6.72*
	Low formality (on Tablet PC)	-4.03*
	Medium-low formality	-3.88*
	Medium-high formality	-1.78*

In order to examine whether other factors affected the quality changes made at each level of formality, between subject effects including design experience, study level and major/specialization were explored. Furthermore, each between-subject factor had only two levels therefore no post-hoc tests had been conducted.

Subjects’ design experience was examined first as it was hypothesized that there will be a difference in the number of quality changes made across levels of formality between subjects who had more or less design experience. Thus, subjects were categorized into two groups: 1) subjects with no experience or some non-computer science/software engineering design experience (n = 15); and 2) subjects with computer science (CS) / software engineering (SE) design experience (n = 15). Table 13 shows mean and standard deviation of quality changes made at each level of formality according to subjects’ design experience.

Results from ANOVA with design experience as the between-subject factor showed that there was a significant formality-by-design experience interaction effect, F (4, 112) = 4.07, p < .005, partial η² = .13, along with the significant between-subject effects of design experience, F (1, 28) = 7.31, p < .02, partial η² = .21, on the number of quality changes made across levels of formality. This indicated that the quality changes made across levels of formality differed between subjects with no experience or some non-CS/SE design experience and subjects with CS/SE design experience. More specifically, subjects with CS/SE design experience made consistently more quality changes across levels of formality compared to subjects with none-to-some (non-CS/SE) experience (see Figure 16).

Figure 16. Multi-line graph of mean quality changes made across levels of formality according to subjects’ design experience: none to some (non-CS/SE) design experience and CS/SE design experience

Two other between-subjects factors – major/specialization and study level, were explored primarily through visual inspection of multi-line graphs due to various reasons: the number of subjects in each group could not be balanced; there were overlapping of subject factors, i.e. explicit, isolative (i.e. nested) grouping of subjects was near impossible in the current study as major/specialization, study level and design experience were all intimately-correlated, and even if it was possible, a much larger sample was needed – therefore subjects were grouped according to one factor only.

Since the experimental task involved HTML (web) form design, it was of interest to see whether quality changes made at each level of formality differed between subjects who had more or less HTML knowledge. Therefore subjects were grouped into two groups: 1) subjects with a non-CS/SE related major (n = 10); and 2) subjects with a CS/SE major (n = 20). Table 14 shows the mean and standard deviation of quality changes made in each group across levels of formality.

Table 14

Mean and standard deviation for quality changes made, and the mean difference between groups, at each level of formality according to subjects’ major/specialization in Auckland University: Non-CS/SE related major (n = 10) and CS/SE related majors (n = 20)

	Major/Specialization
	(X) Non-CS/SE related major		(Y) CS/SE related major
	Mean	Std. Deviation	Mean	Std. Deviation	Mean Difference (Y-X)
1. Low formality (paper)	13.10	5.18	17.05	5.29	3.95
2. Low formality (tablet)	11.40	3.21	13.88	4.19	2.48
3. Medium-low formality	13.30	4.70	12.70	3.40	- 0.60
4. Medium-high formality	10.25	3.76	11.08	4.07	0.83
5. High formality	8.30	4.24	9.38	3.19	1.08

Results from ANOVA with study major/specialization as the between-subject factor indicated that there was a significant formality-by-major interaction, F (4, 112) = 3.00, p < .025, partial η² = .10, and weak significant trends including a linear trend, F (1, 28) = 4.23, p < .05, partial η² = .13, a quadratic trend, F (1, 28) = 4.76, p < .05, partial η² = .15, on the number of quality changes made across levels of formality. However, no between-subjects effects were found. Figure 17 highlighted the significant statistics and showed that there was a strong linear trend of quality changes made across levels of formality and a rapid increase at the low formality (on paper) in the CS/SE major group; where as, in the non-CS/SE major group, there was a weaker linear trend with one non-linear point with respect to other points. Interestingly, at medium-low formality subjects performed at the same level – the mean quality changes made was similar in the CS/SE major group and the non-CS/SE major group. Overall, subjects who majored in CS/SE made more quality changes than subjects who majored in non-CS/SE areas of study. Between-group difference (see mean differences in Table 14) was the largest at low formality presented on paper (mean difference = 3.78) and decreased at low formality present on the Tablet PC (mean difference = 2.05). Next, as formality level increased, the between-group differences decreased – the gap between the two lines was smaller at the higher levels of formality compared to lower levels of formality which further suggested that there was some interaction (illustrated in Figure 17).

Figure 17. Multi-line graph of mean quality changes made across levels of formality according to subjects’ major/specialization in university: Non-CS/SE related major and CS/SE related majors

As study level may have also played a role in producing particular trends among groups, subjects were classified into two groups: 1) undergraduates (n = 22); and 2) graduates/post-graduates (n = 8). Table 15 shows mean and standard deviation of quality changes made in each group across levels of formality.

Table 15

Although no significant formality-by-study level interaction was found from the ANOVA results with study level as the between-subject factor, the between-subjects tests was significant, F (1, 28) = 6.87, p < .015, partial η² = .20, suggesting that number of quality changes made at each level of formality between undergraduates and graduates/post-graduates differed significantly. Figure 18 shows that the number of quality changes made by graduates/postgraduates at each level of formality was significantly higher compared to undergraduates. Visual inspection of Figure 18 also suggested that there was a roughly linear trend in both the undergraduate and graduate/postgraduate. However, trend tests showed that there was a no linear but a significant cubic trend, F (1, 28) = 6.34, p < .02, partial η² = .19, demonstrated in Figure 18 where points of inflection are noticeable e.g. quality changes made by: undergraduate subjects at medium-low formality; and graduate/postgraduate subjects at medium-high formality.

Figure 18. Multi-line graph of mean quality changes made across levels of formality according to subjects’ study level: undergraduate and graduate/postgraduate