Additional Analysis of performance data

3.3.1. Paired comparisons: Total, Quality and Expected changes

One-way ANOVA was conducted to compare the three variables (the number of total, quality and expected changes) across the five conditions, and also to check for internal validity – whether total number of changes and the number of quality changes are comparable with the number of expected changes across the five conditions and that the designs presented to the participants were valid across all five conditions.

Post-hoc paired comparisons were conducted after finding significant differences, from the one-way ANOVA results (p < .003), between total changes, quality changes and expected changes across all levels of formality (see Appendix N). Post-hoc paired comparisons revealed that there was no significant difference between total changes and quality changes at: low formality (paper), low formality (on Tablet PC), medium-low formality and medium-high formality; however, there was a significant difference at high formality (mean differences = 2.26, p = .04). Between quality changes and expected changes, there was no significant differences at: low formality (paper), low formality (on Tablet PC), medium-high formality and high formality; however, there was a significant difference at medium-low formality (mean difference = 2.68). Furthermore, the difference between total changes and expected changes was significant at every level of formality. Figure 23 below shows the total, quality and expected changes made across levels of formality. Overall, the numbers in total, quality and expected changes were different across levels of formality, as can be seen in Figure 22, but relatively comparable in terms of the general negative linear trend and the of number of changes made.

Figure 23. Multi-line graph of mean total changes, mean quality changes and mean expected changes across levels of formality.

After statistical analysis of the three types of changes (total, quality and expected), it was of interest to re-visit the data to count all other changes made to explore the different types of changes made in addition to the expected changes. Therefore, the number of extra changes made, in addition to expected changes (correction of deliberate “errors”), that were of quality (i.e. changes that improved the design) was recorded. Changes that were not expected, nor were considered quality changes were also recorded i.e. “other” changes (total – quality). Each extra change found was grouped according to its type – similar to the main types of functional changes (add, delete, change element type, resize, relocation (refer to Figure 2 in the method section); but for the purpose of such investigation, the “change of element” category was separated into two categories: change of text in labels; and change of element type. Table 23 below shows the number of individual quality changes made (quality – expected changes) in each category of change, in each design.

Category of change	Low formality (on paper)	Low formality	Medium-low formality	Medium-high formality	High formality
Change of text in labels	9	16	11	4	2.5
change of element type	5	3.5	6.5	4	2.5
adding elements	15.5	15.5	9	11.5	10.5
deleting elements	1	1	0.5	1	0
relocation of elements	4	4	5	3	1
resizing of elements	1	3.5	2.5	1	2.5
miscellaneous changes	0	1	1	1	0

Similarly, Table 24 below shows the number of “other” individual changes made (those did not count as quality changes or expected changes i.e. total – quality changes) in each category of change, in each design. For more details on the extra individual changes made in each design (quality – expected; total – quality) and the number of participants making such changes, see Appendix O1, O2, O3, O4 and O5 (from low formality to high formality designs).

	Low formality (on paper)	Low formality	Medium-low formality	Medium-high formality	High formality
Change of text in labels	23	26	16	6	9
change of element type	13	7	9	12	7
adding elements	29	23	11	20	22
deleting elements	8	6	4	5	4
relocation of elements	32	18	16	13	10
resizing of elements	2	3	5	1	7
miscellaneous changes	0	1	1	1	1