Beautification

In order to examine whether other factors affected the total changes made at each level of formality, between subject effects including design experience, major/specialization and study level were explored. Furthermore, each between-subject factor had only two levels therefore no post-hoc tests were necessary.

Subjects’ design experience was examined first as it was hypothesized that there would be a difference in the total number of changes made across levels of formality between subjects who had more or less design experience. The subjects were categorized into two groups: 1) subjects with no experience or some non-computer science/software engineering design experience (n = 15); and 2) subjects with computer science (CS) / software engineering (SE) design experience (n = 15). Table 7 shows mean and standard deviation of total changes made at each level of formality according to subjects’ design experience.

Results from the ANOVA with design experience as the between-subject factor showed that there was a significant formality-by-design experience interaction effect, F (4, 112) = 6.24, p < .001, partial η² = .18, and a significant between-subject effect of design experience, F (1, 28) = 8.49, p < .01, partial η² = .23, on the total changes made across levels of formality. This indicated that the total changes made across levels of formality differed between subjects with no experience to some non-CS/SE design experience and subjects with CS/SE design experience – and more specifically, subjects with CS/SE design experience made consistently more changes across levels of formality compared to subjects with no experience or some non-CS/SE experience (see Figure 12).

Figure 12. Multi-line graph of mean total changes made across levels of formality according to subjects’ design experience: none to some (non-CS/SE) design experience and CS/SE design experience

Two other between-subject factors – major/specialization and study level, were explored mainly through visual inspection due to various reasons: the number of subjects in each group could not be balanced; there was overlapping of subject factors, i.e. explicit, isolative (i.e. nested) grouping of subjects was near impossible in the current study as major/specialization, study level and design experience were all intimately-correlated, and even if it were possible, a much larger sample would have been needed – therefore subjects were grouped according to one factor only, and grouped data was examined through multi-line graphs.

Since the experimental task involved HTML (web) form design, it was of interest to see whether total changes made across levels of formality differed between subjects who had more or less HTML knowledge. Therefore to explore such between-subject effect, subjects were grouped into two groups: 1) subjects with a non-CS/SE related major (n = 10); and 2) subjects with a CS/SE major (n = 20). Table 8 below shows mean and standard deviation of total changes made in each group across levels of formality.

Results from the one way ANOVA with study major as the between-subject factor showed that there was a significant formality-by-major/specialization effect, F (4, 112) = 4.10, p < .01, partial η² = .13, as well as significant formality-by-major/specialization trends: linear trend, F (1, 28) = 5.09, p = .032, partial η² = .15, and quadratic trend, F (1, 28) = 5.37, p < .028, partial η² = .16; however, no significant between-subject effect was found (illustrated in Figure 13). Visual inspection of Figure 13 suggested that there was linear trend across levels of formality, and the total changes made increased more rapidly at the lower formalities in the CS/SE major group, while the other group showed a less consistent linear trend. Interestingly, at medium-low formality subjects performed at the same level – the total changes made was similar in the CS/SE major group and the non-CS/SE major group. This could also explain the non-significant results from the between-subject effects tests. Overall, the between-groups difference decreased as the formality level increased (see mean differences in Table 8) – the gap between the two lines was smaller at the higher levels of formality compared to lower levels of formality, as explained by the significant formality-by-major/specialization interaction.

Figure 13. Multi-line graph of mean total changes made across levels of formality according to subjects’ major/specialization in university: Non-CS/SE related major and CS/SE related majors

As study level may have also played a role in producing particular trends among different groups, subjects were classified into two groups: 1) undergraduates (n = 22); and 2) graduates/post-graduates (n = 8). Table 9 below shows mean and standard deviation of total changes made in each group across levels of formality.

No significant statistics, such as formality-by-study level effect and trends, were found from the results of ANOVA with study level as the between-subject factor, except for between-subjects effects, F (1, 28) = 6.10, p = 0.02, partial η² = .18. However, examining statistics alone was not conclusion there was an unbalanced number of subjects in each group. Visual inspection of Figure 14 suggested that there was a strong linear trend in the undergraduate group, while the graduate/postgraduate group showed a weaker linear trend with a slight increase in the mean total change at medium-high formality, after the medium-low formality condition. It was also visible that the total changes made increased rapidly at low formality (on paper) in both groups. Also, the between-subject differences appeared to be greater nearer the two ends of the formality spectrum: low formality on paper (mean group difference = 5.55) and low formality on the Tablet PC (mean group difference = 3.49); and medium-high formality (mean group difference = 4.43) and high formality (mean group difference = 3.30); and the smallest between-group difference at medium-low formality (mean group difference = 1.33) – see mean differences in Table 9. This also suggested that there was some formality-by-study level interaction. Although differing in magnitude, overall, a rough linear trend was visible for both groups – as formality increased (decreased), total changes made decreased (increased).

Figure 14. Multi-line graph of mean total changes made across levels of formality according to subjects study level: undergraduate and graduate/postgraduate.

The similar trends with the three factors further suggested that they were closely related. The data set was re-grouped according to a combination of design experience, study level and major/specialization (see Appendix K), and multiple regression analysis was then conducted to examine and separate individual effects that contributed to the overall effect of formality on the total changes made. In other words, these analyses sought to discover how much each between-subject factor helped explain the effect of formality on the total changes made.

Formality and the three between-subjects variables (design experience and study level, and major/specialization) were entered one after the other respectively into SPSS. Before looking at the actual results, in addition to the data screening earlier for normality and outliers, multicollinearity was first examined. According to Brace et al. (2006), the closer to zero the tolerance value is for a variable (vary between 0 to 1), the stronger the relationship between this and the other predictor variables; and the higher the VIF value (value from 1.0), the stronger relationship is between predictor variables; and such values becomes a worry. However, results indicated high tolerance values (over .90), and low VIF values (less than 1.08), therefore there was no multicollinearity issues.

Using the stepwise method, a significant model which included formality, design experience and study level, emerged, F (3, 31) = 31.67, p < .0001. The model explained 73%% of the variance (Adjusted R² = .730). Table 10.1 shows the adjusted R square and change statistics of each predictor when added to the model. Formality level (model 1) accounted for 36.1% of the variance (Adjusted R² = .361, p <.0001), and the inclusion of design experience in model 2 resulted in an additional 30.1% of the variance being explained (R² change = .301, F (1, 32) = 30.13, p < .0001). Study level helped explained a further 7.4% of the variance when added upon formality and design experience (R² change = .074, F (1, 31) = 9.95, p = .005). However, study major/specialization was excluded from the model as it did not have a significant impact when added (R² change = .00, F (1, 30) = .003, p = .96) – hence, not a good predictor to explain total changes made across levels of formality.

Model	R	Adjusted R Square	Std. Error of the Estimate	Change Statistics
				R Square Change	F Change	Sig. F Change
1	.616(a)	.361	2.27	.379	20.18	.000
2	.825(b)	.660	1.65	.301	30.13	.000
3	.868(c)	.730	1.47	.074	9.27	.005
4	.868(d)	.721	1.50	.000	.003	.960

b Predictors: (Constant), Formality Level, Design experience

c Predictors: (Constant), Formality Level, Design experience, Study level

d Predictors: (Constant), Formality Level, Design experience, Study level, Major/specialization

Table 10.2

	B	Std Error B	β	t
Formality	-1.217	.176	- .616**	-6.92
Design experience	2.837	.510	.503**	5.57
Study Level	1.553	.510	.275*	3.05