Multiple Regression Analysis

The similar trends with the three factors further suggested that they were closely related. The data set was re-grouped according to a combination of design experience, study level and major/specialization (see Appendix L), and multiple regression analysis was then conducted to examine and separate individual effects that contributed to the overall effect of formality on the quality changes made. In other words, the analyses sought to discover how much each between-subject factor helped explain the effect of formality on the number of quality changes made.

Formality and the three between-subjects variables (design experience and study level, and major/specialization) were entered one after the other respectively into SPSS. Before looking at the actual results, in addition to the data screening earlier for normality and outliers, multicollinearity was first examined. According to Brace et al. (2006), the closer to zero the tolerance value is for a variable (vary between 0 to 1), the stronger the relationship between this and the other predictor variables; and the higher the VIF value (value from 1.0), the stronger relationship is between predictor variables; and such values becomes a worry. However, results indicated high tolerance values (over .90), and low VIF values (less than 1.08), therefore there was no multicollinearity issues.
Using the stepwise method, a significant model which included formality, design experience and study level, emerged, F (3, 31) = 27.15, p < .0001. The model explained 69.8% of the variance (Adjusted R² = .698). Table 16.1 shows the adjusted R square and change statistics of each predictor when added to the model. Formality level (model 1) accounted for 37% of the variance (Adjusted R² = .370, p <.0001), and the inclusion of design experience in model 2 resulted in an additional 24.7% of the variance being explained (R² change = .247, F (1, 32) = 21.75, p < .0001). Study level helped explained a further 7.4% of the variance when added upon formality and design experience (R² change = .074, F (1, 31) = 9.95, p = .005). However, study major/specialization was excluded from the model as it did not have a significant impact when added (R² change = .005, F (1, 30) = .54, p = .47) – hence, not a good predictor to help explain the number of quality changes made across levels of formality.
Table 16.1

Adjusted R Square and R Square change

Model	R	Adjusted R Square	Std. Error of the Estimate	Change Statistics
				R Square Change	F Change	Sig. F Change
1	.623(a)	.370	2.74703	.388	20.960	.000
2	.797(b)	.613	2.15246	.247	21.749	.000
3	.851(c)	.698	1.90283	.088	9.947	.004
4	.854(d)	.693	1.91710	.005	.540	.468

b Predictors: (Constant), Formality Level, Design experience

c Predictors: (Constant), Formality Level, Design experience, Study level

d Predictors: (Constant), Formality Level, Design experience, Study level, Major/specialization

Table 16.2 gives information for the predictor variables (formality and between-subject variables) included in the significant model. The result suggests that formality alone (the manipulated variable) has a strong significant impact on the total number of changes made (β = -.62, t = -6.61, p < .0001). The negative statistics further suggests that as formality level increases, the number of quality changes made decreases. The results for design experience (β = .45, t = 4.68, p < .0001) and study level (β = .30, t = 3.15, p < .005) further indicates that on top of the effects of formality on quality changes made – people with more design experience and/or at a high level of study (e.g. graduates) are more likely to make greater number of quality changes than those with less design experience and/or at a lower level of study (e.g. undergraduates).
Table 16.2

The unstandardized and standardized regression coefficients, and the t-value and significance of each between-subject variables included in the model.

	B	Std Error B	β	t
Formality	-1.50	.227	-.623	-6.61
Design experience	3.08	.659	.447	4.68
Study Level	2.08	.659	.302	3.15

Table 17 below shows mean and standard deviation of expected changes made at each level of formality.

Formality level	Mean	Std. Deviation
1. Low formality (paper)	13.55	4.24
2. Low formality (on tablet PC)	11.18	3.25
3. Medium-low formality	10.22	3.34
4. Medium-high formality	9.02	3.45
5. High formality	8.00	3.30

In order to test whether formality had an effect on the number of expected changes made, one way ANOVA with repeated measures was conducted. Results showed that there was a significant main effect of formality on the number of expected changes made to the designs, F (4, 116) = 29.28, p > .001, partial η² = .50. As one could not tell how levels of formality affected the number of expected changes made, trends test were also examined. A significant linear trend was found, F (1, 29) = 92.70, p < .001, partial η² = .76, over the mean value (expected change) for each level of formality.

Figure 19. Multi-line graph showing mean expected changes across levels of formality which is represented by the black bold line. Each participant’s performance (in terms of expected changes made across levels of formality) is also illustrated – see individual lines.

Pair-wise comparisons (with Bonferroni adjustment for multiple comparisons) revealed that participants made significantly more expected changes when they were presented with the low formality paper design, compared to designs with other levels of formality presented on the Tablet PC: low formality on the Tablet PC; medium-low formality; medium-high formality and high formality. Difference increased as the level of formality increased, as shown in Table 18. On the other end, participants made significantly fewer expected changes when they were presented with the high formality design, compared to designs with medium-low formality; low formality on the tablet PC and low formality on paper. Differences increased as the level of formality decreased; however, no significant difference was found between high formality and medium-high formality. Furthermore, as shown in Table 18, no significant difference was found between medium-high formality and medium-low formality; and between medium-low formality and low formality on Tablet PC. This suggested that subjects’ performance (in terms of making expected changes) was comparable when they were presented with designs with higher formalities (high formality and medium-high formality), and similarly in designs with lower formalities on the Tablet PC (low formality and medium-low formality; plus medium-low formality and medium-high formality).

(I) Factor 1	(J) Factor 1	Mean Difference (I-J)
Low formality (on paper)	Low formality (on Tablet PC)	2.37*
	Medium-low formality	3.33*
	Medium-high formality	4.53*
	High formality	5.55*
Low formality (on Tablet PC)	Low formality (on paper)	-2.37*
	Medium-low formality	0.97
	Medium-high formality	2.17*
	High formality	3.18*
Medium-low formality	Low formality (on paper)	-3.33*
	Low formality (on Tablet PC)	-0.97
	Medium-high formality	1.20
	High formality	2.22*
Medium-high formality	Low formality (on paper)	-4.53*
	Low formality (on Tablet PC)	-2.17*
	Medium-low formality	-1.20
	High formality	1.02
High formality	Low formality (on paper)	-5.55*
	Low formality (on Tablet PC)	-3.18*
	Medium-low formality	-2.22*
	Medium-high formality	-1.02

3.2.3.1. Between-Subjects Factors

In order to examine whether other factors affected the number of expected changes a participant made, between subject effects including design experience, study level and specialization were explored.
3.2.3.1a. Design Experience

Subjects’ design experience was examined first as it was hypothesized that there will be a difference between the numbers of (expected) changes made by subjects who have more or less design experience. The subjects were grouped into two groups: 1) no experience or some non-CS/SE design experience (n = 15); and 2) CS/SE design experience (n = 15). Table 19 below shows mean and standard deviation of expected changes made at each level of formality according to subjects’ design experience.

ANOVA with design experience as the between-subject factor was conducted to examine whether there was a difference in subjects’ performance according to design experience. Results showed that in addition to the significant main effect of formality, F (4, 116) = 29.28, p > .001, there was also a significant between-subject effect, F (1, 28) = 7.64, p < .01, partial η² = .21, suggesting that subjects with CS/SE design experience made more changes across levels of formality compared to subjects with no experience or some non-CS/SE experience as can shown in Figure 20. Furthermore, a weak formality-by-design experience linear trend was found (although not strictly ‘statistically significant’ at the alpha level of .05), F (1, 28) = 4.15, p = .051, partial η² = .13. Visual inspection of Figure 20 further suggested that there was a stronger linear trend across levels of formality in subjects with CS/SE design experience than subjects with no experience or some non-CS/SE design experience – the mean number of expected changes made by subjects with no experience or some CS/SE design experience at low formality was the same in medium-low formality. However, on the whole, there was still a linear trend where subjects made fewer (more) changes as the level of formality increased (decreased). Additionally, as illustrated in Figure 20, there were magnitude differences between the two groups across levels of formality. Although, no statistically significant formality-by-design experience interaction was found, Figure 20 shows that the two lines, representing subjects with different design experience, appeared to be non-parallel, and thus, there was some interaction.

Since the experimental tasks involved HTML (web) form design, it was of interest to see whether the number of changes made across the levels of formality differed between subjects who had more or less HTML form knowledge. Therefore to explore such between-subject effects, subjects were grouped into two groups: 1) those with non-CS/SE related major (n = 10); and 2) those with CS/SE major (n = 20). Table 20 shows mean and standard deviation of expected changes made at each level of formality according to design experience.

No significant effects were found after conducting one way ANOVA with study major/specialization as the between-subject factor. However, since the number of subjects in each group was not balanced (n = 20, n = 10), statistics produced was not conclusive. Visual inspection of a multi-line graph (Figure 21) suggested that the number of expected changes made by subjects who majored/specialized in CS/SE was higher than subjects who majored in non-CS/SE subjects across the levels of formality, except at medium-low level of formality, where there was no significant mean difference (mean difference = 0.13) between the two groups – refer also to Table 20. Furthermore, there was a strong linear trend in the CS/SE major group, while the other group showed a less consistent linear trend with a sudden increase in expected changes at medium-low formality. In addition, this indicated that there was also some formality-by-major/specialization interaction as the lines were non-parallel – gaps bigger at lower formalities compared to higher formalities – illustrated in Figure 21.

Figure 21. Multi-line graph of mean expected changes made across levels of formality according to subject’s major/specialization: non-CS/SE related major and CS/SE related majors

As study level may have played a role in producing particular trends among different groups, subjects were also categorized into two groups: 1) undergraduates (n = 22); and 2) graduates/post-graduates (n = 8). Table 21 below shows mean and standard deviation of expected changes made in each group at each level of formality.

Results from the ANOVA with study level as the between-subject factor showed no statistically significant formality-by-study level interaction, but a significant between-subjects effect was found, F (1, 28) = 5.18, p < .03, partial η² = .16, suggested that the number of expected changes made across levels of formality between the undergraduates and graduates/post-graduates differed in magnitude – graduates/postgraduates made more expected changes across the levels of formality compared to undergraduates (illustrated in figure U). Although no statistically significant linear trend found, a weak significant cubic trend was detected, F (1, 28) = 4.12, p = .052 (slightly above the alpha level of .05). Visual inspection of Figure 22 further indicated that the weak cubic trend was contributed by the strong linear trend in the undergraduate group, and a less consistent linear trend in the graduate/postgraduate group with a steep dip at the low formality condition presented on the tablet PC, followed by a slight increase in the mean expected changes at medium-high formality. The two lines representing undergraduates and graduates/postgraduates also appeared to be non-parallel, suggesting that there may be some formality-by-study level interaction. On the whole, the main linear trend was still visible for both groups but differing in magnitude – as formality increased (decreased), the number of expected changes made decreased (increased).

Figure 22. Multi-line graph of mean expected changes made across levels of formality according to subjects’ study levels: undergraduate and graduate/postgraduate

The similar trends with the three factors further suggested that they were closely related. The data set was re-grouped according to a combination of design experience, study level and major/specialization (see Appendix M), and multiple regression analysis was then conducted to examine and separate individual effects that contributed to the overall effect of formality on the expected changes made. In other words, these analyses sought to discover how much each between-subject factor helped explain the effect of formality on the number of expected changes made.

Formality and the three between-subjects variables (design experience and study level, and major/specialization) were entered one after the other respectively into SPSS. Before looking at the actual results, in addition to the data screening earlier for normality and outliers, multicollinearity was first examined. According to Brace et al. (2006), the closer to zero the tolerance value is for a variable (vary between 0 to 1), the stronger the relationship between this and the other predictor variables; and the higher the VIF value (value from 1.0), the stronger relationship is between predictor variables; and such values becomes a worry. However, results indicated high tolerance values (over .90), and low VIF values (less than 1.08), therefore there was no multicollinearity issues.

Using the stepwise method, a significant model which included formality, design experience and study level, emerged, F (3, 31) = 25.15, p < .0001. The model explained 68.1% of the variance (Adjusted R² = .681). Table 22.1 shows the adjusted R square and change statistics of each predictor when added to the model. Formality level (model 1) accounted for 41% of the variance (Adjusted R² = .41, F (1, 33) = 24.60, p <.0001), and the inclusion of design experience in model 2 resulted in an additional 21.9% of the variance being explained (R² change = .219, F (1, 32) = 19.87, p < .0001). Study level helped explained a further 6.2% of the variance when added upon formality and design experience (R² change = .062, F (1, 31) = 6.62, p = .015). However, study major/specialization was excluded from the model as it did not have a significant impact when added (R² change = .019, F (1, 30) = 2.043, p = .16) – hence, not a good predictor to help explain expected changes (but a better predictor than in total and quality changes) made across levels of formality.

Model	R	Adjusted R Square	Std. Error of the Estimate	Change Statistics
				R Square Change	F Change	Sig. F Change
1	.654(a)	.410	2.87575	.427	24.602	.000
2	.804(b)	.624	2.29388	.219	19.865	.000
3	.842(c)	.681	2.11552	.062	6.623	.015
4	.853(d)	.691	2.08079	.019	2.043	.163

b Predictors: (Constant), Formality Level, Design experience

c Predictors: (Constant), Formality Level, Design experience, Study level

d Predictors: (Constant), Formality Level, Design experience, Study level, Major/specialization

Table 22.2 gives information for the predictor variables (formality and between-subject variables) included in the significant model. The result suggests that formality alone (the manipulated variable) has a strong significant impact on the total number of changes made (β = -.65, t = -6.74, p < .0001). The negative statistics further suggests that as formality level increases, the number of quality changes made decreases. The results for design experience (β = .43, t = 4.34, p < .0001) and study level (β = .25, t = 2.57, p < .015) further indicates that on top of the effects of formality on quality changes made – people with more design experience and/or at a high level of study (e.g. graduates) are more likely to make greater number of expected changes than those with less design experience and/or at a lower level of study (e.g. undergraduates).
Table 22.2

The unstandardized and standardized regression coefficients, and the t-value and significance of each between-subject variables included in the model.

	B	Std Error B	β	t
Formality	-1.705	.253	-.654	-6.742**
Design experience	3.178	.733	.426	4.336**
Study Level	1.886	.733	.253	2.574*