The south Florida Graduate Student

To plan the format and content of the LLMs, the four team members met several times as a group during the Fall 2004 semester. In the proposed design, each LLM was to include an introductory portion in which key terms were identified, followed by an explanation portion in which key concepts were clarified and exemplified with the aid of audio-visual material such as animations, movie and/or sound clips. An application section followed, where students were presented with problems to resolve based on the material presented. Another element in the design was a quiz, intended to assess the students’ knowledge of the material presented in the LLM. Finally, each LLM contained a brief survey to assess the module with regard to comprehensibility of the material, ease of use and effectiveness for learning about the material.

Each team member developed two web-based LLMs for use in the pilot project. The topics of the LLMs ultimately developed were: the nature of language, morphology, syntax, phonetics, phonology, sociolinguistics, historical linguistics and second language acquisition. Technical training for the development and application of the LLMs in a web environment via WebCT was provided by our institution’s Information and Communication Technology (ICT) staff in the form of both group and individual course sessions. Additionally, one team member provided several individual tutoring sessions and troubleshooting for the other team members throughout the grant period. This team member uploaded the prepared modules in a separate master course section and prepared those materials for transfer into the other members’ sections.

In order to test the efficacy of the LLMs, the application of the LLMs was designed to be differentially integrated in courses during the Spring 2005 semester. In the implementation phase, the three sections of the introductory linguistics course, each taught by a different team member, were intended to operate as separate treatment conditions. One section was designed to serve as a control, using WebCT solely as a course administration tool (e.g. posting syllabus, assignments and grades), and not as a means of distributing any course lessons, including LLMs. A second section was to also use WebCT as an administrative tool, but to incorporate the LLMs as an out-of-class supplement, assigning the material and quizzes as homework. The third section was to use WebCT as an administrative tool as in the first two sections, but was to integrate the LLMs into the regular class sessions and assign only the LMM quizzes as homework.

In order to independently assess the overall linguistic knowledge of each section, Three Skills Surveys, (SS), were administered to each section at random intervals. All SS were prepared by the team member that was not teaching one of the sections to guard against any instructor related bias in the preparation of the assessment instrument.

Each SS was a 10 point test, in a multiple-choice, pop quiz format. The topics addressed in each SS were ones that all three linguistics sections covered, but, more specifically, were themes that had been included in the LLMs. The content of the test items ranged from definitions to examples to applications. Skills Survey I included questions on the nature of language, historical linguistics and morphology. Skills Survey II examined sociolinguistics, phonetics and phonology. Skills Survey III examined syntax and second language acquisition.

Although each of the modules was intended to exemplify the same format, it soon became clear that this goal was greatly influenced by the existing technological skills of the team members and their willingness to overcome any technology-based apprehension. These complications altered the intended design of the project in terms of format, distribution, and assessment.

In terms of format, 6 of the LLMs were consistent with the intended design; the remaining 2 provided students with only a list of terms and definitions. Similarly, distribution was affected as team members demonstrated differing abilities to master the WebCT environment and its various component tools over the grant period (e.g. uploading the modules from a master course to their individual section). Consequently, students in the section of Ling 200G that was originally proposed to have access to the web modules as homework assignments were not able to retrieve the information during the testing period. As a result, only one section used the modules on WebCT, while two did not. With regard to assessment, while both the online quizzes at the end of each LLM as well as the student evaluation of the individual modules were potentially available for use on WebCT, two team members were unable to link them to the individual LLMs; thus, they were not able to be completed.

Despite the complications presented by the varying formats of the modules and their differential access, students in all three sections of the introductory linguistics course were given three pen-and-paper SSs as a form of assessment of learning. The average scores for each of the SSs for each class as well as the results of the ANOVA are presented in Table 1 below.

As can be seen in that table, the students in the control class, which intentionally did not have course content support on WebCT as part of this project, most consistently scored higher on each of the Skills Surveys than the other two classes. Moreover, the ANOVA results reveal significant differences between the groups on the Skills Surveys I and II and their overall averages on the three Skills Surveys, but not on the Skills Survey III.

A post hoc measure, Tukey’s HSD, revealed that for Skills Surveys I and II, the significance was attributed to the differences between the control class and each of the other two classes. In the Skills Survey Average, all three classes were significantly different from one another. The significant differences revealed by this measure are starred in the table below.