Faculty use and impressions of courseware management tools: A national survey

Journal of Engineering Education, Apr 2003 by St Clair, Sean, Baker, Nelson C

The draft survey was sent to the faculty involved in survey creation, to campus assessment leaders, and to teaching center directors at both Georgia Tech and Virginia Tech for comment and input on the survey. Based on the comments received from these individuals, some survey wording was revised and additional questions were added.

The desired sample for this survey was a random selection of participants from a pool of all engineering faculty at schools accredited by the Accreditation Board for Engineering and Technology (ABET). To obtain this sample, a list of all accredited engineering programs located in the United States, 314 total universities, was obtained from ABET's Web site [6]. Student assistants then visited all of the engineering Web sites for these 314 schools and collected faculty listings for each. The list of faculty members that was collected from the universities consisted of 23,860 individuals. Because approximately 500 responses to the survey were desired, and a 20 percent return rate was estimated, a random sample of 2,500 faculty members was chosen. The random sample was selected by putting all 23,860 names in a spreadsheet. Each name was then assigned a random number using the random number function built into the spreadsheet program. The names were then sorted according to the assigned random numbers and the first 2,500 names, or the 2,500 names with the smallest random numbers, were chosen as participants and made up the survey sample.

Once the sample of the 2,500 faculty members had been randomly selected, the participants' e-mail addresses were collected. The survey was encoded on a Web page at Georgia Tech and a Perl script was created to send each faculty member a unique e-mail and identifier for the survey. By using these identifiers, responses could be tracked and a follow-up e-mail reminder could be sent to participants who had not completed the survey. Participants for whom an e-mail address could not be found were also mailed or faxed a copy of the survey. For e-mail that was returned as undeliverable (due to wrong e-mail addresses or full disk space), a copy of the survey was also mailed or faxed.

Data collection for the survey was automatically performed via the Web page survey form. Participants, upon completing their answers and submitting them, allowed the storage of their data in a database on the Web server. The responses received back via fax and postal mail were manually entered into the Web server database.

The survey was originally sent to faculty members in early May, 2001. The follow-up e-mail notices were sent in June. Data analysis began in early August, 2001, at which point responses to the survey were no longer being received.

Standard forms of statistical measures were used for analysis of the collected data. The Pearson chi-square test of association was used to compare demographic information to Internet usage to see if there were any relationships between usage and certain faculty characteristics. Cramer's V was used to test the strength of such a relationship, if it existed. If the chi-square test revealed a difference, post-hoc tests were required to determine the nature of the relationship. For this analysis, the standardized residuals (a standardized form of the difference between the expected and actual counts) were compared to standard normal probabilities (also know as "Z" scores) at the 95 percent confidence level ([alpha] = 0.05). If more than one post-hoc comparison was required, the possibility of error increases and so the a level as well as the corresponding critical "Z" score must be adjusted. To compare Likert scale ratings used in the survey, the Mann-Whitney test was used. Difference of proportions tests [7] were used to compare usage rates. All are reported here with the calculated value and the significance level used. For example (Z = 3.54, p