Abstract

According to the University Master Plan "…Truman must become a leader in the integration and use of appropriate instructional technology in the curriculum." Truman has recently made a significant investment in the Web-based course management software, CourseInfo. The purpose of this study is a preliminary assessment of the diffusion, utility and perceived effectiveness of the CourseInfo instructional technology package. Objectives include determination of extent and sophistication of faculty use, as well as measurement of faculty and student opinion regarding the software’s effectiveness. Secondary objectives are developmental in nature and include the collection and dissemination of Web-based curriculum-enhancing ideas within and across academic disciplines. Other objectives include the diffusion of recent computer-assisted survey techniques into the university academic community and the enhancement of student learning by incorporating into this study significant involvement of students enrolled in a Spring 2000 research methods class.

A Web-based survey instrument will be constructed for this project and disseminated online; design of this questionnaire will also be a focal point of student involvement.

Evaluative results will be disseminated by email; suggestions for innovative use will be organized by category and posted on an interactive university Web page constructed for that purpose.

Performance of this study will not only allow the university to better assess the current status of Web-based use of technology to support teaching and learning, but will also make a substantial contribution to the further dissemination and innovative use of that technology.
 

Personnel

I will be the principal investigator for this project. Concerning research principles, I have taught a graduate research methods course at the University of Southern California, been a teaching assistant in doctoral level research methods at USC, taught research methods at Truman (COMM 300), and will be teaching research methods again in Spring 2000. I have formal academic training and some professional experience in survey construction and data analysis.

Regarding computers and Web technology, I have taught an introductory computing course in the business school of Eastern Kentucky University. More recently I have served as the Language and Literature Divisional facilitator of "Web Course In a Box" software (the precursor to CourseInfo), and in 1997 attended the EDUCOM [now EDUCAUSE] conference on educational technology as a representative of the Division and the University; the subsequent 20-page report which I edited is available on the Web and by request. I also served as Division representative on the University Instructional Services Committee, 1997/98, and am presently serving on the University Faculty Development Committee. I attended the in-depth training workshop on CourseInfo this past August.

Student involvement on this project would occur at two stages. First, I intend to engage my entire research methods class in this study in the spring. Although these students will be working on various course-related research projects of their own, I will have all students participate in the initial design, pre-test analysis and subsequent revision of the CourseInfo assessment questionnaire. Students will also be involved in CourseInfo Website inspection and preliminary evaluation. Second, although most of the questionnaire will be in forced-choice format, it will not be exclusively so. I would like to employ one or two students during the summer to conduct a content analysis of responses to the open-ended questions included on the instrument.
 

Current and Previous Support

I have received no prior university grants or external awards. I have no pending university grants or external awards.

Project Narrative

Purpose

The primary purpose of this study is a preliminary assessment of the diffusion, utility and perceived effectiveness of the CourseInfo instructional technology package. Additionally, this project is designed to foster the exchange of ideas related to Web-enhanced curriculum innovations and to help engender the more general use of curricular instructional technologies appropriate to the liberal arts mission of this university. An important corollary purpose of this project is to actively involve approximately two dozen COMM 300 (Research Methods) students in hands-on research design and implementation procedure.

Background and Importance of the Project.

This grant application is being submitted under the Curriculum Development/Instruction program because of a slight change in guideline wording this year, which now includes the phrase "…and use of technology to support teaching and learning." Encouragement of technology-based curricular innovation reflects not only pertinent sections of the University Master Plan but also the interest in student computer literacy as embodied by the new Liberal Studies Program. The larger context, of course, is the astounding explosion in the 1990s of computer-based information technologies across all U.S. educational institutions and throughout American culture generally. While appropriate warnings against the uncritical introduction of elaborate instructional technologies into the curriculum have been voiced (Daniel, 1997; Ehrmann, 1991), there is widespread acceptance of the proposition that new technologies generally—and the Web specifically—will continue to exert major influence on the ways that knowledge is retrieved, stored and shared in educational settings (Chickering & Ehrmann, 1997; Brazell, 1998; Dugan, 1997; Ehrmann, 1991; Rowley, et al., 1998).

This project is a significant first step in the assessment of the educational effectiveness of a Web-based instructional technology into which the University has put noteworthy investment—in both resources and personnel. This project will provide a vehicle for the sharing of information and innovation regarding Web-based curriculum technologies generally and CourseInfo specifically. A planned by-product of this study will be the diffusion across disciplines of a near state-of-the-art, but free, Web-based survey technique—the same technique which will be used in the study itself.

This study will utilize recent Computer Assisted Survey Information Collection (CASIC) survey techniques. CASIC is a Web-based method of surveying populations (Schmidt, 1997), described in more detail in the Methodology section below.

It should be noted that a scientific assessment of the actual educational value added to the curriculum by integration of Web-based technology is beyond the scope of this study. The objective measurement of such value would entail complex research designs involving multiple campuses over several years. The measurement of perceived effectiveness and utility (or lack thereof) is a much more modest goal—and certainly attainable. Such data could nevertheless provide part of the basis for educational outcomes assessment and the formulation of divisional and university instructional technology strategic plans.

Objectives.

The primary objective of this project is the determination of extent and sophistication of faculty use, as well as the measurement of faculty and student opinion regarding CourseInfo’s effectiveness. Sophistication of use will be measured by the number of software features utilized, coupled with level of the integration with faculty "Gold" server Web-site presence, if any. Perceived effectiveness will be measured by survey questions pertaining to four aspects of this software:

• ability to incorporate external Web-based resources into the course;
• ability to incorporate internal documents and assignments into the course;
• capacity to facilitate student/faculty and student/student electronic communication;
• use (and perceived usefulness) of less-traditional aspects of Web-enhanced instruction specific to CourseInfo (e.g., formulaic student Web page construction, the electronic student calendar, and on-line grade sheet information).

Secondary objectives are developmental in nature and include the collection and dissemination of Web-based curriculum-enhancing ideas within and across academic disciplines. The focus in this study will be on CourseInfo, although ideas of a more general application will not be excluded.

One tertiary objective is the diffusion of recent computer-assisted survey information collection (CASIC) techniques to technologically facile computer users in the university academic community. CASIC software allows the researcher to distribute html-formatted surveys on line and gather the resulting information electronically. One such product, Web Survey Assistant, is available on the Web free of charge to academic institutions. I have assisted a colleague in some preliminary work using this software and it appears to be adequate for my needs in this proposed project. This software, and packages like it, could greatly facilitate survey research in all academic disciplines if faculty and students were made more aware of it—an awareness which this study will promote by example. A short subsection of the questionnaire will consist of questions concerning instrument ease-of-use and likelihood of Web-survey technique adoption for future research projects by faculty (or students).

Another tertiary, but important, objective is the enhancement of student learning by incorporating significant engagement of a research methods class in the design and pre-testing of the Web-based questionnaire for this study. The class will also be charged with inspecting all existing CourseInfo Web pages, with the goal of identifying and classifying innovative use. The purpose here is educational; this assignment will task each student with examination of no more than 10 faculty CourseInfo pages in total. Because the emphasis here is on identification of innovative use, this responsibility should be not only interesting to students, but should challenge some of their conceptions of Web-enhanced instruction.

Again, students in the research methods course will be involved in separate research projects of their own; but this project will provide a common focus for real-world issues when discussing questions of variable definition, reliability, validity, sampling, etc. This project will also increase student awareness of cutting-edge (or at least near-cutting-edge) electronic surveying techniques. A planned content analysis of open-ended survey responses would employ one or two students—preferably from the research methods class—part time in Summer 2000.

In summary, the objectives of this project are:

Extent of use will be determined by Web site survey—that is, identification of users on the CourseInfo home page and a click-by-click examination of the home pages themselves; sophistication of use will be determined by a combination of site-by-site examination and the Web-based survey. Faculty and student perception of CourseInfo utility will also be gleaned by Web-based survey, and a statistical comparison of utility perception will be made and tested for significant differences. Descriptive statistical summaries will be made on closed-ended questions. The Web-based survey will likely use the software package "Web Survey Assistant;" This software has been examined by me, appears adequate to the task at hand, and is free of charge. This survey will be formatted in html (hypertext markup language) and posted on my Gold server Website. A link to this survey, with brief cover letter and permission request, will be emailed to all professors using CourseInfo. Once permission is granted by instructors, the same link will be forwarded to all of their students. The target population is thus easily identified and easily reached. This technology eliminates costs associated with traditional survey techniques. There will be no mailing costs, no photocopying costs, and minimal loss of time for distribution and response. Further, data entry is automatic and instantaneous. Traditionally defined data-entry error is eliminated.

Ideas relating to Web-based enhancement of the curriculum, and suggestions for further innovation in this area, will also be gathered by the Web-based survey, as well as by a site-by-site inspection of existing CourseInfo pages. As described above, students in the spring COMM 300 class will assist. Dissemination is described below. The method of dissemination of the innovative Web-based survey technique, in recursive Escheresque fashion, will be the Web-based survey instrument itself (with a question or two calling attention to selected aspects of the technique).

The location for all work related to this project is Truman State University. Summer salary is requested; specific weeks dedicated to this project will be June 15-July 31, less the first week of July.

Evaluation.

The first three objectives outlined above are descriptive in nature. Success will be determined primarily by survey response rates and the quality of those responses. The goal is at least a 50 per cent response rate from faculty using CourseInfo, and 33 per cent from students. Analysis, including statistical comparisons of faculty and student perceptions of CourseInfo, will be straightforward.

Although this is not a diffusion of innovation study, the study does propose to disseminate innovative ideas. Identified innovative usage of Web-enhanced instruction will be emailed to all participants as part of the results report. I project a 100 per cent success rate for this email transmission. This report will also comprise a portion of the follow-up Web site which will be constructed subsequent to the data analysis. Estimates of continued dissemination can be determined in part by logging the number of hits made to the post-grant Web site.

Evaluation of interest generated in computer assisted survey techniques can be determined in part by responses to relevant questions included on the Web-based questionnaire, and in part by follow-up expressions of interest and questions from faculty and students concerning this technique. Although research-method techniques are not generally perceived as exciting, the convenience of computer-assisted Web surveys is truly groundbreaking. For the sake of measuring results against expectations, I set the goal of expressions of serious interest in potential use of this technique at 5 per cent of student respondents, and 10 per cent of faculty respondents. This study does not propose to track future usage.

The quality of student involvement and learning in connection with this project will be evaluated by traditional classroom techniques. Additionally, a question related to student reaction to the active-learning component of this course will be added to the standard course evaluation questionnaire at the end of the semester—as will a question relating to their (potentially heightened) appreciation of the role which formal research methods can play in answering real-world questions. Finally, there will be a portion of the COMM 300, Research Methods CourseInfo Web site dedicated to student evaluation of this project.

Dissemination.

The primary objectives of this project are to determine usage of Web-based course enhancements, the sophistication of that usage, and the collection of innovative ideas pertaining to that usage. But the primary value of this project rests with the effective dissemination of those results. Dissemination has been referred to throughout this proposal. In summary, the dissemination of results will take place via email and a follow-up Web page dedicated to this project. The email, which will contain a link to the related Web page, will be directed to faculty and student participants, my Research Methods class, members of the Instructional Services Committee, the Instructional Technology Center, Computer Services, the division heads, and the VPAA. The possibility of publication in an academic journal will be pursued, with a focus on the comparisons of faculty and student perception of this curriculum-supplementing technology.

Timeline

Brazell, J. (Fall, 1998). Rewiring the way we learn. Bradley University Hilltopics. 10-13.

Chickering, A. W. & Ehrmann, S. C. (last modified August 28, 1997). Implementing the Seven Principles: Technology as lever. Retrieved October 20, 1998 from the World Wide Web: http://www.aahe.org.technology/ehrmann.htm.

Daniel, J. S. (1997). Why universities need technology strategies. Change. 29(4), 10-17.

Dugan, E. S. (Spring,1997). Higher education and the new media age. Liberal Education. 20-25.

Duggan, A., Hess, B. Morgan, D., Kim, S., & Wilson, K. (1999). Measuring students' attitude toward educational use of the Internet. Paper presented at the Annual Conference of the American Educational Research Association (Montreal, Canada, April 19-23).

Ehrmann, S. C. (1991). Gauging the value of a college’s investment in technology. Educom Review, 26(3/4), 24-28. Retrieved Oct. 20, 1998 from the World Wide Web: http://www.learner.org/edtech/rscheval/gauging.html

Katz, R. N. (1999). Dancing with the devil: Information technology and the new competition in higher education. New York: Jossey-Bass.

Palloff, R. & Pratt, K. (1999). Building learning communities in cyberspace: Effective strategies for the online classroom. New York: Jossey-Bass

Rowley, D., Lujan H. L., Dolence, M. (1998). Strategic choices for the academy: How demand for lifelong learning will re-create higher education. New York: Jossey-Bass

Schmidt, W. C. (1997). World-Wide Web survey research: Benefits, potential problems, and solutions. Behavior Research Methods, Instruments & Computers, 29, 274-279. Retrieved October 2, 1999 from the World Wide Web: http://or.psychology.dal.ca/~wcs/hidden/SAdocs/