Between 1996 and 2003
By
The Qualitative Research Class (EDCI 711)
Spring, 2004
This study had three purposes. First, it was designed to provide feedback to the Department of Educational Leadership and Foundations about the quality of the dissertations completed during the first eight years of the EdD program. Second it was an opportunity for individual students to participate in designing a qualitative study. Third, it provided each student with experience collecting, analyzing and reporting qualitative data.
In order to complete this project, the following steps were taken:
1. I attempted to make all ELF dissertations available for this class project.
2. The WCU library only had 15 of the dissertations in their collection so those 15 were sent to the UNCA library for check-out by students in this class.
3. Eleven students checked out all the dissertations during the first two weeks of class. They reviewed the available dissertations and chose one to study.
4. Each student read his or her dissertation.
5. The students brainstormed the criteria against which each dissertation would be evaluated. The result was a list of 14 criteria that were later combined to form the ten criteria that are used in this report.
6. The students designed the study by thinking of it as a quilt. Each of the eleven dissertations represented one of the squares. The ten criteria represented the stitching that held the squares together. Each square (dissertation) had its own features, but the overall quality of the quilt would be determined more by the quality of the stitching (how well they mapped against the criteria) than by the quality of individual squares. Consequently, as we carried out this study, we evaluated the stitching (how these texts stood up against the criteria we created) more than the squares, themselves.
7. As students evaluated their dissertations against the ten criteria, we experienced problems. Initially, some of the criteria were not worded to elicit thick descriptions.
8. Therefore, just before final reporting was due, the criteria were adjusted so students could respond with thick descriptions instead of short responses. (This was an experience in creating high quality criteria/questions for a qualitative study.)
9. Just before the end of the semester, students submitted thick descriptions of their dissertations mapped against each of the ten criteria.
10. I organized the eleven sets of thick descriptions by criteria, and I wrote the accompanying report.
The first issue that arose concerned the availability of dissertations. I chose to use the library reserve system, which resulted in students not having the flexibility needed to do a semester-long study of a text. Some students had to pay library fines because the library had to “check them out” for no longer than a week.
The second issue involved generating the criteria. The main issue with generating the criteria was that the students who were generating the criteria had never completed dissertations, themselves. Still, I felt the criteria they suggested covered many of the quality issues confronting dissertators. We started with 14, but as we reviewed those we realized they overlapped, and they could be combined into ten.
The third issue is also a limitation of the study. Although each dissertation was read many times by the individual students, no individual read all of them. Such a reading by one researcher would have made authorship of this report more meaningful because it would have provided a truer summative picture of the total body of dissertations. Instead, we are left with a summary report generated by me, based on individual class members’ readings of one dissertation.
A fourth issue/limitation is that students were limited to 15 of the total 24 dissertations. And of that 15 we only chose eleven – one per student. Consequently, this report does not present a total picture of the dissertations completed during the last eight years. The dissertations that were studied became part of this research because of their availability and the interests of individual class members.
A fifth and final issue surfaced when students realized they would have to say critical things about the dissertations they were reviewing. They realized that these critical comments might be considered critical of the dissertaters (their colleagues) and the dissertation committee professors. In response to this concern, I explained that we would acknowledge our limitations in doing this study, and this report would only be shared with professors who want to receive it.
We often found that seven or eight of the dissertations were of higher quality than the other three or four, when mapped against a criterion. Upon further examination, it became clear that two of the three or four that were ranked low were the same ones. This has a few possible explanations.
First, it could have more to do with the standards of the individual reviewers than with the dissertations, themselves. We discussed what the criteria meant, but we did not take steps to assure inter-rater reliability. That might have been a worthwhile activity, but time did not permit it.
It could also result from design flaws in the dissertations that were seen as lower quality. Studies that are not well-designed, are likely to have problems that surface at many levels.
Regardless of the reason, a quantitative assessment of the quality of the dissertations suggests that, according to the chosen criteria, six dissertations were consistently high in quality, three were of mixed quality, and two were consistently of lower quality than the others. It should be noted, however, that all dissertations had their strengths, and the general level of quality was high.
Report format
The full report is organized by the ten criteria. After each criterion question, a few sentences describe the eleven-dissertation breakdown for that criterion. For example, as mentioned earlier, according to many of the criteria, seven or eight of the dissertations are of high quality, and three or four are of lower quality. Following a statement that describes the breakdown for that criterion, I report the ECDCI 711 students’ descriptions of how they mapped their dissertation against that particular criterion.
The descriptions of quality within these eleven dissertations are those of novice qualitative researchers. Nevertheless, the words of the student reviewers (the data) might be of interest to those who want to formulate questions about how to improve the quality of our dissertations.
Each criterion has one section devoted to it, and most of the sections have the following sub-headings:
1. High quality in this area
2. Mixed quality in this area
3. Low quality in this area
The ten criteria are:
1.
In what
ways was the literature review connected to the topic?
2.
In what
ways did the dissertation maintain a focus?
3. In
what ways did the dissertation deepen understanding?
4.
What was the
quality of proofreading?
5.
How did the author connect the study’s purpose to
something of value in the field of education?
6. Was the dissertation clearly written?
7. Was the study designed well? When can a reader draw the design of the study?
8. In what ways did the author deal with technical terminology and jargon?
9. In what ways are the data and analysis in Chapter 4 related to the conclusions in Chapter 5?
10.
How are biases addressed?
Contributors
Class members were Betsy
Burrows, Carolyn Franklin, Donna Gardner,