H809 Old vs new themes – Reading 12 (A11.6b)
Ardalan et al. (2007) A comparison of student feedback obtained through paper-based and web-based surveys of faculty teaching
How does the results of both quantitative and qualitative sections of student feedback of faculty teaching differ using the web-based and paper-based surveys?
- Columbia University
- 1415 students total of which 972 provided paper-based feedback (2003)
- 1276 students total of which 392 provided web-based feedback (2004)
Positivism (knowledge theory) is dominant with hypothetical testing and quantitative methods.
- large-scale study that statistically compares the results of both quantitative and qualitative sections of student feedback of faculty teaching using the web-based and paper-based methods
- questionnaire a mixture of scaled, closed-ended and open-ended questions (Appendix 1) used with both methods (paper-based and web-based)
- quantitative section included eight questions, 6 point-Likert scale + not applicable option
- qualitative section – open-ended comment section
- empirically testing and formally testing from 10 hypothesis
- using Qui-square test, t-test,
10 hypothesis were tested to compare the results from web-based and paper-based surveys. The results are not always as predicted and contrast or confirm existing literature. Response rates from web-based surveys are a lot lower as assumed, because of the longer and better accessibility for all, especially for those who do not attend the class often. However, the results are statistically significant lower confirming literature who argue that the lack of perceived anonymity and the traceability of feedbacks upsets students or they are not willing to spend their free time in completing the survey. Overall, there is no difference in the qualitative feedback , but feedback in web-based surveys is longer and more meaningful contrary to paper-based surveys, which confirms predictions that the longer time to finish the survey and the medium itself foster longer responses. Yet, the responses were longer but not necessarily more thoughtful.
The study is cross-sectional and not a longitudinal, thus provides only a snapshot. Although the authors decided to omit the results from fall 2003, due to procedural and technical problems, would a second set of data from e.g. fall 2004 be interesting to compare it with the spring 2004 web-based survey results. Students were used to paper-based surveys, for years, but the web-based survey was novel to faculty and students. Therefore I think the results are not really comparable with all regards.
Another limitation was that although a large number of students participated, only 31% responded in the web-based survey and it is difficult, as mentioned in the ethic section, to draw convincing results.
More research would be also needed with other Universities or in other countries to provide more conclusive results.
- I find it somehow ethically questionable making survey results available to students to help them select the faculty whose teaching method best match students learning styles p. 1088. It is like deciding to enroll in a course because my friend goes there as well, although I am not interested in the subject. Or I go there because I know the tutor is not so strict.
- not sure if this is more a limitation or an ethical issues, how can I compare results of both methods and make conclusions with a response rate that is statistically significant? There is an only 31% response rate for web-based surveys contrary to a 69% response rate for the paper-based survey. How can I deduce results and make claims when 70% of the students did not participate in the survey at all. Non-response makes the research unrepresentative. But high response rates and representative sampling is necessary to draw conclusive results. The author’s view the response rate as limitation and also argue that further research is needed to determine the characteristics of those students, because the validity of the study depend on that cases and controls being comparable in all respects.
The results of the research provide information to faculty and administrators on any differences they may expect as they make the transition from a paper-based to a web-based survey of faculty teaching.
- Interventionist – experimental (case and controls)
- old – paper-based surveys handed out in the classroom to complete in a certain time
- new web-based surveys provided via the universities VLE system, completion in leisure time, with no time limit
Words or concept I don’t understand / Statistical terms or methods that are new to me
I still cannot explain or calculate the chi-square test, but I know the p-value, and already dealt with t-test a couple of years ago.
How convinced I am by the research
I am only partly convinced because,
- the study uses mainly qualitative measures, and the quantification of qualitative feedback data in e.g. three categories positive, mixed or negative seems too simplistic.
- results are less convincing because of the low response rate in the web-based survey (31%) – samples are not comparable with all respects
- web-based surveys were novel to faculty and students, contrary to paper-based survey
- Cross-sectional studies are usually conducted to estimate the prevalence of the outcome of interest for a given population, They are limited, however, by the fact that they are carried out at one time point and give no indication of the sequence of events
Ardalan, A., Ardalan, R., Coppage, S., and Crouch, W. (2007) ‘A comparison of student feedback obtained through paper-based and web-based surveys of faculty teaching’ British Journal of Educational Technology, vol. 38, no. 6, pp. 1085-1101; also available online at http://libezproxy.open.ac.uk/login?url=http://dx.doi.org/10.1111/j.1467-8535.2007.00694.x (Last accessed 02 May 2011 )