Abstract
This paper explores new angles of the student evaluation of teaching (SET) debate. The SET ratings of 179 full-time professors of the University of Puerto Rico-Bayamón were tracked from 1998–1999 to 2003–2004. Information on the characteristics of professors, students, and courses was analyzed. Two principal findings are documented. First, students’ grade expectations formulation process is significantly influenced by their quality and by the mean and variance of their professors’ historical grade distribution. Second, students’ willingness to take additional courses with a particular professor is directly related to the optimism of their relative expected grades. Thus, professors might be able to increase enrollment in their courses by: (a) increasing the students’ expected grades and (b) promoting the academic environment that would transform students’ pessimistic expectations into optimistic ones.
Similar content being viewed by others
Notes
For a literature review refer to Marsh [1984; 1987], Aleamoni [1987], Cashin [1988], Nerger et al. [1997], Stringer and Irwing [1998], Kolitch and Dean [1999], Germain and Scandura [2005], Gump [2007], as well as Theall and Franklin [2001].
For a literature review of the subject, see Gump [2007].
A non-representative sample of such papers includes the following: Dilts [1980], Seiver [1983], Nelson and Lynch [1984], Zangenehzadeh [1988], Krautmann and Sander [1999], Isely and Singh [2005], and McPherson [2006].
The following criteria were utilized: T-G=1, if X̄ GI j<(−S̃ ); A-G=1, if X̄ GI j=(±S̃ ); and E-G=1, if X̄ GI j>(+S̃ ).
It should be recognized that grading behavior could be influenced by variables related to course materials or departmental polices which are unavailable. Furthermore, there could be other determinants of EG, particularly the midterm grade, which cannot be matched to an anonymous evaluation. Hence, readers are made aware along the paper of its exploratory nature.
Matos-Díaz and Ragan [2010] model SET ratings as a function of class EG and its variance, utilizing the course as the unit of analysis. They conjecture that SET ratings will be inversely related to the variance of EG. Conversely, this paper utilizes the student as the unit of analysis und uses the mean and variance of the professor's historical grade distribution, instead of the variance of the course EG, as correlates to model process 1. Thus, the way that both variables are utilized is one of this paper's contributions.
The variable is defined as follows: REG ij =EG ij −GPA i .
PREG=1 if REG i <X̄−σ, RREG=1 if REG i =X̄±σ, and OREG=1 if REG i >X̄+σ.
The scale is as follows: 4=totally agree; 3=moderately agree; 2=moderately disagree; 1=totally disagree; 0=does not apply.
The first set of evaluations corresponds to the second semester of the 1998–1999 academic years. The evaluations of the first and second semesters of 1999–2000, as well as those of the first semester of 2000–2001 are missing. The second set consists of those realized from the second semester of 2000–2001 to the second semester of 2003–2004.
Another frequently used distribution is the logistic. According to Greene [2008, p. 832], both distributions generally give similar results. Hence, the selection of the model is largely a matter of convenience.
Even though β coefficients are not equal to marginal effects, the signs of the probabilities at the end points of the ranking behave as follows: P(y=0) changes in the opposite direction and P(y=j) changes in the same direction of the sign of β̂ k . For technical details, see Greene [2008, pp. 831–835].
For technical details, see Greene [2008, pp. 800–813] and the references cited therein.
It should be emphasized that the simulations coming from the models that do not account for UFH produce probability values somewhat different from those reported and discussed in the text; but all the conclusions reached are robust to the models specifications. Results are available upon request.
A Wald test rejects the null hypothesis of joint insignificance of all academic programs. The computed Chi-squared (χc2)=70.46, its significance level=0.00000. The programs are: Accounting; Biology; Electronics; Office Systems; Chemistry; Economics or Statistics; Mathematics and Physics. The first four offer baccalaureate degrees.
The programs are: Biology, Business, Electronics, Engineering Technologies, English, and Physics.
See, for example: Krautmann and Sanders [1999], Isely and Singh [2005], McPherson [2006], and Nowell [2007].
The same instruments were utilized by Nelson and Lynch [1984], but Krautmann and Sander [1999] questioned their use arguing that they could be correlated with unobserved faculty productivity. They used “Core” and “Graduate” variables as identifiers, but honestly admit that “… although it is impossible to empirically prove that Core and Graduate constitutes a valid set of identifiers, they appear to be reasonable candidates.” Thus, although academic ranks may be endogenous, they constitute my best available set of identifiers.
References
Aleamoni, L.M. 1987. Student Rating Myths vs. Research Facts. Journal of Personnel Evaluation in Education, 1: 111–119.
Anderson, K.H., and J.J. Siegfried . 1997. Gender Differences in Rating the Teaching of Economics. Eastern Economic Journal, 23: 347–357.
Becker, W.E. 1979. Professorial Behavior Given a Stochastic Reward Structure. American Economic Review, 69: 1010–1017.
Cashin, W.E. 1988. Student Ratings of Teaching: A Summary of the Research, IDEA Paper No. 20. Manhattan, KS: Center for Faculty Evaluation and Development.
Cohen, P.A. 1986. An Updated and Expanded Meta-Analysis of Multisection Validity Studies, Paper Presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA.
Correa, H. 2001. A Game Theoretic Analysis of Faculty Competition and Academic Standards. Higher Education Policy, 14: 175–182.
Dilts, D.A. 1980. A Statistical Interpretation of Student Evaluation Feedback. Journal of Economic Education, 11: 10–15.
Germain, M.L., and T.A. Scandura . 2005. Grade Inflation and Student Individual Differences as Systematic Bias in Faculty Evaluations. Journal of Instructional Psychology, 32: 58–67.
Gigliotti, R.J., and F.S. Buchtel . 1990. Attributional Bias and Course Evaluations. Journal of Educational Psychology, 82: 341–351.
Greene, W.H. 2008. Econometric Analysis. New York: Prentice-Hall.
Gump, S.E. 2007. Student Evaluation of Teaching Effectiveness and the Leniency Hypothesis: A Literature Review. Educational Research Quarterly, 30: 55–68.
Hamilton, L.C. 1980. Grades, Class Size, and Faculty Status Predict Teaching Evaluations. Teaching Sociology, 8: 47–62.
Isely, P., and H. Singh . 2005. Do Higher Grades Lead to Favorable Students Evaluations? Journal of Economic Education, 36: 29–42.
Johnson, V.E. 2003. Grade Inflation: A Crisis in College Education. New York: Springer-Verlag.
Kolitch, E., and A. Dean . 1999. Student Ratings of Instruction in the USA: Hidden Assumptions and Missing Conceptions about “Good” Teaching. Studies in Higher Education, 24: 27–42.
Krautmann, A.C., and W. Sander . 1999. Grades and Student Evaluations of Teachers. Economics of Education Review, 18: 59–63.
Marsh, H.W. 1984. Students’ Evaluations of University Teaching: Dimensionality, Reliability, Validity, Potential Biases, and Utility. Journal of Educational Psychology, 76: 707–754.
Marsh, H.W. 1987. Students’ Evaluations of University Teaching: Research Findings, Methodological Issues, and Directions for Future Research. International Journal of Educational Research, 11: 253–388.
Matos-Díaz, H., and J.F. Ragan . 2010. Do Student Evaluations of Teaching Depend on the Distribution of Expected Grade? Education Economics, 18: 317–330.
McKenzie, R.B. 1975. The Economic Effects of Grade Inflation on Instructor Evaluations: A Theoretical Approach. Journal of Economic Education, 2: 99–106.
McKenzie, R.B., and R.J. Staaf . 1974. An Economic Theory of Learning: Students Sovereignty and Academic Freedom. Virginia: University Publications.
McPherson, M.A. 2006. Determinants of How Students Evaluate Teachers. Journal of Economic Education, 37: 3–20.
Moore, T. 2006. Teacher Evaluations and Grades: Additional Evidence. Journal of American Academy of Business, 9: 58–62.
Nelson, J.P., and K.A. Lynch . 1984. Grade Inflation, Real Income, Simultaneity, and Teaching Evaluations. Journal of Economic Education, 15: 21–37.
Nerger, J.L., W. Viney, and G. Riedel . 1997. Student Rating of Teaching Effectiveness: Use and Misuse. The Midwest Quarterly, 38: 218–233.
Nowell, C. 2007. The Impact of Relative Grade Expectations on Student Evaluation of Teaching. International Review of Economics Education, 6: 42–56.
Sabot, R., and J. Wakeman-Linn . 1991. Grade Inflation and Course Choice. Journal of Economic Perspectives, 5: 159–170.
Sailor, P., B.R. Worthen, and E. -H. Shin . 1997. Class Level as a Possible Mediator of the Relationship Between Grades and Student Ratings of Teaching. Assessment & Evaluation in Higher Education, 22: 261–269.
Seiver, D.A. 1983. Evaluations and Grades: A Simultaneous Framework. Journal of Economic Education, 14: 32–38.
Simpson, P.M., and J. Siguaw . 2000. Student Evaluations of Teaching: An Exploratory Study of the Faculty Response. Journal of Marketing Education, 22: 199–213.
Stringer, M., and P. Irwing . 1998. Students’ Evaluations of Teaching Effectiveness: A Structural Modeling Approach. British Journal of Educational Psychology, 68: 409–426.
Theall, M., and J. Franklin . 2001. Looking for Bias in all the Wrong Places: A Search for Truth or a Witch Hunt in Student Ratings of Instruction? New Direction for Institutional Research, 19: 45–56.
Wachtel, H.K. 1998. Student Evaluation of College Teaching Effectiveness: A Brief Review. Assessment & Evaluation in Higher Education, 23: 191–211.
Zangenehzadeh, H. 1988. Grade Inflation: A Way Out. Journal of Economic Education, 19: 217–223.
Acknowledgements
I would like to recognize my deep debt of gratitude to Dr. James F. Ragan, Jr., with whom I had the benefit of discussing these ideas innumerable times. He passed away on October, 13, 2009; but he will be forever missed. I am also grateful to Gilberto Calderón for compiling data for this project, and to Dennis L. Weisman, Dong Li, Paul G. Hare, Alfred J. Crouch, María Enchautegui, Dwight García-Vázquez, two anonymous referees, and Susan L. Averett for helpful comments and suggestions. Any remaining errors are my responsibility.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Matos-Díaz, H. Student Evaluation of Teaching, Formulation of Grade Expectations, and Instructor Choice: Explorations with Random-Effects Ordered Probability Models. Eastern Econ J 38, 296–309 (2012). https://doi.org/10.1057/eej.2011.7
Published:
Issue Date:
DOI: https://doi.org/10.1057/eej.2011.7
Keywords
- SET ratings
- random-effects
- ordered probability models
- relative expected grade
- maximum likelihood methods