Skip to main content
Log in

Student Evaluation of Teaching, Formulation of Grade Expectations, and Instructor Choice: Explorations with Random-Effects Ordered Probability Models

  • Article
  • Published:
Eastern Economic Journal Aims and scope Submit manuscript

Abstract

This paper explores new angles of the student evaluation of teaching (SET) debate. The SET ratings of 179 full-time professors of the University of Puerto Rico-Bayamón were tracked from 1998–1999 to 2003–2004. Information on the characteristics of professors, students, and courses was analyzed. Two principal findings are documented. First, students’ grade expectations formulation process is significantly influenced by their quality and by the mean and variance of their professors’ historical grade distribution. Second, students’ willingness to take additional courses with a particular professor is directly related to the optimism of their relative expected grades. Thus, professors might be able to increase enrollment in their courses by: (a) increasing the students’ expected grades and (b) promoting the academic environment that would transform students’ pessimistic expectations into optimistic ones.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. For a literature review refer to Marsh [1984; 1987], Aleamoni [1987], Cashin [1988], Nerger et al. [1997], Stringer and Irwing [1998], Kolitch and Dean [1999], Germain and Scandura [2005], Gump [2007], as well as Theall and Franklin [2001].

  2. For a literature review of the subject, see Gump [2007].

  3. A non-representative sample of such papers includes the following: Dilts [1980], Seiver [1983], Nelson and Lynch [1984], Zangenehzadeh [1988], Krautmann and Sander [1999], Isely and Singh [2005], and McPherson [2006].

  4. The following criteria were utilized: T-G=1, if GI j<( ); A-G=1, if GI j=(± ); and E-G=1, if GI j>(+ ).

  5. It should be recognized that grading behavior could be influenced by variables related to course materials or departmental polices which are unavailable. Furthermore, there could be other determinants of EG, particularly the midterm grade, which cannot be matched to an anonymous evaluation. Hence, readers are made aware along the paper of its exploratory nature.

  6. Matos-Díaz and Ragan [2010] model SET ratings as a function of class EG and its variance, utilizing the course as the unit of analysis. They conjecture that SET ratings will be inversely related to the variance of EG. Conversely, this paper utilizes the student as the unit of analysis und uses the mean and variance of the professor's historical grade distribution, instead of the variance of the course EG, as correlates to model process 1. Thus, the way that both variables are utilized is one of this paper's contributions.

  7. The variable is defined as follows: REG ij =EG ij GPA i .

  8. PREG=1 if REG i <σ, RREG=1 if REG i =±σ, and OREG=1 if REG i >+σ.

  9. The scale is as follows: 4=totally agree; 3=moderately agree; 2=moderately disagree; 1=totally disagree; 0=does not apply.

  10. The first set of evaluations corresponds to the second semester of the 1998–1999 academic years. The evaluations of the first and second semesters of 1999–2000, as well as those of the first semester of 2000–2001 are missing. The second set consists of those realized from the second semester of 2000–2001 to the second semester of 2003–2004.

  11. Another frequently used distribution is the logistic. According to Greene [2008, p. 832], both distributions generally give similar results. Hence, the selection of the model is largely a matter of convenience.

  12. Even though β coefficients are not equal to marginal effects, the signs of the probabilities at the end points of the ranking behave as follows: P(y=0) changes in the opposite direction and P(y=j) changes in the same direction of the sign of β̂ k . For technical details, see Greene [2008, pp. 831–835].

  13. For technical details, see Greene [2008, pp. 800–813] and the references cited therein.

  14. It should be emphasized that the simulations coming from the models that do not account for UFH produce probability values somewhat different from those reported and discussed in the text; but all the conclusions reached are robust to the models specifications. Results are available upon request.

  15. A Wald test rejects the null hypothesis of joint insignificance of all academic programs. The computed Chi-squared (χc2)=70.46, its significance level=0.00000. The programs are: Accounting; Biology; Electronics; Office Systems; Chemistry; Economics or Statistics; Mathematics and Physics. The first four offer baccalaureate degrees.

  16. The programs are: Biology, Business, Electronics, Engineering Technologies, English, and Physics.

  17. See, for example: Krautmann and Sanders [1999], Isely and Singh [2005], McPherson [2006], and Nowell [2007].

  18. The same instruments were utilized by Nelson and Lynch [1984], but Krautmann and Sander [1999] questioned their use arguing that they could be correlated with unobserved faculty productivity. They used “Core” and “Graduate” variables as identifiers, but honestly admit that “… although it is impossible to empirically prove that Core and Graduate constitutes a valid set of identifiers, they appear to be reasonable candidates.” Thus, although academic ranks may be endogenous, they constitute my best available set of identifiers.

References

  • Aleamoni, L.M. 1987. Student Rating Myths vs. Research Facts. Journal of Personnel Evaluation in Education, 1: 111–119.

    Article  Google Scholar 

  • Anderson, K.H., and J.J. Siegfried . 1997. Gender Differences in Rating the Teaching of Economics. Eastern Economic Journal, 23: 347–357.

    Google Scholar 

  • Becker, W.E. 1979. Professorial Behavior Given a Stochastic Reward Structure. American Economic Review, 69: 1010–1017.

    Google Scholar 

  • Cashin, W.E. 1988. Student Ratings of Teaching: A Summary of the Research, IDEA Paper No. 20. Manhattan, KS: Center for Faculty Evaluation and Development.

  • Cohen, P.A. 1986. An Updated and Expanded Meta-Analysis of Multisection Validity Studies, Paper Presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA.

  • Correa, H. 2001. A Game Theoretic Analysis of Faculty Competition and Academic Standards. Higher Education Policy, 14: 175–182.

    Article  Google Scholar 

  • Dilts, D.A. 1980. A Statistical Interpretation of Student Evaluation Feedback. Journal of Economic Education, 11: 10–15.

    Article  Google Scholar 

  • Germain, M.L., and T.A. Scandura . 2005. Grade Inflation and Student Individual Differences as Systematic Bias in Faculty Evaluations. Journal of Instructional Psychology, 32: 58–67.

    Google Scholar 

  • Gigliotti, R.J., and F.S. Buchtel . 1990. Attributional Bias and Course Evaluations. Journal of Educational Psychology, 82: 341–351.

    Article  Google Scholar 

  • Greene, W.H. 2008. Econometric Analysis. New York: Prentice-Hall.

    Google Scholar 

  • Gump, S.E. 2007. Student Evaluation of Teaching Effectiveness and the Leniency Hypothesis: A Literature Review. Educational Research Quarterly, 30: 55–68.

    Google Scholar 

  • Hamilton, L.C. 1980. Grades, Class Size, and Faculty Status Predict Teaching Evaluations. Teaching Sociology, 8: 47–62.

    Article  Google Scholar 

  • Isely, P., and H. Singh . 2005. Do Higher Grades Lead to Favorable Students Evaluations? Journal of Economic Education, 36: 29–42.

    Article  Google Scholar 

  • Johnson, V.E. 2003. Grade Inflation: A Crisis in College Education. New York: Springer-Verlag.

    Google Scholar 

  • Kolitch, E., and A. Dean . 1999. Student Ratings of Instruction in the USA: Hidden Assumptions and Missing Conceptions about “Good” Teaching. Studies in Higher Education, 24: 27–42.

    Article  Google Scholar 

  • Krautmann, A.C., and W. Sander . 1999. Grades and Student Evaluations of Teachers. Economics of Education Review, 18: 59–63.

    Article  Google Scholar 

  • Marsh, H.W. 1984. Students’ Evaluations of University Teaching: Dimensionality, Reliability, Validity, Potential Biases, and Utility. Journal of Educational Psychology, 76: 707–754.

    Article  Google Scholar 

  • Marsh, H.W. 1987. Students’ Evaluations of University Teaching: Research Findings, Methodological Issues, and Directions for Future Research. International Journal of Educational Research, 11: 253–388.

    Article  Google Scholar 

  • Matos-Díaz, H., and J.F. Ragan . 2010. Do Student Evaluations of Teaching Depend on the Distribution of Expected Grade? Education Economics, 18: 317–330.

    Article  Google Scholar 

  • McKenzie, R.B. 1975. The Economic Effects of Grade Inflation on Instructor Evaluations: A Theoretical Approach. Journal of Economic Education, 2: 99–106.

    Article  Google Scholar 

  • McKenzie, R.B., and R.J. Staaf . 1974. An Economic Theory of Learning: Students Sovereignty and Academic Freedom. Virginia: University Publications.

    Google Scholar 

  • McPherson, M.A. 2006. Determinants of How Students Evaluate Teachers. Journal of Economic Education, 37: 3–20.

    Article  Google Scholar 

  • Moore, T. 2006. Teacher Evaluations and Grades: Additional Evidence. Journal of American Academy of Business, 9: 58–62.

    Google Scholar 

  • Nelson, J.P., and K.A. Lynch . 1984. Grade Inflation, Real Income, Simultaneity, and Teaching Evaluations. Journal of Economic Education, 15: 21–37.

    Article  Google Scholar 

  • Nerger, J.L., W. Viney, and G. Riedel . 1997. Student Rating of Teaching Effectiveness: Use and Misuse. The Midwest Quarterly, 38: 218–233.

    Google Scholar 

  • Nowell, C. 2007. The Impact of Relative Grade Expectations on Student Evaluation of Teaching. International Review of Economics Education, 6: 42–56.

    Article  Google Scholar 

  • Sabot, R., and J. Wakeman-Linn . 1991. Grade Inflation and Course Choice. Journal of Economic Perspectives, 5: 159–170.

    Article  Google Scholar 

  • Sailor, P., B.R. Worthen, and E. -H. Shin . 1997. Class Level as a Possible Mediator of the Relationship Between Grades and Student Ratings of Teaching. Assessment & Evaluation in Higher Education, 22: 261–269.

    Article  Google Scholar 

  • Seiver, D.A. 1983. Evaluations and Grades: A Simultaneous Framework. Journal of Economic Education, 14: 32–38.

    Article  Google Scholar 

  • Simpson, P.M., and J. Siguaw . 2000. Student Evaluations of Teaching: An Exploratory Study of the Faculty Response. Journal of Marketing Education, 22: 199–213.

    Article  Google Scholar 

  • Stringer, M., and P. Irwing . 1998. Students’ Evaluations of Teaching Effectiveness: A Structural Modeling Approach. British Journal of Educational Psychology, 68: 409–426.

    Article  Google Scholar 

  • Theall, M., and J. Franklin . 2001. Looking for Bias in all the Wrong Places: A Search for Truth or a Witch Hunt in Student Ratings of Instruction? New Direction for Institutional Research, 19: 45–56.

    Article  Google Scholar 

  • Wachtel, H.K. 1998. Student Evaluation of College Teaching Effectiveness: A Brief Review. Assessment & Evaluation in Higher Education, 23: 191–211.

    Article  Google Scholar 

  • Zangenehzadeh, H. 1988. Grade Inflation: A Way Out. Journal of Economic Education, 19: 217–223.

    Article  Google Scholar 

Download references

Acknowledgements

I would like to recognize my deep debt of gratitude to Dr. James F. Ragan, Jr., with whom I had the benefit of discussing these ideas innumerable times. He passed away on October, 13, 2009; but he will be forever missed. I am also grateful to Gilberto Calderón for compiling data for this project, and to Dennis L. Weisman, Dong Li, Paul G. Hare, Alfred J. Crouch, María Enchautegui, Dwight García-Vázquez, two anonymous referees, and Susan L. Averett for helpful comments and suggestions. Any remaining errors are my responsibility.

Author information

Authors and Affiliations

Authors

Appendix

Appendix

Tables A1 and A2

Table A1 Grading index parameters distributed by academic programs
Table A2 Empirical estimates for the SET equation

Rights and permissions

Reprints and permissions

About this article

Cite this article

Matos-Díaz, H. Student Evaluation of Teaching, Formulation of Grade Expectations, and Instructor Choice: Explorations with Random-Effects Ordered Probability Models. Eastern Econ J 38, 296–309 (2012). https://doi.org/10.1057/eej.2011.7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1057/eej.2011.7

Keywords

JEL Classifications

Navigation