Case #17: Enhancing Teaching and Learning Through a New Institutional Cascaded Course Evaluation Framework
CAROL ROLHEISER, UNIVERSITY OF TORONTO, CANADA
DISCIPLINE: Teacher Education, Higher Education
RESEARCH AREAS: Teacher education and teacher development, instructional and assessment innovation, leadership, the design, implementation and evaluation of professional development initiatives, system reform and managing educational change
PURPOSE/CONTENT
The use of course evaluations in higher education continues to garner much attention in the popular press and academic circles. After a review of the literature related to course evaluations (Gravestock & Gregor-Greenleaf, 2008), the University of Toronto, Canada, undertook a multi-year process to design, implement, and assess a new online course evaluation framework. This is an example of educational leadership led and managed by a teaching centre in a large research-intensive university.
“Over the last decade this work has necessitated both formal leadership … as well as distributed and shared leadership across the institution …”
CASE EXAMPLE OF EDUCATIONAL LEADERSHIP
In 2009 an institution-wide Course Evaluation Working Group was formed through the Office of the Provost. The Centre for Teaching Support & Innovation (or CTSI, the central teaching-learning hub), played a leadership role in working with senior-level administrators and faculty representatives to “start anew” in our approach to the design and delivery of course evaluations, and to look at how the collected data could enhance teaching and learning across the university. A set of recommendations was put forth to establish a new cascaded course evaluation framework (CCEF) that includes the gathering of data from institution-level core questions that reflect institutional priorities for student learning, as well as context-specific items (e.g., division or faculty level, department level and instructor level).
REFLECTING ON AND APPLYING THE FIVE-PILLAR MODEL
The 5-Pillar Model of Educational Leadership (Fields, Kenny, & Mueller, 2019) is useful for examining the CCEF initiative at the University of Toronto. Over the last decade this work has necessitated both formal leadership (e.g., the Director of CTSI and course evaluation coordinator/team and the Vice-Provost to whom the centre reports, providing ongoing leadership regarding policy, strategies, structures, resources, management, etc.), as well as distributed and shared leadership across the institution (e.g., collaborative design of processes, implementation systems, support mechanisms, etc.) (Lieff & Yammarino, 2017). Reflection on the Five-Pillar Model, the CCEF initiative is action-oriented and intended to bring about long-term transformation regarding teaching excellence through our collective quest to constantly improve teaching and learning, informed by high quality data. Such excellence involves determining the goals we have for teaching and learning across the institution and for particular faculties, departments, as well as at more granular levels (e.g., an instructor’s specific priorities for a given course in a given term).
We set out to bring about large-scale change across an institution of almost 90,000 students, and intentionally chose not to just “tinker around the edges”. This bold initiative necessitates centralized coordination as each division or faculty is brought into the initiative, and decentralized empowerment as each unit (e.g., Faculty of Engineering or Education) examines their local context to remove barriers, builds local capacity for determining teaching and learning priorities, and brings colleagues together for decision-making and ongoing sharing of best practices. Such coordination reflects mentorship, empowerment, and importantly, coaching and collaboration. This combination of leadership has necessitated the use of affective qualities that support dealing with complex and differing perspectives, and, at times, navigating difficult conversations. Such qualities need to be reflected especially when views are different; it is at these times that respect and understanding are most needed. Importantly, demonstrating those qualities can lead to trust and facilitate the relationships across the institution and within divisions that are key to the success of a long-term change strategy. Finally, the evidence-based development of the CCEF and the ongoing quality assurance analyses and pedagogical research that have been priorities throughout the process of implementation and institutionalization, continue to provide context-specific evidence to support our model and its intended purposes, and to inform our next steps at multiple levels of the institution.
IMPACT
The impact of a large-scale and multi-year initiative can be measured in many different ways, with a few highlighted here to illustrate. The University of Toronto now has a policy (2011) that provides important framing of our institutional goals and uses of course evaluations. As well, regular quality assurance analyses have resulted in a range of reports and guidelines that support our collective understanding of the complex issues related to course evaluations. Such analyses have also supported our response to questions raised in the literature and by stakeholders across our institution. Reports and related resources such as the Centre for Teaching Support & Innovation’s, University of Toronto’s Cascaded Course Evaluation Framework: Validation Study of the Institutional Composite Mean (ICM)(2018) and the University of Toronto Course Evaluation Interpretation Guidelines for Academic Administrators (2018) are important documents to support our CCEF and to reflect our evolving understanding and use of course evaluation data throughout the institution. As well, the development of the CCEF and its online implementation have resulted in many presentations at provincial, national and international conferences, and the request for consultations from a large number of higher education institutions around the world. Such activities support our commitment to networked leadership and led CTSI to hosting an inaugural Course Evaluation Institute (2018). This event brought together international colleagues over two days to begin building a collaborative and user-led community focused on course evaluations and teaching assessment and to endeavor to influence an evidence-based narrative around course evaluations.
RELATED REFERENCES
Centre for Teaching Support & Innovation. (2018). University of Toronto’s cascaded course evaluation framework: Validation study of the institutional composite mean (ICM). Toronto, ON: Centre for Teaching Support & Innovation, University of Toronto. https://archive.teaching.utoronto.ca/wp-content/uploads/2019/04/Validation-Study_CTSI-April-10-2019.pdf
Centre for Teaching Support and Innovation (2018). University of Toronto course evaluation interpretation guidelines for academic administrators. Toronto, ON: Centre for Teaching Support & Innovation, University of Toronto. https://archive.teaching.utoronto.ca/wp-content/uploads/2018/10/Interpretation-Guidelines_Final_Oct.1.2018.pdf
Fields, J., Kenny, N. A., & Mueller, R. A. (2019): Conceptualizing educational leadership in an academic development program, International Journal for Academic Development, DOI: 10.1080/ 1360144X.2019.1570211
Gravestock, P. & Gregor-Greenleaf, E. (2008). Student course evaluations: Research, models and trends. Toronto, ON: Higher Education Quality Council of Ontario.
Lieff, S.J., & Yammato, F.J. (2017). How to lead the way through complexity, constraint and uncertainty academic health science centres. Academic Medicine, 92, 614-612.
BIOGRAPHY
Professor Carol Rolheiser has held a range of K-12 leadership positions throughout her career, and the following higher education leadership roles at the Ontario Institute for Studies in Education, University of Toronto: Associate Chair, Department of Curriculum, Teaching and Learning; Associate Dean, Academic Development; and Associate Dean, Teacher Education. For the last decade she has had the privilege of serving as the inaugural Director of the Centre for Teaching Support & Innovation (CTSI), University of Toronto.
CURRENT ROLE
Carol’s current roles include being a Professor in the Department of Curriculum, Teaching and Learning, Ontario Institute for Studies in Education, University of Toronto and serving as Director of the Centre for Teaching Support & Innovation (CTSI) – the University of Toronto’s central teaching and learning hub. The latter role provides her an opportunity to provide strategic direction for educational development across the University, including work that supports programming for instructors and teaching assistants/graduate students (including the support for the Scholarship of Teaching and Learning); coordination and delivery of the Teaching Assistants’ Training program; the implementation of course evaluations and conducting related quality assurance; support for pedagogical innovation, including the use of active and collaborative technologies; and development of communication strategies related to all of these activities.
Among other honours, Professor Rolheiser is a recipient of the inaugural University of Toronto’s President’s Teaching Award (2006), honouring her career commitment to excellence in teaching, research in teaching, and the integration of teaching and research.
Carol’s passion for teaching is at the core of her professional work.
NETWORKS
American Educational Research Association
International Society for the Scholarship of Teaching and Learning (ISSoTL)
Co-Chair: Scholarship of Leading Special Interest Group
Society for Teaching and Learning in Higher Education (STLHE)
Education Developers Caucus of Canada (EDC)
The Professional and Organizational Development Network in Higher Education (POD)