peerScholar

Prof. Steve Joordens, Department of Psychology, UTSC

WHAT IS THE INNOVATION THAT YOU HAVE INTRODUCED INTO YOUR TEACHING PRACTICE?
My particular interest is in use of assessment strategies to enhance learning. On the first day of class we typically tell students how we plan to assess them and on which skills. This signals the learning expectations to them. At this point it is critical that we have some method of assessment that targets meta-cognitive thought process, thereby communicating to students that we are interested in teaching them to think critically and creatively, to be analytic, and to communicate their thoughts clearly. Having at least one assignment of this sort balances the perception students often have, a perception that learning is just about memorizing concepts.

The method we have introduced at UTSC utilized peerScholar, an internet-based tool I created with my Ph.D. Student Dwayne Pare to help develop meta-cognitive skills even in very large class contexts. The general structure of the assignments is as follows:

First students compose a written assignment designed to engage the student in critical thinking through response to an argument, typically on a controversial or particularly engaging topic. The students self evaluate and submit a grade that reflects their perceptions of the quality of their work.
Then the students review five or six of their peers’ submissions – anonymous, randomly selected. The students are asked to evaluate the strength of argument and clarity of presentation, and also provide a “wish” (one thing to change) and “hug” (one thing done very well) for each peer assignment reviewed.
The feedback from peers serves as a formative assessment and after viewing and assessing the feedback given to them for their work, students are then asked to revise and resubmit the assignment. A TA rates final assignment submitted, the appropriateness of the revisions made to it, as well as the quality and richness of the feedback they have provided to others. Thus we grade both the product and the process.

We are able to implement this process effectively in large classes currently using this method in a class of 1500.

WHAT RESEARCH INFORMS YOUR WORK IN THIS AREA?
Our research is two pronged. Some of the research focuses on the tool itself. Essentially this research examines the best ways of implementing this process to insure the highest level of fairness, and to make it an enjoyable experience for both student and professor. Some might call this “best practices” research.

Secondly we investigate the pedagogy, gathering empirical data to demonstrate effects of learning outcomes. For example we have been able to show that students’ abilities to discriminate on the basis of quality consistently improves across four assignments, and that their ability to judge the quality of their own work is enhanced by just a single peer-assessment experience. To the extent that quality-based discrimination taps critical thinking skills, these findings show clear enhancements in critical thought.

Thus, the online peer assessment process is both a tool used to conduct research, and the subject of research. We are currently examining the best way for students to provide feedback to one another in hopes of making the learning experience richer yet. We look forward to doing more of this research and ultimately giving students the best, and deepest, learning experience possible.

PLEASE DESCRIBE THE IMPLEMENTATION PROCESS. WHAT WAS INVOLVED?
Initially I worked with a graduate student, Dwayne Pare, on the design and implementation and the research component became his Master’s thesis. While this was a grassroots project initially, its success has lead us to commercialization, if only as a means to support its use by others. This year it will used in grades 8 to 12 in addition to the University context. Thus it has grown well beyond the Introductory Psychology class in which it was born.

WERE THERE ANY OBSTACLES THAT NEEDED TO BE OVERCOME?
As many people know, when we first began using peerScholar we used it in a summative manner, which means that student’s graded one another and those grades counted towards final marks. Our research demonstrated that this was a fair thing to do … the average of a reasonably large number of undergraduate grades is as reliable as the grade given by a single graduate-level TA … and it empowered students to be part of the bigger learning process. However, despite the fact that we actually added some TA hours during the implementation of this initial system, we had apparently stepped on the toes of the TA union, as their contract states that anyone who marks at University must be paid at the established TA rates. So even if students learned a lot during the process, we would have to pay every student to mark if we continued using it summatively. This was depressing at first because we knew how great of an educational tool this is, and it was mind boggling to think that its wide usage would be inhibited in this way. However, we came up with the formative process described earlier, a process in which TAs do provide the ultimate marks, and one that lets students focus on giving each other the best possible feedback. So the deep learning process was retained, and perhaps even enhanced, and more TAs were included in the process. This comes with some additional cost for TAs, but not so much to be prohibitive.

WHAT ARE THE BENEFITS FOR THE LEARNERS IN THIS PROGRAM?
This method addresses changing demands in higher education. In the modern world content is freely available, and the focus is shifting to acquisition of analytical and learning skills. The university community is talking about metacognitive skills, and this is a tool that actually supports development of those processes.

In particular the student gets the benefits of continual practice in discrimination on the basis of quality, self-reflection, assessing quality of feedback and a sense of where their own work fits relative to the work of peers. The feedback is prompt and timely, which maximizes its learning impact. Students also learn to communicate their ideas in a focused and efficient manner. All of these skills stay relevant no matter which career a student may move on to.

WERE THERE ANY SURPRISES OR UNEXPECTED OUTCOMES?
To some extent, many of the specific learning skills I have highlighted were a bit of an unexpected outcome of a sort. That is, when we started this our goal was to get written assignments back into large classes. I knew that written assignments were an important foil to multiple choice exams, but I hadn’t thought about it much deeper than that. We created the process we did initially just to make it work … to allow assignments to be marked and returned quickly and fairly. It really wasn’t until we began presenting our research on fairness at conferences that we began to realize all of the deep skills that the process was supporting and developing. To while I would love to pretend that peerScholar was initially designed with all these goals in mind, the truth is not quite so structured and focused. It was more good intuition leading to a great innovation.

The other cool thing is that we are able to engage students, coming to “where they are”, using technology itself to “connect” to learners. In a large class setting, service and community are important, and to some extent peerScholar provides both. This approach has the potential to be integrated with Online learning given that it builds community, although it can be undertaken remotely.

DO YOU HAVE ANY PLANS FOR THIS PROJECT IN THE FUTURE?
We will undertake more research on the pedagogical implications, which in turn impacts the practical implementation.

PROJECT URL:
www.peerscholar.com