Multiple Choice Quizzing on Quercus

“I hate multiple-choice tests. And tests that involve matching, fill in the blanks and all other such silliness. To be honest, I’m not a big fan of tests at all. That’s because tests tend not to be designed with genuine learning in mind. If they were, our tests would look very different. And we would rarely, if ever, use tests of the multiple-choice variety.” – M. Mascolo, Ph.D.

Many of us have a love-hate relationship with multiple choice tests and quizzes. We either love them or we hate them. Sure, they require little to no time to mark, but effective multiple choice test items take considerable time to prepare. They are also hard to write well. Even more thought and care is required when designing questions that ask students to grapple with course content using higher-level thinking: analysis, synthesis, and judgment.

Thankfully, when venturing into the world of multiple choice, there are a wealth of resources to guide you. Those resources come in handy both when you need to write test items from scratch and when you want to ensure that items from a text book question bank will serve as dependable indicators of student learning.

Here are a few resources:

Integrating multiple choice quizzes into Quercus, whether you offer them weekly or just a few times a semester, can benefit students. Quizzes can help students get a sense of whether their learning is on track: “Have I really captured the important points from this week’s reading or yesterday’s lecture?” Multiple choice quizzes can also help you as the instructor. Beyond saving time through automatic grading, quizzes can help you to identify gaps in student learning early so you can be responsive in your teaching and revisit difficult concepts long before students complete high stakes assignments or exams.

Want to know how well your quiz and questions are functioning?

Quercus can help. Once students have completed a Quiz, access Quiz Statistics to see overall statistics including the average score, the highest score, the lowest score, the standard deviation and the average time it took students to complete the quiz.

Quercus also provides question breakdown statistics. For each question, you can see the number and percentage of students who selected the correct response as well as each incorrect response or distractor.

  • If all students are getting a question right or wrong, you may want to consider whether the questions is as challenging as you expected.
  • If students are choosing a particular distractor more than others or avoiding another all together, the distractors could likely be made more effective or eliminated. Each of the distractors should be plausible and roughly equally appealing to students.

A discrimination index is included for each question. The discrimination index reveals how well the question reflects the difference between students who performed well on the quiz and those who did not. An ideal discrimination shows students who scored higher on the quiz getting the question right and students who scored lower on the quiz getting the question wrong. On Quercus, low discrimination scores (+0.24 or less) appear in red and higher discrimination scores appear in green (+0.25 or greater).

  • If your discrimination index appears in red, the question is worth a closer look. It could be that one or more of the distractors is ‘tricky’ or confusing or the question is not a good reflection of student learning.
  • If the discrimination index is zero, it means that all students either got the question right or wrong. While there are good reasons for having a few questions that all students will get correct, the question may be worth re-examining.

The Quiz Statistics are often enough to help you determine which questions are working well and which could be tweaked and improved – or thrown out all together. If you would like further and more detailed information, the Moderate This Quiz tool allows you to view each student, their number of quiz attempts, the time it took them to complete the quiz, and their score. Moderate This Quiz can also be used to give students extra attempts or time on the quiz. Two Quiz Analysis Reports are also available. The Student Analysis Report shows you how students responded to each quiz question. The Item Analysis Report presents further statistical data related to quiz reliability, question difficulty, and discrimination – including how well each question’s distractors is functioning. CTSI’s Multiple Choice Quercus Quiz Analysis Reports Guide (PDF) can help you to interpret the information available.

If you would like an in-person consultation about assessment design, including developing multiple choice questions or quizzes, complete CTSI’s General Consultation Request Form. For assistance with Quercus tools contact your Divisional Support Contact or email q.help@utoronto.ca

Reference:

Mascolo, M. (2015, July 27). I hate multiple choice: Why multiple choice is mind numbing [Blog post]. Retrieved from https://www.psychologytoday.com/ca/blog/old-school-parenting-modern-day-families/201507/i-hate-multiple-choice

 

By Carol Roderick, Quercus Liaison, CTSI