[ARCHIVED] Grading Multiple Answer Questions

[pdk4]

As an instructor in a large university physics course, I would like to have an additional grading option for each Multiple Answer question in which either (1) penalty credit deducted for each incorrect answer chosen = total question credit ÷ number of INCORRECT answers (rather than # of correct answers) or (2) the quiz creator can specify individually the amount (%) of credit to be deducted for each incorrect answer (or added for each correct answer).  The reason for this is that when the number of incorrect answer choices exceeds the number of correct answers, it is far too easy to get zero credit for choosing some incorrect answers along with correct answers.  The suggested grading schemes above allow that outcome to be avoided.  For reference, Blackboard's grading scheme for Multiple Answer questions included such options, so they are certainly feasible.  This option would help reduce student confusion and dissatisfaction with the grading of such questions.  My colleagues also support this suggestion.  Thank-you!

[Steven_S]

@pdk4  I am also an instructor, and I understand your point, but the method of only adding a percent for each correct answer requires a maximum possible number of choices in a multiple answer question.  Otherwise, students can simply select every single answer and get full credit without even reading the question.

Similarly, the method of only subtracting points for every incorrect answer requires a minimum number of selected answers, otherwise students could receive full credit for selecting only one of the multiple correct answers.  Subtracting points also requires a minimum score so that negative scores can be prevented.

The normal behavior in classic quizzes was: "To calculate scores for Multiple Answers quiz questions, Canvas divides the total points possible by the amount of correct answers for that question. This amount is awarded for every correct answer selected and deducted for every incorrect answer selected. No points are awarded or deducted for correct or incorrect answers that are not selected. For example, an instructor may create a Multiple Answer quiz question with 9 points possible that includes three correct choices and two incorrect choices. If a student selects two correct answers and one incorrect answer, they would be awarded 3 total points for that question. This would be calculated by awarding 3 points (9 total points divided by 3 correct answers) for each correct answer and subtracting 3 points for the incorrect answer." https://community.canvaslms.com/t5/Instructor-Guide/How-do-I-create-a-Multiple-Answers-quiz-question...

Multiple answer questions are one of the few question types that already have partial credit, and the method used is very much in keeping with classic quizzes: "Partial credit is calculated by dividing the total points possible by the number of correct answers for the question. This amount is awarded for each correct answer selected and deducted for each incorrect answer selected. No points are awarded or deducted for answers that are not selected. Zero is the lowest possible score. For example, if a question is worth 4 points and there are 2 correct answers, Canvas divides 4 by 2 and awards 2 points for correct answers selected and deducts 2 points for incorrect answers. If a student selects 2 correct answers and 1 incorrect answer, the student will receive 2 points total (4 points awarded for correct answers and 2 points deducted for incorrect answers)." https://community.canvaslms.com/t5/Instructor-Guide/How-do-I-create-a-Multiple-Answer-question-in-Ne...

One way to manage the possibilities if you wish to avoid a zero point score for one correct and one incorrect response, Is to limit the number of distractions and in the prompt specify the number of correct answers, "which three of the below..." for example. So in a 3 point question, If there are 5 total answers, and 3 are correct, and students know to select three, then students could select:

• 1 correct and 2 incorrect: 0 points
• 2 correct and 1 incorrect: 1 point
• 3 correct: 3 points
• 2 correct and 1 skipped: 2 points
• 1 correct, 1 incorrect, 1 skipped: 0 points
• 2 incorrect and 1 skipped: 0 points
• 3 correct and 1 extra (incorrect): 2 points
• 3 correct and 2 extra (incorrect): 1 point

It can be helpful for students to have an explanation in the instructions that this a question in which it is better not to guess blindly, because points are both added for correct answers and also subtracted for incorrect answers. 

Another option that is already available is to write a multiple choice question using lists in each possible answer, and selecting vary points by answer.  Then students would only get to select one option, but you could put enough possible combinations to cover typical multiple answer selections, and set the points for each combination to match your preferences. 

There will be work underway soon to add partial credit to other question types, which may be more urgent than adding more ways of calculating partial credit for multiple answer questions, but it would be interesting to see a version of vary points by answer, which would allow assigning a percent of the total points for a correct and incorrect responses to each option. 

That way you could have the option to set a lower percent deduction for wrong answer than percent added for correct answers.  In long multiple answer questions you could even weight the possible answers by difficulty.  A difficult correct answer could have a higher percent added, and an easy to recognize incorrect answer could have a higher percent deducted. 

[pdk4]

Perhaps there is some misunderstanding here. The penalty deduction option we’re advocating does not eliminate giving (+) credit for correct answers. It merely adjusts the credit additionally deducted for incorrect answers in a more flexible way when the number of incorrect answer choices exceeds the number of correct answer choices.

For example, in a Multiple Answer question worth 12 points with 2 correct answers and 4 incorrect answers, here’s how the scenarios play out with the current score calculation algorithms—

(1) Only give credit for correct answers with no penalty for incorrect answers:

Each correct answer = 12 / 2 correct answers = 6 points. No deduction for incorrect answers.

Full credit can be obtained by selecting ALL answers = 6 + 6 = 12. This clearly games the system and should be avoided.

(2) Give credit for correct answers and deduct credit for incorrect answers:

Each correct answer = 12 / 2 correct answers = 6 points. Each incorrect answer = -12 / 2 correct answers = -6 points.

Selecting all answers gives 6 + 6 - 6 - 6 - 6 - 6 = -12, which would be rounded up to 0. Very harsh penalties.

Selecting 2 correct answers and only 2 incorrect answers gives 6 + 6 - 6 - 6 = 0. We would prefer this not to be zero, but rather something > 0, although not full credit. This combination entailed some thought by the quiz taker instead of just blindly choosing every answer with no thought. These two combinations should be scored differently. This is the problem with the current scoring algorithm.

So we propose this additional 3rd option—

(3) Give credit for correct answers and deduct credit for incorrect answers a different way:

Each correct answer = 12 / 2 correct answers = 6 points. Each incorrect answer = -12 / 4 INCORRECT answers = -3 points.

Selecting all answers gives 6 + 6 - 3 - 3 - 3 - 3 = 0, as desired. The system still can’t be gamed by blindly choosing every answer.

Selecting 2 correct answers and 2 incorrect answers gives 6 + 6 - 3 - 3 = 6. There is still a penalty for choosing some incorrect answers, but some credit is now given for correct answers. This combination is now distinguished from blindly choosing all answers.

This is the system we desire for questions with more incorrect answers than correct ones.

Please note that for questions with more correct answers than incorrect ones, option (2) would still allow the system to be gamed by choosing all the answers. Option (3) would give 0 score in that case and also prevent gaming. For anyone who feels that option (3)’s penalties are too harsh here, option (2) is still available.

The reason we advocate option (3) is that it can be programmed using basically the same algorithm calculations and interface as option (2). Just one calculation is altered and added, and one more checkbox is added.

Enabling the ability to assign individual credit to each answer choice would certainly be a wonderful addition, too, but it would require more new computation algorithms and user interface tools than option (3)’s and would entail much more programming. Option (3) is a quick addition that shouldn't require lots of programming overhead and would resolve lots of issues that have been raised concerning Multiple Answer scoring.

I hope this helps clarify our request and reasoning.  Thank-you for considering it.