[ARCHIVED] Bug Report: Multiple answers questions don't score properly

[dbastedo]

I have made up several exams (quizzes) for my online classes.  I tried using the Multiple answer type questions where I was looking for the student to pick two correct answers out of a list of 8.  If the student picks only one answer instead of 2 it corrects that response properly.  But, if the student picks two answers and one is right and one is wrong, it may report 0 points or it may report 1 point.  It is not consistent.   It marks the responses properly.  I see a green or a red indicator next to the student's responses, but it doesn't score the question correctly.  The student may have one green and one red, one choice correct and one choice wrong, but the score is 0 points!!  I have to manually go back through all the students and find this question and correct it manually on all of their exams.  My hypothesis is that out of the two correct answers that I indicated if the first answer is wrong and the second one is right, then it doesn't score the second correct answer. When it goes to assign points, it doesn't assign any points even if the second answer is correct.  Just a guess.  The marking choices as right or wrong is fine, but the scoring is messed up.

[Shar · Accepted solution]

Hi David M. Bastedo,

Hopefully you're talking about Classic Quizzes. To the best of my knowledge multiple answers DOES subtract points for wrong answers. Check out Understanding Multiple Answers Questions

"a student loses the same amount for marking a wrong answer that they get for marking a correct answer."

What their page doesn't say is that a student's score won't go negative and the minimum score for the question is 0.

I tried your hypothesis about which correct answer it selects affects the number of points and I get consistent results with the scoring. Can you provide screen shots?

With multiple answers expecting 2 correct answers and a student answers 1 right and 1 wrong, the score is 0 points. In my example above, I made the questions worth 2 points, meaning 1 point for each correct answer.

And just in case, you're talking about New Quizzes. New Quizzes: Multiple Answer Partial Scoring (2020-04-18 Release) We don't use New Quizzes yet so I have no experience with any of the question types there.

Hope you find something useful, and again I recommend the Blog Post by  @James : Understanding Multiple Answers Questions.

Stay home, stay healthy,
Cheers - Shar

[Shar · Accepted solution]

Hi David M. Bastedo,

Hopefully you're talking about Classic Quizzes. To the best of my knowledge multiple answers DOES subtract points for wrong answers. Check out Understanding Multiple Answers Questions

"a student loses the same amount for marking a wrong answer that they get for marking a correct answer."

What their page doesn't say is that a student's score won't go negative and the minimum score for the question is 0.

I tried your hypothesis about which correct answer it selects affects the number of points and I get consistent results with the scoring. Can you provide screen shots?

With multiple answers expecting 2 correct answers and a student answers 1 right and 1 wrong, the score is 0 points. In my example above, I made the questions worth 2 points, meaning 1 point for each correct answer.

And just in case, you're talking about New Quizzes. New Quizzes: Multiple Answer Partial Scoring (2020-04-18 Release) We don't use New Quizzes yet so I have no experience with any of the question types there.

Hope you find something useful, and again I recommend the Blog Post by  @James : Understanding Multiple Answers Questions.

Stay home, stay healthy,
Cheers - Shar

[James]

ishar-uw,

I looked at New Quzzes a couple of weeks ago when they made the announcement. We don't use it either, but it's available in the beta instance. The partial credit scoring there is the same as was in legacy / classic quizzes.

David's hypothesis is incorrect, which is why it seems confusing. When you understand the rules, whether you agree with them or not, it makes sense.

1. You divide the number of points by the number of correct responses to get what fraction of the points each selected item will be worth.
2. Add up the number of correct responses that were selected and subtract the number of incorrect responses that were selected.
3. Multiply this by the fraction obtained in step 1, but don't go negative.

The order of the items has absolutely nothing to do with it and there wasn't even a need to test it. That document I wrote really does explain everything.

Only selected items are used in grading and every selected wrong answer cancels out a selected right answer.

[dbastedo · Accepted solution]

Thank you both for responding.  I obviously misunderstood the way it is graded.  I didn't realize that a wrong answer was subtracted.  I was trying to get two independent questions operating in one question. When the student has one response correct and one incorrect, I thought that each would earn or not earn a point, not that one answer would penalize the other correct answer.  I was assuming that right answers were worth 1 and incorrect answers were worth 0 not -1.  I am using the classic quizzing.  I guess I wish that I could customize that feature.  I could get a lot more questions into a test with less work if I could put two independent questions in one.

[James]

They are confusing and unintuitive for most people. The first time I tried using them, I thought it would be treated as multiple true/false questions because I hadn't read the guide, I thought I _knew_ how they worked. Most people have a way they think things should work that is based on their experiences and can't imagine any other way because they haven't been exposed to the other way. I only made the mistake of thinking they were true-false questions once, though.

After analyzing them, the way Canvas does it is the only way that makes sense to me now. The multiple true-false way would have given points if a student left the question completely blank. The student is required to take some action (check a box) to get any kind of scoring with Canvas.

I did write a user script that runs in your browser on top of Canvas and can regrade the questions using a different manner. The development of that script is the main reason I wrote the analysis, so that I could explain what was happening and how people wanted different techniques. I think it's linked at the bottom of the analysis document.

With New Quizzes, there is a stimulus question that allows you to ask multiple questions based off a single stimulus. With that, I could ask my question and then make 5 true-false questions instead of 5 responses in a multiple answers question.