New Quiz Tool: Quiz and Item Analysis Report, Is Discrimination Index Valid When Using Item Banks?

[guswatte]

I use item banks for my exams with nested random selection of questions. In the Item Analysis Report, Difficulty Index and Mean values are correct. RPB appears incorrect because it is always 1- for every item. Performance by quintile graph is also off because all students with wrong answers are place in the bottom quintile and all with right in the top quintile. None are placed in 2nd, 3rd, or 4th quintile.

Discrimination Index values appear right but above gives me pause. Curious what your experience is.

[igustafson]

@guswatte 

If you are still encountering this Id recommend contacting Canvas Support to look into it as we would need some examples to take a look. you can find how to contact them by following the steps here https://community.canvaslms.com/t5/Canvas-Basics-Guide/How-do-I-contact-Canvas-Support/ta-p/389767

[leward]

@guswatte .  I share your concerns.  See my recent post, Performance by Quintile Conundrum.    You should also take a look at the frequency counts and percentages for multiple answer questions.  The counts are correct, but the percentages are wrong.  The calculation they are using is:

• number of students who select answer 
  divided by
• total number of selections for all answers

The correct calculation is:

• number of students who select answer 
  divided by
• total number of test takers

Here are screenshots of  the same question with the same student responses in Classic Quizzes and New Quizzes:

Classic Quizzes:

New Quizzes: 

Note that the answers have been shuffled in the report.  Shuffling was enabled in the quiz and regardless, the report should show the answers in their original order.

I created a ticket with support and the first consultant said the New Quizzes graph was correct.  I asked for reconsideration and now a second suppirt engineer is looking at the issue.  Still haven't heard back.

Is this crazy or what?