Discrimination Index vs Point Biserial

[Vastib]

Under quiz statistics>Question Breakdown, CANVAS displays a "Discrimination Index" value. But in the Item Analysis output for a given question, that number is actually the "Points Biserial" value and, as far as I can tell, the actual Discrimination Index is not calculated anywhere. I calculated it myself manually and the real Discrimination Index is very different than the "Discrimination Index" reported on CANVAS. 

Discrimination index and Point Biserial are different things with very different claculations, Yet, unless I am mistaken CANVAS is using the later but reporting it as the former. What numbers are actually being reported and how are they calculated?

[Chris_Hofer · Accepted solution]

Good afternoon, @Vastib ...

Full disclosure...I am not an instructor, and I do not teach courses in Canvas.  I am a Canvas administrator for our school's Canvas environment.  I'm not sure if this will help or not, but I thought I'd provide you with a couple resources just in case you've not had a chance to review these yet.  You may have seen this already, but there is a small "?" icon next to "Discrimination Index" that you can click on within the Quiz Statistics screen, and it explains "The Discrimination Index Chart":

This metric provides a measure of how well a single question can tell the difference (or discriminate) between students who do well on an exam and those who do not.

It divides students into three groups based on their score on the whole quiz and displays those groups by who answered the question correctly.

There is also a link below these two paragraphs to the following Guide: Once I publish a quiz, what kinds of quiz statisti... - Canvas Community.

Beyond this, unfortunately, I won't be of much help.  There may be others here in the Community that understand this data better than I do.  Another resource for you would be to reach out to the Canvas Help Desk staff.  They might be able to give you more detail on how things are actually calculated.  How do I contact Canvas Support? - Canvas Community

I hope some or all of this will be of help to you.  Sing out if you have any other questions about this...thanks!  Take care, stay safe, and be well.

[JohnBriginshaw]

First of all, thank you Chofer for the good faith effort to answer this!

However, looking at the document you referenced ( https://community.canvaslms.com/t5/Instructor-Guide/Once-I-publish-a-quiz-what-kinds-of-quiz-statist... ) I cannot reproduce the discrimination index using the calculation methods shown and the CSV data. As noted by Vastib, what the quiz statistics screen seems to be showing is the point biserial.

There are two sources of information on quiz statistics:

1. Quiz summary screen (obtained by clicking on "Quiz Statistics" when in a quiz) - this includes a calculated discrimination index
2. Item analysis CSV (obtained by clicking on Item Analysis link, when in Quiz Statistics) - this does not list a calculated discrimination index 

For me, it would be great to be able to calculate, or otherwise download discrimination indices, but they are not listed in the item analysis. [However, what is in the item analysis CSV (of which more later) is the "Point Biserial of Correct".]

Looking at the "Once I publish a quiz, what kinds of quiz statistics are available?" link from Chofer, it would seem to be trivial to reproduce the DIs! I quote from the link:

"It (the discrimination index) divides students into three groups based on their score on the whole quiz: the top 27%, the middle 46%, and the bottom 27%. The number of correct answers from the bottom group is subtracted from the number of correct answers in the top group, then the total is divided by the size of the group."

But when you use the data in the item analysis CSV (source 2), the calculated discrimination index is not the same as the one displayed on the quiz statistics screen (source 1).

Specifically, (using the CSV column titles) I'm calculating (Correct Top Student Count-Correct Bottom Student Count)/((Top Student Count+Bottom Student Count)/2)

[Average group size is used because the bottom student count is different (slightly less) than the top student count in my CSV].

The calculated numbers are not the same as the displayed discrimination index in nearly all cases. Sometimes they are close (for 7 out of 37 questions in my quiz, the difference is less than +/-0.02). The average difference is 0.08 (23% of the displayed DIs).

However, the displayed discriminant index from source 1 (screen) is identical to the Point biserial of correct column in source 2 (the CSV). So you can use that column to download the displayed DIs that you see on the screen. 

Some discussion of the strengths of the point biserial are shown here -- Use point-biserial to discriminate high and low performers | GradeHub

[MarcusStrömbäck]

This does not answer the question.
The question is what model does canvas use to calculate the discrimination parameter? 

[feigl]

Vastib is correct. The numerical value displayed on the Quiz Statistics page is the "Point Biserial of Correct" value in the CSV file of the Item Analysis Report.  This is misleading and confusing. The Quiz Statistics page should be corrected to change "Item Discrimination Index" to "Point Biserial of Correct". The "help" page with the question mark should revised entirely.

Furthermore, in the case where 100% of the students answer the question item correctly, no value should be displayed because the PB is not defined. (Any correlation coefficient requires a sample size greater than 1).

Yes, it would be wonderful if Canvas could calculate the Discrimination Index, known as "Cronbach's Alpha", for each item. Indeed, the overall alpha value for the quiz (as a whole) is reported in the CSV file as "alpha". 

Cronbach, L. J. (1951), Coefficient alpha and the internal structure of tests, Psychometrika, 16, 297-334. 

https://doi.org/10.1007/BF02310555