New Quizzes' Item Analysis update

[TamasBalogh]

You’ve likely seen my recent post about the work my team has been developing the New Quizzes API and integrating the Rich Content Editor (RCE) to New Quizzes in the last couple of months. In addition I’ve had some time to dig into Item Analysis.  

Based on conversations with users, I understand that you need faster access to Item Analysis and to be able to download the report through CSV. I’ve also discovered that we could provide a more useful and intuitive interface with additional features to help you or your faculty to get an overview of the quiz and items performance. 

I’ve met a handful of customers to get great insights and based on that feedback we’ve created designs for an improved interface. I’m thrilled with what we came up with, and I hope you’ll like them too! In Q4, we will continue the discovery work around Item Analysis and vet the designs with users in order to understand where we've hit the mark and where we could use some additional fine tuning.

[YoshieRosales]

Your "Benchmark" label article that I reviewed is very impressive and the new eLumen+canvas has taught students like me from the post on J. Williams profile the benefits of this in-structure website. Not to mention the opportunity to continue learning and give space to our assignments as well. Navigating through my semester has been fun too!

[akraft2]

The faculty at our college of nursing need the ability to filter an Item Analysis in New Quizzes by section. The 2023 NQ roadmap suggests various improvements for running an Item Analysis, but I don't see filtering by section explicitly mentioned. Will this feature be added in 2023?

[Hildi_Pardo]

One of our 4th grade teachers started using New Quizzes.   One thing we could do in Classic Quizzes was to see which students responded with which answer, but we can't in New Quizzes.  See the image below -- I expanded a Question on the Reports page to see Item Analysis -- I cannot click the number under Respondents to see which students answered, which would be incredibly helpful.  

I can see individual student's attempts under Moderate (I have to click to open one by one), but under Item Analysis cannot see which were the group of students who answered.

[NHawkins2]

I agree.  I need to be able to see which students are answering the questions correctly or incorrectly in the new quizzes.  I will continue to use the old quiz until this is fixed in the new quizzes.

[TamasBalogh]

Hi @akraft2, @Hildi_Pardo, @NHawkins2  although we haven't included the details in our roadmap, we are improving the analysis with various details, filtering by section list the students under respondents are among our plans and most probably will be included in the improved analysis.

[jon_mason]

Hi @TamasBalogh, I'm glad that you mention a CSV export of the Item analysis. Can I check that you are also planning to enable a CSV export of students responses, the lack of which is currently tying a lot of our academics to Classic Quizzes? Thanks!

[jon_mason]

And just to mention a couple of other analytics-related annoyances in New Quizzes:

• New Quiz attempts by the Test student are not deleted when the Test student is reset. See case #09303543, but support wouldn't accept it's a bug, so there's an idea here: https://community.canvaslms.com/t5/Idea-Conversations/New-Quizzes-Delete-test-student-attempts-when-...
• RPB seems to always be the same for all questions in a quiz, which is obviously possible but unlikely. See case #09293971

[TamasBalogh]

Hey @jon_mason, thank you for feedback. Yes, we have student responses on the roadmap and we plan to make it available through CSV as well:
"Instructors can use detailed student analysis reports for deeper insights of student understanding and activity on quizzes."

Let me reach out to you in private message to talk more.

[YoshieRosales]

Now I see what this means! Thank You. It will help me recognized more in depth writing for my classes too. To start off, calculating and rewriting is what I am attempting to accomplish at right now...

Sounds good! Is the above link the correct address for you as well?

[Tasha_Biesinger]

Hi @TamasBalogh, we just had an instructor run into an issue with quiz analytics with Classic Quizzes because they have more than 1000 students in their course AND because Classic Quizzes Item Analysis doesn't include all questions types.

Can you talk about if/what limitations New Quizzes statistics have? Will they be able to see at least % correct values for all questions in New Quizzes stats even when there are more than 1000 students in the course?

Thanks!

[TamasBalogh]

Hi @Tasha_Biesinger,

The new Quiz and Item Analysis report in New Quizzes doesn't have this kind of limitation and I believe it should support any number of students or questions. I'm sure there is a theoretical limit but it should be way bigger than in any real life scenario. 

The report currently fully supports multiple choice, multiple answer and true or false question types. Other question types have the basic metrics as well, like Discrimination Index, Item difficulty, Corrected item-total correlation coefficient, mean/median score, but don't include Answer Frequency Summary tables for now.