Enhancements and CSV Support for New Quizzes Item Analysis

[TamasBalogh]

In September, we introduced the new Quiz and Item Analysis report, focusing on improving its speed. I am delighted to announce that the 24-hour report generation time has now been reduced to an average of under a minute.

While the report has seen significant improvement, we acknowledge that some key features are still missing, such as Answer Frequency Summary tables for all question types and a downloadable format for further analysis.

Enhancements

Since the initial release, we have continued to make improvements, including:

• Answer Frequency Summary tables for Categorization, Ordering and Matching question types are in beta and will be released to production later this month.
• We introduced a new table to summarize the Answer Frequency Summary tables, by aggregating all the correct and incorrect answers. This is available for Categorization, Ordering, Matching, Hot spot and Formula question types in beta and will be released to production later this month.
• We are releasing the same for Multiple Choice, Multiple Answer and True or False question types today.

CSV support

Today, on January 15th, 2024, we are launching CSV download support for the Quiz and Item Analysis in beta, with the production release scheduled for one month later.

In response to positive feedback received on adding JSON to the CSV report in my last blog post, we decided to add the Answer Frequency Summary table data to the CSV in JSON format and soon will be updating our guides to include instructions on downloading the CSV file and understanding the JSON format.

Next steps

Our commitment to improving the Quiz and Item Analysis continues. The team is currently focused on adding the Answer Frequency Summary table to Fill In The Blank question type, set for release later this quarter.

Discontinuing the old report

The old quiz and item analysis report generation will be discontinued on January 22nd. From that date, only the new process will be available for generating reports. However, previously generated reports for older quizzes will remain accessible until the summer of 2024, giving users ample time to transition and familiarize themselves with the new process and calculations.

For those unfamiliar with the changes, you can compare the old and new reports here or gain a better understanding of each metric and calculation method here.

[KathyPalm]

Thanks for the updates 😃

[RobDitto]

The Quiz and Item Analysis (Old) report version's score distribution histogram includes a 100% (full-credit) bin:

That bin is missing from the (New) version:

Our faculty who have adopted New Quizzes have flagged this as a loss of important functionality.

@TamasBalogh, you note above the (New) report is still being worked on. Given that, can the restoration of the full-credit bin be considered?

[HillaryBurgess]

I am pleased to see that it is on the devs radar to improve exporting/downloading quiz reports. I have worked with support to figure out why my PDF export is cut off at 1 page in new quiz reports, and they weren't able to help me.

This is a big hang-up for my department as we are preparing for reaccreditation and need these reports in PDF format. Would you be able to share a timeline for this feature? 

Thank you! 

[TamasBalogh]

Hi @HillaryBurgess,

We don't plan to release the report in PDF format, only CSV. 

[ptjarnigan]

@TamasBalogh Is there any update on adding Student Analysis to the New Quizzes reports option?  I've seen people asking for this since at least 2021, and the last update I saw was from someone saying it was scheduled for July 2023.  Obviously that has not happened yet.

For example, here is a Canvas Idea, part of the "Improve quiz analysis through robust reporting" theme.

[New Quizzes] Add Student Analysis report with option to download Submitted by: thackead

[mbmacdonald]

I have the same question as @ptjarnigan. I have users asking about this. And I will mention that I find myself once again in a position of having to check and monitor multiple places for info about New Quizzes: both Ideas and Themes and the New Quizzes Roadmap. I noticed that some users were disappointed that a New Quizzes-related theme was selected for development within Ideas+Themes because we had been under the impression that New Quizzes development was a separate workflow from the main product.

[TamasBalogh]

@RobDitto I'm sorry, I've just seen your message. 
We can look into that, but it might take some time to figure out a great representation. Please create an idea in the community page for this! 

[TamasBalogh]

Hi @TiffanyStull2, We do provide a warning of the lack of student analysis report for New Quizzes (actually the warning is outdated as it says Item Analysis can't be exported to CSV neither, which is no longer true since last Saturday, we are working on removing that part of the warning). 

[TiffanyStull]

@TamasBalogh according to that message, "CSV downloads [...] are available through Classic Quizzes."  There is no noticeable connection between that information and the strong recommendation the rest of the message provides for New Quizzes.

All I see for New Quizzes is, "This has more question types like hotspot, categorization, matching, and ordering. It also has more moderation and accommodation features." As an instructor, the tool that only mentions having more things and doesn't warn me that it lacks something is the one I'm going to pick.

[TiffanyStull]

Context for the above two posts (the community boards keep creating a second account for me when I try to log in, and I inadvertently posted with the extra account, so the post has been deleted):

We are also having multiple instructors running into the issue of not being able to export student responses in a CSV file.  Instructors are extremely frustrated to discover this after students have already taken a quiz/exam.  This is especially frustrating because we have recently moved to Canvas and previous LMSes in use here (including instances of Canvas where only Classic Quizzes were in use) all offered response export options.

The New Quizzes tool currently provides no warning that instructors do not have access to a CSV export during the quiz creation process. While the feature is still being developed, can you please add a banner warning in the quiz authoring screen indicating that "Exporting student responses as a spreadsheet file is not possible at this time," so instructors are warned before they create a quiz and end up stuck with this issue after it has been submitted?

[TiemeyerM]

It is incredibly frustrating that the analysis report doesn't always work with New Quizzes. It feels like half of the time the report generation just fails.  IT feels pointless to use New Quizzes until this is fully figured out. 

[paul_fynn]

@TamasBalogh can you confirm what the status of New Quizzes is currently in terms of reporting participations back to Canvas New Analytics please ?

I understand that some UK HEI's have deferred a move from Classic Quizzes until New Analytics can reflect student engagement with New Quizzes.

[mbmacdonald]

@TamasBalogh, I would also like to hear the answer to @paul_fynn's question above. I have found the documentation to be unclear. According to View page view and participation definitions, "Activity data currently includes LTI launches; however, it does not contain detailed LTI tool activity. This is true for New Quizzes and Studio, as well as non-Canvas LTI tools. New Quizzes data is planned for the new Course Analytics offering but a timeline has not yet been determined." What does "Activity Data" in the context mean - does an LTI launch (including new quizzes) currently get listed as a page view or a participation? I also found this is still listed as a Known Issue. Also, I am not totally sure if @paul_fynn is referring to "New Analytics" (the thing in the course navigation menu) and/or "Admin Analytics" (the thing in the Account Navigation menu, which is sometimes, maddeningly, referred to as "New Analytics"). My assumption is that whatever gets counted as a page view or a participation gets reflected in *both* dashboard, but I really don't know because the documentation is so unclear.

[JamesSekcienski]

@paul_fynn and @mbmacdonald,

Based on Tamas's role of "Instructure Alumni", I don't think he is working with Instructure any more.  We may need to wait for the next New Quizzes update to learn who the new Technical Product Manager is for it. 

[JamesSekcienski]

@MPioRoda Are you able to provide any insight into the questions from @paul_fynn and @mbmacdonald 

[MPioRoda]

HI @JamesSekcienski , you're correct, Tamas has left the company and we're interviewing for a backfill position.  Let me get with his team and see if I can get some answers for you.