Faster access to New Quizzes Item Analysis is in beta

[TamasBalogh]

As mentioned earlier, our team has been working on faster access to the Quiz and Item Analysis. We are pleased to announce that this new functionality is now available in the beta environment, followed by the production release a month later, on September 16th.

Important note: Reports in beta are not available for the submissions made in production and copied to beta environment during the weekly beta refresh. Please duplicate an existing quiz or create a new one then create submissions before generating the report.

How does it work?

We have completely reimagined the process of generating reports. Instead of regular intervals, educators now can trigger generating a report when they require it. Upon navigating to the reports page, you will notice a new card with a “Generate report” button. Once the report is ready, the card will display basic metrics and you can view the whole report or initiate a new generation. 

Report card before a report is generated

Report card for an existing report

The time required to generate a report is influenced by factors such as the number of items in the quiz, submission quantity, and concurrent generation occurrences. 

Moreover, our improvements extend beyond the generation process; we significantly improved the analysis as well with valuable input from Instructure's psychometrician. This empowers educators to more effectively identify potential issues or areas for enhancement within quizzes and items.

With the production release in September, we will release a comparison document on the differences between the old and new report, and will also update the guide on how to use Quiz and Item Analysis. Furthermore, we are preparing a comprehensive guide that delves into the interpretation of the analysis.

The old process' availability

While we embrace these enhancements, we acknowledge the value of the existing process. The previous method will continue generating reports until the end of this year, affording educators ample time to embrace the new report, acclimate to its usage, and adapt their established workflows. We will also make sure the old reports generated before the end of this year are available until the summer of 2024. Beyond that point, only the new reports will be available.

[paulamiranda]

This is great news, thanks for the update and explanation. I might be testing too early, but I did try this out in Beta today and keep getting error messages when trying to generate the reports. To confirm, the quiz I am using to test does have submissions, and the New Quizzes feature option is enabled and unlocked in our Beta environment. I understand that there might still be some buggy behaviors for new features, but wanted to report it, regardless, in case there is something we need to do on our end. Thanks!

[Pete5484]

I don't get unauthorised, but do also get the error. New Quizzes, data exists,...

[TamasBalogh]

Hey @paulamiranda, 
Thank you for reporting the issue, we are working on a fix already. Meanwhile, you can avoid the unauthorized issue by waiting a few seconds between navigating to the reports page and clicking on the "Generate Report" button. 

@Pete5484 and all, some of the data which are needed for calculating the report are not synchronized to the beta environment. Even though, as a user, you don't notice the lack of data, but they are still missing for the calculations. Unfortunately, you will need to duplicate one of your quizzes and make submissions in the beta environment in order to test the new process. This is the same for the old report, you can see it always shows "No reports yet". 

I'm going to update my blogpost with this information

[JoeNewman1]

Hello,
You're welcome! It's great to hear that the issue is being addressed, and thank you for the workaround suggestion. Taking a few seconds between navigating to the reports page and clicking the "Generate Report" button sounds like a practical way to avoid the unauthorized issue while the fix is being implemented. This interim solution can help ensure a smoother experience for users. If you have any further updates or questions, feel free to share. TellPopeyes

[TamasBalogh]

Hi @JoeNewman1 and all, the unauthorized isssue has been fix, now you shouldn't see that anymore. Please let me know if you have any feedback on the new proccess.

[jrichter1]

I have been trying to use the new report but am still getting the same error. How many responses do we need to put in to use it and how can we do more than just the 1 test student?

[TamasBalogh]

Hi @jrichter1,

The minimum sample size is 3. I'm not sure how is your account set up, but you should be able to add users to your course as students and then login with their credentials, or if you are an admin, you can "Act as" those users.
We are working on adding more detailed error message in case the issue is the low count of submissions. I understand it is not straightforward now. 

[KathyPalm]

Very exciting! thanks for the tip and notes in this post. I now have generated it in Beta with 3+ responses (I had only used 2 before) and it's very nice to see it able to update on demand and get a feel for how it will work. 😃

Just to confirm, when this is in production the new report can be generated on past responses to assessments, correct?

[vanzandt]

I found this guide, https://community.canvaslms.com/t5/New-Quizzes-Resources/New-Quizzes-Item-Analysis-Report-Comparison... , that was recently posted that compares the two report formats. (very helpful!)  Can you all also add a column that provides the comparison against the Classic Quizzes report? Thank you!

[TamasBalogh]

Hi @KathyPalm, I'm not sure if I know what you mean by past responses to assessment. Can you please elaborate?

@vanzandt I'm glad you find that useful. We'll have a look to see if it makes sense to add a column for Classic Quizzes. With the production release, we will post more (hopefully 🙂 ) useful guides. 

[KathyPalm]

@TamasBalogh I would like to confirm that we can generate the new report based on quiz responses completed prior to the update going live. So if we have a course that has already had a quiz prior to the update, we could still go in and generate the new report after the update.

In beta we have to create new responses in order to generate the report but my understanding is that is because of how production data is brought over to beta, but that's where the question came from.

Thanks!

[TamasBalogh]

@KathyPalm yes, you should be able to go back to your previous quizzes and create a report!

[vanzandt]

Thanks @TamasBalogh .  If it doesn't make sense in this particular comparison guide, maybe it could be a separate comparison with the Classic Quizzes report so that it has items that make more sense in that context.  I think it would be a very helpful resource, since there are still a large number of institutions that haven't enabled New Quizzes.

[kettlebarm]

It's time to let New Quizzes go....Classic is what everyone uses, make that product better.

[GideonWilliams]

A lot of produce development seems to be centered on what happens once a student has completed a quiz and I get that this is important. I would also like to think that Instructure has not forgotten about how to get more people on to the quiz in the first place and to get students engaging more with their learning rather than being passive users (reading, watching, uploading etc)

To that end, my list of 4 feature improvements I am hoping to see:

(i). RCE (editor) in all quiz types - having it for Categorisation and Matching would bring in images (hey Art teachers, try this starter on grouping paintings by style), sound files (imagine taking that to MFL teachers), equations and formulas (come and look at this Maths teachers)

Its already in Ordering - surely its not a difficult step to get to the other two?

(ii). Hot Spots

(iii). RCE (editor) in Studio

(iv). Something that will automatically convert ms-1 into ms(-1 as a superscript) every time I type!!!

To be honest, I'd happily settle for just No 1.