New Quizzes End of Quarter Update for Q3 2024

[DanaOkerlund]

Over the past quarter, we've made strides in enhancing the New Quizzes experience with several updates. One of the highlights is the introduction of Custom Feedback, allowing instructors to provide personalized feedback to students after quiz completion. This feature enriches learning by guiding students on next steps and reinforcing core concepts. We've also expanded the New Quizzes API with scoped endpoints, giving administrators more control over token access. 

Looking ahead, we're excited to bring even more flexibility for managing student results, including the ability to adjust visibility settings and tailor quiz experiences to student needs. Additionally, new features like the ability to add time to in-progress quizzes and seamless migration of question banks will further streamline the assessment process for educators.

Q3 Achievements

Show Custom Feedback with Results

Custom Feedback in New Quizzes allows instructors to provide personalized feedback to students after they complete a quiz. This feature helps reinforce key concepts, guide next steps, and enhance understanding. 

Instructors can toggle the “Show custom feedback with results” option in the Settings, then add feedback using the Rich Content Editor. By default, this option is off. Once enabled, custom feedback is displayed to all students on the Results page after they finish the quiz.

This functionality was released to production environments on August 17. (Release notes)

Scoped Endpoints for New Quizzes API 

New Quizzes endpoints are now available in token scopes when adding an API Key for admins. This update allows admins to restrict API token access to specific endpoints, ensuring more controlled and secure API interactions. These endpoints can be selected within the token scopes when creating an API Key.

This functionality was released to production environments on September 19. (Release notes)

Q4 2024 Objectives

Manage Student Results View

We expanded the options for managing student results view settings. Instructors can now control when to show student responses and whether to indicate if responses are correct or incorrect. Additionally, instructors have the flexibility to limit response visibility to once after each attempt, only after a student’s last attempt, or only once after the final attempt. Time restrictions are now managed with clear date and time fields, providing more options for tailoring the quiz experience to student needs.

Hiding Results from Students is available on the Settings page and disabled by default. When enabled, instructors can control what students see after completing a quiz. Main options include displaying questions, possible points, and awarded points. The visibility of questions and responses can be customized based on attempts and specific time frames. Additionally, there is an option to show correct answers, either immediately or only after the student’s final attempt, with flexible “Show on” and “Hide on” dates.

This functionality is being released to production environments on October 19. (Release notes)

Add Time to In-Progress Quiz Attempts

We’re enhancing the quiz management experience by allowing instructors to add additional time to a quiz in progress. Soon, educators will be able to provide just-in-time accommodations for students while they are actively taking a quiz. This improvement ensures a more equitable assessment process, eliminating the need for instructors to rely on workaround solutions to extend time during an active quiz session.

Question Bank Migration on Course Copy

We’re improving the New Quiz migration process by adding additional support for question banks. Soon, educators will be able to seamlessly migrate their Question Banks to Item Banks in New Quizzes using Course Copy. This enhancement offers a straightforward alternative to the workarounds currently required for those who rely heavily on Question Banks for quiz content management. With this additional bank migration support, all existing quizzes and question banks will be imported as New Quizzes with a single check during course copy. 

Student Analysis CSV Export 

Last year, we introduced Quiz Item Analysis, and now we’re excited to announce upcoming improvements to Student Analysis in New Quizzes. The Student Analysis report will be available via the user interface for CSV export, providing comprehensive insights into student behaviors and outcomes in New Quizzes.

Data points to be included in the report are as follows. 

Student Information:

• Name
• ID
• SISID
• SectionID
• SectionSISIDs
• Submitted
• ElapsedTime
• Attempt

Quiz Item Information:

• ItemID
• ItemType
• Item-specific columns for each item in the quiz (Question, EarnedPoints)

Quiz Performance Summary:

• numberOfCorrect
• numberOfIncorrect
• NoResponse
• OverallScore

Reporting API

We’re developing additional API endpoints to give access to Quiz Item and Student Analysis reports, making it easier to facilitate internal reporting and analysis.

Canvas Commons

We are continuing our efforts to make it easier to share your New Quiz content with other educators. As part of that effort, we are working to support sharing through Canvas Commons. 

Course files in Item Banks Known Issue update

Updates on the Known issue related to course files in item banks can be found in this blogpost. 

Blue Printing & Course Copy Issues 

Over the past few months, we have noticed that our customers have had an inconsistent experience as they use the blueprint, send to, and copy to features with New Quizzes. We have identified root causes and plan to deploy code in the next release to resolve many of the issues. 

Fixes targeted to be released by mid November:

• Common Cartridge exports will complete even if New Quizzes fail to export
• Quizzes shared using the 'Send to' workflow will import correctly
• Resolved import issue that lead to failures or quizzes imported with no items

Issues that we are currently investigating:

• New Quizzes syncing from Blueprints without questions
• New Quizzes are failing to duplicate and instead return an error of "Oops! Something went wrong with making a copy of "
• A New Quiz is stuck in “Copying assignment...” after syncing from the blueprint course

Quarter 3 Completed Fixes and Features

Here is a list of the related fixes or features that we completed in the past quarter:

• The quiz migration alert is now displayed only when there are quizzes to migrate
• An info page is now shown to students when they open an untaken quiz in a concluded course
• Quiz results in concluded courses can now be viewed by students
• Formula questions with multiple variables are now imported and exported as expected
• Strikethrough icons are now used for Choice Elimination
• Accessibility
• Clear error messages are now provided when creating stimuli
• Clear error messages are now provided when adding questions to an item bank

[jmerlenbach2]

Regarding the Student Analysis CSV Export - will that report honor the SIS Data - Read permission for user roles? Meaning, will users who do not have that permission be able to run the report, or if they run it will it not include the SIS information for users/courses?

[patrickdorls]

Great news! We're especially looking forward to "Add time to in-progress quiz attempts". This has been a huge problem for us since 2020!

[JoshAllenCC]

I see in the release notes that the extended time got pushed back until the 30th - did the same thing happen to the results view update? I wasn't seeing it this morning.

10-22-24 update: I am seeing it this morning. @patrickdorls 

[patrickdorls]

I contacted the Tier 1 support about this yesterday and received a message that there was a slight delay. The result view has been updated for us now

[paul_fynn]

@DanaOkerlund 

Many thanks for this

1. are you able to confirm that the longstanding issue with where Quiz images were stored has been resolved, please
2. there have been multiple threads and discussions which have referenced the fact that student New Quiz attempts do not appear as Participations in Canvas Analytics - this is a major limitation impacting both New Quizzes and Canvas Analytics, and I understand that some UK HEIs have still not transitioned from Classic Quizzes because of this.
   • One third party analytics vendor has suggested that they did have access to the New Quiz LTI background data and could produce participations

Is this on the roadmap to be resolved please ??

[audra_agnelly]

Mid-November to fix the Send To issue??? We have been reporting this behavior since August. The turn around time to fix this very disruptive bug has been extremely disappointing. That is on top of the back and forth I had to go through with Support to make someone believe that this was an issue, then followed by more back and forth where I was told the issue was resolved only to see the first Send To/Import issue I tested failed to copy correctly. We are fortunately not seeing issues with Copy To or Blueprints as those have been a lifesaver to get content out to my teachers at the start of the year. At this point my colleague and I have used Copy To to individually push out New Quizzes and Modules to 100s of courses. I don't want to think about how many hours of work I have lost trying to support my teachers as well as provide documentation of the behavior to support. It is absolutely not acceptable for a key product feature to be non-functional for 3 months.

11/1 Edit - Somewhere I was told the Send To fix would be released on 11/6, I think may have come from our CSM. We've been trying to confirm in Beta that this is working. My team member tested last week and could use Send to to share an individual quiz, but when sharing a module with a New Quiz, that quiz was coming through as just a title. I have tried testing this week and I can't even access New Quizzes in Beta. On Monday I was receiving a We're sorry, but something went wrong message when I tried to create a brand new New Quiz and when I try to open the Build screen of an existing quiz I see a 503 error pop-up message at the top of the screen. That goes away after a second and the build screen is just stuck "Loading...". I tried again today (Friday) thinking this was perhaps a temporary issue in Beta but I'm still seeing the same behaviors. Is the Send To fix still scheduled for the upcoming deploy on the 6th? 

I would also like more information about "supporting sharing through Commons". Up until August when the send to issues cropped up, my instructional leadership offices had shared assessment modules to Commons that contained new quizzes. I can find older modules in Commons with New Quizzes that import now without issues, but the ones that were shared since August do not contain the quizzes at all. I have shared this with support only to be told Commons is not supported, but again we have been using Commons for years without any issues, and used Commons heavily during the pandemic. But, when school started this year, we were left scrambling to get core content out to our teachers. What are you adding support for when this functionality had been working just fine?

[cdoherty]

@DanaOkerlund Will the Question Bank Migration on Course Copy feature also apply to the Import Course Content method? We don't use Course Copy, as we auto-create all courses, so we would need this to exist in Import Course Content or it will be useless to us.

[GideonWilliams]

What news of the RCE editor for all quiz types and partial scoring for categorisation and ordering questions?

[RobDitto]

This is a very exciting update!

One concern about Student Analysis: it seems that the student's answer to each item in an attempt unfortunately isn't targeted for the initial release of this report. Is that a "forever" limitation, or just what was possible for the initial release?

At our institution, exporting student answers to quiz and survey attempts is at the core of our use case for Classic Quizzes' Student Analysis. I believe lots of customers will be expecting to have that in New Quizzes.

[isabel_anievas]

@DanaOkerlund  Any updates on the known issue related to course files (images and media files) in Item Banks? Is the fix being implemented by the end of October? Will it be retroactive? 

[MARISSASCHRADER]

I would like to reiterate what @audra_agnelly posted as we are experiencing the same issues with Send To and Commons. Further, we have had issues with New Quizzes using Copy To and Importing Course Content for a lot of instructors and it has been an nightmare finding workarounds. Mid-November is along time for this to be resolved. 

[ckeigher]

Add Time to In-Progress Quiz Attempts was put into production today 10/30 and in testing I'm getting very inconsistent results (Canvas Release Notes (2024-10-19)).  I really wish someone involved from Instructure could explain in more detail beyond the minimal info in Canvas Guides: How do I moderate a student's quiz attempt in New Quizzes?

For example, when I've added time to a student's attempt... when exactly is supposed to be applied?  Immediately, is there some delay?  In my testing it is not immediately and therefore sometimes  even though I've given extra time to a student's "in-progress" attempt, their timer and thus their attempt expires before the extra time is even given/applied??  Additionally, I've found that I can now re-open a submitted attempt, but when I do the time allowed for the student continuing the attempt doesn't make sense and the timer itself is just missing altogether.

Is anyone else able to test this and confirm the behavior(s) they are seeing?  Our nursing department in particular is understandably wanting to know exactly how these features now work on their high-stakes exams.

-update- in my most recent test, I watched the timer precisely on a student's in-progress attempt and it took 5 minutes from the moment I gave them extra time to the time that their timer updated to actually reflect/update that. 

[afurr0026]

@ckeigher, I have the same request about additional details needed for this update. I have also experienced similar results in my minimal testing this morning.

Giving additional time on an in-progress attempt does not display immediately for a student. It takes several minutes (@ 3 or more) for the timer to adjust in the student’s view. Anything past 5 minutes, and I ended up refreshing for the timer to display the correct time.

When reopening an attempt with time remaining, the timer displays for the student. However, the time is a few minutes off. Most of my testing shows @ 1 or 2 minutes more than what the student actually had when the quiz was submitted.

My main concern with the way that this feature has been released is that I will be instructing our faculty that the feature for adding time to an in-progress attempt, that we’ve never been able to do until now, is the same process that they’ve used to add time to an individual quiz in the past before any attempts were started. Now, this same feature allows them to also enter time in an in-progress attempt after I have shared numerous times in the past that this feature doesn’t add time to an in-progress attempt. But now, it handles both.

Or it may be that I’m not fully understanding how this new feature works due to the vagueness provided in the Canvas Release Notes (2024-10-19).

[Pete5484]

Same. I lodged a Field Admin ticket last week based on similar testing when this was still in the 'beta' environment. I noted that we were continuing to experience oddities such as displaying incorrect time remaining for a learner, on occasions, when additional time had been added. (Sometimes it updated the student's time; other times it didn't and the quiz auto-submitted at the original time despite having added time). 

Other comments on our ticket were:

• Attempting to adjust Current New Quiz Time does not add time to countdown clock and sometimes the countdown clock disappears from screen.
• There appears to be no way of ascertaining elapsed time (doesnt appear in Moderation which I though it previously did) so it's difficult to ascertain how long student has been completing the quiz.
• The student's quiz log should indicate that additional time was added by [user_id] at time x:xx
• Speedgrader 2: Steel - cannot edit previously submitted but ungraded quiz attempts (can edit if revert to standard Speedgrader UI)

 Instructions and operation still remain unclear. I sent in two videos, haven't yet received a reply to the last msg from us from nearly a week ago.

[ckeigher]

@afurr0026 @Pete5484  Just wanted to add that we seem to now be able to re-open a quiz attempt where the time had expired (thus auto-submit).  We could technically do this before, but with no time left or available, the re-opened quiz would immediately expire again.  Now it seems like re-opening an expired quiz allows a student to resume their last attempt with another full allotment of quiz time.  I have not seen this new behavior documented anywhere by Instructure.  For what it's worth, we do prefer this new behavior, but we need it documented to be sure.

[Pete5484]

@ckeigher Actually, this too doesn't seem to be functioning correctly (and requires documentation).

The dialog to the instructor indicates that if they reopen the quiz then the timer, if applicable, will continue with the remaining time. But, as you note, it resets the timer to the full quiz time plus any accomodation or time adjustments. So either the dialog box is wrong or the timer being reset to the full quiz time is unintended.

Either way, at the moment restarting the full quiz time if you reopen the latest attempt is actually really challenging for us, since we still have students that manage to accidentally submit their attempt (despite the warnings). We will need to monitor and then manually submit the reopened attempt at the correct expiry time... despite the student's timer showing the full time being available. Which.... isn't a great user experience for the student or educator.

[paulamiranda]

Unfortunately, our team has been unable to successfully add additional time to in-progress attempts. While an instructor is able to add the time and have that additional time be noted under the Accommodations column of the Moderate table, students do not see that additional time applied to the timer above their quiz, and the quiz will auto-submit after the original allotted time expires.

Our CSM confirmed during our check-in with them yesterday that the engineering team overseeing this update is still looking into it, though the Notes still indicate a production availability date of 2024-10-30.

More generally, I am concerned that adding time for one particular attempt means that the extra time will be applied to all New Quizzes attempts for the user.

While being able to add more time for some users on all their quizzes is helpful for those who need accommodations throughout the entire course, there are many instances where some students only need the extra time for just one quiz. The instructor would have to remember to remove this additional time for some students and, in a large enrollment class, this would add to their already full administrative workload.

[annalindsay]

Hello everyone! Following up on a few folks' comments:

@ckeigher thank you for bringing this to our attention! I was able to reproduce the issue where the quiz timer doesn't immediately reflect the updated time limit for an in-progress attempt, even though the total time allowed is updated correctly. I've raised this to our engineering team for further investigation, and will post another update here as soon as I have one.

For further updates on this behavior, please also feel free to reach out to Canvas Support with your specific quiz example, and they can attach your ticket to this behavior. CCing @Pete5484, @paulamiranda  and @afurr0026 so they can be updated as well. 

[annalindsay]

@jmerlenbach2 hi there! Regarding your question: "Regarding the Student Analysis CSV Export - will that report honor the SIS Data - Read permission for user roles? Meaning, will users who do not have that permission be able to run the report, or if they run it will it not include the SIS information for users/courses?"

I am looking into your question with our product team, and will be in touch again as soon as I have information to share. Thanks for your patience! 

[annalindsay]

Hi @isabel_anievas, the long-term solution for the file issues has been pushed to November 12*. The most recent update can be found here. 🙂

[annalindsay]

Hi @cdoherty, yes, the Bank migration solution will apply to both the course copy and course import processes!

[Pete5484]

@DanaOkerlund 

Wondering if you might be able to clarify the response I've received on our support ticket for Add time to New Quizzes. 

L1 Canvas support: Regarding your issue with testing for New Quizzes Add Extra Time | Manage Student Result View. Generally we’d advise against attempting to adjust the time limit once an attempt is in progress as it’s not garnered (sic) to work. More often it doesn’t as generally once an attempt is started it uses the settings for when it started and keeps them.

As I was trying to check the links you sent, it seems like Beta was reset and we lost all the work there so the links are showing Page not found error. 

---

This is very confusing, since adjusting time when an attempt is in progress is the exact functionality that we'd understood this release was intending to add (ie providing equivalent functionality to Classic Quizzes'  'Add Extra Time on Current Attempt').

If this release doesn't do this at all, or isn't (I think the reply intended to say) 'guaranteed' to work, or more often doesn't work, then what does it do?  It's very opaque as to how this is to be used, or what limitations apply. 

The other issues raised:

• Attempting to adjust Current New Quiz Time does not add time to countdown clock and sometimes the countdown clock disappears from screen.
• There appears to be no way of ascertaining elapsed time (doesnt appear in Moderation which I though it previously did) so it's difficult to ascertain how long student has been completing the quiz.
• The student's quiz log should indicate that additional time was added by [user_id] at time x:xx
• Speedgrader 2: Steel - cannot edit previously submitted but ungraded quiz attempts (can edit if revert to standard Speedgrader UI)

also weren't addressed.

Wondering if we can get some clarity on the purpose and actual functionality of New Quizzes Add Extra Time (and whether there are any currently know issues)?

cc @ckeigher @afurr0026 @paulamiranda 

[EmilyCottrell]

Bugs with “New Quizzes: Add Time to Existing Quiz Sessions”:

As a member of a team of folks who manage Canvas accounts for multiple professors, we were all looking forward to the opportunity to add additional time to an existing timed quiz session (meaning the students were to finish the assessment within a specific number of minutes/hours)—but, as with all things, we wanted to test it out before we presented it to faculty. Glad. We. Did…. There are so many bugs and issues with how this works that I wish Canvas would halt the push of it so that it can be remedied before it really causes issues. 

THE GOOD NEWS:

In testing this MULTIPLE times on MULTIPLE days after the 10/30 launch, I have found only 3 possible ways for this feature to work:

1. If the time is added BEFORE the assessment has been launched, the time will add appropriately, and will correctly present itself in the assessment clock at the launch of the assessment.

NOTE: This is all well and good IF the issue that causes the need for additional time happens BEFORE the student has even launched the assessment.

2. If the time is removed COMPLETELY (aka: unlimited time), the clock is immediately removed from the assessment, even if it has already been launched

NOTE: This cannot be the only solution since there is a LARGE difference between adding a few minutes to a timed assessment and giving UNLIMITED access to a timed assessment.

3. If you DO NOT have anything set in the “Available Until” section of the Assign-To AND you add the time BEFORE the student reaches the 2 minute warning mark

NOTE: This is problematic because “Available Until” is use by individuals to ensure that students who are absent/late cannot access an assessment outside the testing window.

THE BAD NEWS:

Here are a few ways I tested that DO NOT work to add additional time to a timed New Quiz in Canvas:

AGAIN: THESE DO NOT WORK AND WILL NOT ACTUALLY ADD ADDITIONAL TIME (all of these are in the scenario that the assessment had a time-limit and had already been launched by the student when additional time was needed)

• Time is added to the assessment after the 2-minute warning has already shown on the student’s screen—assessment will auto-submit at the end of the original time amount (despite showing the additional time in the assessment launch screen)
• There is an “Available Until” time listed in the Assign To (I tested this in multiple scenarios, and every time an “Available Until” time was there, it would not add additional time)—assessment will auto-submit at the end of the original time amount (despite showing the additional time in the assessment launch screen)
• Creating a new “Assign To” for a specific student who needs additional time—especially if you add an “Available Until” time—assessment will auto-submit at the end of the original time amount (despite showing the additional time in the assessment launch screen)
• Adjusting the “Available Until” time for the entire class after adding time for a specific student (having the “Available Until” prevents the time from adding, no matter where it is or what the time is—EVEN if the student would finish the assessment before the “Available Until” time expires) assessment auto-submit at the ORIGINAL time (despite showing the additional time in the assessment launch screen)
• Exiting and re-launching the assessment—UNLESS it is option 2 or 3 listed above, it will still have the original clock time within the assessment AND auto-submit at the end of the original time (despite showing the additional time in the assessment launch screen)

Again—ALL of the above listed tests (those that worked AND those that did not) will still SHOW the additional time in the launch screen – for example: if I added 5 minutes to a student’s assessment in Moderate and the student opened (or re-opened) the assessment, they will SEE the new amount of time listed on the launch screen—but for all scenarios, except the 3 listed at the top, students will NOT see that time reflected in the assessment clock. In Scenario 3 listed above, the additional time will ONLY show in the assessment clock once the 2-minute warning has appeared on the screen.

Needless to say—there are a lot of issues with this new feature that should be addressed by Canvas. It is problematic, to say the least. I see above that even Canvas has said that they don't recommend adding it once the assessment has started, but that is the main scenario we have when additional time is needed. This feature is not helpful if it won't function how the users need it to.

[annalindsay]

Hi @Pete5484: I've connected with your Customer Success Manager to clear up any confusion from this recent support response. They should be reaching out soon to discuss this further.

I also want to confirm that your original understanding of this feature is correct. This new feature is intended to allow instructors the ability to add time to an in-progress quiz attempt. As noted in my reply late last week, our engineers are working on a solution to adjust the behavior being reported here.  I'm continuing to follow their internal ticket and will update the comments in this blog post once there is tangible information to share. 🙂

@EmilyCottrell I want to tag you here as well to confirm what the intended behavior will be once a fix is implemented. Once resolved, time can be added via the "moderate" tab of a new quiz for an in-progress attempt (screenshot 1). Additionally, instructors can reopen an attempt if the time has past and add additional time to that attempt (screenshot 2). 

I hope this helps to clarify what the behavior should be (and will be). I'll update the group again once a fix is released.