Quarterly Update from New Quizzes - Q3 2022

[MPioRoda]

New Quizzes End of Quarter Update - Q3

We are entering the last quarter of the year (October, November, December) with affirmation from Instructure leadership that New Quizzes is one of the top priorities for the entire company.  What this means is that we are focusing our resources and initiatives to ensure that we continue the work at the pace we’re going in order to achieve milestones as set forth in our roadmap.

For reference: Link to New Quizzes End of Quarter Update Q2

In this End of Quarter Update:

• New Quizzes End of Quarter Update - Q3
• New Quizzes Transition Resources
• Q3 Achievements
• Q4 Objectives
• Other Updates

New Quizzes Transition Resources

New Quizzes Hub

The New Quizzes Hub kicked off on October 3, 2022 to be sure users can find everything they need about New Quizzes in one place. Blogs from Product and a timeline of upcoming features will be highlighted on the main page, keeping it easy to find the Instructure-Official items. Blog posts and questions from the current groups will also make the move to the New Quizzes Hub, making it possible to check for discussions and blogs you have participated in previously.

Quizzes Transition Feature Flags

Updates were made to the Feature Flag blog post with information on the newest Feature Flag and release update information. As described in the blog post, the new flag is called “Default to New Quizzes” which makes it so that the only option when creating a Quiz will be to create it in New Quizzes. Users will still be able to edit, import, and migrate Classic Quizzes. While many customers who have been using Classic Quizzes may not be looking to enable this feature option at this time, it offers new customers or newer New Quizzes customers the option to limit users to only one quizzing engine.

For future updates, the Feature Flag post can be found in the Transition Tool Kit on the New Quizzes Hub.

Q3 Achievements

For Q3, we projected the following items: 

• Blueprint Course locking on Blueprint courses in New Quizzes
• FITB in Word Banks with multiple answers
• New Quizzes Outcomes in Learning Mastery Gradebook
• Infrastructure and Performance Improvements
• Accessibility Improvements

Updates on the RCE Integration, please see this blog post from Tamas Balogh on the RCE integration work related to Phase 1: Core functionalities like editing, A11y checker, upload/record media, math editor, HTML editor.

Blueprint

Blueprint Syncing after initial sync and item banks went into production on September 17. Previously, after the initial sync, changes to a New Quiz in a Blueprint did not sync to associated courses. Now, subsequent syncs will update changes to New Quizzes across all associated courses. As well, New Quiz Item Banks are now supported in Blueprint Courses which allows institutions to provide consistent Item Banks across courses.

Quiz Locking for New Quizzes in Blueprint courses allows for users to set a quiz as locked in a blueprint course. From the Blueprint Course details page, a user would go under General Locked Object and check off Content, which allows the user to lock quizzes in the course. The other way to lock a quiz in a Blueprint course would be to go into Locked Objects By Type, under Assignments drop down, check off Content. 

General Locked Objects:

Locked Objects by Type:

The next step is to go to the Assignments index page and lock the specific quiz using the blueprint lock icon on the right. 

Development work for this functionality is in its final stages and we expect to be completed in the beginning of Q4.

FITB in Word Banks with multiple answers

As mentioned in the last update, this work supports a common word bank where the answers disappear as they are used or they may be used in more than one blank. An option will be added to the Options section of the question where the user can choose to “Allow word bank choices to be reused” We are in the final stages of development work and anticipate completion in the beginning of Q4. 

New Quizzes Outcomes in Learning Mastery Gradebook

There were two very thorough blog posts, Part 1 which goes over the three alignment types in New Quizzes and Part 2 which goes over how outcomes-related scoring and reporting happen in New Quizzes. The work to include data from outcomes-aligned New Quizzes in the Learning Mastery Gradebook and the admin-level outcome results report that was mentioned in these posts is nearing end..  The development work is in the validation phase right now and will be released into beta environments on October 27 and production environments on November 9. 

Infrastructure and Performance Improvements

We completed major improvements in Q3 related to performance and stability such as multi-levels of database updates. In Q4, we will continue improvements related to stability and integration. 

Accessibility improvements

In our efforts to maintain accessibility in New Quizzes we have nearing completion of all bugs related to the internal and external NQ audits.  We have also taken on Responsive issues such as how the screen adjusts or handles small screens.

Q4 Objectives

Our objectives for Q4 cover feature development work, back-end work, and planning work.  This mix of objectives represents our focus on all aspects of Quizzes from front-end to performance and stability improvements. 

Blueprint Quiz Level Locking, FITB Word Bank, and Outcomes Data development work begun in Q3 will be completed and deployed in the first part of Q4. 

Speedgrader

SpeedGrader continues to be a favorite feature among our users and we often hear how helpful it is in saving teachers time as they provide meaningful feedback to students. This quarter, we will be working to improve the experience of grading New Quizzes in three ways. First, when an instructor clicks on the update button on a New Quiz in SpeedGrader, the score should sync to the score input in the right-hand column in SpeedGrader which shows the overall score based on the total of any points added to the various questions. Second, we have heard feedback that instructors need easier access to SpeedGrader from within their workflows. We’ll be adding links to open SpeedGrader in additional menus. And, finally, we know that a favorite, time-saving feature from Classic Quizzes is the ability to grade one question at a time. We’ll be working to add this functionality to SpeedGrader for New Quizzes also. 

Un-select Multiple Choice Answers

Every quarter we engage in a hack week to get our creative juices flowing and work on enhancements that are not in progress on the product roadmap.  Product and engineering might get together to review some discussions happening in the community or to propose enhancements to features already live.  In the last hack week, one of our engineers took on the concept of un-selecting choices in a multiple-choice question.  One use case is that for programs that offer negative point questions, a student might opt to not answer at all and take the 0 for that question instead of the penalty. 

Quiz Export 

As you can see in the roadmap, Exporting a New Quiz outside of Canvas account is in the Possible Later column.  We will begin the technical planning stage in this quarter which is an important part of the process to define the scope and objectives, identify constraints and risks, establish roles and responsibilities, and divide program scope and objectives into discrete elements.

Other Updates

This Course Filter

As noted previously, after release, we found an issue with the filter on courses with large numbers of assignments so we had to pull it back to add operational improvements to support very large courses. To enable Course filtering and improve the overall experience, some significant changes needed to happen on the backend of our New Quizzes. These changes take time to architect, implement, and build. However, they position New Quizzes to iterate more effectively on future changes beyond Course filter.

At the moment, the "This Course" filter shows any Item Banks that are shared to this course.  For now, the part of the functionality that we pulled back was for that filter to show what was USED in this course, but the focus is still on the course the user is currently in.  It is important to note that the difference between “This Course” filter and “Banks Shared to Course” is that “Banks Shared to Course” shows all banks shared to ANY course that a user can access - not just this course. The use case for Banks Shared to Course is if you know you used an item bank in a different course but weren’t the one who created it, you could use this filter.

Quarter 3 Completed Fixes and Features

Here is a list of the related fixes or features that we have completed in the past quarter:

• Migrated surveys imported as classic quizzes but all questions are lost (fixed)
• Import assessment as new quiz isn't importing questions using the Copy a Canvas Course option (fixed)
• Migration using Import -> Copy a Canvas Course, or using Canvas Course Export Package (either IMSCC or QTI), or by individual/manual migration will migrate to New Quizzes as expected
• Results for randomly generated variables in formula questions provide answer as zero instead of NaN (fixed)
• Duplicate FITB questions cause answer field to lose association to question when viewing submission (fixed)
• Migrating to New Quiz from Classic Quiz can cause errors in Formula questions (fixed)
• After migration, NQ can generate solutions for formula questions such as those with decimal precision
• Migrated quizzes with fill in the blank questions overwrite answers when editing the question(fixed)
• When migrated, correct answers are not overwritten and text in backticks inherits correct answers
• New Quizzes matching answers can get cut-off at the bottom of the page (fixed)
• Sufficiently small and large values cause blank screen when editing formula questions in New Quizzes (fixed)
• Moving questions from one Item Bank to another does not update counter or move all/random button (fixed)
• Matching questions drop-downs are hidden when scroll bar is used to expand menu (fixed)
• Adding and removing item bank shares in a single action skips some of the shares (fixed)
• All sharing changes are applied when an item bank has shared added and removed in a single action
• Duplicated questions attached to stimulus remain associated to stimulus questions when moved (fixed)
• Duplicate FITB question cause possible answer field to lose association to question (fixed)
• Regrading FITB questions in New Quizzes does not allow modifications to answer settings (fixed)
• [Classic Quizzes] Some values in formula questions are not rounded to the nearest whole number correctly (fixed)
• Numbers are rounded to the nearest whole number when the “decimal places” field is set to 0
• [Classic Quizzes] "Update Scores" button can bump late classic quiz submissions out of needing review (fixed)
• Late quiz submissions that need review remain in pending_revew status

[BethCrook]

Thank you for once again keeping us updated, thank you for the entire team for all of the ongoing incredible hard work, and thank you for this --> "the ability to grade one question at a time" !

Did I mention thank you?

From all of us 🐼❤️

[KathyPalm]

Great update, thank you! 😃

[Hildi_Pardo]

thanks for the update!

Is there a date (month/year) for when media recorder will be available, please?  This is one of the biggest reasons many of our teachers are not using New Quizzes yet.

[MPioRoda]

Hi @Hildi_Pardo , media recording is part of Phase 1 of the RCE work which is nearing completion. The latest update on that work was posted here. 

[Hildi_Pardo]

Thank you @MPioRoda -- yes, I've read that.  I'm really asking for an approximate date (month/year). Do you know, please?

[cvalle]

Thanks for the update to FITB! I really like the new plan for how word banks will work.

[lph]

Thank you for the update and reassurance that New Quizzes is important. 

Here are the biggest "glitches" to using New Quizzes that upset teachers at the high school level.

Sharing item banks takes way too long. There should be an option to select multiple banks and sharing done in one move.

Copying an item bank of questions should be possible. Along this line, selecting multiple questions to move to another bank should be possible. In other words, multiple questions should be able to be shared in one move. 

Formula based questions are okay except you cannot add a dropdown as a fill-in-the-blank style. For example, teachers want a formula question so numbers are generated, but they want the SI units available as a dropdown next to the answer. 

Printing of a new quiz doesn't work properly if questions are pulled randomly from the item banks. If a teacher prints a copy, then tries to print an answer key, they will get different questions. 

A search across all of the item banks is still impossible. If there are over 28K questions in banks, then a teacher will spend an inordinate amount of time trying to find that one question that needs to be edited. 

The item banks are not linked in the build portion. This means a teacher may want to modify question in the item bank and have to leave the build, go to the item bank, make the changes, etc.

Adding a mathematical formula into a fill-in-the-blank is impossible. This means a^2 has to be done by the student. Worse, questions cannot be built for students to add the math into formula based questions as well as fill-in-the-blank. 

The above are the top complaints I hear almost daily --- to the point of frustration and with some teachers abandoning Canvas for other programs. For example, Positive Physics allows formulas and SI unit dropdowns in one question. They also allow formulas. Carousel Learning is another more modern quiz engine. 

Personally, development of key areas of concern regarding New Quizzes seems too slow for most high school teachers. They don't have teams doing the backend and don't have prep periods and PD times. This means they are creating content on their own and get very frustrated. I hope you understand. Canvas is a great tool -- A healthy New Quizzes can make it an even better tool.

Good luck. 

[simone_laughton]

Thank you for this update and it is heartening to see that progress is being made on this important part of Canvas functionality.

Is it currently possible to download a CSV file of all quiz responses (with one student per row), similar to what it is now possible to do in Canvas Classic Quizzes?  This is one functionality that is hindering widespread adoption of this tool at our university campus.

Our instructors need to be able to download a permanent backup record of how each student has responded on a quiz.  In addition, this functionality enables our instructors to do more detailed analyses of student quiz responses to improve future iterations of the quiz.

Another request from our instructors is to have the ability to remove a question from a quiz after it has been completed by students.  Perhaps it would be possible to have the functionality to not count a question after a quiz has been completed by students in the final score in the Gradebook?  That way the original record of the quiz is kept, and questions that are later found not to be an effective assessment of student competence / knowledge can be removed and improved upon in later iterations of the quiz.

Thank you for considering these requests from our University of Toronto Mississauga instructors.

Best,

Simone