New Quizzes: We're listening

[SuSorensen]

As we have been getting closer and closer to the time for Classic Quizzes sunset, we have heard many clear, articulate, and thoughtful voices talking about just how difficult this transition would be in the current climate where, to borrow a common refrain, teaching faculty are “barely able to keep their heads above water.”

We are currently exploring adjustments to the Classic Quizzes sunset that we hope will give you and your institutions more control over the transition. To help us with this effort, would you be willing to share your top three considerations or requirements for a smoother transition?

[dkpst5]

1 - Complete the currently-planned feature set of NQ first, with sufficient time for us to train people on it. Training for a moving target is not feasible. Yes, features will get added later. But the big list of needed-for-parity features should come before any attempts at training or quiz system forcing.

2 - Top of those features still missing that are critical: (a) full features of global RCE [this alone is a show-stopper of anything brought up in this post; there's no training possible for NQ without the normal RCE] (b) exporting reports/full item analysis to CSV.

3 - Bulk transition from CQ to NQ, including banks, and quizzes that use said banks, also including hopefully an easy comparison/preview or call out to see where the import may need intervention.

[atcarver]

@shaun_moon - thank you very much for your (and the team's) attention and effort towards altering the planned transition off Classic Quizzes. It is greatly appreciated, as is your team's work on the tool. I look forward to hearing more on an adjusted timeline.

The top three requirements, for us, are:

1. One year of transition time from a "fully-baked" New Quizzes solution to transition our instructors
2. Fix noted workflow issues
3. Add tooltips and unify method/implementation

[kroeninm]

The guides are helpful as is the comparison document, I use this often and is my main source of reference.  So for me, the features is the key aspect of what would make our transition smoother.  So my #3 features which may not be exactly what you are looking for 🙂

#1 Course based item banks as a default - There is not a way to backup, export, copy, or efficiently share banks in the way that supports intellectual property in the ways we use them now and training faculty to a new way of using banks, naming banks, sharing banks, will not "stick" the first time with all the other changes so I see a lot of faculty inadvertently deleting or changing banks of colleagues or of their own courses which is my biggest concern.

#2 Essay question notification - Notification in the Grades and/or Todo list for faculty to know which questions/submissions have been graded or not.

#3 Consistent RCE - Worth waiting for to have integration with our video (Kaltura), same Equation Editor, same Image and File upload process, and HTML access.

I'd actually give up the bulk migration tool(!)and #2&3 which are also important if we could just get Item Banks working in a course-based default way so that we don't have to create extra processes to manually track and shift bank ownership to ensure the correct editing rights.

Melanie

[snugent]

My top three are 

Integration with Commons - Our faculty use Commons a lot and the lack of support for Commons is going to make this switch difficult. 

Full Import/Export options - Fully integration with Respondus (exam creation tool with Canvas publishing options) and publisher resources to new quizzes. Along with this the ability export quizzes from new quizzes. 

RCE Editor - I agree with other about this. It has to be the same in new quizzes. 

[mmyers3]

I appreciate the rethinking of the timeline especially when institutions and faculty are so stretched  

1. Outcomes transition and parity for institutional assessment continuity: A rock-solid understanding of how the transition from question banks with Outcome alignment and classic quizzes to New Quizzes and item banks will occur.

An ability to control and lock certain item banks to preserve institutional data collection while giving faculty control of others.  The ability to test to the transition in test to make sure there are no issues with Outcome transition  

2. At least one year of fully built-out capacity to plan and make a transition. Fully built out capacity, documentation, and flexibility with timing on implementation will greatly reduce financial costs and stress levels  

3.  Accessibility issues must be resolved | there is no accessibility checked in RCE and the hot spot question type is not accessible.  That means we have to train faculty NOT to use that question type -  this must all be resolved prior to launch  

[chriscas]

Thanks, Instructure, for listening to the feedback that's been given about the New Quizzes timeline. I know a multitude of issues (some major, some minor) still exist, but here would be my top 3 items around the transition.

1. The ability to use other LTI tools (specifically Kaltura or other editor-based tools) within new quizzes.  It's been said this will happen when the new RCE is integrated into New Quizzes, but I want to make sure this is identified as a somewhat separate item in case the new RCE doesn't magically solve the LTI issue.  We have a lot of quizzes with embedded Kaltura videos, so until those work properly in new quizzes (both migrating and creating new questions), we can't recommend the tool.
2. Partial credit *options* for all applicable question types.  At a minimum, anything that had partial credit in old quizzes needs to have it (and function the same way) in new quizzes.  Teachers will be very upset if their migrated quizzes suddenly are not graded as they expect and they have no option to fix it.
3. To go somewhat along with my #1 and #2 items, we need a 100% seamless migration from classic to new quizzes.  This isn't a case where having 95% of things work right and 5% (or even 1%) go wrong is going to be acceptable for teachers.  On a very edge-edge-edge case where something does go wrong, a very clear message needs to be displayed to the teacher with info on what quiz/question did not migrate right and exactly what is different between the new and old versions.

I'll just throw in one more comment, which is about the timeline...  From an admin perspective, we need the issues with new quizzes to be ironed out well before any forced migration.  It's very hard for us to explain to our faculty that there is a transition period starting now, but the new tool is still not quite in a final usable form with parity.  Once we get to the point where all of the major issues pointed out are addressed, we would like to have a 6-12 month transition period where both classic and new quizzes are available just to find any edge cases and give faculty time to verify the migration when it best fits into their schedule.

Hope this helps out in planning for the future!

-Chris

[darlene_schaefe]

Hello!

1) Full RCE capabilities. The New Quizzes RCE is very limited, especially for quizzes/courses that require math equation editor (we MUST have an advanced equation editor when moving LaTex from one platform to another).  Also, we need to replace stimuli and images all the time (including alternative text for images). The RCE needs to be comparable to existing RCE.

2) The option to "cherry pick" import and run a Blueprint update (not from initial synch, but consistently)  when additional new quizzes are added to a course master or Blueprint course (we have to do partial developments ahead of instruction all the time).

3) 12 month transition period once all "roadmap" features are fully implemented.

[cvolpe1]

Assuming full capability of the RCE and a bulk migration tool is available, below is my input:

1. Ability to Batch Download Submissions
2. Improved workflow as well as a mature product with time to test
3. Survey tool as well as missing question types (Text, Stimulus not the same, Fill in Multiple Blanks, Multiple Drop-Downs)

[lisa_wathen]

Thank you for continuing to listen!

Our top three are:

1. RCE fully functional in New Quizzes

2. Outcomes data from New Quizzes generating in the Learning Mastery Gradebook

3. Longer transition period once New Quizzes has fuller parity with Classic Quizzes. 

[milesl]

Thank you very much for this engagement. Lots of great ideas mentioned already that I won't repeat:

1. Full feature parity with classic quizzes: Over the past 8 years (and especially in the past 18 months) our teachers have whole-heartedly adopted the classic quiz tool, and are using it in every way imaginable. There is no feature, no matter how small, that could be removed without negatively impacting a valid pedagogical function. Recently, Instructure has been clear that parity is not a goal of NQ, but I strongly suggest you reconsider, and use the extra development/migration time others are requesting to deliver on this ideal goal.
2. More confidence around scalability: a few years ago, we encountered issues with classic quizzes in very large courses. This was challenging, as it meant classes with thousands of students were having very visible and impactful failures. Ultimately, we successfully worked with Instructure engineers to adjust our compute resources so that we could be confident there would be no issues with classic quizzes. We were told that new quizzes was built to scale, and would not face the same issues, but recent developments (e.g. reporting and moderating features do not work in realtime in new quizzes) have not inspired confidence. I would like to see more evidence that NQ works in very large courses before we adopt it, so that our instructors and students do not have to go through the same painful failures we did with classic quizzes.
3. Classic quizzes remains functional for 1 year after NQ is feature complete: OK, I lied when I said I wouldn't repeat other's requests. 😁

[audrey_crawford]

1. Ditto the list from dkpst5.

2. Ditto the concerns from chriscas, especially integration with Kaltura.

3. Publish from Respondus 4.0 must work. Changes and limitations must be clearly documented.

[cory_little]

1. Essay question notification.

2. Quiz bank migration

3. Math editor improvement

[rshultz]

Thank you so much for considering extending the sunset of Classic Quizzes. Please see our 3 top priorities for the transition: 1. New Quizzes need to be 100% working for one year prior to the sunset of classic quizzes. 2. Free Canvas provided training videos for teachers on the transition from classic to new quizzes. 3. Assurance that the teachers can easily tell which quizzes need grading, that partial credit is available, and that ungraded quizzes are displayed on the teacher's to-do list when using new quizzes.

[Anya_]

1. A 12 month transition between New Quizzes being fully baked and Classic Quizzes turning off. I can only speak for how it happens at my school: a university. Being a university, not every class is taught every semester and not every faculty teaches a course every semester. So there are some semesters where they (or their assistants) have no Canvas course to work in. Which means if the "start using NQ" period happens in a semester they are not teaching, say Spring, (and thus don't have a course where they can have the classic quizzes while crafting new quiz versions), then in July the classic quizzes can't be created anymore in crafting for Fall courses, they didn't really get time to have them side by side or maintain classic quizzes as needed while they work on transitioning them to new quizzes. So having a two-semester transition would be helpful.

2. All questions must migrate appropriately and if they don't, the migrated quiz needs to have a huge red flag, icon, error-notready-text, SOMETHING to indicate questions did not go correctly. I think I read some work is being done in the beta migration tool about this, but I really want to hammer it home that I need this. I just tried migrating a quiz that has a few "Text" set-up questions. I then previewed the quiz and none of them showed up; I thought they didn't transfer at all. I went to Build and saw the text did transfer as stimulus questions, but since they had no "sub-questions" (?), they didn't show in the preview. There needs to be something that tells people "Needs to be Reviewed" or "Alert: Empty Stimulus Questions", something! Also, the migration tool did not preserve blank lines in any of my questions, so they are all harder to read (and this is compounded by the quiz questions taking the entire width of the page, increasing eye travel distance....not to go too off-topic since I tried to stick to migration admin/faculty showstoppers instead of student/UI ones, but I really do find the New Quizzes hard to read, so will be making my own abundant line breaks, but if we could also get some consistent margins/spacing, that would also help:

3. New Quizzes Must Respect the Mute. I don't know if this is scheduled to be changed, but as of right now, it does not. Everybody has been trained that as long as the assignment is on "Manual" posting, then students cannot see their grades, comments, correct responses or even the questions themselves (even if show answer immediately is checked). I relied on this heavily when we had quizzes open to take within a 24-hour period, then once it was over we could unmute and they could see everything. This is not true with New Quizzes, even with the posting policy on manual, the migrated quiz still showed the right answers to the questions after Test Student submission while the Classic version does not. If NQ will not respect the mute, then the migration tool needs to ascertain the mute status of a quiz/course-wide policy when migrating and adjust the checkboxes accordingly to achieve the same effect. But ideally, it would just respect the mute. I feel without this, no matter how much training is done to get this point across, people are going to assume the mute is taking precedence as it has in the past and they are inadvertently going to be showing answers when they did not mean to.

[n_adamson]

As others have said, NQ should be fully developed before heading into a sunsetting phase. We'd like to have confidence in it, and be looking forward to promoting it, rather than being apprehensive about its release.

1. RCE: we acknowledge the planned developments with this, but the timing of its release is unclear.
2. Migration: bulk migration of quizzes and easier migration of question banks.
3. Navigation: i.e., not losing the course navigation menu and breadcrumb etc whilst in NQ.

Thanks for listening! 🙂

[themidiman]

All the comments so far are right on par, but I'm going to repeat some just for making sure they are getting acknowledged:

1. RCE Parity - This is already an acknowledgment made by Instructure, so I'm linking to this additional article for the benefit of the other commenters on THIS article
2. Item Banks "Belong to" and "Travel with" the Course and when sharing course with other instructors/ designers, newly added Instructors edit the banks by virtue of being in that role. 
3. Essay Questions or other questions not able to be automatically scored alert the teacher when they need to graded. 

If I also had a 4th, the addition of ample transition time would give our team the ability to sleep better at night. Think of other enhancements that went relatively well such as the New Grade book, and other LTI side-car apps like Attendance that weren't such a burden to trainers and new adopters. They just worked and made working with Canvas an improved experience. 😁

[hesspe]

1) No sunset: Continue to support both old and new quizzes into the foreseeable future.  This is based on the fact, which I have heard no one convincingly refute, that because new quizzes was developed as an external tool, there can never be full parity between classic and new quizzes. I agree that even in its present incomplete state, new quizzes is probably the better stand-alone quiz engine.  Even so classic quizzes has many attributes (many of which are listed above, and surely more will be listed below) that people, for very sound reasons, have come to rely on and expect.

And just to show I am playing by the rules, here are two and three (but boy it was hard to pick just two).

2) Full integration with SpeedGrader, including grade by question and the orange dot to indicate submissions that still need grading.

3) The ability to automatically release and retract quiz results at specified times.

[lesya]

We appreciate that Canvas is listening! I have seen several much-needed
requests for improvement that have been prominent on our campus:

1. Ownership of item banks: Canvas assumes that instructors build their
   courses in Canvas independently, without input from others or the
   institutional support structures. This is almost never the case in large
   undergraduate/graduate universities. Ownership of item banks in its
   current form creates many obstacles for instructional designers to offer
   just-in-time assistance to instructors.
   
   What happens when the owner of item banks leaves the institution and the instructors who pick up the courses of others need to be able edit the questions in item banks? How much effort is required to troubleshoot ownership?
2. Images in New Quizzes are uploaded into the cloud, so there is no way to re-use them since images are no longer stored in the course-specific file structure.
3. The RCE editor in New Quizzes:  not only the current RCE does not work with LTIs, such as Studio in Canvas, but also formula equations migrate as images. Instructors must then re-create formula equations from scratch using roundabouts.
4. Questions placed within Item Banks do not appear to link to Outcomes, even though questions outside of Items Banks can be connected to Outcomes. This is at least the experience of our institution - we have looked into the permissions, and the issue us still there. 
5. The integration with Speegrader is lacking, especially for large enrollment courses in which instructors ought to be able to conveniently view the quizzes that require grading (say, when quizzes contain Essay questions).
6. No way to efficiently search for questions using key-words: Since Item Banks follow the user from course to course, it has become really hard to remember which item banks contains which questions, even if instructors follow naming conventions strictly (first course title, next the title that reflects a concept/chapter/unit around which questions are organized). There is no way to do any question search efficiently without navigating into each item bank. 

[sharris]

We have a very large online operations - 750+ master shells/templates and 7,000+ students. Our ID's work with a SME to develop the course. Then, that content auto-copies (via API) to live sections every 8 weeks. The people teaching the course is almost NEVER the person who developed. I have two big concerns about New Quizzes for our model.

1. Quizzes copy unpublished - This is a support/QA nightmare. We run classes every 8 weeks with a heavy adjunct population. 

2. Item banks aren't tied to courses - This is an even bigger issue than #1. I've recorded a video that outlines my issue with this process (http://www.kaltura.com/tiny/08e0w). But - the people who teach the course will NOT have the ability to easily edit quiz questions without having the original creator, or support staff, share those question banks. We aren't staffed for this. Again - support/QA nightmare. 

3. We need 12 months of time between the completion of these roadmap items in production and the sunset date. 

[briddell]

Thank you.

1) New Quizzes with any Ungraded questions show in Instructor's "To Do" list.

2) Partial Credit for Matching, Multiple Answer question types.

3) Ability to grade one question at a time.

[jenny_ericsson]

This is our three most important issues that needs to be fixed before even considering transitioning(but it is almost impossible to choose only three because we have plenty of concerns). With these points in mind, we assume that ALL bugs are first fixed, and NQ are stable and reliable!

1: Migration: We wish that there would be a fully seamless migration from classic to new quizzes, done by Instructure, both quizzes and question banks. The responsibility should not be on the teachers. This function (quizzes) is so much used now, that it needs to 100%. No errors or unexpected things will be acceptable for teachers. Would  something still go wrong, this needs to be displayed extremely clear (how) to the teacher.

2: Better integrated LTI: We need the RCE to be fully functional, to be able to access course images, course files, math tools, and Kaltura LTI for example (as well as other LTI available from RCE in CQ). We need Speedgrader to work seamlessly with NQ.

3: Question banks: Teachers are often working together with question banks, so the qwnership should by default be to the course, not the individual. We worry about what happens when the owner of item banks leaves the institution and the instructors who pick up the courses of others need to be able edit the questions in item banks? And are they possible to share on Commons?

[candice_mcphers]

To echo what many others have contributed, we need at least a full year after critical elements are completed for new quizzes. Some of the most critical elements in our context includes:

1. Inter-institutional export capability (common cartridge)
2. New RCE successfully integrated (to resolve video and media challenges and HTML editing ability along with third party integrations like MathType, etc.)
3. Increased capacity with the number of tests and students in a course and how that affects score passback.

[venitk]

Thanks for asking. I echo all the thoughts above. 

1. Maybe I'm missing something, but there are no question groups. In Classic, if I want students to answer 20 questions, and each of those 20 questions have 3 options to preserve academic integrity), I would keep all those questions in the quiz and create 20 groups. However in NQ, I would need to make 20 item banks. 
   That's so many banks. If the course has 3 exams, each with 20 questions, that's 60 item banks for a single course,
   There isn't an intuitive way to create an item bank in the course. I just tried to do it without referencing the guides and couldn't (so we'd also need to train people to do this). Additionally, the way banks are stored and accessed across an account is unsustainable. Admins/IDs who have access to every bank will be completely inundated with banks.  
   Unless stimulus questions are modified to act as question groups, this will be absolute chaos and immense cognitive overload. 
2. We need a way to quickly see what questions don't exactly migrate 100% perfectly; otherwise we have to manually check each question in each quiz to make sure everything--formatting, question choices, literally everything--migrated perfectly. 99% isn't good enough. So we need some sort of flag or warning--that needs to be 100% (again, 99% isn't good enough) reliable--telling us when a specific question needs attention (and it needs to identify specifically which question). If we don't have this, those of us who are admins who support whole programs (rather than individual faculty who just have to worry about their own courses) will need at least 12 months of really intense work to complete the migration. Even with this, 12 months would be ideal because everything in higher ed moves slowly. 
3. We really need a solution to "keep" practice quizzes and graded surveys (ungraded surveys would be nice, too). We use these two tools to control student access through the courses, following Canvas advice. When Canvas personnel trained us on, for example, Mastery Paths or locking down progress through a module, etc., practice quizzes and graded surveys were touted as a way to do it. All of our courses start with students taking a practice quiz Code of Conduct. I also have classes where students have to submit "worksheets" that were completed as graded surveys in order to unlock an assignment later in the module. It's not the same just saying these items don't count towards the student's final grade or making them worth 0 points because they still show up in the gradebook.  

[DeletedUser]

Thank you for continuing to ask about these issues. 

1. The lack of RCE is a complete deal breaker. Most of our quiz questions are based on image stimuli. 
2. A math equation editor 
3. Export capability for schools in different instances

[snmargeson]

1. Fully developed partial credit must be available for New Quizzes. Asking teachers to go back to hand grade matching questions is not reasonable. Any partial credit available in Classic Quizzes should be available in New Quizzes.

2. Question bank migration needs to be made available. We have endorsed creating question banks in the Classic Quizzes and the fact that there is no migration available to Test Banks is frustrating.

3. Indicator for quizzes that require grading. Currently, grades are given to students even if there are questions that need to be reviewed. It should be indicated in Grades whether a student has an ungraded submission for a New Quiz.

[mfernandez]

There is a lot that needs to be dealt with that Canvas is calling "enhancements" which if they were teachers, would be called "major flaws in the new quizzes set up." 

1. Allowing multiple point totals for matching sections. What teacher counts all 15 wrong if a student gets one wrong? NONE

2. Quizzes needing grading not showing up on a teacher's to-do list. 

3. Students getting an immediate email that their quiz is graded when it is really not.

4. On Speed Grader, students have a green check mark even when a quiz is not graded. If you miss grading a question, you have no clue you have done so because it is a check mark. 

5. An annoyance, but I can live with it. When Speed Grader shows you which ones to grade, it does not show them in numerical order, just a jumble of numbers with no order whatsoever. 

Mind you, all of these things worked with classic quizzes. I am going to have to show teachers how to get around all of these flaws, and with a school of 100 teachers, they are going to go back to pen and pencil because of these MAJOR FLAWS. 

[lgarmire]

There are many common themes with this request:

1. Notification for teachers that there are manually graded questions and a score does not go in the gradebook until the teacher published it.
2. RCE needs to be similar to Classic quizzes: ability to record audio/video within an essay question, edit a question using the HTML editor, and the same math equation editor. New quiz's equation editor leaves out a lot of functions for higher math.
3. Ability to migrate item banks and also questions with embedded videos. With uploaded videos and images, where do they reside? No place to easily replace the file.

[mskoch]

Really there are too many to name, these three are ones that could save a lot of time:

1) Item Bank - migration from Question Bank to Item Bank and ownership of item banks shifting to the course instead of user. The current Question Bank to Item Bank workaround requires the use of Classic Quiz, so it makes no sense to even talk about deprecating or sunsetting CQ before we have the tools to do the work without needing workarounds.

2) RCE - the Canvas RCE needs to be in place, and it needs to be in all questions, including FIB. 

3) Respondus 4.0 compatibility - either make NQ backward compatible to QTI 1.1xx which is what Respondus 4.0 currently outputs, or wait until Respondus 4.0 is able to directly publishing to NQ. 

Also, I am having a difficult time tracking what fixes are supposed to come and when. I keep seeing reference to the Product Road Map and New Quizzes timeline, but they don't have the info we need.

[michelle_coots]

Thank you for taking input and exploring adjustments to the sunset of classic quizzes. Here are my top three categories of requirements with notes on each.

1. Scoring Improvements - instructors MUST see a notification in their to-do list when a question needs to be scored; muting must be available so that scores can be hidden until questions are graded; instructors need to be able to filter in speed grader including by accurate grading status and scoring of individual questions; partial credit must be given on matching questions

2. Full Canvas RCE - need to be able to include images & video in questions and items; need to be able to use LTI tools (including Studio); edit with HTML editor & same math equation editor; accessibility checker

3. Migration - the ability to migrate ALL quizzes in a course; zero migration errors (and notification if there is a unique occurrence of an error); item banks (including course ownership); use of outcomes that feed into learning mastery gradebook; training resources for teachers that are free from Canvas to navigate migration; 12 months for transition AFTER all functionality needs are addressed

[chelsey_laningh]

I would echo many that have been previously stated on this thread. 

1. Notifications - Students should have notifications appear in their "To-Do" list to complete these quizzes, as well as notifications for instructors to grade short answer responses in Speed grader. 

2. Accessibility - "New" Quizzes does not function well with the Canvas App - is this being addressed? 

3. RCE - Having the same RCE options as other assignments/Classic Quizzes 

[lsickler]

• Accessibility — New Quizzes MUST be reliable in iOS app. Working with K-5 students, the app MUST be fully functional.
• Migration — We must have reliable migration of all quiz items from Classic Quizzes to New Quizzes. 
• Fully developed RCE
• Fully developed partial credit
• Free training videos from Canvas to help ease the transition

Please consider the huge range of technology ability levels in your users -- both students and course designers. This is going to be a major change; and unless NQ is fully-functional (and allows teachers time to learn and transfer content), I foresee many teachers jumping ship and returning to traditional methods.   

[gncrum]

Thank you for offering this forum for feedback

1. Feature Parity with Classic Quizzes

2. One Year Transition Time line once New Quizzes is complete and "fully Baked"

3. Partial Credit on all question types

[ewest]

1. We need to have adequate time to train faculty members
• Most of our faculty are off contract in the summer, making the current implementation timeline challenging.
• We ideally need two semesters for training for major changes, and for the tool to be in a consistent state that includes key features during this time frame.
2. The Rich Content Editor
   • We would require this feature be ready for when we start training to maintain consistency with training experience for users.
   • The RCE is crucial for having the accessibility checker. It is essential to have this measure in place to check accessibility.
   • RCE also is needed to develop more robust questions
3. The lack of student analysis report and CSV export for stats.
   • This helps us measure and demonstrate compliance with department of education guidelines for online assessment.
     
   • This is also an essential piece for our institutional assessment team in their process for handling program assessments.
     

[mjennings]

Thank you, thank you, thank you to listening and responding to your users Canvas! This has always been what sets you apart from other platforms. I appreciate your willingness to re-think and adjust your plans based on the needs of your users. I understand that there is no way to make everyone happy, but you are trying and for that, I thank you again.

My top 3 would be:

1. 6-12 months between full release of New Quizzes and turning off the ability to create Classic Quizzes to allow for proper training and testing for edge use cases.

2. Ownership of items banks need to live with the course by default.

3. Reporting from New Quizzes needs to support exports and be available without waiting for 24 hours. Classic Quizzes stats are near real time and faculty depend on this to correct any mistakes or mis-keyed questions quickly so that student can receive timely feedback on their quizzes. New Quizzes analytics need to be available within a reasonable timeframe.

Thanks you again Canvas for listen to your users and giving us a place to share our needs with you. Keep being awesome.

[smathews]

1. Parity with Classic Quizzes (including RCE). 

2. Direct Respondus 4.0 QTI compatibility  -- this is an outstanding tool to use with Classic Quizzes for quickly loading banks.

3. Video (and text) resources for training. 

[tarnold]

Thanks for the chance to weigh in:

1. Partial Credit Option needs to be available first please.

2. Rich Content Editor (with LTI availability) needs to function consistently.

3. Full migration from old to new (including banked items).

[alissaarcline]

Thank you so much! For (1), I agree with so many others that it would be great to have a 6-12-month transition period AFTER NQ is "fully baked." (2) We need the grade by question functionality to work in NQ--this is really throwing off some of our profs/graders who have already made the transition. (I will sneak an extra item in here--full compatibility with the gradebook would be ideal--right now, if we have an NQ muted, it doesn't actually mute scores, and doesn't show that there are any quizzes to be graded on the to-do list for the prof).) (3) Having the RCE work consistently, and also having it available at all points in a question (e.g., titles, question stems, being able to create tables for students to fill in for fill-in-the-blank questions, etc.), is really important for us. I will second what "ewest" said above about the accessibility checker--it would be great to have that in NQ. Finally, I will second what some have said above about LTI compatibility with the RCE in NQ--this would be so helpful.

Thank you so much for listening, Canvas! 🙂

[jeastham]

1) The RCE, especially with the media recorder.  We use this for special education testing and has been a major hang up with my teachers.  Along with the RCE, when we migrate a quiz to new quizzes, the recordings do not integrate seamlessly. 

2) Partial credit option on the question types that it applies to. 

3) New Quizzes MUST be reliable in iOS app. Working with K-5 students, the app MUST be fully functional.

Also, need to have a mass migration option instead of each quiz one-by-one. 

[mary_jackson2]

This is not a concern with the transition so much as a comment about features that Classic Quizzes has that New Quizzes does not have:

1) Moderate Feature - New Quizzes has moderate features that are very nice, but Classic Quizzes has the feature that tells teachers when/if students leave the testing page.  This is very nice, and I would like it to be added to New Quizzes.

2) Quick Scoring - in Classic Quizzes, you could go through and grade short answer and essay questions one question at a time in speed grader.  Once the question was selected, SpeedGrader would file through the students' answers on that question and it could be graded and saved automatically, all at the same time.  Right now, in New Quizzes, You have to scroll through the students and then find the question, grade it, scroll down to save/update and then move to the next student. This becomes tedious with 180-200 students.

3) Partial Credit - The option of giving partial credit on certain question types (multiple select, categorizing and fill in the blank) is really nice. Matching, ordering and hotspot should also have this option.

4) Hot Spot Questions - There should be an option to put multiple hotspots on a question.  As a math teacher, I would love to use the hotspot feature to test if my students can graph a line with multiple points or system of equations, but that is impossible with the state of the current hotspot types of questions.

[bbennett2]

I'm going to echo most people with these, but these are definitely the items that really keep me from talking about new quizzes with staff:

1. Submissions notifications/gradebook display. Having the quiz grade go right into the gradebook, even with unscored items, bucks every other workflow in Canvas. Classic quizzes, discussions, and assignment submission all show pending submissions, not scores. New quizzes need to follow the same pattern. As a subpoint, getting to the SpeedGrader for a NQ submission is a nightmare.
2. Outcomes and New Quizzes need some work. I know you only asked for three, but I'm going to break this into a couple sub points for clarity.
   1. Outcomes on NQ do not report to the LMG. Right now, there is a gnarly workaround where staff can manually add a rubric to the NQ once it's created, but now we need two screens to manually do what CQ already does - send Outcome results directly to the LMG. The whole point of Outcomes is to check student progress. NQ improves that by allowing staff to add Outcomes to questions on the fly, but since they don't go anywhere, they're not helpful at all. I also want to echo this comment about losing the usefulness of question groups. 
   2. Course outcomes are not available in NQ. We have all our course outcomes stored at the root level so all instructors A) have up to date standards, and B) can compare student progress across courses against the same baseline. Currently, Outcomes are not available in NQ, even when they're imported into the course from root.
3. Public API tooling. We use the API to manage several aspects of our courses. Right now, NQ is opaque behind the API. Full documentation with samples, especially because it's moving to GraphQL over REST, means a longer dev cycle to get systems into place.

I still think NQ has good ideas, but they were announced at InstCon 2017. I'm glad you're asking for more feedback and considering stopping because the migration timeline detailed this year is still fast, especially given our needs around Outcomes. It's frustrating to see Instructure continue to promote and announce without really considering what the needs are. Building a tool via LTI to replace a core feature still smells of rebuilding tools to better support a la carte pricing rather than core systems updates. 

I think my final word will echo many others - either a 12 month period once feature parity is reached or an optional adoption entirely for institutions where forced switch will cause major issues is in line. NQ is a product - it's built with added value and can be managed as a plugin. Using that approach, it cleans up all this uncertainty about readiness from all parties involved and allows Instructure to pick and choose what to fix and what to push until after launch.

[lthompson2]

1.  We need partial credit options for all applicable questions.  It is unreasonable for teachers to have to go in and regrade every matching, multiple answer, and other types of questions that would have partial credit available.  This is my teachers' number one problem with New Quizzes and why so many resist it.

2.  The RCE needs to be on par with the RCE in Classic Quizzes.  We need the ability to put math and science formulas and equations in without teachers having to do a lot of extra work.  We need images to show up year after year instead of being housed only in that course for that year.  We also need the media recorder.  This is crucial for our special education students and their accommodations (not having to leave the room and being able to work at your own pace and hear your own teacher read it makes such a difference!), and it's also something our primary teachers rely on for students who are just emerging readers.  To ask them to go to another program, record each question, save it, and then upload it is an unreasonable ask.

3.  Grading needs to be substantially improved.  Not only is it not convenient to get to the SpeedGrader for New Quizzes, teachers don't get alerts that essay questions still need graded.  That needs to be on their to-do list like it is in Classic Quizzes.  It also shows the current grade in the gradebook and then sends it over to our SIS when a sync is done, even if the teacher isn't done grading it, whereas with Classic Quizzes (and Discussions and Assignments), it shows an icon in the gradebook to indicate it's been turned in but hasn't been graded, giving a signal to both student and teacher that it needs graded.  This current system creates added confusion and makes what had been a simple, stream-lined process far more complicated and frustrating than it had been--it's not easy to sell teachers on that when Classic Quizzes works so well for nearly all of their needs.

[embleton]

#1 - Get Item banks back into courses. There is no practical way College support staff can continue to provide routine quiz building services if New Quiz Item Banks are associated with specific user's login ID, course ownership/rollover would be a nightmare, and instructor's ability to manage their Item Banks for multiple courses will be severely compromised as it currently stands. (The Item Banks don't even provide details to instructor about which courses and quizzes use them!) ... this is a show stopper right now.

#2 - Allow all question types to have partial grading. Realistically, instructors now have to remove all matching questions from their existing quizzes and have zero incentive to use the new question formats (categorization, sorting) since they don't allow partial grading.

Tie for #3:

#3a - Get the Rich Content Editor fully functional. 
#3b - Work with Respondus to get tool for importing entire question banks!
#3c - GROUPS in quizzes.
#3d - Let instructors know when there is something to grade! (seems so obvious ...)
#3e - Don't release partially scored quiz grades back to students! (seems so obvious ...)

[jgarriott]

Fully developed partial credit must be available for New Quizzes. Asking teachers to go back to hand grade matching questions is not reasonable. Any partial credit available in Classic Quizzes should be available in New Quizzes.

Question bank migration needs to be made available. We have endorsed creating question banks in the Classic Quizzes and the fact that there is no migration available to Test Banks is frustrating.

Indicator for quizzes that require grading. Currently, grades are given to students even if there are questions that need to be reviewed. It should be indicated in Grades whether a student has an ungraded submission for a New Quiz.

RCE missing. This needs to be added as it is in Classic Quizzes.  We need the ability to put in formulas, equations, and media recordings. Accessibility checker is a must-have.

[Jeff_F]

Here goes:

• Item banks - these need to be owned by the course, not the exam author. Not being 'Chicken Little' here, but this is super critical for all the reasons already mentioned. 
  
  
• The lack of question groups - suppose you wish to set up an exam in such a way to ensure a student receives a set number of questions for a given topic that match an objective/ outcome.  With NQ it is necessary to create separate Item Banks for each of the learning objectives.  Given the noted issues with naming conventions and not seeing what assessment the item bank supports, this can get messy. That will be so many Item Banks.  Here is an idea:  perhaps there could be a means to group Item Banks that were created for an assessment.
  
  
• RCE - functionality and feature parity with classic quizzes.

Thanks, Jeff

[nwilson7]

Thank you Canvas for soliciting feedback!  You have a tough job ahead trying to address what you can from everyone’s three as each institution is passionate about what is make or break for them.  The majority of our faculty have already started working on their Spring courses which means several weeks ago would probably have been when we needed to have New Quizzes ready if we are thinking of a July 22 cut off on Classic Quizzes.  As many others have pointed out, it is impossible to force faculty to switch to a product that is not ready for prime time yet.  A 6 month overlap where New Quizzes is “fully developed” is a minimum but having the switch be mid-year is not desirable so I would echo the full year of overlap so the cut off could still be during the summer months.  This is assuming New Quizzes is ready in June 22.

It is extremely hard to pick just three items as each of our faculty have very different ways of using quizzing that is integral to their teaching style.  That said, here they are:

1. Question Banks/Item Banks.  There MUST be a way to get current question banks over to item banks, especially if a quiz is using a question bank and then is migrated over to new quizzes, you just end up with a blank quiz.  Item banks also need to somehow be attached to the course and not the user so as there is turnover (either a person completely leaves or is just not teaching that course anymore) the next person needs to be able to have access to the previous quizzes.  Putting any more work on the Canvas Admins is NOT a solution.  It is also extremely annoying for the admins to see every bank.
2. Survey being removed.  We use this across the university similarly to how others have mentioned where it is used to unlock something or an easy way to track things that do not happen in Canvas.  For example, many of our courses have components such as attending an event and we have these setup as surveys where (we use the essay question type) the student goes in and enters the date they attended the event and sometime a few other details and that automatically gives them the points.  This mostly happens in courses of several hundred students where it is not feasible to ask someone to actually go through and grade each essay each time (One class in particular has about 800 students that may end up attending at least 16 events, that would be 12,800 manually graded essays just in that 1 course).
3. The RCE needs to be the global one and all normal Canvas RCE functions need to fully work.  This includes third party tools being able to be integrated.

For Students, our top one would be:

4. Essay Question Type automatically being scored as 0.  While this may not be the top three for faculty, it would probably the top item for students so I am adding it as they are our biggest stakeholder. 

We have run into several issues while trying to use New Quizzes now and the answer from support continues to be “well it is an LTI so it won’t work like that”.  This needs to be addressed as that can’t keep being an answer.  This would be things like 24 hour delays in data, outcomes not connecting to the mastery gradebook, other LTIs playing nicely, etc. 

[jcox1]

1. Complete all features, especially partial scoring for category, order and matching

2. Add to To Do List for instructors

3. Bulk Migration from Old to New

And please don't make a change this big in the middle of an academic year. To think that Semester I quizzes may not operate in Semester II is too much in this current environment. 

[olden043]

1. The Designer role must be able to edit New Quizzes. Whether this is done via an updated permissions management option or the Designer role is mapped as a Teacher makes little difference to us. Proceeding with New Quizzes without Course Designers being able to edit New Quizzes would be a challenge.
2. Item Bank sharing/editing must be addressed as others have already stated. 
3. External Tools must be made available. Presumably, New Quizzes API will be made available and address this issue.

[lynchjo]

For my institution, it is absolutely critical that the RCE is available in all text input fields, including question text, answer text (both correct and incorrect answers), and feedback fields, for both instructors and students.

Last I checked, Fill-in-the-blank and Categorization didn't have the RCE built in to all such fields.

Allowing for partial credit with Matching is important, too, as is allowing bulk import.

[patrickdorls]

Thanks for listening to the community's concerns and adjusting the sunset process. The following requirements are necessary for the transition process

#1 Ensure that no content is lost, damaged or inaccessible

We have experienced before that not all questions / item banks were migrated correctly. Even though some bugs were fixed (e.g. true/false questions gave wrong results), there is still uncertainty if all content is migrated correctly and if the correct teachers have access to it, especially when it comes to large item banks that belong to the courses in Classic Quizzes, but are on personal level in New Quizzes.

#2 Seamless integration of New Quizzes into core Canvas

The decision to introduce New Quizzes as an LTI has led to several problems when there are similar settings in both core Canvas and the LTI itself.

If the teacher choose in the gradebook to publish the results manually, the results will be published automatically anyway if the teacher forgets to change this inside the New Quizzes settings. Same with anonymous grading, the teacher can easily access the students' submissions and real names through the moderate page inside the LTI.

#3 Bugs

There are several bugs, some of them have been reported more than 1,5 year ago and still haven't been fixed, e.g. the setting that teachers can give students extra time thorugh the Moderate page when a New Quiz already has been started. Everything looks correctly for the teacher, but the student doesn't receive any extra time anyway. If this bug cannot be fixed, why don't you just inactivate that setting? (Canvas support case no. 05553597)

[rdelp]

I will keep the simple... Why change the classic quiz option? Canvas has plenty of "infrastructure" to leave it as is... Why take something away from a teacher that works for them? In education, it is always on to the next shiny new thing. If classic works for teachers, have at it! If you want to use the new fancy quiz interface, get nuts! A teacher needs all the tools in the ol' toolbag, so please don't steal my tools...