New Quizzes End of Quarter Update for Q1 2024

[MPioRoda]

For reference: Link to New Quizzes End of Quarter Update Q4 - 2023

New Quizzes News

While the last quarter has allowed us to finish up the Item Analysis work including the CSV download, we’ve also been hard at work on two big features that don’t have a lot of flashy customer-facing changes, both of them bring value by increasing the portability of New Quizzes and giving greater control over sharing item banks to departments within an institution.

This quarter we will begin work to round out and enhance the setting to manage a students ability to view their results.  While New Quizzes provides more granularity in what types of information students can view after their quiz attempt, the missing piece has to do with limiting that view by dates.

Looking forward on the roadmap, we will continue the portability work by making New Quizzes available to share through Canvas Commons and start planning out the API work for scoped endpoints and accommodations. 

Q1 Achievements

• Item Analysis updates
• Moderation enhancements

Item Analysis updates

The Answer Frequency Summary table displays the number and percentage of students who selected each answer choice.  The Answer Frequency Summary shows the total number of correct responses, incorrect responses, and no responses for the following question types: Categorization, Formula, Hot Spot, Matching, and Ordering. The following question types have now been added: Fill in the Blank, Essay, File Upload, Numeric. 

Here is an example of how the Answer Frequency Summary would appear for a numeric question:

Answer Frequency Summary for Numeric Question

For Essay and File Upload question types the table is called a Score Frequency Summary rather than Answer since there is only one answer that is scored.  The information displayed shows the distribution of respondents per total point value and the percentage of correct, graded, incorrect, or no response.

Here is an example of how the Score Frequency Summary would appear for an Essay question:

Score Frequency Summary for Essay Question

Quiz and Item Analysis reports can be downloaded in CSV format, encompassing all metrics and Answer Frequency Summary tables accessible through the User Interface.  With the CSV download, instructors are able to more efficiently share data for audit and review, compare different reports, or further analyze the data with third-party solutions.  More information can be found in the release notes (2024-02-17).

Moderation enhancements

This past quarter we were able to release to production the following small features related to the moderate page:

On the moderate page, instructors are able to toggle between viewing scores as percentages or raw points to give greater flexibility and alignment to grading systems.  A configure icon was added in the score display column

Moderate View Score Toggle 

From the View Log link for each student, the question position has been added for each answered question to make it easier for instructors to verify the question position as students answer them.

Moderate View Log with Question Position

Q2 2024 Objectives

Export New Quizzes by Common Cartridge

We are nearing the end of development on the Common Cartridge work.  While the workflow and user interface will not change when exporting as Common Cartridge and importing course content, the change is in the files that make up the export package.  This change makes the export of New Quizzes more reliable when there are changes to the quizzes.  Users will be able to use the export packages as backups for their data regardless of what happens to the original quizzes.

Please stay tuned for a more detailed blogpost on the technical make-up of these files in Q2 when we release to Beta environments in the next few weeks. 

Update on Sub-account Item Bank Sharing and Outcomes alignment

As this work is in its final stages of development, the outcomes side is currently in testing mode and the sub-account bank sharing work continues to plug along, we are still looking to release this functionality by the end of this quarter.  At that time we will be publishing a blogpost to describe how outcomes can be aligned to individual questions in item banks and how we have expanded the functionality to share item banks by subaccount with a new sharing modal that allows users to search by subaccount.

Manage Student Results View

We will begin planning out the work to manage students’ results view this month. While the main goal is to limit student view of correct answers between certain dates, after a certain date, or before a certain date, we are considering the settings to show correct answers only after the last attempt or once after each attempt.

Course files in Item Banks Known Issue update

Last September, we posted about New Quizzes Migration - Question Banks to Item Banks. We identified a known issue related to course files in item banks.  Copying a course does not automatically change the links in an item bank, so when you use a question from an item bank in the new course, the links would still refer back to the original course. The current workaround is as follows:

If someone wants to put an image in item bank content (as part of a question), they are either loading it from their user files or from their course files.  If they are loading the image from their own user files, as long as they set a user file as readable, then the links will work in item banks when they copy a course. A warning appears only if it is course linked content, so if they remove it as a course file and move to a user file then the warning should not come up. Once they move the file over to their user files. They will need to delete the old link to the course file and re-insert the new link to the user file the way they would normally insert a link with the Rich Content Editor in the question.

The bigger picture around this issue is that this work entails improving Canvas file handling to allow for better support of files not directly associated with a single course or user such as those that are in use by item banks.  We consider this a top priority as we work to optimize the capabilities of item banks and the portability of quizzes. As senior engineering leadership is discussing the work, please stay tuned for an update once they have a plan for a path forward.

Quarter 1 Completed Fixes and Features

Here is a list of the related fixes or features that we completed in the past quarter:

• Fill in the blank questions in an exported New Quiz fail to import if it was migrated from classic previously
• Blank print previews are cutting off or missing questions in Chrome
• Embedded external tool is removed when exporting and importing with New Quizzes content
• Grading notes don't copy from build new quiz export
• New Quiz QTI file exports import media embeds broken
• New Quizzes print blank quiz displays "Typesetting math: 100%" on each page
• New Quizzes with Lockdown Browser enabled return an error with student view(Respondus)
• Accessibility fixes
• When creating a Stimulus the RCE steals focus from the title field
• Separate "No Response Tooltip" from the Table Header in the Answer Frequency Aggregates Summary Table
• Duplicate info presented to screen reader users in the Answer Frequency Summary Table
• Unable to access headings on Quiz index page via screenreader

[SeanPBA]

@MPioRoda Thank you for the update.

I noticed that the "API enhancements" feature for customizing "test-taking accommodations in New Quizzes" has been pushed back to "Q1 2025 and Beyond" on the Canvas Product Roadmap. 

This is disappointing since this feature would solve the last problem keeping us from fully embracing New Quizzes at our institution. Until we are able to programmatically update test-taking accommodations in New Quizzes using an API endpoint similar to the functionality of the Quiz Extensions and/or Course Quiz Extensions endpoints that currently exist for classic quizzes, we are unable to use New Quizzes.

Please reconsider placing more importance on this feature. This was originally planned for Q1-Q2 2024, then it was pushed back to Q2-Q3 2024, and now it seems like Instructure is sending the message that this is a very low priority.

[leward]

Thank you, @MPioRoda , for raising the profile of the Course files in Item Banks Known Issue.  This issue has occupied the better part of my time for the last several months , and I would like to add some observations, questions, and recommendations based on our experience with the bug here at Indiana University. We've already communicated most of these points to our CSM, but I think it's important for you as the product owner to be aware of our questions (many of which remain unanswered) and concerns:

Why this bug is so egregious

Indiana University has been a Canvas customer since 2012 and this bug is among the most egregious we have encountered over the course of our 12 year relationship with Canvas and Instructure for the following reasons:

• Unlike most bugs, which tend to break a feature only while the bug is in effect, this bug is causing faculty to unwittingly create bad data in their item bank questions.  The longer the bug persists, the more bad data is created.   Fixing the bug itself will not fix the data. 
  
  Here at IU, over 700 new question banks have been created since we first started investigating this problem in mid-February and at least 40 of those banks contain questions with broken image links.  We have notified the faculty at IU who are using New Quizzes and provided them with a workaround, but there's no guarantee that they will read or heed our message.  The number of item banks with broken links will undoubtedly continue to grow at IU and elsewhere until the bug is fixed.
• Since the links work in the course where the bank questions were created, most faculty won't even be aware there's a problem until a future term when they attempt to use the same bank in a different class.  We have already had several reports from faculty whose students encountered broken images while taking an exam.  And that brings me to mt last point...
• This bug affects the integrity of online assessments--it results in students taking quizzes and exams with questions that can't be answered because the images that are the subject of questions or responses are not visible.  Indiana University was a major contributor to the Sakai LMS and the owner of many core tools.  Had a bug like this occurred in the Saki Quizzing tool, our developers would have dropped everything until it was fixed.  Basically anything that affected the gradebook or quizzes was considered urgent.  According to your blog post, Instructure has been aware of this bug since September of 2023 (although I suspect the bug did not make its full blown appearance until February of 2024).  If that is indeed the case, how is it possible that the issue has not yet been fixed?  And in the meantime, with the exception of your September 2023 blog post (the topic of which was item bank migrations, which is completely unrelated to the context for this bug), Instructure has done nothing to communicate the nature of this bug and its full ramifications -- particularly the creation of bad question data-- to customers.    
  

What happened to the verifier parameter?

After the release of the full Canvas RCE in NQ on June 17, when instructors embedded images or added links to documents to an item bank question, the files were indeed uploaded to the course Files tool, but a “verifier” parameter was added to the file location, which allowed the file to  be visible to students in courses other than the course where the file was stored.  Here is an example of the HTML that appears in these bank questions:

<img role="presentation" src="/courses/2178757/files/163718916/preview?verifier=bxCsuSRvCKyBHmWCZt9Kcspk6n3cG5dLai1CfcfI" alt="" width="592" height="333" />

Most of our bank questions with images created before mid-February 2024 include a verifier parameter and work perfectly well. 

After mid-February, however, all new bank questions were missing the verifier.  So, it seems there was initially a solution (albeit an imperfect one) for this issue. Then, for some reason, it disappeared.  Does anyone know why this happened and whether it might be possible to bring it back, at least as a stopgap measure while a more robust solution is developed?

Also, while investigating this issue, I noticed that when images are uploaded into question bank questions in Classic Quizzes, the image is first uploaded to course files,  but as soon as the question is saved, a copy is placed outside of the course in a location named “/assessment_questions/…” (here is a sample embed tag: <img src="/assessment_questions/140071915/files/170597828/download?verifier=dh0LSkR6hBaBnD5ayk1MTzzNkZfEpjEhdAV9XYN7" alt="custom link grid.jpg" />.   So, apparently the original Canvas quizzes team was confronted with the same challenge and devised a solution.  Given this, why wasn’t the need to store item bank assets outside of the course recognized as basic requirement of the full RCE implementation in New Quizzes?

When did this issue actually begin, September 2023 or February 2024?  

When I scanned your blog post New Quizzes Migration - Question Banks to Item Banks last September while testing item bank migration, I remember seeing a paragraph with the heading “Course Links in Item Banks.” I probably didn’t read the content beneath the heading, however, because the “Course Links” tool in the Canvas editor is for creating links to native Canvas content like pages and assignments, and I wouldn’t necessarily expect links to course content to work in item bank questions.  Moreover, around the time you made the post, embedded images and links to documents in item bank questions were not broken here at IU because of the aforementioned verifier parameter.   I can find a few isolated cases from fall 2024 with missing verifiers, but those questions may have been created by a different method such as copying the question from a quiz to an item bank rather than uploading the file into a new bank question.  I can provide numerous examples of bank questions with verifier parameters added to links in banks from summer 2023 through early February 2024, which tells me, at minimum, if this bug was present in fall 2023, it only occurred intermittently and did not affect most users or bank questions with links.

Beginning in mid-February 2024 and after, however, it seems that all new bank questions with links to course files are missing verifier parameters.  I have many examples from this period. Unfortunately,  I've had to resort to finding these examples manually because New Quizzes data is not yet available in CD2, so my findings does not take all of our bank data into account.  Regardless, the preponderance of evidence seems to suggest that something changed in February 2024 resulting in the verifier parameter being dropped from all links to course files created in new bank questions

Inadequate warning message

Last fall, the following warning message started appearing when instructors attempted to save item bank questions containing a file or course link that referenced the current course:

“You are attempting to save content containing course links which may not work when used in other courses.”

While this message does alert instructors to the fact that there may be a problem with the question, it is not particularly helpful for the following reasons:

• It uses the term “course links”, which, as a mentioned above, is a feature for creating links to native Canvas content, not files.  Updating the text to something like  “...links to course content or files…” would at least be more accurate.
• The wording is ambiguous at best, and the message appears regardless of whether the file is available to students in other courses.  For example, the message appears even when the link includes a verifier or the file has been made public and is therefore available in all courses.  The message would be far more helpful if it only displayed when there was an actual problem.
• The message appears in plain text and is easily ignored. Additional formatting through the use of color, borders,  icons, and font weight might draw greater attention to the message.
• Most importantly, the message provides absolutely no information as to how to determine whether the links will or will not work, and if not, what to do to remediate them.   Adding a link to the full details on how to detect, work around, and remediate this bug would be very helpful.
  

Who is responsible for remediating broken links? 

As I mentioned at the top of this post, after a fix is deployed, the broken links will persist unless someone or something (i.e. an automated script) remediates each and every broken link.  Manual remediation is extremely tedious work (see the documentation we have prepared for our support staff and faculty).  Aside from not having the time to do this work, the process is simply too complicated for many instructors.  It doesn't seem reasonable to expect instructors or support staff to spend hours correcting a problem caused by a Canvas bug.  Our CSM was unable to say whether Instructure will fix the data as well as the bug.  Can we count on Instructure to remediate all links in bank questions affected by this bug?

What is the ETA for the fix?

We were thrilled to learn that the priority of this issue has been escalated.  Still, we realize it’s a difficult nut to crack and may take some time to resolve.  Is it possible to provide an estimate of when a fix will be available?  Are we talking weeks, months, the better part of a year, or even longer? Also, if the long term goals for the fix cannot be achieved in the next few weeks, is it possible to put a stopgap solution into place.  For example, might it be possible to find and restore the code that adds the verifier parameter to item bank links to course files.  Alternatively, when an instructor uploads a new image into an item bank question, could the permissions on the file be automatically adjusted to set the availability to “Only available with link” and the visibility to Public  We realize that neither approach will solve the related known issue, “Images shared to item banks can break if original course/item is deleted”, but this bug affects far more users than the related one.

Dedicated Known Issue needed

We have repeatedly asked our CSM to try to get a separate Known Issue posted as well as meaningful documentation on how to cope with this issue until it is fixed.  Our CSM was made to understand that a separate Known Issue will not be created because this issue is under the same Jira as this “Images shared to item banks can break if original course/item is deleted. He explained that Instructure only publishes one Known Issue per Jira. That may be a useful internal rule, but it is not helpful to your customers who are counting on you to alert them to import bugs through the Known Issues pages  The published known issue does nothing to warn customers about the fact that links to files are broken regardless of whether the course has been deleted or not, which is much more common occurrence than the deletion of courses  At minimum, it seems reasonable to modify the published Known Issue to incorporate information about this, much more egregious and widespread issue.   Ideally, a separate known issue with considerably more detail would be provided.

As an aside, the description in the published known issue is overly specific.  It states that the problem occurs when a bank is "shared with the institution", which, while true, is not entirely accurate.  This problem affects any bank whether shared with the institution or not.  If the bank owner attempts to use a question with an image stored in a deleted course, the image will not be available, period, regardless of the sharing status.

Updated documentation needed

We also asked our CSM if information about this bug could be added to How do I add content to an item bank in New Quizzes?, and were told that this isn’t something that Instructure ordinarily does.  In my view, this bug is serious enough that Instructure should be taking proactive steps to alert customers, even if it means breaking from standard procedure.  There is nothing on the Community site that provides a full explanation of how to detect, work around, and remediate for this bug.  This puts the full burden on your customers, which doesn’t make a lot of sense since we don’t have all the information needed to communicate accurately and effectively on the topic, not to mention the fact that each institution would need to reinvent the wheel.

But, if mentioning the bug and workaround in the Canvas guide isn’t in the cards, why not simply add a note along the following lines (this would eliminate the need to mention the bug while providing accurate information about how to create a successful link as things stand today):

Note: When embedding images or linking to files in bank questions, take one of the following actions to ensure the files are available to students in all of your courses:

• Before creating the question, upload the files to your personal User Files and set the availability to “Published” or “Only available with link”
• If you would prefer to upload during question creation, after saving the question, set the availability on the file to “Only available with link” and the visibility to “Public”.  Note that the link to the file will break if the file or course is deleted.

Conclusion

From my perspective, Instructure's handling of this issue has been less than optimal to say the least.  Personally, I would like to see Instructure be far more proactive and transparent when it comes to addressing and communicating about far-reaching issues like this one.  There was a time when we received regular communications from Instructure about serious bugs and performance issues, which was very much appreciated, but unfortunately these notifications were discontinued in 2018.  Perhaps its time to resurrect this practice.

[audra_agnelly]

I am so frustrated by Item Banks and content sharing in New Quizzes. From the get-go, there has been so much confusion about where things live once a bank is created and propagated out to courses, and issues with media links living in courses students don't have permissions to view. Bank questions are created by creating quizzes in a course....when an instructor adds an image from their desktop to a question through the RCE the image is placed into Course Files. Adding images to User Files just isn't a workflow instructors are used to and has never been necessary to embed images anywhere else in Canvas and still have them work following a course copy. Copying content should. just. work. Instructors should not have to go through unintuitive hoops to move files to User Files and re-do their previously embedded quizzes. We do not actively promote the Item Banks feature because of the unreliability around embedded media content.

Do not get me started on the Support Agent that escalated a ticket to me in which they told the instructor that the institution admin (me) could write custom code to fix the issue for them? Huh??? Please advise Support on the correct responses to the issue, no one should be telling my instructors that I can write custom code for them when this is a design issue the New Quizzes team needs to correct.

[dbrace]

Reading @leward's and @audra_agnelly's posts (thank you, Thank You, THANK YOU for sharing it and sorry for what you and everyone else has been going through) makes me glad that my institution (a two-year community college with small Department of Online Learning team using Canvas fully since Fall 2019) has not yet enabled New Quizzes (except in beta).

[pjonas1]

Please just give us an item bank editor that lives outside of a specific course and stores its files in user files with correct permissions in a sensible layout. The whole point of item banks is to use them in more than one course.

While you're at it, feel free to make the item bank editor better:

1. Whether a question has feedback written should be visible without having to open individual items.
2. Item Analysis should flow back into an item bank across all courses and quizzes that use that question. And then reset when you create a hopefully improved duplicate.

[Charles_Barbour]

Bravo @leward! If only Instructure put as much effort into understanding and documenting the problem...

While our use of NQ is thankfully limited, I can't think of many situations which are more stressful for students and instructors than having problems during an assessment. You get one shot at an assessment and having technical issues can fluster even the best students and instructors. Other than system outages or data issues (loss/corruption/disclosure), almost any other LMS problem can be resolved after the fact with minimal impact.

Not only are there problems today, but the problem scope continues to increase every day.

Yet Instructure has:

• not provided even a rough estimate for a fix
• not come up with a workaround or temporary solution
• somehow made the situation worse by removing the verifier parameter (without explanation?)
• not added adequate warnings to help instructors to understand and avoid the issue
• not updated their documentation adequately
• not created a detailed remediation plan
• not really demonstrated they grasp the severity of the issue and view it as a top priority

Based on the way this issue is being handled, one might assume Instructure doesn't think problems with assessments are a critical issue.

I certainly hope that's not the case.

[jchapes]

I can say that I support everything that Leward added in the comment above. I've communicated with our colleagues at Indiana about this issue and our own testing at the University of Kansas supports the timing that something has really broken with files/images in Item Banks in February 2024. New Quiz Item Banks is essentially an unusable broken product until these file handling issues are addressed. If Item Banks can't be used reliably in different courses other than the one it was originally built in, then Item Banks have no point in existing. This must be fixed.

[jmurray6]

To add to @leward 's post above - I would like to mention that this bug can impact quizzing even if one isn't using item banks.

We've run into it in several instances with courses that are using Lockdown Browser. Several of our instructors build directly in New Quizzes in a particular course without using Item Banks because they are building quizzes that will be used for just that semester. When building a quiz in the course, the images used do no get the verifier - and in general, that's ok because those images become course images and students in the course have access when taking a "new quiz." However, once Lockdown Browser is engaged, if an image link does not have the verifier - it's broken in Lockdown and does not display.

The workaround we've had to implement is for instructors to put these images in their user files and then add them to the quiz - so that they do get the verifier and thus work in Lockdown.  The problem is that there is no way for an instructor to know if they've got an image that is going to be broken in Lockdown or that needs to be modified/moved to their user files because they're generally not technical enough to be able to switch to the raw html of the question and verify if the image has the verifier in the html code or not. So my office (which is just myself and one other admin) ended up with a significant amount of calls from faculty who had students who couldn't complete quizzes because the images were broken.  Since Lockdown is essentially an LTI, student view does not work, so there is no real way for faculty to verify that everything is working correctly before students take the quiz.

This is also going to create other issues for us because now we're going to have all of these images in the faculty user files - and we've had cases before where faculty have done this (sometimes accidentally) and then several months later ended up deleting files from their user files because they don't know why they are there - or don't understand what user files are. . .and thus, they've broken their assignments or quizzes because the source file that was in their user files is no longer there. OR, they've shared quizzes / course content with another instructor - and then deleted the user files (because of the reason above) and now the new instructor has broken quizzes that we cannot fix because - the images were in the other faculty's user files and the other faculty no longer has those images. It has been an absolute mess for us.

[dbrace]

Thank you for sharing those details, @jmurray6. My institution also is a Respondus LockDown Browser + Monitor customer. I am so glad that we have not enabled New Quizzes yet.

[isabel_anievas]

Ok, let me see if I understand this correctly. 

I have a number of classic quizzes with short media recordings that I am starting to migrate to New Quizzes. 

The current migration tools do not migrate my original media files (I teach Spanish, these are short recordings in Spanish that were created on the quizzes themselves).

I have started to migrate these quizzes one by one and re-record the audio in New Quizzes.

Every time I record a new media recording into a New Quiz I get a warning that says: "You are attempting to save content containing course links which may not work when used in other courses".

So, now, what I am hearing is that a new copy of my course will have no access to these recordings? Is this correct?

As for the proposed solutions, I have no idea how to implement them. 

I am recording my audio directly into my question bank. I cannot see where these recordings are stored. I have looked carefully at all my user files and course and I do not see them anywhere. 

So, basically, I should stop trying to migrate my classic quizzes into New Quizzes until this issue is resolved?

[LauraJiménez]

Hi everyone,

I 100% agree with you that this bug is serious enough to be a priority, and it's been several months since it was detected.

We've also been dealing with some issues regarding broken links in New quizzes.

When a Bank Item is created from the blueprint course, when you execute the synchronization to its related courses, the links (images, for instance) will still point to the blueprint (broken links).

We've been doing a few tests and what we've found out is that if you create the Bank Item from the sub-account where your target courses are instead of one course, the images will display correctly.

I hope this helps 🤞

[leward]

@MPioRoda .

Another observation.  In your blog post,  New Quizzes Migration - Question Banks to Item Banks, under the heading Course Links in Item Banks, you mention the impact on teachers and TAs, but not on students.  A teacher or TA who reuses their own item bank in a different course will be able to see the images embedded in their bank questions.  It is only students who will experience the problem, because they are not enrolled in the original course.  It is critically important to describe the bug in a terms that help end users and support staff understand the full ramifications of the issue. If I were an instructor reading the description in your blog post, I would assume that as long as I can see the images, everything should be okay.

[leward]

@isabel_anievas ,

If you are using the native Canvas tool for creating media recordings in bank questions

the videos should be available to your students in any of your classes.  Unlike embedded images and linked documents, Canvas media is stored outside of the course, not in course Files.  I just tested adding a media file to a bank question and I was able to access the video as a student in another course.

On the other hand, if you are using a third-party LTI tool like Kaltura to store and share class videos, the links may not work.  In the case of Kaltura, for example, we can't use the LTI integration because it authorizes users based on their membership in the course where the video was embedded.  As an alternative, we use a different method of embedding the video that does not rely on LTI.

[llettie]

Hi Everyone, 

This has also been issue at our school as well.  Fortunately for us not everyone is using New Quizzes yet, because of issues like this. 

And now because of issues like this, departments at our school are losing confidence in the Canvas assessment platform and are starting to use solutions like ExamSoft. 

[isabel_anievas]

@leward 

Thank you so much for the response! 🙂

Yes, I am using the native Canvas tool to create these media recordings. 

I think I have seen some of these media recordings uploaded to my Course Uploaded Media file. I will double-check!

Great to know that you were able to access the video as a student in another course. That's very reassuring! 

Thank you so much 🙂

[Mikee]

A massive and heartfelt shout out to @leward for the post above. 

Instructure seems to have missed the ferry with this entire functionality - and continues to chase the sunk-cost fallacy with what looks like (from the outside) a fundamentally faulty foundation.

[mbmacdonald]

@MPioRoda, regarding "Manage Student Results View": is this going to include any ability to customize the results page? Just to clarify, I am talking about the page shown below:

At the very least, we would like the ability to add a message (explaining what to do next, for example). Best case would be different messages based on the score.

[MPioRoda]

Thank you all for this feedback. We greatly appreciate the time you have taken to write very thoughtful and thorough comments. Your insights into the impact of this bug are valid and are being discussed amongst engineering and product leadership. Thanks again for your patience. We'll keep you posted on progress.

@SeanPBA  We have had to make some adjustments to the timeline to prioritize the API work in the following order: Scoped Endpoints, Accommodations, Reporting, and Submissions so that we can be realistic about our current capacity.  However,  we are looking to add resources in the next few months that could help us to accelerate and pull the API timeline forward.

@leward We are discussing the verifier parameter work internally to understand why it was removed and the possibilities for remediation.

@isabel_anievas When the warning comes up, it means it "may not" work and users will have to ensure the file is publicly accessible.

@mbmacdonald Manage Student Results View is just about the settings around when and what the students can view.  The work you are talking about is actually part of a small project we're trying to push through soon which does exactly that, adds a way for teachers to add content for students to see when they finish the quiz.  I will connect with the engineer who developed that and discuss the possibility of adding messages based on score.

[Charles_Barbour]

While I appreciate the response, I feel it is not sufficient. It doesn't really indicate an increased understanding and acknowledgement of the scope of the problem. Nor does it imply that it is being prioritized more highly or that any additional effort and resources will be deployed to resolve it.

@leward wrote over 2000 words on the topic. She asked 4 questions and made 3 specific requests or observations.

You only responded to one of those.

Do you plan to answer or address the others?

Questions:

1. What happened to the verifier parameter?
2. When did this issue actually begin, September 2023 or February 2024?
3. Who is responsible for remediating broken links?
4. What is the ETA for the fix?

Requests or observations:

1. Inadequate warning message
2. Dedicated Known Issue needed
3. Updated documentation needed

[jon_mason]

Thank you @leward for highlighting this issue. While we at Oxford don't use New Quizzes that much (largely because of the various other ways in which it lets itself down), I've just spent a good chunk of time identifying instances of this issue and applying the workaround within the ~250 New Quizzes in the courses that I have oversight of. It's worth noting that I think it is sufficient, certainly in our instance, to change the visibility of the image files to "Institution Members" (not "Public") as users have to be logged in to take a quiz. 

However, that has only fixed existing, broken, banked questions with images, not those that are created in the future, or have already been created without the verifier and are added to a bank later. 

I just wanted to add my voice to those asking for more clarity and action on this, and for Instructure to commit to taking remedial action to fix all questions with images that lack the verifier (or whatever the ultimate solution for sharing the images is), for all questions created since it was removed, whether that was Feb 24 or Sept 23. 

As a side note, identifying instances of this would have been much easier if the full New Quizzes data was available in Canvas Data in the same way as Classic Quiz data is. I was quite hopeful when I asked "Ask Your Data" for all New Quizzes questions shared with a bank and containing an image, but was quickly disappointed again when I discovered it had just given me the Classic Quizzes questions. I don't understand enough about what the mentions of the API in the New Quizzes roadmap (https://community.canvaslms.com/t5/New-Quizzes-Hub/ct-p/new_quizzes_hub) actually mean to know whether this is included within that, but even if it is, it doesn't look like it's coming until 2025, and based on what's happened with New Quizzes so far, that's probably a very optimistic best-case scenario!

[leward]

@jon_mason Thanks for the tip about sharing with the Institution instread of making the images public.  Originally, I was advising instructors to use Instructure's recommended workaround--placing the files in User Files instead of course files--but that requires extra steps.  In another community thread, someone mentioned setting the availability to public, and I latched on to it because it was so much easier without even considering the possibility of Institution.   I did, however, ask our CSM and L2 support whether setting visibility to Public had any downsides and neither offered up any, so I assumed there was reasonable security through obscurity .  However, after reading your post, I decided to do a little testing and, to my horror, I discovered that public files in Canvas are indexed by Google.   So, I'll definitely be modifying my documentation to recommend setting visibility to Institution rather than Public.

[sara_weaver]

@leward Yes- we've recently brought this issue with public links being indexed by Google, and I'll add on, permanently (even if the visibility within Canvas is switched back to something else). Unfortunately, Instructure doesn't see this as a problem. They can be removed by either deleting the file in Canvas, or resetting it via the API. Of course, there's no way to identify any files that have public links within Canvas. These links also persist if a course is copied. We offered some suggestions to them:
1. That file link is created when a file is made public. It should be reset when a file is no longer listed as "public". 

2. These publicly available links should be time-bound, and expire after a certain period of time.

3. There should be an easy way via the UI to reset them.

It's a huge headache that Instructure does not seem eager to acknowledge, let alone solve.

[leward]

@sara_weaver , OMG, that is absolutely terrible, particularly the fact that they continue to be available in Google after the visibility is restricted.  IMO, I think all Canvas content should not be indexed by default, even if it is public.  Making public "discoverable" via Google should be a course-level option.

[Charles_Barbour]

Why would making them public also make them available to Google to index?

At a minimum, why is there no way to make a file public but not index-able by search engines? Panopto offers that option.

[sara_weaver]

Excellent questions, and I wish I had answers or any faith that Instructure prioritizes any sort of fix.

[dspiel]

It is time to admit that New Quizzes is a failure and start from scratch. I can give SO many reasons why our institution will not move to New Quizzes, but will only give a few.

1. Less bells and whistles, Classic Quizzes works better and far less clicks to accomplish anything.
2. Converting Classic Quizzes that use question banks takes forever just to prepare to make a copy, but after doing that work, faculty then has to recreate their quiz.
3. With Canvas raising costs, charging for training, and then making a new product that takes additional time to use, I wonder if they will be covering the costs to train all the faculty in using it as well as the extra time it will take for them to use it within their course(s).

Our Nursing program made a decision to all try and move to New Quizzes in academic year 22-23 which means they've been using it for about 2 years. The Dean is now to the point of having everyone move back to Classic Quizzes for 24-25.

[canvas_admin]

We refuse to use New Quizzes as well.  The LTI approach is a fiasco. Instructure admitted mistakes when it comes to community idea voting. Why won't they admit New Quizzes LTI was a mistake?

[Hildi_Pardo]

I have not encouraged teachers to use New Quizzes and we hesitated to turn it on for the longest time, but since Instructure Announced Classic Quizzes were going away (thank god it did not), we had no option.  And we had seen some positive things.  But I was always confused about the Banks and could never get a straight answer on best practices for using them.

Thank you @leward for the detailed write up and advocating for us all!

Just this morning a teacher reached out because images were not showing when they tried Student View (I know they are not supposed to use Student View with Quizzes and New Quizzes, and should use Preview, but that's another story).   They discovered, just one hour before students were supposed to take the test, that images don't work!   This is the last unit test of the year!!   The teacher had imported the Test from another course managed by the CLT, where the Quiz is pulling from Item Banks.    As a System Admin, I'm trying to figure out how to help them while they can continue teaching --- yeah, teachers can't stop teaching and attending to their students to figure these things out -- so as I'm trying to find the Item Banks, I click and they are not available to me for some reason.  Teachers don't have time to stop in the middle of the school day to troubleshoot and try out work arounds, even with the support of the System Admin or Canvas Support.    So now what happens with the Final Unit Test of the year??   Now they have to scramble to figure something else out. 

Priority must be given to correcting THIS problem before other improvements.  

[dbrace]

Tagging Update on the Known Issue Related to Course Files in Item Banks so followers of this post and see the latest information.