Outcomes, Item Banks and New Quizzes

[jmarrapodi]

I am really hoping someone has figured this out. I built a 108 question item bank at the root level for my university. It only shows in All Item Banks. I don't remember choosing anything to get it there. I just went into Item Bank in my Admin login, and started a new one. Each question is coded and tagged so I can find them by date added, and outcome code. I attached the matching outcome to each question.  

I built a quiz in a course using New Quizzes in the master course, importing the questions from the item bank, randomizing the questions. That all worked beautifully. We copied the section and the students took the exam which also went fine. 

Here's the problem. When I run the Outcomes Analysis it shows the total score under every outcome rather than the X of Y score I was looking for.

I've been back and forth with support on this and got escalated to L2 and was told 

Our guide states "Outcomes associated with questions from item banks do not display in the Outcomes Analysis report." Only the outcomes directly linked on the quiz will be in that report. https://community.canvaslms.com/t5/Instructor-Guide/How-do-I-view-reports-for-a-quiz-in-New-Quizzes/...

I have 2 questions.

1. What did I do wrong in the setup? Was I supposed to build the bank in the course? Can you even do that?

2. Is there a way to correct the reporting on the students who have taken the exam? Fortunately its only 9 students.

Thanks.

[Ron_Bowman]

@jmarrapodi -

What about them?

[jmarrapodi]

@Ron_Bowman Thanks for asking and the quick response. I hit save after ruling out the possible already asked questions, assuming it was starting a question but it actually posted it. I just added the content and hope your brilliance can shed some light on this.

[Ron_Bowman]

@jmarrapodi -

Thanks for the update.  Unfortunately, I will be of no help with your question.  However, now that the actual information is posted, someone else may be able to help.  The only reason I made my post was for exactly what you stated.  Or to see if it was just another person posting an empty post for no reason.

[chenshall]

@jmarrapodiI'm afraid I can't help but I have the same question - is it even possible to build question banks within a specific course? I built one that ended up in All Item Banks, quite unintentionally it seemed to me.

[pcharles]

I am not 100% sure what your are asking, but I am going to assume it is the same problem I ran into a year or so ago.    My interpretation is that when you create questions in a bank, you cannot see the outcomes that you created for the class.  Is that correct?  If so, read on because I had the same problem and after working with many people at Canvas they did not seem to know what was going on.  This, I have learned, is normal.

Anyway, what I noticed was that if I created a quiz, and then created a question in the quiz, the outcomes for my class were available. If I tagged the question in the quiz with an outcome and moved it to a bank, the outcome remained attached to the question.  How this works, and why Canvas has no idea why it works is a mystery, but I was able to convert about 1200 question and get them all tagged.   Here are the steps I used:

1. Create a quiz and give it a suitable name - I teach chemistry, so my name might be "Ch02Q01-Naming Type 1 Ionic Compounds". I use a chapter and number so that I can sort them later. 

2. Add questions to the quiz, one at a time, from the bank.  DO NOT tell it to select X number, you must add each question, one at a time because this copies the question in to the quiz.

3. Delete the question from the bank. Yep, seems drastic, but you cannot edit questions in quizzes if they are in banks. So, after you make a copy in the quiz, you must delete the one in the bank.  I recommend you run a test with one question and check that everything is working. 

4. In the quiz, you should now be able to edit the question and add the outcomes from your class. 

5. Once you are happy with the result, click the button to send the question to a bank. When I did this I made sure the new bank had a different name to the old bank so that I could keep track of things. 

6. You should now be able to go to the bank and see the question and that it has the correct outcomes attached. 

Moving Forward:  What I do now is write all of my questions in an appropriately named Quiz and then copy it to a bank when it is tagged and done.  I have even created a separate canvas course called "New Quizzes" where I write all of my questions. 

I haven't done this conversion since summer 2021, so I am working from memory, but hopefully the steps are correct.  It sounds tedious, and is, but once you get in to the swing you develop the muscle memory and can move pretty quick.  

Paul

[pcharles]

Re-reading, I think I have not answered your question. Sorry.  There is a page of settings for outcomes, and how they are calculated.  I'm not familiar with the stats, so I just use the default.

[jmarrapodi]

Hi @pcharles.

Well maybe this is the workaround. Thanks for your reply. It sounds like you rebuilt everything on the course side, THEN added the outcomes, and pulled things into the bank. That seems backwards to me, but honestly, I don't really understand the logic of how this is working. 

I built the questions, tagged things, and added the outcomes, starting in the root level (I have admin privileges). When I imported the questions into the course quiz, they show with the questions, show in the item analysis (see screen shot), but the outcome analysis is not sorting the questions by outcomes, and only showing the quiz score under each outcome. This appears to be a known bug. I got this from Douglass Healey, one of the L2 Canvas support folks. It's documented in the linked post.

Our guide states "Outcomes associated with questions from item banks do not display in the Outcomes Analysis report." Only the outcomes directly linked on the quiz will be in that report.

https://community.canvaslms.com/t5/Instructor-Guide/How-do-I-view-reports-for-a-quiz-in-New-Quizzes/...

It seems to me, though I'm not a programmer, that the report is missing a sort by field, since outcomes are available in the course with the questions. @CommunityTeam do you have any more insight on this? Maybe there's an Outcomes group I need to poke around in. 

As to editing them, when you go to edit an item that is part of a quiz, it throws a message that you're editing a copy of it because students have already taken the exam. That works sort of like the rubric edit message that tells you you can't edit the existing rubric because it's in multiple courses, but makes a copy for you to work with. For the exam questions, that makes sense. It keeps the record of the way the quiz was when the student took it, and changes are made to the question moving forward, kind of like versioning. I was able to edit things in the root level when I needed to fix something. It would make sense that you can't edit at the quiz level, I think. 

Jean

[pcharles]

Yep, it seemed like a completely backwards solution to me, but that was the only way to work around the problem.  If I create the question directly in a bank I do not have access to the outcomes I created and I needed access as part of the university assessment process because you cannot use the quiz results for assessment when you randomize from banks and do not use multiple guess questions.  

Paul