[SpeedGrader] Warning for Incomplete Rubric grading

[willmoin]

We have been having an issue whereby grading staff are failing to fully complete rubrics when grading work in SpeedGrader. Because the rubric is often applied with the setting 'Use this rubric for assignment grading turned on' an incomplete rubric has a dramatic effect on the students grade for the assignment.

What we would like to see is a Warning appear when the grader hit's Save on the rubric (when the total score copies from the rubric into the assignment grade). The warning should read that the rubric grading is incomplete and whether user wishes to Save or continue editing.

Screenshot showing the rubric tool in the SpeedGrader which has been saved in an incompleted state

I think the problem is exacerbated by the move away away from on-campus exams, where graders are using the rubric to generate an exact grade for things like multipart scanned exam scripts, also where submissions are anonymised and a grader has a large number of scripts it can be like a needle in a haystack to identify problems. We have tried looking at the API, but the solution there seems very unwieldy for what is required. 

I would welcome the thoughts of other Canvas users, and if you support the idea please log-in to this system and click on the five stars above to vote on this idea. You can also speak to your administrative team about raising with your CSM at Instructure which can also apparently help.

Many thanks, Will 

I note that Belen reported the same issue (does not appear to have a feature request): https://community.canvaslms.com/t5/Canvas-Question-Forum/Incomplete-rubrics/td-p/492841

And a related Idea Conversation to implement autosaving of the rubric: https://community.canvaslms.com/t5/Idea-Conversations/Auto-Save-Rubrics-or-Give-Save-Warning/idi-p/3... 

[a1063023]

I agree, I've had to correct this on a number of occasions. I now use the API to check for it automatically and send out messages to staff, but it's like using a bazooka to kill a mosquito.

[Gregory_Putman]

Hi,

First semester with live students and TAs in Canvas courses.  I've had my first couple of the same.  This will be an ongoing issue.  Please address this issue.  I manage many sections of multiple courses with nearly 800 students and 25 or so TAs in any given semester so any help reducing incomplete rubrics is very important and helpful.

Thanks.

[ethomas48]

Great idea. Please implement.

[James]

Please no.

I use a rubric to incrementally keep track of assignments. That is, one homework assignment that has a rubric criterion for each section of homework for the chapter. Almost every rubric is incomplete, on purpose, until the end of the chapter.

Although I don't do it, I could see where multiple people could be responsible for assessing different parts of a rubric [Dr. Smith is the statistics expert, so he grades that part of the comprehensive exams, but Dr. Johnson is in charge of the first question, and Dr. Taylor is in charge of grading the second question.]

For those grading question at a time, it is reasonable to expect that they might complete a rubric in parts as well.

As mentioned, this information can be checked with the API and Canvas already has too many places where I have to tell it that I meant to do what I told it to do.

[Vide1955]

Hello

Different assignment types may require different grading scales and different numbers of levels. As you develop your rubric, decide how many different levels it should have. For example, you may choose a rubric with three or four levels for an essay assignment, while a one-level rubric (or credit/no credit) may be useful for smaller assignments and save you time when grading.

mybkexperience.com survey

[a1063023]

@James — I see your point but I think there are two responses to your comment:

* Just because it might be inconvenient to you, it causes actual problems for students now where students get incorrect grades because of staff oversight — I think the tradeoff would be worth it

* It would also be great if the rubric could ALSO allow you to "save as draft" (I don't think this has been opened as a feature request yet, but it's certainly necessary when you are grading an assessment after the marks have been posted). That would suit your use case nicely

[James]

@a1063023 

I believe I have seen a request to autosave rubrics here somewhere. I quick search found at least three for autosaving when completing a rubric and at least one for autosaving when designing a rubric.

I do not want my assignment as a draft. I want the students to see it so they can tell that they have turned in and I have graded their homework. I drop the lowest grade in that category, so the grade doesn't affect their homework grade for the chapter. I also have the ability to mark the assignment as not counting towards the final, which is what I do early in the semester when there are few grades. Saving as a draft doesn't help me. Historically, we had a much bigger issue (and I'm talking about things I've read in the community) with students having incorrect grades because people would forget to unmute grades than having incomplete rubrics.

I've been using Canvas for just over 10 years now. Many of the issues people raise in the Community are because of [lack of] training. My recommended solution then, as now, is education. Train people in the areas where they underperform. If possible, don't hire back people who cannot do their jobs.

I disagree that you should make life harder for everyone because a few people have issues -- even if those few people are students. I would suspect that more students are damaged by poor instructors than by incomplete rubrics, but I haven't done an analysis on the second (the first is purely anecdotal).

Canvas has traditionally designed for the middle 60% of users. As a math professor, I already have to suffer through inadequate quizzing because there's not enough return on investment to develop a complete solution that understands mathematics or science. I'm outside the middle 60%.

Another thing to consider is that perhaps the design of the rubric is too blame if people cannot  consistently complete them. If people have to scroll through 30 items, it's easy to miss some. If people put lengthy long descriptions in the boxes so that the instructor has to scroll to complete the rubric, they're more likely to miss stuff. That's something your institution can work on. I wrote a script to hide the long descriptions since the instructor that asked me for it knew the ratings and could grade without the long descriptions, but it left them there for the students. It made for a lot less scrolling during grading.

There are provisions in place for students with incorrect grades. Notably, the student goes in, notices the low grade and contacts the instructor to find out what's wrong. If your instructors are conscientious, they will check the rubric (the student could have done that) and notice that something is wrong.

Since you check through the API, you could automate the process of notification. Rather than just contacting the instructor, send a weekly report to the Dean so they are aware of the issues. Use that information to redesign rubrics where appropriate.

Although I think this is a terrible idea, Canvas has a history of making it harder for faculty to do their job by implementing pop up notifications to warn us about things, so they may well implement this.

[a1063023]

@James — thanks for the detailed comments!

I think we must be at cross purposes with the idea of a "save as draft" version of the rubric. Autosaving of rubrics is definitely a recurring problem in that staying on the page too long can lead to data loss, an inadvertent swipe can lead to data loss, etc. When I am marking a 100 page honours report having any kind of data loss is very demoralising... I understand there may have been some improvements to this recently so I hope not to run into that issue again but I'm not holding my breath.

100% agree that unmuting grades is all too easy to forget. I really wish that they could be scheduled or automated (I don't think there's anything in the API about that).

This is getting off-topic, but we do indeed struggle with the Canvas interface for the rubrics for our honours / masters project grading. We've tried to design them down to around 12 rows, and we end up with quite long descriptions because there's quite a lot we want to consider in each one. We could send people off to an alternate rubric, or even using a different tool like Google Forms, but currently have decided that despite the interface problems it's better to stay within and rely upon Canvas as much as possible.

[willmoin]

Thanks @James and @a1063023 - I think using the API to detect an incomplete rubric would be a big timesaver here, where I think staff are still eyeballing the rubric on each submission - they may well still do this but having an API flag would be belt-and-braces. So far I hacked something together to return entries for specified rubric criteria (having to know in advance the id of each criteria row), ending up with a spreadsheet I would then scan through. I'm not a coder but is the approach you use along these lines? 

When I encountered the first person using the rubric I had to catch myself before I channeled my inner Steve Jobs ("you're holding it the wrong way"). Previously I'd seen rubrics used for marking essays and assignments with a handful of criteria, rather than as an exam grading sheet. We have zero appetite for moving away from SpeedGrader, but a recent demo by CrowdMark looked good for allowing grading by question https://crowdmark.com/help/grading-a-remote-exam/. Since I posted this request I'm meeting more staff who use the rubric in this way, mainly from science and engineering areas.