Student Performance Texts

[kona]

If you follow the Release Notes‌ then you probably noticed that Canvas is trying out something new called "Student Performance Texts." These texts are to help students better manage their course work and applies to students who have provided their phone number in their Canvas profile. For right now there are only two types of notifications: (1) reminder to turn in any unsubmitted online assignments that are still available after the due date and (2) reminder to retry a quiz if a quiz contains multiple attempts and the highest score was less than an A or 95%.

I'm pretty excited about this because  @James ‌ had his statistics students research "what students want" in terms of communication from the College, including Canvas. Overall the majority seem to want information via text message and in terms of this type of automated text notification, they would REALLY like to know if an assignment due date is coming up and they haven't submitted the assignment. So maybe if it's 24 hours (or 12 hours) until a due date and the student hasn't submitted the assignment then they get a text notification. Hopefully this is coming, but for now we had our first report from a student that they got one of the text notifications (see below)

This definitely got the student's attention and if the student hadn't already submitted the assignment then I think it would have prompted them to get it done! As it is there appears to possibly be a bug of some type since the student got a text before the assignment was even due and even though the assignment was already submitted. This was part of a group assignment (graded as a group, not individually) so I'm wondering if it's because this specific student didn't turn in the assignment - even though someone from the group did submit it.

Regardless, I think there are a lot of benefits to student performance texts and I hope someday soon they include more performance/grade scenarios and the ability for Instructors and Admins to see which students received a text (and the type of text).

[laurakgibbs]

This sounds really useful, Kona! I am wondering about the contents of the text. A message like "any score is better than a 0" ... do admins set a default? Do instructors get the tweak the default message? I would definitely want to phrase my message a bit differently than that, but for students to get a text message after the "due date" but before the end of quiz availability would really be perfect.... and it would also be great if instructors could flag as items as NOT getting text message reminders. I have a bunch of extra credit assignments that truly are "extra" and I would feel bad if students were getting a bunch of texts about those when they are totally optional and not something to stress about.

Pinging  @kevin1 ‌ about this new thing; here's the Canvas Production Release about it:

https://community.canvaslms.com/docs/DOC-9721-canvas-production-release-notes-2017-04-22#jive_conten... 

[kona]

As of right now, no, there is no customization or tweaking of the texts or when they are sent out. Hopefully that will be coming in future updates!

[ryan_corris]

 @kona ‌ What a great feature!  Thanks for sharing your thoughts.  These reminders would be very beneficial.  Many of our teachers use Remind.com for sending messages to students.  It is nice that this is an automated feature!

[snugent]

Thanks for sharing a screenshot. That is helpful. I had a student submit a ticket because of one of these texts. Unfortunately, the student received it for a course that allowed late submissions and the teacher purposefully does not enter grades in the gradebook. ARG!!

[laurakgibbs]

That's another good example of why we need some customization and control, snugent‌! I hope some customization and control features will be coming along soon now that they have the basic texting functionality in place.

[andreasampson]

I had just forwarded one of these texts that a student received to our university's Canvas engineer yesterday. I do NOT like this feature as it is. First of all, the student assumed that this text was something that I intentionally and personally sent, which was not the case. I have a strict policy on assignment deadlines. These texts imply that the student will earn a grade above 0 if they turn in work late, and my students will not (without a valid excuse and advance notification). "Any score is better than a 0", except when it's still a 0. I don't want my students diverting time and attention into completing a past assignment that they aren't going to get credit for, rather than working on current assignments where they can still earn a grade. I've seen that snowball on students where they never get caught up.

That said, this feature could be useful to me if it were customizable. I would love to be able to change that text to "You've missed the deadline for XYZ assignment. Please pay attention to your course deadlines to maximize your grade in the future." Or, even if we could simply turn those text messages off in each course so that the student won't be mislead into thinking that they can turn late work in for credit. 

[James]

I will add some details about this. The student who got the text was from one of the groups who was working on phone communication from the college, including phone calls and text messages and I had let them know about this new feature so they could incorporate it into their presentations if they desired. She was probably more aware of what this text message was than other students in the class might be.

Students have to go in and setup text messaging as an communications channel. How do I add a text (SMS) contact method in Canvas as a student?  When the project started in March, I checked Canvas Data and found that just 3 out of 42 students in the course had set Canvas up to receive text messages. This particular student was one of them, having set it up back in January 2014. On a positive note, 4 more students have signed up for SMS messages since I told the class that Canvas would be adding the Student Performance Text capability, so it seems that students really do want this information.

Despite being used to receiving texts from Canvas, this one must have really thrown her for a loop because

• this message is supposed to be delivered after the due date for unsubmitted assignments [release notes]
• the assignment was due at 11:59 pm CDT on May 7 [assignment settings]
• the text message was sent at 7:55 pm CDT on May 7 [image of text message]
• the assignment had been turned in by one of her group partners at 4:25 pm CDT on May 7 [Speedgrader]
• when she clicked on the link, it took her to the group's page (I inferred she meant it did not take her to the assignment page) [student's email to me at 7:59 pm CDT]

I filed a trouble ticket for her with Canvas support.

The first response was this indeed was the new feature, then provided the link the release notes, and said that if we have any questions about the new feature that we should contact our CSM as the release notes explained. I knew all of that and told Canvas Support that when I responded and that I didn't have questions, I was reporting a bug (actual behavior didn't match described behavior).

The next time Canvas Support responded, it was a different person and they could see the conversation between the students but "we don't actually have record of the message being sent from Canvas". They asked if I could send an image of the text. I had already sent it once, but sometimes things get lost in translation, so I attached it again and included a link to it. That was late yesterday afternoon so it's too early to expect more information yet.

The first thought I had when Canvas said they didn't have a record of it was that perhaps someone was spamming students pretending to be Canvas. The fact that Google just went through a similar issue with their Google Docs earlier in the week may have played into that fear. I imagine that the phone number can be spoofed, but the text message came from a phone number that was listed in the Salt Lake City area, and the link was to the current assignment in my course, so I'm assuming it was a legit text message from Canvas.

The thing I first thought of when the message came in is that the assignment is overdue IF you ignore the time and the timezone part of the due date. Central Daylight Time (CDT) is 5 hours behind UTC, so in UTC, the assignment was due at 4:59 am on May 8. The message was sent at 7:55 pm CDT, which would be 12:55 am on May 8 in UTC. So, it was technically May 8 and past due, but only if you ignore a bunch of stuff that shouldn't be ignored. And that still doesn't address the issue that it was a group assignment that had already been submitted.

[kona]

andreasampson‌, if the assignment locks at the due date (so if the until date is the same as the due date) then student's won't get this notification. I think the assumption by Canvas is that if you don't have the until date set then students can still technically submit their assignment after the due date - whether or not the Instructor will give a grade above a zero for an assignment that is turned in late is a completely different story...

[James]

andreasampson, did you specify the "Available Until" due date on the assignment? I normally do, but this was one assignment that I needed for them to proceed to the next step -- even if it was late, so I didn't. Part of the criteria for sending the message is that they can turn in the assignment after the due date because the "Available until" wasn't specified.

[andreasampson]

No, and the reason for this is that I have an FBI agent in this particular class that has to turn work in late due to a work assignment he was on. I can't lock him out of it. If we could allow exceptions on a student to student basis, that would fix the problem, but I'm not aware of any way to lock the assignment but leave it open for one particular student. If there is a way, please let me know!

[Stef_retired]

andreasampson, I will second and third what  @kona ‌ and  @James ‌ have said about the "available until" date. For assignments on which I would not accept late work, I set the due and available until dates and times so that they were identical. This will solve the problem of students receiving these "Any score is better than a 0" messages after the fact.

For those few assignments where I would accept late submissions (albeit with a late penalty), I set the available until date and time to be X days after the due date, with the specifics of the late penalty spelled out in the assignment instructions and in the course syllabus. In these instances, I would be happy for my students to receive that reminder message, because "Any score is better than a 0" would apply. In fact, I use "Message students who" the day after the full-credit due date to say essentially the same thing as the text does as well as reiterate the policy for late submissions.

That said, being able to customize those messages would be super cool.

[Stef_retired]

Good news: You can absolutely give a single student a different due date on an assignment, andreasampson. Check out https://community.canvaslms.com/docs/DOC-9973 

[snugent]

You can do that with differentiated assignments. 

https://community.canvaslms.com/docs/DOC-10036 

[andreasampson]

AH, excellent! Thank you!

[kona]

andreasampson‌, actually you can set the due date differently for just your FBI student (how cool is that!!). Check out the instructions on this guide - https://community.canvaslms.com/docs/DOC-9973 

[hilary]

Thanks so much, all, for the great conversation!  As  @kona ‌ noted, this functionality is something we're trying out and we have two goals:  first to send relevant, timely, useful information to students, and, second, to provide that info with as little heavy lifting from instructors as possible.  Ultimately, we'd like Canvas to do this so well that it feels effortless for faculty.  

As you can imagine, it'll be a process.  We'll need to experiment with the timing, the text of the message, and also learn a lot about the students individually.  Does this particular message make this student more or less likely to take a helpful action?  What time of day should the message appear to maximize the likelihood this student will respond?  

I'm really excited about the data we have coming in and how we can use what we know about students, behavioral economics, inferential statistics, and machine learning to individualize for every student.  However, we have zero tolerance for sending bad information to students so we're proceeding very carefully.  It sounds like we have a bug or two, so we'll go on the hunt and may stop this machine while we work on it.  

To answer your questions above: 

 @kona ‌, the "hey, you have a due date approaching" is one of the next messages in the queue to build.  Also, the next update will send via Notifications in Canvas.  Admins will be able to see them with all the other Notifications that are sent to users.  It does sound like there's a bug with group assignments and we'll likely exclude those while we work on how the heuristics for group assignments need to change.

laurakgibbs, instructors aren't able to tweak the message yet, but that's something we're absolutely planning for a future update.  Also, you make a great point about extra credit assignments--are you doing that by making the assignment worth zero points?  If we can get an understanding about how faculty manage extra credit, we can make the machine smart enough to tell the difference.

snugent‌, one of the triggers for the assignment message is that there's not a student submission when the due date elapses.  It shouldn't matter if the instructor has graded it or not, which makes me wonder if there's a timing bug in there somewhere.  Did you make a support ticket, by any chance?

andreasampson‌, the next version of this will allow it to be enabled (or not) at the course level.  Instructors will need to opt-in so no one will be surprised by messages that contradict their course policies.  I love your suggestion for changing the message and hope you won't mind if I steal parts of it

 @James ‌, thank you for all that great detail!  It does look like we have something unfortunate going on with time zones and group assignments.  You may notice fewer of these messages going out while we figure out where our conditions need to be tweaked.  The phone number the text came from is actually Canvas--as I mentioned above, the very next iteration puts those messages into Notifications so they'll look like all the other communication from Canvas and everyone will be able to see the history.  (If you're coming to InstCon, you'll hear lots more detail there!)

Please let me know if I've missed your question or if you have any new ones.

Thanks!

[dhulsey]

Hilary, how does Canvas know I am going to accept the late submission just because I haven't locked down the due date? 

 Even if I lock down most things with available until dates, I cannot lock external tool assignments (Turnitin, Softchalk, perusall) because, if I do, it locks students out of any content or feedback in those external tools. Does that mean Canvas will send out alerts to students telling them they can still submit these assignments? What if my syllabus says they can't submit these assignments late?  What if a student shows up in the Dean's office with a Canvas alert that says she can still submit the assignment (in contradiction to my syllabus)? Isn't it possible that these alerts could directly conflict with an instructor's grading policy?

[snugent]

Deactivated user‌, The assignments in the course are TurnItIn assignments which are LTI (external tool) so the student isn't really submitting in Canvas. Additionally, the instructor is purposefully not entering grades in the gradebook. This has been an ongoing issue for us but instructors can choose what they want do in their courses but this practice has been very confusing to this instructor's students long before this service was enabled. So I am assuming the student got the message because of the missing grade and no set Until date on the assignments. I have trained our faculty NOT to use an Until date with TurnItIn assignments since it removes the TurnItIn assignment dashboard from students' view once the date passes. So I guess my question is how will this notification service handle LTI assignments? 

[snugent]

We must of been on the same wave length! LOL!

[kona]

Thank you for this information and for taking the time to update us!

[hilary]

Sorry folks, I'm slow today!  Copying my response to Dallas from the release notes:

Great questions, Dallas E Hulsey.  We're looking specifically for assignments with the "online" submission type, which should exclude any LTI assignments.  The online ones are the only case where we can tell for sure if a submission has happened or not.  The heuristics look at not only the submission type, but are looking for a status in Canvas Data of "unsubmitted."  It filters for due dates in the last 24 hours with "lock at" dates at least 24 hours away (+ some number of hours to allow for the delay between when we're scraping the database and when the message gets sent.)  We're also filtering so a particular assignment only ever triggers one notification to students (in case a due date gets moved.)  There are some rules around group assignments, also, which may have a bug in them as observed in a discussing happening here.

Your scenario where a student complains about differing messages between a syllabus and one of these messages is well put, however.  I'm going to get this functionality paused so we can tweak the text and will let you know what we're thinking for a new message and when it'll be un-paused.

Please let me know if you have further questions.

Thanks!

[hilary]

Thanks, snugent‌, that's super helpful.  Let me dig around and see what I can discover about how we handle that with TII.  In the meantime, we paused the engine.  I'll let you know when the issues are resolved and we've started it up again.

[lindalee]

Just adding to the chorus of voices ( @kona ,  @sanders3 , snugent) supporting differentiated assignments as a solution here! This is one of my favorite features to introduce to faculty, for exactly the sorts of needs you outline, andreasampson‌.

You could put your FBI student in their own "assign to" area with the same due date as everyone else but no available until date -- that would allow just the one student to submit late but no one else. I really appreciate the flexibility provided by differentiated assign to areas and due dates/availability dates.

[James]

Adding more information ...

I talked to the student today in class. She didn't know what I was talking about with groups. I showed her the message she sent me where she talked about her group project page and she didn't have an answer.

She also showed me more texts that she had received from Canvas. These were from a quiz that she had taken but not scored 95% on. They were locked though, so she couldn't submit after the due date. These texts had the same "Anything is better than a 0" message, which is weird for someone who already has a grade. I don't know if it ignored the lock date or if the message is the same for both kinds, but it's kind of confusing.

I got a response back from Canvas Support once again pointing me to the announcement about these as an indication that it's a newer feature and "may need direct review by the team(s) overseeing it".

[hilary]

Thank you,  @James ‌.  That certainly sounds like it's not working as intended--we're reviewing right now.  These messages have been paused and we'll let you know when they'll be back.

Thanks so much for your patience

[dhulsey]

Yes! The only thing is that I don't know why I didn't snap to this issue when the texts were first announced. I thought it sounded awesome at first! I am sure Hilary will sort it out and make something awesome for us to use to help students.   

[laurakgibbs]

Thanks for all the information, Deactivated user‌! 

Extra credit: no, not zero points; instead, the extra credit is a way for my students to EITHER make up work they missed in the past week OR to bank some extra credit if they know they might miss some upcoming assignments. So, to Canvas, they look just like regular assignments. If I could flag them as "optional" or "extra" or something like that, I would certainly do so. Otherwise, there is no way for Canvas to know that they really are different from the other assignments since they go into the points total just like anything else.

Here's how a typical week works for me; I use a points-based system in Canvas so all points are created equal, but the students know which work is expected and which is optional/make-up based on how I present that to them:

Online Course Wiki / Week12 

It would be counterproductive and even anxiety-producing to send students reminders that urge them to rush and do assignments that they are actually NOT expected to do. And I would prefer to avoid anything that produces unnecessary anxiety. School is anxiety-producing enough as it is.

And that "any score is better than zero" part of the message is also completely inappropriate for my classes. It implies that there is partial credit and/or a late penalty, neither of which is true for my classes. So, it introduces a confusion that I would really prefer to avoid.

I admire your faith in building individual student analytics, but I would also guess that instructors have a lot to offer in terms of configuring communications. I really don't want communications to be automatic; instead, I want them to be effective and appropriate, and for that, I need more control. If people want to go with defaults, using canned texts and automatic algorithms, sure, that's great; if it works for them that way, then go for it! For me, though, especially since I teach fully online, communications with my students are really important, and I need to be able to shape the messages in my own way. I believe in communications that use data, yes, but not data-driven in the sense that I have no ability to add a layer that reflects my own understanding of the context. Adding in that layer of instructor control doesn't mean every instructor will use that layer, but I would! 🙂

[laurakgibbs]

Aha, so this is not going to happen for quizzes, is that right? If so, that means it actually doesn't apply to me at all, since the only thing my students do at Canvas is log on and take a "quiz" (not really a quiz; just a single true-false question) which is their declaration of work completed just to get the points in the Gradebook. If I understand your comment here about assignment types, quizzes would not be even part of the texting strategy right now, is that correct...?

[hilary]

laurakgibbs‌, it is supposed to happen for quizzes, too, but the quiz text should only happen when:

• it's a quiz with multiple attempts
• and it only has questions that are auto-graded
• the student has completed at least one attempt
• they scored less than an "A"
• and it's at least 24 hours before the "lock at" date

There may be a wire crossed somewhere, but we're reexamining all the heuristics right now.  When it's re-enabled, that's how it will work.  

[laurakgibbs]

Aha, I would be out of the loop: my quizzes are set up for multiple attempts, but students don't need multiple attempts (they always get 100% because it is just one T-F question about the work they completed). 

I was thinking at first they would get a reminder about flat-out not having done the quiz, but if this is just about boosting scores on multiple attempts, then I think my class is not in the algorithm target. 🙂

I really appreciate the chance to ask questions and get answers like this: thank you!!!

[hilary]

Thanks so much, laurakgibbs‌!  That's great feedback.  You're right--from your description, I can't see a simple way for Canvas to figure out which assignments are optional.  The next functional update will make it so a course opts-in, but we're planning more instructor controls after that.  We'd really like to have a loose framework where faculty are able to input not only the text of the message but also make decisions about timing and score cut-offs, etc.  

Selfishly, the data we get from teachers using their good teacher judgement to modify these messages will let us make them much smarter much faster (;  It will also prevent us from overriding that good teacher judgement when the message just doesn't work in your context.  

Thanks again for talking me through how your course operates.  Everyone's feedback is going to make the final product so amazing I get goose bumps when I think about it!  

Oops--this was supposed to comment on your description at the end!  Sorry about that.

[thompsli]

I would like to add that it's really, really important for us to be able to set our own thresholds for things like the 95% re-take nag. I use short 4-5 question auto-graded quizzes so students can do a few math problems (one per type of problem covered in the lesson) get feedback on whether or not they are correct, and then decide if they need to do more "homework" problems or not, and the target I usually set (for the "green check" in Module requirements) is in the points range of "no more than one incorrect" rather than "completely correct".

These quizzes are 10% of their final grade with the lowest 3 dropped, so I really don't want them hammering on each quiz until the computer says they're perfect. Instead, I want them to take the quizzes until they feel they understand the topics assessed and can successfully answer those kinds of questions going forward. (They get 3 tries by default, and can message me to get another try after getting help on the problems if they need to - the default 3 is just to slow them down and keep them focused on working on the math rather than guessing their way through the homework until they get lucky.) If they only miss one problem, it's often more productive for them to figure out why their answer is wrong and re-do the problem on their own rather than re-do the entire quiz to chase something that has little impact on their final grade, and I don't want them to feel pressured to be perfect all the time.

I'm trying to get them to see progress in my class in terms of learning math rather than scoring lots of points and "getting done", and I want to be careful about what message they get and how they help build that mindset.

(Next year, I may put the online quizzes in a category 0% of their grade and have them instead turn in weekly "Reflection and Evidence" sheets where they summarize how they're doing on each of the learning targets for the week, what tools they used to work on learning them, and how they assessed whether or not they understand them yet for that "homework" 10% - I haven't quite decided if I can make that work or not for my population given how many students join my class in the middle of the term and thus miss a lot of my class culture orientation materials and habit-building assignments.)

[laurakgibbs]

That is a great example of the kind of modifications that teachers should be able to make,  @thompsli ‌! I teach writing (which has a lot of the same issues as teach math, where people sometimes feel like they "can't write"), so I actually don't put grades on anything, and I really don't want students stressing about that. Even if Canvas's intentions are good, I don't want any communications going out with the language of grades and grading that I cannot adjust to fit the approach I'm taking in my classes.

[hilary]

Hi All, 

We've reexamined all the heuristics around these messages and made some tweaks.  We are now filtering out all group assignments and any assignment that has plagiarism detection enabled.  Also, we were sending the assignment text to students when we should have been sending the quiz text--that's been repaired.  

One additional change was made to the wording of the assignment text.  It now reads "You missed the due date for an assignment, but you can still turn it in.  Go there now [link to assgt]."

We're going to turn the reminder engine back on this afternoon.  

Thank you so much for all of your input!

Hilary

[dhulsey]

Hi, Deactivated user‌! How do these changes solve the conflict I mentioned earlier between my syllabus and the notifications Canvas sends out?  

[ProfessorBeyrer]

Thanks  @dhulsey for asking that. In one of my classes I give a courtesy point for late assignments -- it helps me distinguish those who made the effort when I'm deciding how fuzzy the barriers are between letter grades. It would be great if this feature became both optional and editable for instructors.

[James]

When you look at the Student Guide https://community.canvaslms.com/docs/DOC-10589-4212717407 , you see this:

The To Do section [1] shows all assignments with a due date in the next seven days, including ungraded quizzes and assignments that do not require a Canvas submission. Each item in the To Do list displays the assignment name, the course name, the number of points, and the due date for the assignment. Once the due date has passed, items remain in this section for four weeks.

I don't see how this is any different except it's only sent once, within 24 hours, and it doesn't hang around for four weeks.

The text message just reminds students that they can turn it in, which is true, but any policies that the instructor had already put in the syllabus would be in effect. It's up to the instructor, not Canvas, to decide what to do with late work.

The wise thing to do is to put it in your syllabus and then perhaps inform students that Canvas will accept late work, but that doesn't mean that I (as the instructor) do.  Be clear in your syllabus and you tell the students that Canvas may remind them by text message or in the To Do list that they can still turn it in, but that you won't accept it late. Then there should be no confusion, or at least none that reaches the level of grade appeal. You could even tell them how to make sure they get these text notices for their other classes and use that to remind them that it doesn't apply in your course -- kind of like notifications are across the board and not set at a course level (at least not yet) because that's what this is -- a notification.

[dhulsey]

I already do say that in my syllabus. Still, this is more active than a to do list. This is the student actively receiving a message encouraging them to submit a late assignment.

What in the message makes it clear that it is an automated message from Instructure  and not a direct communication from the instructor offering an exception to the policies in the syllabus?

This  is not the same as a to do list that contains past due items.

[hilary]

I'm glad you asked,  @dhulsey ‌, because we had a long internal conversation about that.  Here's the whole message (old on top, newest one last.)  As we test this, I end up getting a lot of them (;

Those of us who have taught before thought it was a good balance between needing to have a strong call to action to motivate students to turn missing work and trying to not imply an instructor would give credit for that work.  What do you think?  

[dhulsey]

Hi, Deactivated user‌. I believe the new message still implies that the instructor is offering to accept the late work. By student reasoning, why would there be a message encouraging the late submission unless the instructor was offering to accept the assignment? Something like "Check the course policies to see if you can submit this late assignment" makes more sense given that Canvas has no way of knowing what the course policies are. Although, yeah, the message loses clarity, but my point is Canvas can't be clear unless it can find a way to be consistent with each instructor's policies.

In the long run, it would work best for these messages to operate on the settings with each assignment group. Add settings to each assignment group about whether late submissions are accepted, for how long, and with what penalty. Use the information from the settings in each assignment group to dictate timing and content of the message:

"You missed an assignment but may still submit within 72 hours for a 10% late penalty." If late submissions are not accepted at all for that category, no text gets sent. Obviously, that coding doesn't happen overnight; I know this is not an easy or quick fix, but it would allow the instructor to customize how these alerts work to be consistent with the course policies. That would make these notifications truly useful.

Looking further forward, build an infrastructure that later allows further settings like the ability to customize messages before the due date. The instructor could flag assignment groups as "major" or "minor." Major groups would send out a preemptive text: "Exam II in American History is due Tuesday night at 11:59 PM MST."  -- or something like that.

Time will tell how students receive and interpret the texts Canvas currently sends, but I am still worried that the texts are sending a message for me that I do not wish to send for all assignment groups in my courses.

[straussi]

Can't agree enough with the need for customization and control, or the ability to opt-out of these features. We have other projects in progress that will overlap with some of these features.