Overview
The University of Minnesota's Academic Technology Support Services (ATSS) unit partnered with Instructure to conduct a usability study on some of Instructure’s most recent updates to the Canvas Discussion tool. The study aimed to understand user experiences to implement changes and/or improve training.
Usability partnership & study process
Since 2015, the University of Minnesota academic technology teams have conducted usability studies to understand how instructors and students use different learning technologies. This work is carried out with internal and external stakeholders such as University of Minnesota academic technology leadership, the academic technology community, and our learning technology vendors.
For this study, the Canvas Discussions tool was selected for two reasons: Instructure was redesigning the tool, and it’s used in approximately 25% of University of Minnesota academic courses as shown in the graph below. Discussions are heavily used in those courses: as often as 14-15 times during a 15-week semester-long course.
In conjunction with the University of Minnesota Usability Services team, our project team developed a focus question to guide our study: What prevents people from engaging with one another using the new Discussion tool in a satisfying way? We then determined the criteria for participant recruitment and tasks that participants would complete during the usability testing sessions. While the participants were engaged in the usability testing sessions, project team members took notes that captured observations about what the participants did and said, specifically noting any points of confusion. After completing all usability testing sessions, the project team met to debrief and review issues that the observers captured in their notes.
One of the distinguishing features of this study was the involvement of the Canvas Discussion Product Manager, who was a part of the project team and provided input on the scenarios and recommendations.
Participant recruitment
After the focus question was developed, our project team created the recruitment parameters:
- Instructors from across the University of Minnesota system who had created or graded a Canvas discussion were invited to participate in the usability evaluation sessions.
- Students from across the University of Minnesota system who had participated in a Canvas discussion were invited to participate in the usability evaluation sessions.
Participant scenarios
During usability evaluations, participants complete a series of tasks created by the project team. For the student participants, their scenarios encompassed the following tasks:
- Complete a discussion post in response to an instructor prompt.
- Complete a reply to another student’s post and tag a student in it.
- Find a reply that an instructor or student made to your post and respond to it.
- Find your grade for a discussion.
- Find feedback from your instructor on your discussion.
For the instructor participants, their scenarios encompassed the following tasks:
- Create a discussion with multiple due dates.
- Create a discussion that requires students to post before they can read others’ posts.
- Find the post(s) from specific students.
- Review student posts to grade them.
- Create a small group discussion.
- Allow students to create discussion topics.
Key Findings
Finding #1: Mentions Feature
One new feature we tested with students and instructors was the “Mentions” feature. The way this feature works is that when a user types the @ symbol in the Rich Content Editor (RCE) of a discussion post, a drop-down menu with all students in the class appears for the user to select the name of the person they want to mention or “tag” in a post. The saved post will have the mentioned user’s name highlighted within the discussion as an additional visual cue that the person was specifically mentioned in the post, as illustrated in the screenshot below. If the user has notifications turned on for Discussions - New Mention, they are notified that they were mentioned in the post.
In our scenarios for instructors and students, we asked them to try this feature. Instructors were not completely sure what this functionality did. Those more familiar with social media recognized that it seemed like a tool that would call out a specific person in the discussion. Still, they were unsure how the person mentioned would be notified. The value of when this feature would be useful also seemed to be lost on instructors. They wondered how calling someone out specifically in the discussion may affect the overall use of the Discussion tool. Students seemed to immediately recognize this feature from social media apps and assumed this feature would work similarly to how it does in those platforms. They also wondered how or where they would get notified if someone mentioned them and vice versa.
Recommendation
After observing both students and instructors use this new feature, we decided that we want to target some communications to both user groups. We want to make sure that both user groups are clear that this does indeed work similarly to how other social media platforms use the tool but that the individual user must have the specific “Discussions - New Mention” notification turned on to receive any notifications. Instructure has already made a knowledge article about this feature, detailing how it works and adding a link to the article that instructs users how to turn on this notification, so we will be using that knowledge article in our communications.
Finding #2: Anonymous Discussions
The Discussion redesign also comes with the ability to create an anonymous discussion. There are three new settings that users can choose, as shown in the screenshot below:
- Turning off this feature (which is the default setting),
- Selecting partial anonymity (where students can choose to reveal their name and profile picture or interact anonymously in the discussion),
- Creating a completely anonymous discussion where student names and profile pictures will be hidden.
It is important to note that a discussion set for partial or total anonymity cannot be graded or set as a group discussion.
Recommendation
Other similar feature options in Assignments and Quizzes are still editable after saving the initial activity. Features will lock once a learner has engaged in the activity. Still, as an instructor first sets up the activity, the options stay available past the initial saving of the activity.
The product manager of the Discussion tool saw the inconsistency of those options being unavailable to edit after the instructor first made the discussion. She returned to her team with this issue, and they changed the settings to be consistent with how options work with Assignments and Quizzes. All options are now available for editing after the instructor has saved the activity as long as no learners have participated.
Our team will introduce the new feature options for making a discussion anonymous and note that this type of discussion is not available for graded or group discussions. We will use any knowledge articles that Instructure makes about this feature option to communicate it to instructors further.
Finding #3: SpeedGrader Confusion
One of the instructor scenarios was to grade and give feedback on a student’s post. The instructors were able to find Speedgrader fairly easily. However, when an instructor views a discussion post in Speedgrader, there is the option to “View the full discussion” as noted by the arrow in the screenshot below.
In the usability sessions, instructors would click on the “view the full discussion” link to see the student’s response within the context of the whole discussion. The problem arose when the full discussion was open within Speedgrader, as shown in the screenshot below.
Suddenly, an instructor can post a reply to the discussion right next to the “Add a comment” box within Speedgrader. Instructors unknowingly sent personal feedback to students via a Posted Reply instead of in the comment box within Speedgrader.
Recommendation
Our Instructure partner saw how this could confuse instructors, and she sent this feedback to the team who works on Speedgrader.
In the meantime, our team planned extra communications to help instructors more clearly know the difference between replying within a discussion vs. giving feedback via Speedgrader. We also added information to the Canvas Essentials course for new instructors and TAs about grading discussions, specifically highlighting this situation. We use the screenshots above to further communicate the importance of putting student feedback into the Speedgrader comment box and keeping student feedback out of the Discussions thread in the center of the page.
Finding #4: Group Discussions Feedback
The final finding from our UX study was more of a “bonus find” in that we weren’t specifically testing this area of Canvas but received feedback when we asked instructors to set up a group discussion.
Instructors talked about how the Groups feature is generally dissatisfactory. Here are two quotes from participants that we heard:
- Participant #2: "I've had trouble with setting up groups...peer review in groups was a nightmare"
- Participant #3: creating group discussion forums is “slow and ponderous.”
Recommendation
Again, we were not specifically testing this feature, but the feedback was important to document. Our Instructure partner took the feedback back to her colleagues. As of August 2024, Instructure has started an exploration of new group options.
Our team recognizes that the Groups feature is not easy for instructors and students to use. We already have a self-help guide for extra support on how to use the tool, and we will communicate to the UMN community as additional changes are made from Instructure.
Conclusions
We plan to continue to partner with Instructure on future usability studies on Canvas features. Having someone from Instructure as a part of the study team allowed us to gain further traction and improve the user experience with the tool. Although we don’t expect any vendor to fix every issue found in our UX studies, it was positive to see that some of the findings elicited changes in the product design.
Resources
2024 InstructureCon presentation slides
Contributors
This post is cross-listed on the University of Minnesota Extra Points blog.
