Improvements to Studio Insights Needed

[ghalsell]

Having used Arc (now Studio) pretty extensively last year, I was hoping that with the new name and icon, we might see improvements in the functionality of the Insights area. Alas, such is not the case. Here are my complaints (and also my suggestions for improvement):

1. Students appear in the Insights area apparently in the order that they first accessed the first available video in that course. There is no provision to reorder them, sort them by groups, etc., and judging from others' comments, if they drop the course, they still appear in Insights.

2. In my experience, the viewing data collected is frequently inaccurate. I've had students report that they watched a video in its entirety despite Insights reporting they watched either only a portion or none at all. In some cases, I'm sure I was being lied to, but in others, students whom I trust implicitly also claimed to have watched the videos, and looking at their Canvas usage during the time period in question seems to have backed them up.

3. There is no easy way to grade students on having watched a given video. I already have more quizzes than I like in my class, and I don't want to embed another one for each video I post. So, my solution has bee to grade students on how many minutes of a video they watched divided by the total time, but that approach is problematic given the inaccuracy of the recorded data, which seems to be affected by what browser was used, whether the video was viewed on a mobile device, etc. There is also no provision for the viewing data to flow to the Canvas Gradebook - I have to open the course gradebook in one window and Insights in the other, then manually enter a score for each student, after having to hunt through the disordered names to find the data I need.

4. If a student goes back and watches a video after I've scanned the data and entered a grade (which I allow), there is no means for me to be automatically notified that is has happened. The only way I know is if a student lets me know, or else I go back and periodically review all the data for each video, which again is a very laborious process, given the random order of the students' names and the inability to tell by looking at their icons that there is new viewing data available.

Studio would be a lot more useful if the problems I described are fixed. Right now, if I could avoid using Studio at all, I would, but I haven't found any other viable option to deliver my self-produced videos.

As a footnote, I've discussed these problems with Canvas techs in the past, but so far, nothing seems to have changed. This shouldn't be a votable item, but rather a necessary bug fix.

[Jeff_F]

Yes, viewer drop rate is always a concern.  The quiz option is a useful approach but this is just one strategy.

Perhaps there could be an assignment type 'View Video' which shows complete/ incomplete based upon a set percentage of the video played and viewed by the student (e.g. 80%).   Then again, in that scenario a student could press play for a video and subsequently step away to do something else.

[ghalsell]

Hi Jeff,

Thanks for your ideas. Actually, I just spoke to a Canvas tech about this yesterday, who confirmed that if students access the videos through a browser on a mobile device instead of using Canvas Student, then Studio won't record their viewing. Other than that, there don't seem to any confirmed situations where Studio fails to record a viewing. However, I have several suspicions that could stand to be checked out, including whether recording is affected by using an iPhone or iPad, and whether switching between Wi-Fi and cellular service or losing and regaining a cellular connection during a viewing affects Studio's recording.

I don't have the time to confirm any of these scenarios, nor should it be my job. I just know that I get a number of complaints every semester from students who tell me they watched a particular video but received a zero because Insights didn't record their viewing. And yes, I know some students might have been untruthful, but others are ones I've trusted to be honest, and in some cases, their overall viewing habits would tend to support their assertions, such as Insight recording their viewing of the third installment of a series of videos, but not the first two. I've also had students watch the same video multiple times before Insights recorded their viewing - not the behaviour one would expect if the only interest was in getting an A for that assignment.

I agree that there's always the possibility that a student can turn on a video, then go do something else while it runs. I've even mentioned this to my classes, with a caution that the material on the videos will of course be on the test, so it's in their best interest to pay attention to them. Your idea of grading them on a complete/incomplete basis based on their watching 80% will help those few students who have one or more small gaps (likely caused by stopping and restarting a video mid-viewing), but for most of my students who seem to be affected, Insights records only a small amount of viewing time or none at all.

Thank you again. I just hope Instructure has major improvements to Studio in the works. The Insights issue is only one of a number of problems with this add-on that I'd dearly love to see fixed.

[Jeff_F]

Ironically, I was just now fielding an insights question as a direct link is not available under the embedded video for instructors if the in-video commenting feature is disabled.   

You insights add tremendous value and if there are any 'gotchas' such as with the mobile viewers I believe the info should be summarized and added as notes to the page - 

How do I view user insights and analytics for Canvas Studio media in a course as an instructor? 

But I think it is odd that the mobile browser is the pain point and not the app.  I would have assumed it to be the other way around.