[ARCHIVED] How are you measuring 'success' of Canvas at your institution?

[glparker]

I was in a meeting last week demo-ing Canvas to some folks from SOCOM, and they asked a question I feel I should be able to answer, but was not. 

Is Canvas successful at my institution?   More to the point, by what metrics are we measuring Canvas?  

I could craft a few metrics to measure by:

- # active courses in Canvas

- tool usage per course

- average # student interactions / course

But I'm not sure those answer the question "Is it a success".   

Do other institutions have a similar metric?   What are you measuring that lets you go to your senior management and say "Canvas is a success" (or not, whatever the data shows)

Thanks< Glen

[kona]

 @glparker ​, I determine it based on our adoption rate (how many faculty and students are using Canvas; it's not required at our Institution except for online & hybrid courses) and survey feedback from faculty and students.

We survey faculty & students immediately after they finish training - asking if they feel like the training prepared them to use Canvas, asking if they feel like they can successfully use Canvas for their courses, etc. We also survey faculty & students at least once a year (more when we were first implementing Canvas) asking them to provide feedback about their experience with Canvas. This comes from a couple different questions and even one where we ask them to rate their use of specific features.

For me this has been great information because it shows how happy (and yes, they are happy) faculty and students are with Canvas and also shows to some extend how much use they are getting out of the different Canvas features. We can track this over time to see how things are going and this information also allows us to determine where we need to focus our extra training and informational emails.

Hope this helps!

[shane_ohara]

 @glparker ​, I am just beginning this task as we recently rolled out our pilot of Canvas. I am trying to determine similar metrics such as tool usage, student tool use, discussion board usage, time spent, etc. Have you come up with a method to obtain this data from the system yet?

Thanks!

Shane

[nprueda]

Glen, It's nice to hear from someone that is going through the same as my institution. We are finishing our first month using canvas, and we are starting having some feedback from parents. We are a k-12 school and it's been an amazing experience. I think that having representative groups from each actor of the sistem that can tell you their experiences, troubles or success experience could be great for you. It would be gret if we can work together to have a rubric that help us measure.

Thanks.

[canvas_admin]

Glen,

We are measuring success on our distributed campus model by whether an instructor is using Canvas for something beyond "Dropbox."  In other words, does an instructor use his or her Canvas course for more than simply providing documents (syllabus, handouts, etc.) to the students.  It could be anything from announcements to module usage for a weekly schedule. It doesn't have to be anything fancy but we want them to be using Canvas beyond something that could be accomplished with a Dropbox-like system. 

This means we take the following into account, the published/unpublished state of the course and the presence of something like an assignment, quiz, discussion, announcement, etc.  I also go through the modules to see if they were used.

[brendaa]

Since we are in the first semester of adoption, we are currently looking at adoption rate by faculty for success.  Faculty still have a choice of Blackboard vs Canvas until Summer 2016.  For now, the most important stat is how many people are trained and how many are publishing courses.  We are tracking in person attendance at workshops and participation in our online Canvas Training Center to track training.  So far 54% of our faculty are using Canvas to some extent--at least publishing their course.

As we move forward I agree with some of the previous posters that tracking use of LMS tools such as Assignments, Quizzes, Discussions, etc. will be more important.  We know from trainings and discussions with faculty that more of them are trying out assignment submissions and grading online.  I'm hoping we can verify that through some of the statistics available from Canvas.

[gorsi]

I wish we were measuring success in more detail than we are- we rely on the metrics that come with Canvas but there are some aspects of those that just can't / don't answer more subtle or sophisticated questions... e.g., is a teacher just doing the minimum with Canvas or doing more?

[kmeeusen]

We have been with Canvas for three years now, and have used some of the metrics described above: faculty adoptions, numbers and percentage of courses in Canvas, student and faculty satisfaction, etc. However, our most important metrics are student course retention and successful completion stats and their comparison with our traditional courses. Because our online courses are coded in our student information system (03= fully online, 08=hybrid, 09=web-enhanced), we are able to get this data from our Institutional Research Department. For the record, our online courses are significantly outperforming our traditional courses (by more than 7% for the hybrid category).

We also measure Canvas success through non-traditional uses - clubs, administrative groups, departments, professional development etc.; and this use of Canvas is growing at a rapid rate on our campus.

[rrichards]

***UPDATE***

I developed a framework for defining and measuring success with Canvas, and a tool for quickly tracking Canvas component usage. See attachments.

I'd welcome any feedback you might have.

Best,

-Ryan

***ORIGINAL MESSAGE***

Hi Glen - I'm wrestling with this question as well.

Our ultimate goal with Canvas is a richer student learning environment and a more empowering instructor environment.

How might we measure if that has been achieved?

• Survey students and instructors about satisfaction (qualitative)
• Pull analytics data on how much each Canvas component is used (ie Files page views per student per course)
• Survey/Assess instructors for know how and motivation to use Canvas

That is my best guess for now.

[kbickell]

Ryan, very interesting report! Where are you pulling the student & course data from? We're just starting to jump into creating our own success measurement of Canvas and would love to create a report like yours.Thanks.

[QC99_tsilvius]

This sounds like it would make an excellent panel discussion. Colleges or K12 would have similar concerns I'm sure.