Evaluator Role and Integration with 3rd Party Textbook Site

[mmoore1]

Does Canvas have a built in Evaluator Role that allows access to an integrated 3rd Party Textbook?

I can't seem to find an evaluator role or a way to have an evaluator have access to a third party textbook tool so does this exist is my first question and my second question is can Canvas require 3rd party Textbook companies to allow this access for evaluations.  

A lot of times good teaching is lost, missed, or not evaluated during an evaluation because the evaluator/observer does not have access to the third party tools.  It is like asking an evaluator to look at an outline of the course without seeing what the individual items in the outline do. 

[tdelillo]

Does the Observer role not allow access? In my experience, the observers can access any integrated tools, with the only limitation being that some external tools require a paid account. If the textbook is something the students are paying to use, the observer would not be granted access unless you negotiated access with the publisher. 

[mmoore1]

I am glad you stated that because I am thinking there should be a Canvas requirement for third party vendors that allows for evaluators to easily access those materials.  If Canvas does not require it then it will never happen.  The problem that arises every term is that an evaluator can not observe those third party tools and so to be totally legitimate and transparent in an evaluation process this should be mandated.  

Currently we run into the issue of the third party tool denying an observer access to evaluate. 

[kona]

 @mmoore1 , from my experience facilitating peer reviews the Observer role doesn't work for the 3rd party/textbook resources. The only thing we've been able to do in cases like this is to log in as the Instructor (masquerade) or have the Instructor log in and show us around.

Kona

[mmoore1]

Yes, while this is a great idea, we are finding that the masquerade feature would not be great to give out to our faculty for various contractual reasons. 

[jbuchholz]

We had that same issue here and we created an evaluator account that we were then able to connect to the publisher side of things so that the evaluator would have access to all of the same course materials and experience a student would have access to.  We also created a sub-account where a copy of the courses live so that the evaluators do not have access to any actual student data. The evaluator accounts are given a generic password and upon completion of the review process, the passwords are changed.  This keeps things as secure on both ends. 

I am happy to share our process in greater detail if you or anyone else thinks that would help.
Jesse

[kona]

 @jbuchholz , yes! I'd appreciate hearing more about your process!

[jbuchholz]

 @kona ‌,

Our online program adopted Quality Matters (QM) this year and before the first course went through review I looked at how we could bring a reviewer into Canvas without giving them access to things they didn't need, but also creating an environment where they had access to all the things a student would have access to.  After looking at the different roles that we have in Canvas I concluded that there needed to be something different for these courses. This is what I came up with.

#1. I created a new sub-account where a version of these courses live that allows the reviewer to interact with the content in the same way that a student would have in a regular course. I named this sub-account "Course Reviews" and placed it under our Manually Created course sub-account. This allowed us to basically create master shells for the courses that were going through review and it allowed for the course to live "outside" of the other sub-accounts, courses, and students. 

#2. I created three reviewer accounts that had a generic login and password. Each password is specific to an account and it logged in a Google Doc that my team has access to. A calendar event was created to revisit the accounts and change the password at the end of each review cycle.

#3. The reviewers use a non-authenticated URL to gain access to Canvas. This allows us to bypass the AD tree that everyone else needs to use.  We could create the accounts through AD and we may in the future if we feel this is becoming a security risk.  Right now those accounts only have access to what we put them in so I feel that it is a low-risk situation.

#4. Each reviewer account is added manually to the course that needs to be reviewed. I then change the role of the reviewer to a student and the professor links all of the textbook content much like they would if it was a "real" course.

#5. We work with each publisher and give them the reviewer account information and explain that this account needs access for review purposes only. The publishers then grant access on their end and we make sure the connections work before the actual reviewer logs into the course. 

#6. Once the review is complete the reviewer accounts are removed from the course. This ensures that even if the reviewer tried to access our LMS they wouldn't have access to anything.

So far this is working for us. It is not the best solution, but it is a solid workaround for the time being. I could simplify things by adding the reviewer accounts to the course through a CSV, but for me, it is easier and faster to add them manually and then change their role, accept the course invitation, and make sure everything works.

Jesse

[kona]

This is great!! Thank you so much for sharing! It will definitely help us improve upon our online peer reviews! 🙂

[mmoore1]

Yes, these are great ideas.  I guess that I have always viewed a difference between the course review process and the evaluation process (teaching an online class).  But thanks for the ideas.

[mmoore1]

We have an evaluator account too, but it does not allow the evaluator to evaluate the third party tools.  Consequently, evaluations which happen frequently are limited.  It would be nice if this swimmingly integrated into third party tools and third party tools were required to allow that connection.  If you evaluate a course without students in it, you are saying that only the content is valuable and ignoring the humanization of a course.  We strive to evaluate the actual total teaching on our campus. 

[jbuchholz]

Matthew,

Are your evaluators looking at content or student experience?

[mmoore1]

Great Question!  Both. 

[jbuchholz]

Matthew,

I think I am tracking with you. Usually, we have an evaluator look at one aspect at a time. This allows for better clarity and also helps us keep evaluators in check. FERPA is always a big concern and we try to make sure that we don't do anything to even skirt that line.  

Essentially, that is what prompted us to create evaluator accounts and have deep discussions about privacy and necessary information.  I am happy to talk more with you if you want.

Jesse

[mmoore1]

Thanks for the ideas.