[ARCHIVED] Course evaluations

[spaul]

Good afternoon everyone:  ​Lorain County Community College will begin using Canvas for the summer semester beginning May 26.  I am in need of your collective expertise.  What are you mechanism are you all using for course evaluations?  We have been able to do so in ANGEL And are brainstorming for future ideas.    Thank you for your time.  Susan

[awilliams]

We have been using a third-party solution from http://evaluationkit.com/. It integrates quite well with Canvas and provides a "turn-key" solution for course evaluations. I feel this is a good choice for community colleges or other smaller institutions that lack the in house resources that would be required to code something entirely custom. However, Canvas does offer customized solutions through their "ProServ" department. Contact your CSM (Customer Success Manager) for more information about the ProServ options.

[Cindy_Masek]

I will chime in with Adam.  As a small college with little resources to devout to customization, we went to EvaluationKit this last spring and it has been awesome.  Because it integrates so well inside Canvas and it can create a pop-up box that students get when they enter Canvas to remind them to do their evaluations, our completion rates have vastly improved.  The time spent to create the evaluations semester to semester is much, much less than our previous product. It uses the enrollment in the actual courses in Canvas for the evaluation enrollment so the evaluation enrollment is always correct - whew!  We've used another system before, SurveyMonkey and we did evals within ANGEL prior to this through separate Evaluation Courses that duplicated the live courses.  This is by far the best option I could have asked for.

[kona]

Deactivated user, good to see you in the Community! I know I already responded to you via email, but I'm going to include my response here as well.

For all course evaluations (f2f, hybrid, and online) we use a homegrown system - which prevents students from duplicating their feedback, automatically creates the evaluation for each course, and automatically releases it to the right instructor after the end of the semester.

Before we developed our own system we used Survey Monkey for our online courses and paper/pencil for f2f and hybrid. The biggest problem with Survey Monkey was that we had to manually create a survey for each section being evaluated (it wasn't too bad because we would just copy the master and rename it), but we still had a unique URL for each survey that then needed to be sent to the right instructor with information about which course it applied to or manually added to the correct course. This took up a LOT of time and energy.

Looking forward to seeing what other people are doing for course evals!

[Robbie_Grant]

Deactivated user

We have been using EvaluationKIT since the Summer of 2013 and it has been an awesome experience for us.  As awilliams​ mentioned it is a "turn-key" solution that now has CAS authentication.  We have a large number of departments using it with an average response rate of just below 50%.  We do have some departments that have upwards of 90% to 100% response rates.  We are using it for all online and any face-to-face courses that want online evaluations.

Robbie

[MLentini]

Hi Susan,

We've been using an external evaluation app that we built using PHP/MySQL. It's authenticated through our campus single sign-on, so students log in and see evals for courses they're in, and then can only fill out the evaluation once. We provide instructions to our faculty about how to refer students to the eval tool... a combination of announcements, adding to the module, etc. Response rates are pretty good, and since it's the same login as Canvas, there's usually little loss there.

Marc

[Cindy_Masek]

We have done all our course evaluations electronically for several years. First I used Survey Monkey and it became a huge, complex task with a lot of work required for cross-sectional reporting.  Next I went to CoursEval - easier than Survey Monkey but still required a lot of imports/exports.  SInce this is only one of many hats I wear at our small college, I needed something faster, and I kept having error issues.  The errors are more likely on my part, but I also had a lot of complaints from students that they couldn't get logged on at high use times.  Now we have moved to EvaluationKIT.  I've only used it one semester so far but I love it.  Even just learning the ropes it took me half the time of CoursEval and a small fraction of the time for Survey Monkey.  I am not getting any error reports about who is in what course because it is fully integrated into Canvas!  I can't goof that way!

Our end of semester evaluations have only been open 3 days and I already have a 20% return rate which is incredible.  I always shoot for 75%, but at best get 60%. (Faculty still long for the 100% completion of paper evals but are adjusting their expectations gradually. I just remind them of the work of hand compiling by peer faculty and how much they loved that!)  I am anxious to see what % of return we will have this semester.  One of the options in EvaluationKIT is the ability to create a pop-up-box that appears in the course, once students enter the course with an open evaluation.  You can even set it to require completion before continuing into the course.  I've enabled the pop-up for the last week of classes but allowed the user to click to do later.  After the last Friday of class though, I've set it to not allow student's to continue until they complete their evals.  (I didn't want anyone to have that required response to appear if they were entering a final exam!)  Students typically check courses frequently for final grades until the SIS has the official grade the following Wednesday, so those checking grades over the weekend and until Wednesday are likely to complete and I hope to see improved responses. 

[miltonv]

Hi Cindy,

Do you mind sharing your final spring course evaluation return rate using EvaluationKit?  We are currently reviewing options.  Any comments , suggestions, or issues using EvaluationKit would be much appreciated!

[awilliams]

We've been using it since Fall 2013. Here's our response rate:

The red line is when I took over full-time as Instructional Technology Specialist and by virtue of having admin access to EvaluationKit became the de-facto course evaluation Nazi. I did everything from getting PR to plug evals on the college's social media accounts, posting them on our (the eLearning Department's) social media feeds, posting flyers, setting up recurring non-respondent emails, turned on the pop-up for students with an eval left ( both of these are things done in EvaluationKit) and getting an image added to the slider on our homepage announcing them. I think it is safe to say I had an impact. I also strongly encouraged the faculty to mention the course evals to their students and explain to their students how their feedback is valuable because:

The take-away for me is that students are happy to respond to the survey IF they A) Know about them, B) Don't have to spend more than 10 minutes doing it (less if possible), and C) Have a clear understanding of the value of their feedback to the instructor and the institution.

[kona]

WOW! I'd definitely say it's safe to say you've had an *impact* on the response rate!

[Jeff_F]

I just received an 80% response rate using the same tool.  Good vibes ~ !

[Cindy_Masek]

Our spring evaluations were 63% which really is very good. That was with no effort beyond what was set up through evaluationkit on our part to Encourage participation.  Even more significant were our summer returns which historically are abysmal. We have received only about 25-30% of summer evals in the past and we averaged 55% this summer.  Our courses start and stop at all different times all summer so capturing these varied dates has always been a nightmare. Because EvaluationKit uses the imported course start and stop dates and you can set the evaluations to start and stop based on these dates we were able to make certain that all courses were captured in a timely manner.  And with the pop-ups set to remind students it appears to be much more effective than emails used in the previous system for catching student attention.

[John_Lowe]

Baylor University also uses the EvaluationKIT system.  When we started, we deployed that simply LTI integration that showed "Course Evaluation" links in every course menu all year long regardless of whether or not that course is actually being evaluated.  With their new integration using some custom CSS and JS code that you add to your main Canvas account settings, the "Course Evaluation" link shows up in the course menu only in those course with an active evaluation.  In addition, when students enter a course with an open evaluation, they get a friendly reminder to complete their evaluation.  Once the evaluation window closes or they complete their evaluation, they no longer see the course menu link or the reminder.  It is a really clean and simple (for the end users) implementation.

My only complaint about the system is that EvaluationKIT uses the short name for matching to their side of things instead of the SID ID.  If you are used to working in SIS IDs for most integrations, using the short names takes a little getting used to with EvaluationKIT.

[PSU_Tony]

We also use Evaluation Kit. With the LTI plugin and the pop up in Canvas, it is hands down the best product I have seen. We set the popup so the students can opt out the first week, but at the beginning of the second week they students must take the evals before it lets you into the course. Also the automated email communications have a great link that takes students straight to the evaluation. There are also mobile apps as well.

[chriscas]

UM-Dearborn is also using EvaluationKit.  We started a pilot in Fall 2014, and has a response rate of around 65%.  The system was adopted fully by 3 of our 4 colleges (the 4th remained in Pilot mode) for Winter 2015, and we ended up with a 56% response rate.  The setup process was pretty easy, even with our somewhat complex organizational structure here.  I would definitely recommend checking them out.

[canvas_admin]

Reformed Theological Seminary uses SurveyMonkey. Our Distance Education courses are on rolling enrollment so we have a "Course Survey" and a "Professor Survey" that we use for all of our online courses.   Our residential Hybrid courses also have a survey.  SurveyMonkey's latest update includes a better Graphic User Interface (GUI) for designing and branding your surveys and allows for more granular control over the design. They also have several different question types, question logic, etc. It's been very useful for us.

You can export the results based on a date range or other filters. An exports can be done as csv, pdf, ppt, etc. You can also include open ended comments in the export and have the option to have one question per page for a cleaner export.

[John_Lowe]

How do you enforce students only completing the Survey Monkey survey one time?  Does an LTI integration handle that for you?

[canvas_admin]

 @John_Lowe  Since the same student will be taking multiple courses online, they can take the same survey over and over. SurveyMonkey does provide an option to only allow one taking of the survey based on IP address.   I manually comb through each month's survey and delete duplicate surveys based on IP address. 99% of the time the student hit the submit button 2x by mistake.  We do have a question to indicate which course they are taking so I can verify that the student duplicated the survey for the same course rather than a student who submitted the survey 2x due to taking 2 different courses.

[lecia_sims1]

Our institution is beta testing a small number of courses this semester and I was looking to use SurveyMonkey for course evals. 

Were you able to embed the survey in Canvas?  Or did you just email the link to students?  I would prefer having the survey within the lms - if possible. 

Any help appreciated.  :smileyconfused:

[miltonv]

Hi Lecia,

We went with EvaluationKit for our course evaluations beginning Fall 2015. Our response rates have exceeded 50% each semester. We are very pleased with EvaluationKit and the ease of integration with Canvas. I highly recommend it!

Vikki Milton

Chipola College

[bacherts]

Susan,

At San Juan College we have been using EvaluationKit as well since Spring of 2014 and are very happy with the way it works and integrates with Canvas.  Each semester we have been tweeking it and are almost to the point that it is working like a well oiled machine.  Would be happy to share/answer any questions if you all decide to go that route.

Thanks,

Sam Bachert

[mfranz]

Howard Community College uses the IDEA survey which we have been doing on paper.  We are just transitioning to the online version which, from what I understand, is hosted on Campus Labs. We are hopeful to be able to integrate Campus Labs into Canvas at some point to allow for ease of access. Particularly given the fact that most students don't check their .edu email addresses so likely wouldn't see any notification to participate through that mechanism.

[DaleDrees]

I was wondering if anyone has done any Course/Faculty review with anything inside of Canvas, such as the Anonymous Survey, or does this not provide enough information?

[Robbie_Grant]

dreesd​,

We started off using the Quiz tool and setting up surveys inside of Canvas, but found that the reporting functionality was not quite up to par with what our departments were use to or needed.  We then look at different third party applications and chose EvaluationKIT as our solution.

Robbie

[PSU_Tony]

Evaluation Kit is hands down an awesome program. The LTI in Canvas drives very high return rates.

[bacherts]

dreesd​ - I wanted to echo  @Robbie_Grant ​ and  @PSU_Tony ​ comments regarding EvaluationKit - we have been using the tool since SP2014 our first semester with Canvas and have really loved the tool.  We have actually moved away from using other survey tools on campus because EvaluationKit allows for both end of course evaluations as well as web based surveys like Survey Monkey.

[DaleDrees]

Thanks Canvas Community!  @bacherts ​  @Robbie_Grant ​  @PSU_Tony ​  @John_Lowe ​ chriscas​  @Cindy_Masek ​ awilliams​

I just wanted to say thanks for the feedback on EvalutionKit!– Since Jul 8 we reached out to EvaluationKit and they just finished doing a demo to our Online Learning department and some administrators at our school today and they Loved it! The reporting, the integration and ease of use was great. Half-way through the demo I started thinking “I wonder if someone from Canvas built this?”

Thanks again!

[canvas_admin]

Our problem is that EvaluationKit is cost prohibitive for our institutional size so we use SurveyMonkey.  It may not be as pretty or intuitive but they do allow you to design your own surveys so we tied it to our institutional branding and we collect responses via a url which we then put into our end of course module as an external url.

[PSU_Tony]

I understand about cost, but the LTI pull in from Canvas and the pop up fir students, which drives incredibly high return rates, makes it invaluable for us.

Sent from my iPhone 6

[canvas_admin]

I would mention that we have a 50-60% response rate with our current setup.   Our 2 surveys per course require about 3 minutes each and are similar (with some tweaks for online and technology) to surveys at our residential campuses so there is a familiarity between what's online and what students might have at their campus.

[christian_gray]

Hi, just how expensive was EvaluationKit? We currently use SurveyMonkey, but are in the early stages of exploring other options.

[buellj]

 @christian_gray ‌ - the pricing for a small to medium size school of around 7K enrollments is about 10-15K/year.

[christian_gray]

WOW!

[kmahome]

Christian,

What size is your school? I thought Survey Monkey had privacy policies in conflict with the security of these type of data of higher education institutions in the US, how did you navigate that issue? We still use Blue by Explorance and it is on par or more than EvalKit depending on the license type and features, but their features and services suit our needs.

[kmahome]

Hi Susan,

Indiana University in Bloomington is using an evaluation system called Blue by eXplorance for all of our midterm and end-of-term course evaluation needs. Although we are a rather large university (46k students on this campus), we've found that this evaluation software works for our smaller campuses, too. Several of our smaller campuses in IU system have switched over to using Blue from CourseEval and paper evaluations. The evaluation system is a centralized system that integrates well with canvas, and we have not encountered any issues with LTI integration. Students also have the ability to access the system via canvas through their phones or just use the link directly on their phones.

There are several evaluation tools on the market, so I suggest shopping around to see which company will meet your needs and provide your college the best service during the implementation process and beyond. I recently reviewed CollegeNet's evaluation system (what-do-you-think) which seems to be a good decentralized system for small colleges with limited special needs, if you are thinking decentralized, check them out, too. We just switched one of our larger schools over from CourseEval to Blue after doing our final analysis of both systems. I have also reviewed the evaluation tool that Scantron offers Class Climate. All of these systems integrate well with Canvas, but you may have to sit through a view demos before you find the system that meets most of your needs. IUB is happy with the service that we are provided around the clock from eXplorance Blue and the free support to assist us with our diverse needs. We also like that this company takes our feedback in consideration in their quarterly updates that often reflect our requests.

Good luck in your search, Susan!

[candace_girard]

Krisy,

What was it about CoursEval that made you switch to Blue?  They both appear to be excellent systems, with Blue being more modern, in both architecture and presentation.

Thanks!

[kmahome]

Hi Candace,

The customer support, services, features (including dashboards), additional products (released formative feedback tool etc.) have been a step ahead of other vendors that have presented demos to me thus far. I have heard from colleagues that CourseEval has improved and expanded their features and products to be more in class with the previous releases from Explorance Blue, but I have not seen a recent demo. I also heard that CourseEval would be releasing their own version of a formative assessment tool (I have not seen this demo either). From my perspective, it seems as though the unique features that large institutions with diverse needs require, if not thought of originally, are implemented quickly with Blue instead of as a response to competition. I know that the centralized option carried weight in the decision making for our flagship campus and the university system followed. We are all now operating in the same evaluation system. We are functioning as both centralized and decentralized within the same evaluation system and able to adjust the features at each level of operation. Sorry for  the delay, I hope this information helps.

[amberh]

At the University of South Florida, we also use Blue by eXplorance for end of the semester course evaluations.

[palmarinich]

Since this is a nice thread mostly about EvaluationKit, I thought it would be a good place for my question...

We are currently exploring SaaS-type evaluation products, and like EvaluationKit.  However, our IT department has expressed concerns over potential impact to staff.  We anticipate having an "application administrator" that is not in IT, so their role would only involve anything related to data connections, SSO, and other backend tasks.

Can anyone share their experience in terms of the degree of IT resources required to implement and maintain EvaluationKit (or other similar products)?  If you can say anything about the overall workload of managing the application, that would also be helpful. 

Thank you.

[chriscas]

Hi Chris,

I think the answer to this will partly depend on how you plan to get data into EvaluationKit...  If you plan on importing from Canvas, and have your courses well named and structured in Canvas, I'd say the resources probably won't be that heavy.

EvaluationKit (and most other similar platforms, I think) has a one-time setup to establish your department structure, and how courses map to those departments (this is where you hopefully already have things well named from your SIS to Canvas).  I was able to do this for UM-Dearborn in a couple hours.  You'll then build your evaluations in the system, and attach them to a particular department or top-level.

Each semester, you'll then make a new project (or copy an existing one), where you setup evaluation dates, and import the courses from Canvas.  This step usually takes an hour or so for me at this point, since I'm mostly just copying existing settings, a first-time run through may take a few hours. I also have our departments do a final step where they'd emove any of the courses they don't want to have evaluated (independent studies, internships, etc).  We don't have university-wide rules around this part, which is why I pass it down to department staff.  I think this takes each department an hour or so to do.

Your experiences might be different depending on the structure of your school and your current evaluation processes (and how closely you want to try to stick to them).  I've found EvaluationKit to be pretty easy to use, and actually somewhat more responsive to change requests (with good justifications included) than Instructure.

Hope this helps a bit!

-Chris