Thoughts so far?

[leward]

Hello fellow smart search testers. What are your thoughts so far?  At this point I am not finding the smart search feature to be particularly useful and I probably won't continue until someone from Instructure announces a major improvement.  In my experience, far to many irrelevant items are returned for every search expression, and often the item I'd expect to see at the top is several documents down.  Maybe I am too old fashioned, but I would like to be able to do things like:

• search for an exact match to a quoted phrase
• search item titles only
• filter by date, content type, name of creator, contributor, etc
• sort results by date

Also, in order to be useful to instructors, smart search needs to include course files.  

I know this is just the first iteration and I am hopeful that the tool will improve dramatically before it is released to the masses.  In a courses with large numbers of canvas content and file items, it’s critical that users of the tool have enough control to find what they are looking for.  I suppose this could be done be creating a long and detailed prompt, but built-in controls for refining results are much more efficient and will yield much more predictable results.

I would be interested in hearing the experiences of other testers.

Lynn

[rmartini]

In my limited testing so far, I agree with everything you've mentioned. It hasn't been as useful as I had hoped. For it to really be useful, it needs to search course files (or at least titles of course files). For better or worse, many of our faculty primarily share files in their course sites and content isn't written on pages/assignment instructions. For those courses, Smart Search won't be very useful in the current iteration. I've also seen it return more results than expected (even when using quotes to try to limit what's returned). 

[KFalbo]

I've been piloting it's use with various faculty. Here's some of my observations.

1. Ranking of content is not clear to me yet. I feel like this would be a recognizable trait of a tool like this.

2. It's context was useful but not as robust as I would have expected. Instructor uses "Schema" in their course content, we search for "archetype" and results for "Schema" appear. This to me is the bread and butter of using AI in search. I recall from last summer's demo of such a feature where the search was for "mandolin" and results showed "guitar".

3. It's not clear to me yet how this will drive changes in Instructional Design. The first place for me is to think about time-stamp linkable transcripts of video content.

[emilyk23]

Thank you all for the feedback!

The development team is also anxious to begin querying additional Canvas features, such as files. I have also passed along the insight into wanting to filter search results based content type, date, etc. As development continues, keep your eye out for updates in the Blog area and the Change Log for the Smart Search group. 

[d_j_corbin]

I agree with all of the feedback above, and can confirm that our users and Canvas coordinators at our university have the same frustrations with the limited functionality. Since it's not possible to search for file titles, module item titles, or the contents of files our users are saying that it is not very useful at the moment. Not every teacher needs or uses HTML content in Canvas, some just use it to upload lots of files and module item links to external websites, much to our dismay.

@emilyk23  In addition to the feedback already given by others, I have two of my own:

1. How does the search functionality work in regards to the upcoming "Assign to" changes? From what I understand it will be possible to assign modules, pages and some discussions to specific sections or individual students, something similar to what already exists for assignments. Will the search results be tailored to each individual user's access to content? Honestly I haven't done any testing of this in regards to assignments, so I'm throwing it out there hoping Instructure already has taken this into account 🙂
2. I have a suggestion that will drastically change the user experience and be very helpful for larger universities, but will likely be very unpopular with the development team: Smart search should also be available from the left navigation menu or dashboard, and users should be able to receive results for multiple courses, including both active courses and courses from previous semesters, and be able to filter results per course if they want. There are currently a lot of steps to activating and using smart search, and search results are limited one course's content -- most often our content is spread out over a number of courses throughout degree programs that can last from 1-6 years.
   
   A bit of background on why this is beneficial: Our university's way of using Canvas is the following: one course from the SIS can often be more than one course in Canvas, and in some cases we see even up to 15 courses in Canvas for one course in the SIS. In addition, there are multiple teachers per SIS course spread out over multiple Canvas courses. Some study programs (for example dentistry) even have a totally different course structure in Canvas, where courses are study area- or skills-based and are active the entire span of the student's program (3-5 years). And each course has quite a bit of content. So as I mentioned above, content can be hard to search for, whether or not you know where to look.

Hope you will take this into consideration 🙂

[emilyk23]

Hi @d_j_corbin - wonderful suggestions! The team looks forward to adding additional features (i.e. files, etc.) to the returned results as development continues. 

As far as your question about Assign to... Smart search respects the users permissions/visibility. So if something is not yet published or visible/assigned to to a specific student, that student will not see that item returned in results. Teachers will see unpublished content across the course, but student users will only have results returned to items that they currently have access to. 

Regarding account-level searching, that is certainly feedback the team has heard and understand the desire for that type of functionality. They are prioritizing search within the context of a course first, adding additional features within course searching, and then exploring expansion in the future. 😊

[rmartini]

@emilyk23 : This is great to hear! Just something to note: as I was looking at the new Assign To/Modules features, it looks like content may be assigned to select students/sections as well as "everyone" depending on module settings (see my comment on the Latest on Releasing Modules to Students & Sections blog). I expect the team is already looking into those issues but wanted to flag this as it may affect Smart Search results until that's resolved. I'm not able to look at Smart Search in our beta instance to test this at the moment.

[emilyk23]

Oh good call out! Yes, if the student can see an assignment (bc it's assigned to a subset of the class AND everyone), they will be able to search for it.   

[j_avneri]

Perhaps the formatting of the result paragraph could be improved, as it can sometimes look a bit jumbled. While I understand the need to present information about the result, enhancing the formatting could make it clearer and more readable.

[patrickdorls]

Hi!

At our university we have enabled SmartSearch only for a sub-account where we can do testing. So it's not activated in any "real" course yet.
I did some testing in the end of June and tried the same prompts today again. I can see some improvements in how SmartSearch understands my prompts, but still the results are far from acceptable.

I tried the following things:

1. Searching for a single word by writing a whole sentence instead of just the word I'm looking for. Earlier, SmartSeach didn't get it and showed results for all words in my prompt, but today it's working better, just highlighting the word I really was looking for. Still the order of the results isn't satisfying (I gave my thumbs up for the best result)
2. Asking questions about content (How many assignments are there in the module "Test"?). This one is not understood by SmartSearch at all.
3. Search for unpublished content (as a teacher). This one works, but I'd prefer to see from the result list of it's published or unpublished material
4. Search for single words inside an assignment description. This one still doesn't work. The correct result is shown as result nr 17, and the word I was looking for wasn't highlighted.

I hope this summary can help you developing SmartSearch. We're really looking forward to implementing it when it works!