Oddities with Discussion Summaries

[amcallister]

We turned on the AI features and then had our teaching and learning folks kick the tires, and we discovered some very odd responses and turned them off. We’re concerned that the AI is conflating information from other Canvas institutions or other sources we’re not clear on.

Example 1

We asked the AI tell us from a discussion in course with a number of faculty from different Faculties, “Which Faculties are most represented”

The AI responded with:
“The faculties most represented in the course discussion are:

•          Faculty and Curriculum Development Centre (FCDC), represented by instructors (Redacted - OCAD U person) and (Redacted - OCAD U person)
•          Faculty of Art, represented by instructors (Redacted - OCAD U person)
•          Faculty of Design, represented by instructor (Redacted - Faculty member from California State Long Beach)
•          Environmental Design, represented by instructor (Redacted - OCAD U person)”

The problem with this response is that the instructor from California State Long Beach was not in the discussion, not in the course, and they do not, nor have ever taught at OCAD U. California State Long Beach who apparently are also a Canvas school. So all that we can figure out is that the AI model has absorbed that information and is serving it to us. It suggests to me that the AI model is not sandboxed to our institution. 

Example 2

This example is less clear. We asked “who loves sailing” in a course discussion. It correctly answered with: “(Redacted - OCAD U person), the (person's role) at OCAD, loves sailing. He shared a photo of himself sailing J24 sailboats on the Hudson River in October 2015, suggesting this is an activity he enjoys and looks forward to continuing in his new role.”

However, we have no idea how the AI determined it was a J24 sailboat, or that the boat was on the Hudson River (we think it conflated this or analyzed the image) because there’s no metadata in the file to suggest the date or content, and the user did not mention the Hudson River, J24 Sailboats or October 2015.

[zachpendleton]

thanks for the feedback, @amcallister! I’m so sorry to hear that you had some weird experiences with discussion summaries. would love to connect further (I’ll get in touch with your account team), but I think both of these are instances of hallucination/the model falling back on prior training and aren’t indications that the model is looking at data from other institutions.

1. the model is only given the text of the current discussion when running summarizations and does not have access to other data from your institution or data from other institutions.

2. similarly, the model does not inspect attached images or files.

the relevant code paths are in open source canvas and you can see both the prompt and the context sent. the prompts are available at https://github.com/instructure/canvas-lms/blob/master/config/llm_configs/discussion_topic_summary_ra... and https://github.com/instructure/canvas-lms/blob/master/config/llm_configs/discussion_topic_summary_re....

the code that interpolates the prompt and injects context is available at https://github.com/instructure/canvas-lms/blob/master/app/models/llm_config.rb, and the context sent is available in the discussion API controller (https://github.com/instructure/canvas-lms/blob/master/app/controllers/discussion_topics_api_controll...) in the `summary` method — line 122 as I type this. you can see there that only the topic is passed after being serialized to XML via the PromptPresenter available at https://github.com/instructure/canvas-lms/blob/master/app/models/discussion_topic/prompt_presenter.r...

I hope that clears up that the model is not, in fact, relying on data from other courses, institutions, or sources outside of (1) the text of the discussion topic being summarized; and (2) the model’s prior training (which does — to be clear — include the internet at large, and is likely where you’re seeing this information coming from).

the goal on our end is to constrain model outputs down to only text in the discussion, which is always a challenge given LLMs’ proclivity to rely on past training and their non-deterministic outputs. we’ll continue to work on adjusting the prompt and improving the feature over time. again, I hope this gives you some context and look forward to continuing the discussion. thanks for raising the issues!

[amcallister]

Also, the prompt says not to use names, and it did.

[amcallister]

Hello Zach,

Thanks for these links. I had a look at your prompts for the AI, and might I suggest you add:

- Do not hallucinate.

- Do not invent, add, embellish or include any content that is not explicitly mentioned in the discussion thread.

Thanks,

Andrew.

[amcallister]

Hi Folks, I just want to follow up to this post indicating that in Example 2, Instructure observed that the AI was reading the ALT text of the image that included that information. So, this has been explained.

In Example 1 however – in discussion with Instructure – we can only explain this as a hallucination on the part of the AI.