Roadmap for Canvas Live Events

[jeff_longland]

Tagging @sgergely since this is mostly a question for him, but I suspect it'll be of interest to folks like @ColinMurtaugh and likely others.

For a few years, we've been using a mixture of CD1 requests and Canvas Live Events. They both have their pros and cons, but in the last year we've found ourselves using Caliper events more often than requests, especially when looking at engagement. Using `NavigationEvent` or `asset_accessed` as a proxy for a page view is vastly easier than trying to do the same using requests. It's not impossible, but it's so much easier.

Back in 2021, there was an Instructure presentation to the now defunct Internet2 LRS SIG (kudos to Tom if you're lurking here!). At the time, a number of comments were made about Live Events. Our notes were:

• Canvas Live Events are to be used for supporting applications, not to be relied on as a data source
• Using Canvas Live Events as a data source is characterized as a stop-gap measure until Canvas Data 2 rolls out with the 4-hour latency

There was subsequently an "Event Architecture & Event-Based Development Group" where Instructure was trying to better understand how clients are using events.

I haven't heard much about the roadmap for Live Events. Could we get an update? Are we safe to rely on Live Events for analytics projects?

Thanks in advance!

[sgergely]

Hi @jeff_longland!

Thanks for the question! I have tried to look up "Event Architecture & Event-Based Development Group" but I couldn't find any information so it might not have been formed.
Regarding the roadmap for Live Events I have also looked it up and for the current API there are no large-scale improvements or further meaningful updates are planned.

Is there any particular reason you are asking? Is there a problem, feedback or request regarding Live Events?

[jeff_longland]

The focus group was run by Casey Sato, Director of Customer Success - Higher Ed. I don't think there's anything sensitive in the re-cap that was shared with the group so I'll share it here. If it's considered sensitive, please let me know and I'll edit my post.

Our team capacity was limited at the time the group was active, so I wasn't able to fully participate. Generally, what's summarized is consistent with our usage of Live Events. We join events with CD1/CD2 to fulfill data requests, investigate operational incidents (ex. problems with SIS integration), learning tool reporting, and as a data source for tools (ex. instructor dashboards). We use the Caliper version of events and unlike some of the feedback, we are definitely interested in near real-time data.

The reason I asked about the roadmap is that we want to understand the status of Live Events in the broader context of the Data Access Platform. We don't see it mentioned much and given that we are increasing our reliance on the data (e.g. shifting from requests/web_logs to NavigationEvents) we want to make sure that we aren't using it in an unintended way or relying on something that will disappear.

We have a few items where improvements could be made to events. Is the best route to open support cases? Generally, it would be nice to see Instructure adopt Caliper 1.2 and see Live Events documentation/roadmap alongside DAP.

[pgo586]

I second @jeff_longland question on the state/roadmap for Live Events. As he suggested, we need to understand whether we're still safe depending on Canvas Live Events for analytics projects. As other parts of the Instructure platform evolve over time (in particular, CD2), it would be good to know what's the current Instructure plan for Live Events, which hasn't received any attention (or mention by Instructure) in a long time. 

[sgergely]

As I have written above Live Event stays as it was, no roadmap update is expected though.

What is your analytics use case? Where do you want to evolve it to?

[pgo586]

Thanks @sgergely for your reply.  Our use case for Live Events is various homegrown learning analytics apps (LTIs) which are course based, and use Live Events (in conjunction with CD2) as a data source for visualizations. These visualizations display event analytics related to quiz nd assignment submissions and file downloads for now, but might be extended to other types of Canvas content (e.g. grades, etc.) in the future. This streaming transactional information is for us much easier to work with than the CD2 weblogs. While I'm not looking for a specific upgrade to the feature at this point, I want to make sure that it continues reflecting the state of the world in Canvas as Canvas' own functionality evolves (e.g. New Quizzes, etc.), and that I can continue depending on it as a (non stale) data source for such learning analytics applications. 

[jeff_longland]

I replied up-thread, but yeah, similar sentiment from us: streaming transactional data is useful and we'd like to see it evolve as part of Canvas product development.

[marco_divittori]

We're also interested in how use of Live Events is being positioned as we were advised by Instructure some time ago that it was not intended for learning analytics so we've steered clear of incorporating it into our Canvas Data solution. It would be good to know if that recommendation has since changed.

[jeff_longland]

Agreed. It would be good to get clarity.

[pgo586]

Interesting @marco_divittori , my recollection is in fact quite different: analytics was one of the *only* things that Instructure was agreeing they could be used for, since Canvas Live Events could not be guaranteed to be 100% complete, so other, more authoritative uses (e.g. academic integrity) were out of the question. Moreover, I recall Instructure surveying institutions a couple of years back to see if anybody was using Canvas Live Events for something *other* than analytics (I assume this happened at the same time the group that @jeff_longland mentions was created).

[marco_divittori]

Hopefully we didn't receive some bad information but I'm certain that on more than one occasion, it was explicitly mentioned that Live Events was intended for app integrations (e.g. teaching/administrative tools) that might need real time data from Canvas as opposed to learning analytics use cases (e.g. to build reporting datasets).

[pgo586]

Yes, we seem to have been given potentially conflicting info apparently @marco_divittori . My understanding is that Live Events can be used for both of those purposes actually, as long as one made it clear to users that the data set can potentially be incomplete. This is why analytics uses (in which 'estimates' of usage can  be displayed to users) are less risky propositions -at least in my view- than tools that may depend on (potentially missing) real time event information. But all of this depends on your particular use cases of course, and how much data quality and reliability is required (as @jeff_longland mentions). Having said the above,  my personal experience with Live Events has been pretty good (I haven't found the data set to be incomplete in my use cases). 

[jeff_longland]

Getting clarity on Live Events is similar to the question @craig_thompson posted about requests/web_log

It's important for Instructure to be clear on data quality and reliability.