Improving Pageview Load Times – Temporary Solution in Place outside us-east-1 region

[MateLipcsei]

 ## Updated 02.06.2025 6:06PM CET:

We know how important fast and reliable access to your data is, and we appreciate your patience as we work through this issue. The initial fix has been successfully rolled out to all non-us-east-1 regions, and performance has significantly improved based on our monitoring.

That said, we’ve identified a separate issue that may still cause similar symptoms (UI errors, List Page View API delays), even though the underlying service is now responding as expected. Our teams are actively working on a hotfix to fully resolve this, and we’ll keep you updated.

We truly appreciate your feedback and patience, and we’re committed to ensuring a smooth experience for you.

## Updated 02.04.2025 7:10PM CET:

The team has made excellent progress, and testing has gone well. We are on track to deploy the fix tomorrow morning Budapest time to address the performance issues impacting non us-east-1 regions.
Deployment will begin in Frankfurt (eu-central-1) at 9 AM CET, followed by production testing. If testing results are positive, we anticipate deploying to the remaining regions around 12 PM CET.
This fix specifically aims to resolve the performance issues that have affected query response times. The team has thoroughly tested and validated to ensure stability before rollout. We will continue to monitor closely the performance post-deployment.

##

As part of our response to the incident in December, about slow-loading pages when accessing activity data in Canvas on the ‘User Profile’ page and ‘History’ tab and slower data access for our User API’s PageView objects. Our engineering team has implemented a temporary solution to improve performance. The update is being rolled out gradually, expected to arrive on Feb 4 on Feb 5. This interim update shifts the data source, allowing for faster access while we work on a permanent fix.

ℹ️ Important: Sites in the IAD region are NOT affected by this issue or the temporary solution. This update applies only to sites outside the us-east-1 region.

As a result, there are two temporary changes to be aware of:

1. Activity data will be available from 2024-07-01 00:00 UTC
2. The event_time timestamp may show slight variations for the same request_id, as it now reflects a different step in our server response process. (The request_id continues to serve as the unique identifier for an activity event)

We understand the importance of accuracy and accessibility, and we’re committed to resolving this fully. Thank you for your patience and continued feedback - your experience remains our top priority!

[jb2483929]

Could you clarify what IAD region is (US - East?)? How do we know what we are in - is it from our instances's status page?

[MateLipcsei]

@jb2483929 ,

yes, us-east-1 is NOT affected. Modified the post to clarify the AWS region.

Thanks.

[dbrace]

I have two questions/requests, @MateLipcsei:

1. I believe I know the answer (but others might not), using a method supported by Instructure, how does someone find out what region they are and if they are/were impacted? Should we go to https://statushistory.instructure.com and review the specifics for our instance or is this provided somewhere else?
2. When resolved, will the new "Page Views" UI allow for admins to place their mouse over a row's "User Agent" to reveal the more specific information? I ask because: (1) it is no longer possible, (2) relying on the downloadable CSV file (which is only 300 rows, even when looking at a specific day) is not helpful (mostly because of the increasing number of rows that exist from LTIs and Canvas mobile apps), and (3) the API is complicated (for one off needs, unless involved in a detailed research) and only supports the most recent 100 entries (when the API url is something like https://CANVAS_DOMAIN/api/v1/users/USER_NUMBER/page_views?per_page=100, anything more requires using bookmark links found in the API's headers).

Thank you.

-Doug

[MateLipcsei]

@jb2483929,

as @dbrace pointed out, you can find your AWS region at https://statushistory.instructure.com/ by entering your instance's name pretty quickly.

[MateLipcsei]

@dbrace,

1.) yes, this method is a pretty convenient way to find it out. Thanks for your comment!

2.) sounds like a valid use-case. do you mind DMing me your contact details so that I can reach out to you for a quick discussion?

Thanks,

Mate

[NickChevalierUT]

@MateLipcsei, appreciate the update! That said, where can measurable impact be seen from these temporary changes? I'm still getting 500 internal server errors anytime I run the "List user page views" API, and still getting an endlessly spinning wheel when trying to view a specific user's page views through the UI.

[MateLipcsei]

Hello @NickChevalierUT,

thank you for your feedback. The changes are getting gradually rolled out globally, the estimated completion is EOD Feb. 4.

Should you experience any issues on Wed, please don't hesitate to reach out to us.

[MateLipcsei]

Hi @NickChevalierUT,

I've shared the most recent update I got from the engineering with the detailed roll out plan. Please find it above.

Thank you for your patience and continued feedback!

[NickChevalierUT]

Hi @MateLipcsei -

Thank you for the continuous updates!

Based on these updates, it sounds like two actions should have taken place by now:

1. The temporary solution had an estimated rollout completion of EOD Feb. 4 (yesterday).
2. "If testing results are positive, we anticipate deploying [the fix] to the remaining regions around 12 PM CET [6 AM EST]."
   • Does this mean the temporary fix that was supposed to be rolled out EOD Feb 4, or a permanent resolution? And this should be deployed at 12 PM CET on which day?

Can you confirm whether these fixes are in place? As of this writing, I'm still getting 500 internal server errors anytime I run the "List user page views" API, and I'm still getting an endlessly spinning wheel (followed by "Could not retrieve page views") when trying to view a specific user's page views through the UI.

My institution is in the PDX/us-west-2 region.

Thank you!

[MateLipcsei]

Hello @NickChevalierUT ,

Thank you so much for your continued feedback.

Yes, unfortunately I needed to add a new update as you can see above.

The temporary fix was successfully rolled out on Feb 5, fixing the underlying service's performance, but unfortunately we have identified a separate issue, causing the same symptom. Our teams are actively working on fully resolving this.

Thank you!