Canvas weblogs operation issues (North Virginia)

[Edina_Tipter]

In the past two days, our weblogs (including user_agents table) data processing experienced some issues while completing the query jobs for customers in the North Virginia region. As an effort to prevent a wider incident and maintain Canvas Data 2 main functionality and uptime, the ability to query Weblogs (including user agents) was temporarily turned off for customers in North America (us-east-1 AWS Region, or IAD). Thus any queries targeting Weblogs will result in Error 503: "The namespace 'canvas_logs' specified in the request is currently not available for querying due to internal errors."  Queries of tables from the Canvas or Catalog namespaces are completed successfully. The issue is contained to the North Virginia region and impacts less than 40 customers.

The team is currently investigating the impact of queries targeting Weblogs on overall performance of CD2, which might require a few days even.

Please bear with us, I will keep you posted. 

--------------------------------------------------------

Update on weblogs (5 Oct, 2023): the fix is on its way and we are targeting next week to release it after it undergoes a testing phase.

--------------------------------------------------------

Update on weblogs (13 Oct, 2023): the fix we wanted to release has not successfully passed the tests last weekend. Throughout this week, we have undertaken revisions, and today we will subject it to further testing and continuous monitoring throughout the weekend.. Should it meet the anticipated standards, the release will be scheduled for Monday.

--------------------------------------------------------

Update on weblogs (17 Oct, 2023): We wish to inform all consumers of CD2 weblogs in the Northern Virginia region that both the weblogs and user_agents tables are now available for querying. Yet, we have identified a gap in the data for Sunday, October 8th. We are currently assessing potential solutions for data restoration for this day, although we anticipate a longer resolution timeframe due to the development effort required on Instructure’s part. In the interim, should anyone require immediate mitigation for this data gap, the CD1 requests table can be utilized as a temporary workaround. We apologize for any inconvenience this event may have caused.