Canvas Data: In with the New, Out with the Old

[BobODell]

As the end of 2023 is approaching, we offer you a brief retrospective of what’s new with Canvas Data, an important reminder about migrating data to Canvas Data 2 as Canvas Data 1 reaches end-of-life this year, and references to useful resources for current and new Canvas Data users. 

2023 Retrospective

First, a new leader.  Last quarter, I joined Instructure as Sr. Director of Product Management for Instructure Data and Analytics.  I am so excited to join at this time when Instructure has laid the foundation for data and analytics with the 2023 launch of both Canvas Data 2 and Admin Analytics and several exciting announcements to come in 2024. I look forward to meeting many of you to discuss your data and analytics priorities. 

Next, we launched Canvas Data 2 in March after beta testing with some of the world’s largest education institutions. Canvas Data 2 streamlines data access, empowering customers and partners leveraging data to make informed decisions to improve student learning.

Specific enhancements include: 

• Fast, reliable access to trusted data (refreshed in  <= 4 hours )
• Scalability - built to high volume, variety and velocity of data
• Efficiency  - incremental queries to reduce unnecessary data movement 
• Manageability -  toolkit to support robust and sustainable analytic solutions e.g. Documentation, API specs, ERD and schema versioning 

By now, we expect you’ve gotten the message that Canvas Data 2 is a modern data analytics warehouse that will support innovative solutions to be delivered by Instructure, customers, and partners.  This will be the Instructure Data Access Platform for the future.  As such, we have been communicating that Canvas Data 1 will be deprecated at the end of 2023. 

Canvas Data 2 will be the Instructure Data Access Platform for 2024 

There is a process to onboard your data from Canvas Data 1 to Canvas Data 2 and effort required to migrate your analytics to the new schema.  If you haven’t started yet, please reach out immediately to your Customer Success team for how to proceed when access to Canvas Data 1 will be denied.  For more information about Canvas Data 2,  see: 

• Canvas Administrators Guide: What is Canvas Data 2
• Getting started with Canvas Data 2

We wish you all the best in the holiday season and peace and joy in 2024.

[Charles_Barbour]

I'm surprised that there seem to be multiple people still experiencing issues with CD2/DAP and concerned about the shutdown of CD1, yet Instructure apparently still seems to be planning to shut down CD1 in 18 days.

If there are still unresolved issues, will there really not be any overlap between a production ready and feature complete CD2 and CD1?

Since there are only about 5-10 working days left at most US universities, this seems almost inconceivable.

[bfelix]

Charles, we are in that situation, and were asked by our CSM to file for an extension, which we did. It does make me nervous, and I do wish the data team was more transparent about the current issues and offered some sort of timeline for a fix.

[HenchiFellig]

this change is going to have a huge impact on our business, we are surprised that this change was not communicated better and in advanced.
we reached out to  our CSM and we hope to get a response soon. 

[sor1]

Are Live Events (How do I install Canvas Data Services using Live e... - Instructure Community - 184 (canvaslms.com)) affected by CD1 end-of-life ?

[ryucali]

Resonating with reponses above, this is causal detrimental to the continuity of our business intelligence processes as we do still have problems with CD2. Also some institutions seem to get later deadlines  in 2024, and it is unclear how that is determined. It would be beneficial if an extension can be an option for all instututions. 

[BobODell]

@Charles_Barbour @bfelix @ryucali - Can you please share the specifics of what issues you are experiencing with CD2? Please include your open support case numbers as well so we can make these a top priority in addressing. CD2 is production ready and is serving more data, customers, and use cases than CD1 was. We are aware of some issues with the CD2 CLI and client library but these are provided today as beta. We are working to make them GA with improved performance and stability in February 2024. If you are interested in trying the GA version ahead of time please let me know.

[bfelix]

Thanks for the post Bob and Happy New Year. 

The Case number is 10088208 and dates back to Sept 6.

The basic issue is that we have found significant data gaps in CD2 compared to CD1.  For example about 3600+ user records in CD1 not in CD2

Same issue also exists in CD2 tables like Submissions, Quiz_submissions, Submission_comments and Context_module_progressions. There are records in CD1 but not in these tables above from CD2.

We inspected our data pipeline. Our current process is

  1.  Obtain access tokens to start data pulling job through endpoint “dap/query/canvas/table/{table}/data”
  2.  Inspect job status to ensure the job is complete before data extraction through endpoint “dap/job/{job_id}”
  3.  If job complete, obtain all the presigned URLs through “dap/object/url” for this table
  4.  Download files through the presigned URLs
Then we cannot find anywhere in this process that data might have been leaked.
We have also tried to snapshot loading for all the aforementioned tables, and reloaded all the records that we can download through snapshot from CD2, no luck.

L3 support has replicated this behavior back when we filed the ticket on Sept 6. Since then, we've basically heard nothing despite repeated pings. There was some mention of a possible fix around Nov 22, but the date came and went with no further updates or intel. 

It has been a frustrating experience to say the least.

[BobODell]

@bfelix - thank you for the information, very helpful. I was able to track this down and it looks like the issue was indeed resolved by a Canvas update that went out on November 22nd. In the engineering ticket notes it looks like it was also validated on December 7th as fixed specifically to your accounts and the examples provided. 

Can you please confirm this on your side and let me know if in fact it's resolved. 

I apologize about the gap in notification as well. I'm not sure why this wasn't immediately communicated given the open support ticket or why it took so long to resolve. I'll continue digging in to help us improve both of these going forward.

[bfelix]

Thank you for the prompt response @BobODell much appreciated. 

The communication gap is certainly confusing, as we had a drumbeat of pings almost weekly, with our CSM and Sales rep involved as well.

Regardless, this is fantastic news, and we will test this out asap!

[jsimon3]

IS
sis_section_id
anywhere in CD2?

I would have asked in the main discussion board but apparently Instructure removed it maybe related to CD1 removal

[David_L]

Greetings,

While our university has made the conversion to CD2 without issue for the most part, when our data systems engineer looks for the web_logs table (the documented successor to requests), he is not seeing it in the list given to him as an available table.
Could you please advise if there is something we're missing, or is web_logs on a different timeline for availability? I did not see references to it on the road map, only for mobile pageview data in Q2/Q3 later this year.

[stimme]

@David_L The web_logs table is in a different namespace: canvas_logs. There is a relevant discussion here: Canvas Data 2 - Weblogs .