Canvas Data 2 schema version errors

[mclark19]

@sgergely It seems that schema versions for a number of tables in the the canvas namespace have reverted. For example, "rubrics" had been at version 2 in mid February, but shows version 1 when downloaded today from DAP. The following tables seem to have issues:

access_tokens
assignment_overrides
assignments
attachments
conversation_messages
developer_key_account_bindings
discussion_topics
grading_standards
learning_outcomes
master_courses_master_content_tags
rubrics
submission_comments
submissions

We aren't using the DAP client to load the tables, but we do check versions numbers as part of our process.

[phanley]

I'm also seeing errors ('cannot drop values in an enumeration: pending' ) with the same tables this morning

Sample error log:

Running command: ['/usr/bin/bash', '-c', 'set -e; dap syncdb  --namespace canvas --table access_tokens || dap initdb  --namespace canvas --table access_tokens ']
Output:
Python version: 3.11.11 (main, Feb 25 2025, 02:38:55) [GCC 12.2.0]
Platform: uname_result(system='Linux', node='canvas-data-sync-dap-init-or-sync-db-i6kk27gj', release='5.15.0-131-generic', version='#141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025', machine='x86_64')
Package versions: {'pysqlsync': '0.8.2', 'json_strong_typing': '0.3.7', 'instructure-dap-client': '1.3.1', 'aiohttp-retry': '2.9.1', 'types-aiofiles': '24.1.0.20241221', 'tsv2py': '0.7.1', 'aiofiles': '23.2.1', 'PyJWT': '2.10.1', 'aiohttp': '3.11.11'}
pysqlsync.postgres:connecting to cd@dbserver:5432/canvas
pysqlsync.postgres:PostgreSQL version 16.0.6 final
Database version: PostgreSQL 16.6 (Debian 16.6-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
INFO: pysqlsync.postgres:connecting to cd@dbserver:5432/canvas
ERROR - operation not permitted; cannot drop values in an enumeration: pending
INFO - Python version: 3.11.11 (main, Feb 25 2025, 02:38:55) [GCC 12.2.0]
INFO - Platform: uname_result(system='Linux', node='canvas-data-sync-dap-init-or-sync-db-i6kk27gj', release='5.15.0-131-generic', version='#141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025', machine='x86_64')
INFO - Package versions: {'pysqlsync': '0.8.2', 'json_strong_typing': '0.3.7', 'instructure-dap-client': '1.3.1', 'aiohttp-retry': '2.9.1', 'types-aiofiles': '24.1.0.20241221', 'tsv2py': '0.7.1', 'aiofiles': '23.2.1', 'PyJWT': '2.10.1', 'aiohttp': '3.11.11'}
INFO:pysqlsync.postgres:connecting to airflow_user@canvas-data-rw:5432/canvas
ERROR - operation not permitted; cannot drop values in an enumeration: pending
INFO - Command exited with return code 5
ERROR - Task failed with exception

[StevenWalls]

We are getting those errors this morning as well on multiple table syncs.

[reynlds]

I'm seeing something very similar on these tables:

access_tokens
assignment_overrides
assignments
attachments
conversation_messages
custom_gradebook_columns
developer_key_account_bindings
discussion_topics
grading_standards
learning_outcomes
lti_resource_links
master_courses_child_subscriptions
master_courses_master_content_tags
rubrics
submission_comments
submissions
submission_comments
submissions
web_conference_participants

Here are the errors I'm seeing...everyt weekend things go to heck...:

2025-03-03 06:05:27,773 - ERROR - operation not permitted; cannot drop values in an enumeration: pending

2025-03-03 06:08:21,369 - ERROR - operation not permitted; cannot drop values in an enumeration: Course

2025-03-03 06:08:38,755 - ERROR - operation not permitted; cannot drop values in an enumeration: outcome_alignment_cloningfailed_to_clone_outcome_alignment

2025-03-03 06:09:26,514 - ERROR - operation not permitted; cannot drop values in an enumeration: public

2025-03-03 06:16:03,345 - ERROR - operation not permitted; cannot drop values in an enumeration: audio/aacaudio/amraudio/flacaudio/mp4audio/mpegaudio/oggaudio/vnd.dlna.adtsaudio/wavaudio/webmaudio/x-m4aaudio/x-ms-wmaaudio/x-wavvideo/3gppvideo/avivideo/mp4video/mpegvideo/oggvideo/quicktimevideo/webmvideo/x-m4vvideo/x-matroskavideo/x-ms-asfvideo/x-ms-wmvvideo/x-msvideo

2025-03-03 06:19:40,419 - ERROR - operation not permitted; cannot drop values in an enumeration: deleted

2025-03-03 06:21:46,117 - ERROR - operation not permitted; cannot drop values in an enumeration: not_threaded

2025-03-03 06:25:41,398 - ERROR - operation not permitted; cannot drop values in an enumeration: archived

2025-03-03 06:28:34,728 - ERROR - operation not permitted; cannot drop values in an enumeration: weighted_averagestandard_decaying_average

2025-03-03 06:29:29,780 - ERROR - malformed HTTP response:

<TITLE>ERROR: The request could not be satisfied</TITLE>

<H1>504 ERROR</H1>

2025-03-03 06:29:29,799 - ERROR - malformed HTTP response

2025-03-03 06:30:46,728 - ERROR - malformed HTTP response:

2025-03-03 06:30:46,747 - ERROR - malformed HTTP response

2025-03-03 06:31:02,721 - ERROR - operation not permitted; cannot drop values in an enumeration: MediaTrack

2025-03-03 06:37:35,275 - ERROR - operation not permitted; cannot drop values in an enumeration: archiveddraft

2025-03-03 06:38:51,185 - ERROR - operation not permitted; cannot drop values in an enumeration: audio/aacaudio/amraudio/flacaudio/mp4audio/mpegaudio/oggaudio/vnd.dlna.adtsaudio/wavaudio/webmaudio/x-m4aaudio/x-ms-wmaaudio/x-wavvideo/3gppvideo/avivideo/mp4video/mpegvideo/oggvideo/quicktimevideo/webmvideo/x-m4vvideo/x-matroskavideo/x-ms-asfvideo/x-ms-wmvvideo/x-msvideo

2025-03-03 06:39:32,565 - ERROR - operation not permitted; cannot drop values in an enumeration: audio/aacaudio/amraudio/flacaudio/mp4audio/mpegaudio/oggaudio/vnd.dlna.adtsaudio/wavaudio/webmaudio/x-m4aaudio/x-ms-wmaaudio/x-wavvideo/3gppvideo/avivideo/mp4video/mpegvideo/oggvideo/quicktimevideo/webmvideo/x-m4vvideo/x-matroskavideo/x-ms-asfvideo/x-ms-wmvvideo/x-msvideo

[a1222252]

Hi all, I just checked the rubrics table and I'm seeing version 2.

[mclark19]

@a1222252 Did you get any errors yesterday? Perhaps this is (was? -- things seem OK so far this morning) a regional issue.

[jwals]

Hi @mclark19, we had most of our tables fail yesterday morning (Monday, March 3). Around 80 tables failed with some version of "operation not permitted; cannot drop values in an enumeration" and a couple failed with "malformed HTTP response." We ran into the "cannot drop values in an enumeration" issue almost exactly a year ago when Instructure made significant changes with DAP 1.0. By yesterday afternoon all of the tables were syncing successfully again without us having to drop and reinit any tables or types. Rather than a regional issue, I'm wondering if Instructure (accidentally?) made some breaking changes that were quietly rolled back.

[StevenWalls]

We had multiple table syncs fail yesterday with those error too, none have failed today except one with "malformed http." Haven't heard an explanation.

[sgergely · Accepted solution]

Hello everyone! Thanks for reporting the issues!

There was an issue while we were deploying a new version of our application in the background without the intention of making any changes to the schema. The released version got corrupted and unfortunately published "version 1" for the tables reported above. Once I have seen @mclark19 mention I instantly went after this and notified the team and they have quickly fixed it.

Sorry for the issue we have caused! We are implementing additional tests to catch such issues before they go live to prevent these in the future. 

[mclark19]

Thanks for getting it fixed, @sgergely !