Multi-Table Support for Canvas Data will Streamline your Efforts with DAP CLI 1.4.0

[sgergely]

We’re excited to announce the release of DAP CLI 1.4.0, which significantly boosts efficiency and ease of use by introducing multi-table operations. This release addresses a major customer pain point by allowing operations on all or selected tables within a namespace in a single command, saving valuable time and reducing repetitive tasks and additional coding tasks.

What’s New?

Namespace-Level Operations

• Execute commands across all tables in a namespace with a single command:

dap initdb --namespace canvas --table all

• Optionally, specify a subset of tables for more targeted operations.

dap initdb --namespace canvas --table accounts,courses,collaborations

Supported Commands

• Snapshot: Generate snapshots for all or selected tables in a namespace.
• Incremental: Perform incremental updates for all or selected tables.
• Schema: Retrieve schemas for all or selected tables.
• Database Management: Easily manage databases with these commands:
  • initdb: Create schemas for all or selected tables.
  • syncdb: Synchronize data for all or selected tables.
  • dropdb: Drop schemas for all or selected tables.

Improved Error Handling

• Receive detailed logs indicating which tables were processed and which encountered issues.
• If there is an error with one or multiple tables, the application continues with the rest of the tables.

Use Cases

1. New User Convenience

Simplify the onboarding experience by allowing users to initialize, sync, or query all tables in a namespace at once.

Example: 

dap initdb --namespace canvas --table all

2. Bulk Table Management

Streamline operations for data engineers managing multiple tables.

Example: 

dap syncdb --namespace canvas_logs --table accounts,courses,collaborations

3. Enhanced Automation

Enable seamless integration into automated workflows that require data from multiple tables.

Example: Set up nightly incremental updates for all tables with 

dap incremental --namespace canvas --table all

Why This Matters

• Saves Time: Eliminate repetitive tasks and focus on deriving actionable insights from your data without hitting the API limits.
• Improves Workflow: Reduce complexity and simplify automation for data engineers and analysts.
• Addresses Feedback: This feature directly responds to customer requests (1, 2 and numerous solutions on github) for easier multi-table management.

Try it now! To update to DAP CLI 1.4.0, run:

pip install -U instructure-dap-client

For more details, check the documentation. As always, we look forward to hearing your feedback!

[reynlds]

Nice update and welcome additions. I don't see much time savings, however, as the tables are still sequentially processed. Maybe a few seconds of "administrative overhead" when (not if) things fail and require a clean up.

[sgergely]

@reynlds  Thank you for the kind words! We’ve seen many users struggle to script multi-table synchronization, often creating unnecessary load on both their systems and our API. The sequential processing approach was chosen primarily for easier debugging, allowing users to quickly identify where and why a failure occurred. Additionally, it helps prevent overwhelming the API with simultaneous requests, which could degrade performance for all CD2 users. That said, the CLI does include built-in retry logic to handle failures and minimize disruptions. We appreciate your feedback and are always looking for ways to improve!

[StevenWalls]

@sgergely   Since we updated our CLI to ver 1.4.0, we've noticed several tables throwing an error. "ERROR - Failed export of table" on outcome_proficiencies", and outcome_proficiency_ratings.  Today we also got the same error on grading_period_groups and discussion_entries.

Is anyone else seeing this?

[sgergely]

@StevenWalls thanks for writing! Could you please share those logs before you end up on that log line which says "ERROR - Failed export of table"? So we can take a look? Thanks!

[StevenWalls]

@sgergely   This is the same verbiage no matter what table fails. It's only one or two per run, sometimes none.

2025-03-19 09:52:51,515 - INFO - Starting export of table 'groups' (5 of 44)
INFO:pysqlsync.postgres:connecting to dw-inf-svc@dwpgsrv/Canvas_CD2
INFO:pysqlsync.postgres:PostgreSQL version 15.0.12 final
2025-03-19 09:52:53,402 - ERROR - Error during export: type `Union[TableJob, CompleteSnapshotJob, CompleteIncrementalJob, FailedJob]` could not be instantiated from: {'id': 'd18910c9-c9ff-405e-96f4-91786ad23266', 'status': 'complete', 'objects': [{'id': 'd18910c9-c9ff-405e-96f4-91786ad23266/part-00000-2d2cce3d-b41e-401c-91ad-7cd0b86dbcd4-c000.tsv.gz'}], 'expires_at': '2025-03-20T14:19:48Z', 'since': '2025-03-19T13:30:52Z', 'until': '2025-03-19T13:30:52Z'}
2025-03-19 09:52:53,402 - ERROR - Failed export of table 'groups' (5 of 44)

[sgergely]

Hey @StevenWalls thanks for the logs, there has been an incident and it has been resolved: https://status.instructure.com/incidents/mhkszp5zx2bm