Data Access Platform (DAP) CLI and Client Library v1.1.0 is now available

[BobODell]

Data Access Platform (DAP) CLI and Client Library v1.1.0 is now available. 

Version 1.1.0 is a fast follow to our GA release to address some community identified issues with a new feature enhancement to prepare us for future updates. The list of fixes can be found below:

• Fixed error “You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'groups TO canvas__groups' at line 1” when automatically migrating the database schema from version 0.3.x to 1.0.x in MySQL databases. This occurred when the groups table (from the canvas namespace) was already present in the database.
• Fixed an issue with the client library not cleaning up the temporary directory used for storing downloaded files.
• Fixed error “ERROR - module 'socket' has no attribute 'TCP_KEEPIDLE'” by removing faulty/unnecessary TCP socket settings.
• Fixed an issue with the client library not decoding URL-encoded (a.k.a percent-encoded) characters in the database username and password in the database connection string.
• Database connection string no longer logged in plain text at debug log level.
• Fixed error “Fatal Python error: none_dealloc: deallocating None” occurring in some cases when replicating data to a database, running the client on Windows using Python version lower than 3.12.

In addition, v1.1.0 introduces extensible enumerations. Extensible enumerations are types in which the set of values is not defined in advance but values are added to the set as they appear in data. They will be most apparent in the case of web_logs in the namespace canvas_logs where we don’t have a well-defined list of web_application_action and web_application_controller values.

Today, web_application_action and web_application_controller are represented as enumeration types in PostgreSQL and MySQL. Going forward, they will become two new tables, maintained by our tooling. Whenever new values are received from DAP API, the client library will add them to the table of enumeration values. Because an enumeration type is transformed into a table, the change entails an automatic data migration on the client side. This new capability will be triggered when the schema advertised by DAP API changes in a future version.

You can find the release notes here: 

https://data-access-platform-api.s3.amazonaws.com/client/RELEASE_NOTES.html#release-notes

The new CLI and Client Library can be immediately downloaded from pypi.org - here: 

https://pypi.org/project/instructure-dap-client/

We greatly appreciate the community feedback provided and the validation that v1.1.0 fixes those issues. Please continue to let us know how v1.1.0 works and what features and enhancements you'd like to see next.

[StevenWalls]

I installed the new CLI and I'm using the same URL, client_id and secret as on our other two servers.

We're running Python 3.12.

When I run a simple DAP list I'm getting:  ERROR - Cannot connect to host api-gateway.instructure.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)')]

[marco_divittori]

What is the estimated timeline for creating the new web_application_action and web_application_controller tables and which namespace will they be in?

[RichardCarballo]

Installed new dap tool on new VM 
Old VM could run dap syncdb --table accounts 
no problems

new dap tool with dap_api_url set to https://api-gateway.instructure.com
dap_client_id matching old VM, dap_client_secret matching old vm, dap_connection_string matching old vm
REQUIRES adding namespace now?
also when using name space get massive print out mentioning commnent on table "instructure_dap", print out as follows:

C:\Users\....>dap syncdb --table accounts --namespace canvas
INFO:pysqlsync.postgres:connecting to ......../canvasdata
INFO:pysqlsync.postgres:PostgreSQL version 14.0.9 final
INFO:pysqlsync.postgres:connecting to ........./canvasdata
INFO:pysqlsync.postgres:PostgreSQL version 14.0.9 final
INFO:pysqlsync:synchronize schema with SQL:
CREATE SCHEMA IF NOT EXISTS "instructure_dap";
CREATE TABLE "instructure_dap"."database_version" (
"version" bigint GENERATED BY DEFAULT AS IDENTITY,
CONSTRAINT "pk_database_version" PRIMARY KEY ("version")
);
COMMENT ON TABLE "instructure_dap"."database_version" IS 'Table for storing meta-information about the database schema version.';
COMMENT ON COLUMN "instructure_dap"."database_version"."version" IS 'The version of the database schema.';
CREATE TABLE "instructure_dap"."table_sync" (
"id" bigint GENERATED BY DEFAULT AS IDENTITY,
"source_namespace" varchar(64) NOT NULL,
"source_table" varchar(64) NOT NULL,
"timestamp" timestamp NOT NULL,
"schema_version" bigint NOT NULL,
"target_schema" varchar(64),
"target_table" varchar(64) NOT NULL,
"schema_description_format" varchar(64) NOT NULL,
"schema_description" text NOT NULL,
CONSTRAINT "pk_table_sync" PRIMARY KEY ("id")
);
COMMENT ON TABLE "instructure_dap"."table_sync" IS 'Table for storing meta-information about tables managed by the client library.';
COMMENT ON COLUMN "instructure_dap"."table_sync"."source_namespace" IS 'The namespace of the source table exposed by DAP API (e.g. `canvas`).';
COMMENT ON COLUMN "instructure_dap"."table_sync"."source_table" IS 'The name of the source table exposed by DAP API (e.g. `accounts`).';
COMMENT ON COLUMN "instructure_dap"."table_sync"."timestamp" IS 'The timestamp of the source table that can be used by the client library in incremental queries during subsequent `syncdb` command executions.';
COMMENT ON COLUMN "instructure_dap"."table_sync"."schema_version" IS 'The latest schema version of the source table at the time point when it was last initialized or synchronized with the DAP API.';
COMMENT ON COLUMN "instructure_dap"."table_sync"."target_schema" IS 'The name of the target schema in the local database if applicable (e.g. in case of a PostgreSQL database the tables can be grouped into schemas).';
COMMENT ON COLUMN "instructure_dap"."table_sync"."target_table" IS 'The name of the target table in the local database. In might differ from the name of the source table. For example in case of a MySQL database, the tables cannot be grouped in schemas. In this case, the implementor can use a prefix that reflects the namespace of the source table. For example, the qualified name `canvas.accounts` would become `canvas__accounts`.';
COMMENT ON COLUMN "instructure_dap"."table_sync"."schema_description_format" IS 'The format of the schema descriptor (e.g. `json`).';
COMMENT ON COLUMN "instructure_dap"."table_sync"."schema_description" IS 'The latest schema descriptor of the source table at the time point when it was initialized or last synchronized.';
2024-04-18 12:16:56,892 - ERROR - error executing query:
CREATE SCHEMA IF NOT EXISTS "instructure_dap";
CREATE TABLE "instructure_dap"."database_version" (
"version" bigint GENERATED BY DEFAULT AS IDENTITY,
CONSTRAINT "pk_database_version" PRIMARY KEY ("version")
);
COMMENT ON TABLE "instructure_dap"."database_version" IS 'Table for storing meta-information about the database schema version.';
COMMENT ON COLUMN "instructure_dap"."database_version"."version" IS 'The version of the database schema.';
CREATE TABLE "instructure_dap"."table_sync" (
"id" bigint GENERATED BY DEFAULT AS IDENTITY,
"source_namespace" varchar(64) NOT NULL,
"source_table" varchar(64) NOT NULL,
"timestamp" timestamp NOT NULL,
"schema_version" bigint NOT NULL,
"target_schema" varchar(64),
"target_table" varchar(64) NOT NULL,
"schema_description_format" varchar(64) NOT NULL,
"schema_description" text NOT NULL,
CONSTRAINT "pk_table_sync" PRIMARY KEY ("id")
);
COMMENT ON TABLE "instructure_dap"."table_sync" IS 'Table for storing meta-information about tables managed by the c...


*********with installing the new DAP ensure you are running with admin privileges to install new directories and schemas that were not mentions in the DAP update documentation. In my organization we used a standard account with read/write privileges for executing dap commands, this new update pushed changes and tried to create new schemas and tables in the database and needed admin privileges to execute

[RichardCarballo]

this version is riddled with issues

[StevenWalls]

We continue to get the error below when running our DAP syncdb processes. It is sporadic and happens during or right after the file download. Is there some known issue that causes this?

WARNING - Received error in response: Middleware error
- ERROR - Middleware error