DAP CLI Tool Imports Structure but No Data into Tables

[AzJedi]

Hello Community,

I'm currently working with the DAP CLI tool to import Canvas Data 2 into a database, but I'm running into an issue where the tables and column headers are created successfully but the table rows remain empty. This is after running the “initdb” command (dap initdb --namespace canvas --table TABLENAME).

I am using the Command prompt as an admin in Windows 10 and Python version 3.12.2. I have tried both PostgreSQL and MySQL and have the same issue. It even says that “Data has been successfully retrieved” with a ‘.tsv.gs’ file mentioned. But when I look at the table, it just has empty rows. This happens with any of the tables I try to import into the database.

Has anyone else experienced this issue, or can anyone offer guidance on what might be going wrong or what I should try next?

Any insights or suggestions would be greatly appreciated!

[LeventeHunyadi · Accepted solution]

Officially, Instructure supports Windows 11 for DAP client library but not Windows 10. You need to switch to one of the supported operating systems.

At a more technical level, the issue seems to originate from an open-source library that DAP client library uses called tsv2py. This library is responsible for efficiently parsing text data (specifically, tab-separated format) into Python in-memory data structures, which are then sent to the database driver. (Text manipulation done directly in Python is generally slow, and can easily become the bottleneck, this library brings approx. 5x speed improvement.) The latest version of this library published to PyPI is built for Windows 10.0.19045 (i.e. 2022 update of Windows 10) but if you have a less recent version of Windows, the library may fail (and hence no data be inserted/updated). Recently, we have contributed a patch to target Windows 8.1 and later for this library, which would be included in a future release. Likewise, this library uses AVX2 CPU vector instructions (SIMD, or single instruction, multiple data) on Intel/AMD machines. If you have an old machine, which lacks these CPU instructions, you might run into issues. All modern hardware for Windows has these instructions.

[AzJedi]

Thank you for the detailed reply. The machine I'm using is indeed pretty old. Am I correct in assuming that I might have more luck if I run this on Linux in a virtual machine?

[LeventeHunyadi]

Intel has CPUs with AVX2 support since around 2014, and AMD has CPUs with AVX2 since around 2015.

You can also choose to install tsv2py from source. By default, the package manager pip will prefer a binary wheel to the source code. However, if you pass the --no-binary option (or the PIP_NO_BINARY environment variable), pip will download and install the source distribution. You will need a C compiler on your machine (e.g. gcc, clang or MSVC) but this approach will work regardless of whatever CPU architecture you have.