Data2 CLI for TSV Export

[canninga]

We are trying to use the CLI to run snapshot queries that result in a download of TSV files to a local server. We are not looking to have the files go straight to a database at this time. We tried the sample code below but are getting a syntax error. Does anyone have any advice? I apologize for my ignorance. Neither I nor the one working with the code is familiar with Python.

[jwals · Accepted solution]

Hi @canninga, yes unfortunately the examples that Instructure provides are non-working code. Python's asyncio library requires that things look like my code below. However, you can also use the dap CLI utility directly from a command line and then you don't have to work with Python at all -- just run a command like dap snapshot --namespace canvas --table access_tokens --format tsv. Hope that helps!

import asyncio
import os
from dap.api import DAPClient
from dap.dap_types import Format, SnapshotQuery

async def download_files():
    output_directory = os.getcwd()
    async with DAPClient() as session:
        query = SnapshotQuery(format=Format.JSONL, filter=None)
        await session.download_table_data("canvas", "access_tokens", query, output_directory)

if __name__ == "__main__":
    asyncio.run(download_files())

[canninga]

This worked perfectly for what we needed. Thank you so much!

[J-J-Jason]

I think you want to put your code in a function and then call it asynchronously using asyncio. You'll have to install asyncio using:

pip install asyncio

I just put your code in an async method and call that method from main() using asyncio. Async code is just kinda different to work with.

Here's an example that uses a function and is called from main(). It downloads data for two tables.

Jason

import os
from dap.api import DAPClient
from dap.dap_types import Format, SnapshotQuery
import asyncio

async def download_table_data(namespace, table):
     query = SnapshotQuery(format=Format.JSONL, filter=None)
     output_directory = os.getcwd()
     await session.download_table_data(namespace, table, query, output_directory)

def main():
     loop = asyncio.get_event_loop()
     loop.run_until_complete(download_table_data("canvas", "accounts"))
     loop.run_until_complete(download_table_data("canvas_logs", "web_logs"))

if __name__ == '__main__':
     main()

[canninga]

Thanks, Jason. This is very helpful.

[LeventeHunyadi]

I would refer to the section Code examples in the DAP client library documentation, which writes:

DAP client library is following the asynchronous programming paradigm, and makes use of the new Python keywords async and await. The examples below have to be executed in an asynchronous context. You can enter an asynchronous context by invoking asyncio.run. By default, a Python script runs in a synchronous context; you must wrap the examples below into an async function, or you will get a syntax error.

In other words, you should wrap each of the code examples in an async function (e.g. my_func), and call that function with

asyncio.run(my_func())

This is nothing specific to DAP client library per se but generic to how asynchronous programming in Python works. There is Python documentation available that explain the concept in more detail.

[canninga]

Thanks, Levente. I appreciate all the resources and am definitely been trying to sift my way through them to learn as much as I can. As someone with minimal programming experience, it ends up being very overwhelming. I'm happy to have this community for additional support.