Hi @canninga, yes unfortunately the examples that Instructure provides are non-working code. Python's asyncio library requires that things look like my code below. However, you can also use th...

We use the DAP Python library within an AWS solution. On October 19, we had 78 tables fail with the "Middleware error" and 12 succeed in syncing. October 20 we had 65 tables sync successfully ...

Hi Jeff, If your goal is to get the Parquet (or other format) files into S3, this is fairly straightforward by using the DAP API endpoints. We do this with the web_logs files and simply store them in ...

Hi @BobODell , We have a daily job that refreshes the 90 tables in the canvas namespace, stored in a PostgreSQL database. This job typically takes under ten minutes. We also have a weekly job that...

We had 66 out of 90 tables fail this morning. We have implemented an AWS-based solution that kicks off a Lambda process for each table. We have the processes configured to time out after five minutes,...

We typically use last_activity_at and total_activity_time off of the enrollments table rather web_logs for student engagement questions. Instructure usually disclaims that web_logs is not guaranteed t...

We had not been bothering with meta.action for web_logs on the assumption that the number of deletes should be statistically insignificant for looking at broad usage patterns and aggregates, which is ...

Hi Jeff, If your goal is to get the Parquet (or other format) files into S3, this is fairly straightforward by using the DAP API endpoints. We do this with the web_logs files and simply store them in ...

Hi @StevenWalls, yes for better or worse this is par for the course. Skimming through our log history (not guaranteed to be exhaustive), we had some number of tables -- anywhere between 1 and 26 -...

A screenshot of our web_logs state machine is above in this thread; it's really pretty simple, and the underlying Python code isn't much more complicated. I'm happy to share that privately...