Incremental Data and API Usage Workflow

[MeredithAdams]

Hello! I am a business analyst working on questions from stakeholders about the incremental data system in Canvas Data 2. The questions are as follows:

1. Can data continue to be uploaded once a day, or does it need to be every four hours? If once a day, would that still be "incremental data"? We are wondering if we could skip the "compare schema" step on the workflow if using incremental data (instead of snapshot data), but only uploaded once a day.
2. On the API Usage workflow, what is Canvas comparing the schema version against? Does this comparison happen frequently (every time there's an upload)?
3. In the broadest sense, how is Canvas determining what is "incremental" data?

Thank you very much for your help!

[jerry_nguyen · Accepted solution]

@MeredithAdams 

Firstly, I would recommend using the CLI tool to download your data, it will do all the tasks as described in the workflow. (Including authentication, API calll to request data, loading data to your local database and manage incremental update)

1. You can download data at any time (Except for web_logs, which only retain data for 30 days)

2. It's comparing against the last time you pull the data

3. Incremental data means instead of having to download the full data set (e.g. Canvas Data 1) you can request data at any particular point (E.g. last 15 days)

If you need to help setting up Canvas Data 2, feel free to send me a PM.

[MeredithAdams]

thank you so much for your help!