Tips for CSV Speed Processing

[yves_schelpe]

Hi,
We're currently uploading our data to CANVAS via CSV files filled with data assembled from sis and some configuration tooling to populate some extra's on the precalculated sis-data.

We're in the process of developping a delta-based model (so that we don't have to push the full state, but only the changes) to CANVAS, but at the moment this is still in the works.

Does anyone have tips on how to improve CSV processing for large uploads. Currently we're taking 9+ hours and this is hurting, especially if we want to test certain features. In comparison, our previous ELO processed this in less than two hours (and we're not even close to the data we need to push).

Y

[pklove]

If you do the import using the API, rather than via the GUI, you can use a diffing_data_set_identifier.  Canvas will then compare your upload with the previous ones and only apply the changes.  

That is, it will generate the delta for you and apply it.

[robotcars]

I do nightly imports which take a considerable amount of time and then hourly differentials during the day, the query time is the same, but the import process is greatly improved.

Some more details I posted in another thread.

https://community.canvaslms.com/thread/22550-peoplesoft-canvas-real-time-enrollment-sync#comment-998...