Participation on Canvas Analytics API, call on the most current date

[jolvera]

Hello

Need some help when using the Analytics Canvas API.

For the participation endpoint - how can I just make a call with the latest creation date. When I make the call I get all the dates and participations ; but that is to much to loop for every student; I was wandering on a way to make some delta where I just call the end point where the date is greater than the last date I got.

Any help appreciated.

[James · Accepted solution]

 @jolvera  

Sorry, but that API endpoint doesn't support filtering by date.

Most endpoints do not. Some of the endpoints of the submissions API will allow you to specify things submitted or graded since a particular time, but that's the exception rather than the rule.

Maybe rethinking the process can help. For example, if you fetch the enrollments for a course, then it has a last_activity datetime. Activity is not participation, but if there is no activity, then there is unlikely to be participation. Perhaps you could fetch the enrollment data first and then use it to only fetch the participation data for those who have shown activity. You should cache this locally if you're going to do this, so that you have a complete record for everyone.

Another thing that can help with the time issue is using a programming language that allows for asynchronous API calls. I started off with PERL and PHP which were both synchronous (without special libraries I wasn't using) and it takes way longer than my more recent stuff that uses Node JS and JavaScript.

[jolvera]

Thank you James;      Filtering with those with no participation helps. For asyncrhonous API call - you mean when you get the call back then use the date that is returned on JSON? or  can you give me an example of how the asynchronous call would help on the delta? 

[James]

I think I'm using asynchronous correctly, but I might have it confused with synchronous. What I mean is that you make the call and go on processing and then when the API is done, the system processes the data. The real benefit is that it allows me to make parallel or concurrent requests.

With PHP and PERL, I had to make an API call and wait for it to finish to make the next call. I fetch a bunch of information for every course, section, assignment, submission, and enrollment each night. Part of that is the student summary from the analytics API. I don't care so much about the participation count as the time in the course and the last time in the course. I know those are meaningless, but I get participation data through submissions. The other benefit is that the student summary is just one call per course rather than one per student.

It did help me identify a student the other day who was showing up as being active in the course (because she was using the mobile app and it was registering phantom accesses) but the time in course wasn't changing so I knew she wasn't spending any time in there doing anything. When I filled out the referral for our student success system, I used the time she had last done anything in the course rather than her last activity or her last submission -- which was before spring break and COVID-19 because they were not submitted through Canvas and so they weren't showing up in Dropout Detective.

Anyway, what I can do is get a list of courses and then make 5 or 6 API concurrent calls to get the student summaries at the same time. That's what I meant by asynchronous -- I can have 6 going at the same time without having to wait for one to finish before starting another.  In reality, that API call is a cheap one and I can have it set to make up to 50 at a time, but I stagger each of them 100 ms apart to make sure that it doesn't overload the system. The lowest my threshold got last night was 687. It starts at 700 and Canvas stops you when you get down to 0. It took me 667 seconds to process our 398 active courses. The student summaries took 39.873 seconds to make for the 398 courses, so the 100 ms was the limiting factor there on that call. I could tweak it even more, but it takes less than 12 minutes to run and I feel better not pushing the system to its limits. When I tweaked it too much, I was hitting the limit and getting cut off.

Because I can make concurrent calls, I normally limit the per_page setting and make more but cheaper calls. I found it seems to take longer to request per_page=100 even if it never reaches it than to specify per_page=50 and every now and then have to make some additional calls. I picked that trick up from Canvas Gradebook where they were only downloading 33 results at a time. It made for more API calls, but they could start displaying the results faster than if they did 100 results at a time.

I don't do it, but I could start the processing of courses as they came. Instead, again to keep the system from becoming too taxed, I wait for all of the courses to download before I attempt to do anything with them.

[jolvera]

thank you for this. I was submitting my requests in parallel.