Inconsistent Grade Change Event Results Through the API

[AaronWong]

Hello -

I'm new to the Canvas API, and I wanted to learn it, so I thought I'd pick an easy target: Get the grade change log and recreate the gradebook. Fast forward one week, and I'm starting to go crazy trying to figure out what's happening because I'm getting an inconsistent result depending on my approach, and it's not making any sense.

Approach #1: By Student and Assignment

1. Get a list of all students (api/v1/courses/{courseID}/enrollments?type[]=StudentEnrollment)
2. Get a list of all assignments (api/v1/courses/{courseID}/assignments?per_page=100)
3. Get all grade change events for the course by student and assignment (/audit/grade_change?course_id={courseID}&assignment_id={assignmentID}&student_id={studentID}&per_page=100)

Runs extremely slowly (as expected) and recreates the gradebook successfully after yielding 10743 entries for the particular class I selected to try this on.

Approach #2: By Student

1. Get a list of all students (api/v1/courses/{courseID}/enrollments?type[]=StudentEnrollment)
2. Get all grade change events for the course by student (api/v1//audit/grade_change?course_id={courseID}&student_id={studentID}&per_page=100)

Runs slowly (but not as bad) and recreates the gradebook successfully after yielding 10743 entries.

Approach #3: By Course

1. Get all grade change events for the course (api/v1/audit/grade_change?course_id={courseID})

Runs reasonably quickly, but only yields 10710 entries. And there is no rhyme or reason to the missing 33 entries that I can find. The missing entries seem to be randomly scattered throughout, not linked to any one student or assignment, or missing the last page of changes, any other pattern that I can identify.

I tried it with another course, and got 6951 entries in the first two approaches and 6939 for the third one (missing 12 entries). But when I did it with a smaller course, I got a consistent 751 items across the methods. Is there something happening with these larger requests maybe?

Here is a link to the Python code on Github to see if the problem replicates elsewhere. Obviously, you'll need to change the institution, key, and course number. 

https://github.com/AaronWongNSC/CanvasAPIHelp

Thanks for any insight you can provide!

[chriscas]

Hi @AaronWong,

Just wanted to put a quick initial reply here...  I think you may have inadvertently chosen perhaps one of the most complex tasks as your learning experiment here.  I can say that I ran your script in a course in our canvas environment, and the by student and by student assignment had the exact same count, while the by course had less entries. If I have some free time, I may try to dig into where the differences come in.  I could imagine student status (active/inactive/concluded) could come into play, but the difference could also be caused by students in multiple sections or that changed sections in a term, or perhaps even other things I'm not thinking of yet.  I've never used this particular API before, but course/section enrollment issues can be a "gotcha" in all kinds of areas of Canvas, so that's where my mind went to first.  Someone else who has used this API before may be able to come in and shine some light on their experience too.

-Chris

[AaronWong]

It's good to know that I'm not crazy, and that it's unintentionally a difficult task. Although, conceptually, it seems that "Get the data out of the log" *shouldn't* be complicated. I know it gets complicated when it comes to calculating grades -- dropping grades, excused assignments, weightings, etc. -- but the grade change log itself shouldn't be impacted by this. (At least, that's how it would work based on how I imagine that data is kept.)

The particular classes I was looking at are all single-section courses, and so there shouldn't be anything about changing sections impacting things. And it does not seem to be tied to student status, as it's not as if I'm getting no data from particular students, and the missing items in the audit are scattered throughout different students and assignments. It's also not about "excused" assignments or dropped grades in calculating the final grade, or things like that. As far as I can tell, it's just log entries that are not being pulled up by the API.

If you mess with the code that exports the gradebook, you can get all three spreadsheets. And if yours is similar to mine, you'll just see random scores missing.

[chriscas]

Hi @AaronWong,

So after some more digging (at least with data from my own instance), I can say a couple things...

1. Multi-enrollments can cause duplicate data in your first two methods (by student, and by student assignments).  I see this where students enrolled in one section, then dropped that (making them inactive for us), but enrolled in another section of the same course.  Since you said this is a single section course, it may or may not contribute to what you're seeing, as there are also ways a user would have multiple enrollments in a single section (multiple roles, could both be student-based).
2. The third method (by course) will include data for the test student (student view), whereas your first two methods will not include this data because while the test user can get graded, it's not generally included in enrollment reports.

One or both of those may be what you're seeing, but without getting a copy of your exact data, it's hard to say.  I exported the data from each method to a CSV file so I could then sort and compare line by line in excel to find where things were different, then looked at the data for that line to spot the unique things that could be happening.  At least form what I see in my instance, my guess is you'll find the differences are from a very small set of users which you can start investigating for unique circumstances (like multiple enrollments, different statuses, etc).

While I know it seems like this should be a really simple task, this might highlight some of the things that you'll run into when working with Canvas APIs.  Your methods were pretty sound, but gotchas like multi enrollments and the test student account come into play in many API calls and need to be accounted for (especially when you're comparing one method vs another for the same task).

Hope this helps a bit!  If you have any questions, I'll try to give some more responses (and others may jump in as well).  I may not be able to do a ton more work on my own for this though since I've already put in quite a bit of time on it.

-Chris

[AaronWong]

Hello @chriscas -

I appreciate that you put in time to try to figure this out. Unfortunately, I am not getting a coherent result when I look at the differences in the data.

I'm wondering if there's a pathway of escalating this. I could still be wrong (and I really hope I am), but the deeper I look, the more I suspect there's potentially something off in the API itself.

1) You have reported that you're getting duplicate data. That seems to be a peculiar oversight unless you mean that the student data is being replicated but with different grade change ids when a student changes sections. (Basically, it makes a copy of the grade changes for the new course.) But then, that data should also appear twice in the full course search. (If you're getting repeated data with the same ID, that's a little weird but fixable. You can always just clean up the list, I guess, but that should really be done on the back end.)

2) Even if the test student is in the course query but not the student or student/assignment queries, that would make the course query LONGER, no shorter. So that doesn't seem to address what's happening (though I think it's a good insight into thinking further about what other quirks are in the system -- in this case, it's buried in the enrollment type of the course API).

As far as my data is concerned, both the student and student/assignment results are identical. There are no entries in those two results that are not in the course-wide query. The only thing that is happening is that entries are being dropped in the course-level query. And even after a deep dive into the dropped items, there's no discernible pattern. It's not any specific student, assignment, timestamp, or anything.

Incidentally, instead of exporting straight to CSV, here's a fast way to get the mismatches:

mismatch = []

for event in by_student:
    if event not in grade_change_events:
        mismatch.append(event)

From here, you can export to CSV however you might typically do it.

[chriscas]

Hi @AaronWong,

For #1, what's happening is that enrollment in Canvas is always attached to a section.  When you're calling the enrollments API for a course, it may list one particular user more than one time if they have more than one role in a section, or have enrollments in multiple sections.  This causes the number of entries for these methods to perhaps be higher than expected (and higher than the "by course" method where duplicates would never be returned).

For #2, you're correct in that factoring out the test student would make the number here even smaller.  But when factoring out the duplicates from #1 and the test student from here, I get identical event counts among all three of your methods.

You may have something else coming in to play in your environment, but without being able to access your data and Canvas environment, it's going to be rally hard to determine what else might be happening for you.

I'd generally suggest contacting Canvas Support for issues, but when getting into the depths of API calls, I haven't always had the best of luck.  Based on what I'm seeing in my own environment, the API calls are returning correct and consistent data, but your environment could have something else going on.  For the entires that aren't showing for one method or another, I'd definitely check the user's enrollment status, the assignment settings, etc to see if you can find any commonalities.  I personally believe there's got to be some consistent difference there that just needs to be discovered.

-Chris

[AaronWong]

Hello @chriscas -

Regarding enrollment, I see it now. It's not that the grade audit appears multiple times in the audit log, but that a student is listed multiple times in the enrollment side, causing it to be called out of the log multiple times.

Since I *really* don't want to try to write an enrollment tracker right now, I'm going to do it again but without any reference to enrollment. I've generated a new code that (again, in my head) should get the same logs.

1. Get all grade change events for the course.
2. Identify the first and last dates of grade changes.
3. Cycle through all of the days from first to last and get grade changes from each day (24 hours - 1 second periods to avoid potential overlap of two grade change events being exactly 24 hours apart).

This code is here: https://github.com/AaronWongNSC/CanvasAPIHelp/blob/main/by_date.py

So now I have the following:

• length of log by course: 6939
• length of log by date: 6948

There are 3 items in the by_course log that are not in the by_date log, and 12 items in the by_date log that are not in the by_course log. I checked for weird boundary cases and things like that, and it still looks somewhat random.

The only observation is that some of the items in the by_date log that are not in the by_course log have the same timestamp. (This is because I'm giving credit for assignments where they get their points just for turning something in, and so those grade entries go in very quickly.) My mismatch list (in by_date but not by_course) has three elements that share the same time stamp.

6b0d6011-b2a3-46c7-a10c-6315cbecef9b, 2023-01-31T16:50:43Z
566877ca-0a5b-4f15-9191-a3472cedf6dc, 2023-01-31T16:50:43Z
68ecdbfa-117e-4581-9b43-58cbbcab9475, 2023-01-31T16:50:43Z

When I run the API at this specific time stamp (audit/grade_change?course_id=xxx&start_time=2023-01-31T16%3A50%3A43Z&end_time=2023-01-31T16%3A50%3A43Z&per_page=100), I get this result:

566877ca-0a5b-4f15-9191-a3472cedf6dc, 2023-01-31T16:50:43Z
68ecdbfa-117e-4581-9b43-58cbbcab9475, 2023-01-31T16:50:43Z
68f8d725-c3ad-4ea0-952f-1a7060fabf26, 2023-01-31T16:50:43Z
6b0d6011-b2a3-46c7-a10c-6315cbecef9b, 2023-01-31T16:50:43Z

 And you might think that it has something to do with multiple elements at the same time stamp, but if I go one second later, I get another four entries.

37b3c4d8-cda4-4ddf-bc7e-335b85f79ba3, 2023-01-31T16:50:44Z
7a68c49e-f131-4960-a526-0b08d78e0abb, 2023-01-31T16:50:44Z
10017ddb-20ea-45ec-bdcf-05aede62f58e, 2023-01-31T16:50:44Z
5872f510-0b3f-47dc-baf5-40e17d37aa27, 2023-01-31T16:50:44Z

All four of these entries appear both data sets. 

And all of this happens without touching the enrollment API. It's just the grade change audit API querying what should be the same data in two different ways.

I would really hope some more people can try the code to see if this is something specific to my campus' instance or if there's something else going on.

[chriscas]

Hi @AaronWong,

I'm hoping someone else ,ay be able to jump in here and look at this, as I unfortunately do have a lot on my plate between my regular university job and being a community coach here (and then all of the non-work things).

With that being said, can you tell us what your end goal ultimately is here?  In your original post, it sounded like you just wanted to learn a bit about the API, but in the most recent post it seems like you have something specific in mind that you're trying to accomplish.  If you can tell us your goals, someone might be able to lay out some decent guides or suggestions.  As a programmer, digging into the fine details like you're doing is always interesting, but it's very time consuming, and if you can accomplish goals without having to dig into this kind of stuff, it would probably be easier all around.

Asa final suggestion, make sure you're looking at the user and assignment ids of the anomalous events, as thats what I did with your original dataset and that lead me to find the both test student issue and the enrollments being double counted issues you encountered.

I apologize for not being able to continue to dig into this right now, but If I end up having a lot of free time at some point, I may be able to return to it.

-Chris

[AaronWong]

Hello @chriscas -

I completely understand the constraints on your time, and I also hope someone else jumps in. And I'm also grateful for the insights you have already provided.

I have a number of broad goals, most of which is trying to figure out what things I can accelerate/automate in my processes and potentially some research that I might be starting up with one of our instructional designers. For example, my classes have a lot of little assignments that have similar but not identical prompts, and I'm slightly annoyed at the number of mouse clicks that are required to set those things up. And if I change something from one semester to the next, I have to go through each assignment individually and update it. I don't like that. I've tried the Multi-Tool or Design Tools (I can't remember which) that does some of this, but doesn't quite do what I want. I've got a lot of programming experience, and so building my own thing seems like the best chance of getting to my destination.

Specific to the gradebook, I am mostly able to accomplish one of my goals with what I know (identify students who have a grade below X on some assignment, compose a personalized-but-also-boilerplate message, and send it). It's the inconsistent results that keep giving me pause to try to make sure I really know what's happening. Recreating the 

I have a smaller class that has fewer than 1000 (I think it was 750-ish) items in the log, and that one came out completely consistent. The medium-size course is off by about a dozen items out of around 7000 items. For the largest class (~10000 entries), I get the same number using the latest two methods, but there are 33 elements that don't match between the two lists. 

I did go through the assignment IDs and students, and there's no pattern that I saw. It has happened on some assignments that have been unchanged for the entire semester, and students who were enrolled from the beginning. It happens sometimes in the middle of a string of students/assignments like what I had indicated above. I've got to put this down for a couple days, but I intend to give it one more deep dive next week.

[JamesSekcienski]

@AaronWong 

Is there a reason you are using the Grade Change Log to generate your own version of the gradebook?

Based on your end goal of re-creating the gradebook, have you considered using Submissions instead?  I know this doesn't solve the issue you discovered with the Grade Change Log, but Submissions may be easier to work with and not result in these same inconsistencies.  In addition, the Gradebook view in Canvas is generated in part by using Submissions.

[AaronWong]

Hello @JamesSekcienski -

Creating my own gradebook is currently just a learning tool, though I have a couple things I'd like to try if I can succeed at it. I went to the grade change log because when I looked through the API call list (doc/api/live/) it was the only thing that looked like it connected to the gradebook. I'm going to feel kind of dumb if there's a more straight-forward way of doing this.

Submissions won't work entirely for me, as I have a number of in-class activities that are not submitted online, though I can definitely see some potential use cases.

[JamesSekcienski]

@AaronWong 

Ok, that makes sense for why you were looking at using the Grade Change Log.  However, Submissions is a bit misleading of a name.  Even if a student hasn't made a submission to an assignment in Canvas, there is still a submission for each student associated with each assignment.  So as long as the in-class activities correspond to an assignment in Canvas, you will see Submission data for each of the students even if they didn't submit anything in Canvas.

[AaronWong]

Hello @JamesSekcienski -

That's interesting. I would probably not have discovered that for quite a while just based on the word choice. I'll have to look into that next week when I sit down with it again. I appreciate that insight.

In the meantime, I remain convinced that something is up with the API (whether globally or just our instance, I don't know). It would seem that a grade audit by course should get *every* grade change associated with the course. And it's not at all clear why it isn't doing that for me.

[AaronWong]

Good news:

• The suggestion from @JamesSekcienski to look at Submissions worked in terms of me getting the gradebook values I was looking for. It was good to learn that Submissions aren't what I thought they were based on the name.
• The observations from @chriscas regarding enrollment in sections vs. course makes sense for why the first two methods could sometimes yield repeated students and things like that. Since I only have single-section courses, I would not have known that I should even think about course vs. section.
• The by-student and by-student-and-assignment loops through the grade log seem to work correctly.

Bad news:

• Trying to get the full grade log for a course still does not seem to work correctly. The logic in my head says that if I get the grade log for the course, I should get all items, and that all other calls that slice things up in different ways (by student, by assignment, by date, etc.) should all be subsets of this master list. But that's not what's happening for me. It's not clear whether this is a global issue or just a local one.

General Observation:

• I wish there was more documentation for understanding how the data in Canvas is organized/created on the backend. I still don't know how I would have learned that the creation of an assignment leads to the creation of submissions for students and that those submissions would contain the current grade information other than by someone coming along with that knowledge and telling me. The same is true with just understanding how enrollment works. (I also know that this is a big project, and one that very few people would want to do.)