Get files by course is returning a 403

[lindstromc]

I am writing some course automation and auditing tools using the API.  I am able to call most of the endpoints without problem.  I need to place a JSON file in the course that will contain some versioning information that I need to read to perform the correct audit checks etc.  When I use the Get Files by Course Id endpoint I get a 403 error.  I am a designer for the course.  What permissions do I need to retrieve the list of files then eventually the file from the course.  Or is there a better API or method to do this?

Thanks
Craig Lindstrom

[James · Accepted solution]

@lindstromc 

A 403 status code means forbidden, but it may not mean that you don't have the correct permissions within Canvas.

If you are a designer for the course, you almost certainly have the proper permissions within Canvas. Otherwise, it would be difficult for you to do you job. If you can view the files page from the course using the web interface, then it's not a Canvas permissions issue.

You can verify that it works with the API through the web interface as well. Let's say I'm in a course with a Canvas ID 896851. That means that my pathname to my files is /courses/896851/files. If I add /api/v1 in front of that to get /api/v1/courses/896851/files, then I should get the first 10 course files. That is the call that your program makes to get the list of files.

The 403 status code can show up if you have the wrong ID. Sometimes people transpose a couple of digits. If you have numeric SIS IDs for your courses, then you want to make sure that you're using the Canvas course ID and not the SIS ID unless you tell Canvas you are using the SIS ID for the course. If you put in a course ID that does not actually exist, you get a 404 code rather than a 403 code. But it's possible to use a numeric SIS ID for your institution that is a Canvas ID for another institution and that will give you a 403 message.

Double check the ID and make sure it's the right type.

If your code works for one course but not others, then add some debugging to see which course is failing. Then verify that course ID works through the web interface.

If you can access the files from within the Canvas web interface but not your program, then we know it's not a Canvas issue, it's a problem with your code.

The Canvas REST API uses throttling to make sure that you're not hitting the system too quickly. This happened to me earlier this week with some pretty complex code that had been working perfectly and then all of a sudden it stopped and gave me the 403 error. I was using Node JS to write my code and used the Bottleneck library to limit my requests to 10 simultaneous connections with at least 50 ms between them. This was a rather simple get function and should not have hit limit. I got looking, and I had missed using the limited fetch() function in one spot in my library and used the native fetch() function, which is not limited. I fixed that and then the code worked.

You need to make sure that you don't hit the API too quickly, which can happen if you're making a bunch of calls without sufficient time between them.

For me, I was trying to get course information for 409 courses within milliseconds of each other. When I ran the command with a single course (I added a break statement in my loop of courses), it worked fine.

Hitting the throttling limit can also happen with just one course if you're using pagination and try to speed up the pagination process. The files API uses numbered pagination with page=2, page=3, etc. Some people (myself included) look at the link header and get the last page and then request all the pages at once. For my sandbox course, my link header tells me the last page is page 7, so after I fetch the first page, I then make 6 additional simultaneous requests rather than loading page 2, then loading page 3, then loading page 4, etc. It goes a lot faster and it works because I have the Bottleneck throttling library limiting the number of requests that actually get sent out. My code can make them as fast as I like, but the library makes sure that Canvas doesn't get upset about it.

Another possibility is that the API token you're using is not for your account and doesn't have permissions to that course. This doesn't happen often when people are developing their own code, but it might happen if you're using a system account that someone else created. Try making a single (not within a loop and not using pagination) request using the token your program is using. If it works when you just make a single simple request, it's likely a throttling issue or a case that you don't have access to some particular course you're looping through and I've talked about both of those already. If it doesn't work for a single request, then it may be that your token doesn't have the right Canvas permissions.

As for the question about whether there is a better API endpoint to use, I think the list files endpoint you mentioned is probably the best way. However, since you uploaded the file, you probably know the name of it. That means you can use the search_term parameter for that endpoint. By searching for that name, you can save time by not having to work through the pagination for all the files in the course. Also, with a search parameter, you can specify per_page=1 as a query parameter. If you were trying to get all of the files, you might use per_page=100, but it seems to take longer the more information you request, so limiting it can speed it up more. The search parameter may slow it down a tad, but it should be faster overall than getting all of the filenames.

Even with those optimizations, you'll still need to employ some kind of throttling.

Realize that throttling may not be an issue depending on the programming language you use. My understanding (I don't use Python regularly) is that the Python is synchronous and so one API request has to finish before you make another. You should never hit the limit when doing it that way. However, JavaScript is asynchronous and I can make multiple API requests at one time and the throttling is a real issue.

[James · Accepted solution]

@lindstromc 

A 403 status code means forbidden, but it may not mean that you don't have the correct permissions within Canvas.

If you are a designer for the course, you almost certainly have the proper permissions within Canvas. Otherwise, it would be difficult for you to do you job. If you can view the files page from the course using the web interface, then it's not a Canvas permissions issue.

You can verify that it works with the API through the web interface as well. Let's say I'm in a course with a Canvas ID 896851. That means that my pathname to my files is /courses/896851/files. If I add /api/v1 in front of that to get /api/v1/courses/896851/files, then I should get the first 10 course files. That is the call that your program makes to get the list of files.

The 403 status code can show up if you have the wrong ID. Sometimes people transpose a couple of digits. If you have numeric SIS IDs for your courses, then you want to make sure that you're using the Canvas course ID and not the SIS ID unless you tell Canvas you are using the SIS ID for the course. If you put in a course ID that does not actually exist, you get a 404 code rather than a 403 code. But it's possible to use a numeric SIS ID for your institution that is a Canvas ID for another institution and that will give you a 403 message.

Double check the ID and make sure it's the right type.

If your code works for one course but not others, then add some debugging to see which course is failing. Then verify that course ID works through the web interface.

If you can access the files from within the Canvas web interface but not your program, then we know it's not a Canvas issue, it's a problem with your code.

The Canvas REST API uses throttling to make sure that you're not hitting the system too quickly. This happened to me earlier this week with some pretty complex code that had been working perfectly and then all of a sudden it stopped and gave me the 403 error. I was using Node JS to write my code and used the Bottleneck library to limit my requests to 10 simultaneous connections with at least 50 ms between them. This was a rather simple get function and should not have hit limit. I got looking, and I had missed using the limited fetch() function in one spot in my library and used the native fetch() function, which is not limited. I fixed that and then the code worked.

You need to make sure that you don't hit the API too quickly, which can happen if you're making a bunch of calls without sufficient time between them.

For me, I was trying to get course information for 409 courses within milliseconds of each other. When I ran the command with a single course (I added a break statement in my loop of courses), it worked fine.

Hitting the throttling limit can also happen with just one course if you're using pagination and try to speed up the pagination process. The files API uses numbered pagination with page=2, page=3, etc. Some people (myself included) look at the link header and get the last page and then request all the pages at once. For my sandbox course, my link header tells me the last page is page 7, so after I fetch the first page, I then make 6 additional simultaneous requests rather than loading page 2, then loading page 3, then loading page 4, etc. It goes a lot faster and it works because I have the Bottleneck throttling library limiting the number of requests that actually get sent out. My code can make them as fast as I like, but the library makes sure that Canvas doesn't get upset about it.

Another possibility is that the API token you're using is not for your account and doesn't have permissions to that course. This doesn't happen often when people are developing their own code, but it might happen if you're using a system account that someone else created. Try making a single (not within a loop and not using pagination) request using the token your program is using. If it works when you just make a single simple request, it's likely a throttling issue or a case that you don't have access to some particular course you're looping through and I've talked about both of those already. If it doesn't work for a single request, then it may be that your token doesn't have the right Canvas permissions.

As for the question about whether there is a better API endpoint to use, I think the list files endpoint you mentioned is probably the best way. However, since you uploaded the file, you probably know the name of it. That means you can use the search_term parameter for that endpoint. By searching for that name, you can save time by not having to work through the pagination for all the files in the course. Also, with a search parameter, you can specify per_page=1 as a query parameter. If you were trying to get all of the files, you might use per_page=100, but it seems to take longer the more information you request, so limiting it can speed it up more. The search parameter may slow it down a tad, but it should be faster overall than getting all of the filenames.

Even with those optimizations, you'll still need to employ some kind of throttling.

Realize that throttling may not be an issue depending on the programming language you use. My understanding (I don't use Python regularly) is that the Python is synchronous and so one API request has to finish before you make another. You should never hit the limit when doing it that way. However, JavaScript is asynchronous and I can make multiple API requests at one time and the throttling is a real issue.

[lindstromc · Accepted solution]

Thanks for the information.  I think it was a timing issue.  I have it working now.