Slow API responses for modules and module items (even when cached)

[jbeast]

I'm having an issue with some API requests being very slow to respond. The requests are being made on course homepages.

The request in question is specifically for the modules and module items for the course. This requests take ~4-6 seconds each (see Network screenshot attached).

Interestingly, the responses don't even include any content, but instead return a 304 Not Modified (as a If-None-Matched header is sent along with the ETag).

This doesn't seem to occur on all courses. We also have another separate Canvas environment with the same content, where the responses come back quickly (< 500ms).

What reasons could there be that the Canvas API is taking such a long time to determine that the already cached version of the resource is okay to use and return the 304 Not Modified?

[James]

@jbeast 

There could be several things causing the problem.

The thing that first jumps out is that it looks like dev.iheed.org is a test or beta instance of Canvas. From your main website, it takes you to learn.iheed.org as the production instance of Canvas. Beta and test are always going to be slower than production. Sometimes they are faster than others, but rarely is beta slower than production (unless production is having issues or network congestion). That alone could be an explanation of why it's slower here than on another system with the same content.

I just tested the same course in beta and production. It took 1.234 s in beta, 0.856 in test, and 0.544 s in production to load. Those numbers vary, of course, and re-executing the GET statement will return different times. It does not appear that refreshes are automatically quicker, though. In some areas, Canvas has a cache that gets loaded and then used when you fetch the same content. Then I tried it in a course that had a lot of modules and it took 0.892 s for production, 1.517 s for beta, and 1.650 for test.

None of that was fetching module items, it was just asking for a list of the modules. When I request the module items, the first course took 1.343 s in beta, 1.297 s in test, and 0.976 s in production. The second course took 1.226 s in production, 1.883 s in beta, and 1.723 s in test.

There may be more going on than just that.

I noticed you had per_page=500 on your first request. In general, Canvas doesn't support more than 100 at a time. You can verify this by looking at the link header in the response. When I try per_page=500, it comes back with links that contain per_page=100. Some programmers have chosen "what does it matter? The sky is the limit. I'll just ask for the moon and accept a two-story building if that is all that Canvas will give me."

However, in my experience, reducing the number allowed speeds the process up, even if the limit is never reached. I cannot seem to duplicate that behavior here, but I have seen it in other places. One of those is when I download information for each student every night for our early-alert system. The enrollments API seems particularly costly. The x-request-cost for something with 100 (or 500) per_page is higher than if I'm requesting 20 or 50 items. When making multiple asynchronous requests, you are more likely to use up the x-rate-limit-remaining that starts at 700 if you request larger per_page values.

While I see that most requests complete faster, the downside is that you might have to make more requests. I pick a medium value that I'm happy with to minimize that. I mean, do most courses really have 100 modules in them? Remember that per_page on the modules request applies to the number of modules, not the number of module items. My first course only had 10 modules and my second course had 65 modules.

I pulled up our Canvas Data and did a query to see how many modules were being used per course. I know that we're a small institution, with only 8274 courses in this query, and your course design will likely differ from ours, but here's what I found. The default per_page is 10, but only 44.95% of our courses had 10 or fewer modules. 92.40% of our courses had 20 or fewer items. 95.13% had 22 or fewer items. 98.30% had 30 or fewer items, 99.07% had 35 or fewer items. 99.50% had 40 or fewer items. 99.88% had 50 or fewer items. The most that any course had was 80 module items.

You may be able to speed things up by specifying a lower per_page setting. Specifying 500 accomplishes nothing because the max it will take is 100.

Loading module items is only necessary if Canvas does not return them with your modules. If you have a lot of module items in a module, then it may not return them. I mention this not because of the speed issue, but just to make sure you're not blindly downloading information you already obtained. Reducing the per_page for the module request may increase the likelihood of getting all of the module items. I don't have any evidence to back that up, but for those modules where the items weren't returned, you may want to try lowering the per_page and see if then gives you the items.

Depending on what you need, you may be able to use the GraphQL interface to fetch your information. It only returns the information that you request, rather than a bunch of information you don't need, and can be quicker. Unlike many of the GraphQL endpoints, the one for modules only returns those that are active (others return all and you have to manually filter on the status). One of the courses I queried with Canvas Data had 97 modules, but only 16 were active. The GraphQL only returns the 16, which is good.

It is likely that you would be able to fetch all of the module information much faster, as well. Getting just a list of the modules with their id, name, and position took 0.105 s my course with 65 modules (this is on production). That took 0.892 s through the modules API.

That doesn't contain any of the module information, though. That module information is a little trickier to get as you have to specify what you want for each type of module item. You cannot even get a name for every module item, you have to go into the content type and specify what you want. Classic Quizzes aren't really supported well (I would get the information for quizzes from the assignments type). The speed-up may be there if you need the module items. I asked it for just the ID and name/title for each of the items and it took 0.919 s. Compare that to 1.226 s previously and it could easily be a difference due to network traffic. The modules?include[]=items is a lot simpler and probably gives you all that you need.

There are other possibilities. I experience more delay when loading my content than other people do. I had Instructure look at it back in April 2017 and when they viewed my courses as them, it was quick. When they viewed it as me, it was slow. The more modules and the more items in the modules, the longer it takes to load. I've seen 6+ seconds in production for some of my courses until I went through and optimized my content. I don't think that's what is going on with you. The most likely culprit, based off what you've given us, is the Canvas instance you're pulling the information from.

[jbeast]

Hi @James,

Thank you for taking the time to do such a detailed and thorough response.

You're correct in saying this is one of our test envrionments. The same request on our production instance does indeed take about 10x less time. I did try your suggestion of using `per_page=100` but unfortunately it did not make a difference.

Thanks for pointing me towards the GraphQL API. It did produce responses far quicker returning them in < 500ms. Unfortunately, it doesn't seem to include the Completion Requirement information, which is available on the REST API, which is one of the attributes we need. If this was available in the response this would have been a perfect solution, as it's so much quicker, and only requires the one request.

I did also realise that the slowness is only happening for 1 particular module. This module has had significant changes over the past month, where dozens of module items have been created and then deleted many times. I was wondering if perhaps Module Items were one of the entities that Canvas does a soft delete on? And that if this was the case, perhaps every single module item was being loaded for this module, in order for Canvas to do its cache check? Would there be a way to do a hard delete on module items?

Again, thanks again for taking the time to help with our issue.

Best regards, Chris

[James]

Chris ( @jbeast ),

per_page=100 won't be much faster (if at all) than per_page=500. Canvas changes your 500 to 100. I mention the 500 because we often copy code that others use without fully understanding what's going on. per_page=25 may be faster than per_page=100 if you are doing this for a lot of courses or if you're working in a browser environment where you need to return some responses quickly so the user isn't waiting and then can load the rest in the background. If it's a program running outside the browser and you have plenty of time, then the 100 should work fine.

Canvas typically does not hard delete things. There's not a way to tell it to do so.

I haven't seen any evidence that Canvas is including the module requirements in the cache. It doesn't cache everything, but when there is a cache hit, you can make the same request twice within a short period of time and see a substantially quicker load on the second. It makes request optimization more challenging when hitting the cache, but it seems one technique I used to bypass the cache was to vary the per_page used. I don't see any of that here. The requests seem to take roughly the same amount of time.

The number of module item changes in the last month might be a clue, but I'm not convinced it's the cause. I would think that it would take a lot of changes to have an impact. The request to the database is going to filter by the workflow_state and that's likely to be indexed to speed things up. If your changes were made by a rogue script and you had hundreds or thousands of soft deleted items, I might be more concerned.

I think that having lots of module items or lots of module items that had requirements would have more of an impact. It's possible, but doubtful, that the type of module items would matter. If this module has lots of item types that are different from other modules, that might be another clue.

There are two ways I can think of to check the hypothesis about the number of deletions/creations.

One thing you might try is to create a new module and then duplicate the module items and requirements in it. Do not try to shortcut the process and copy the module back to the same course. It makes copies of all of the assignments in the "imported assignments" group when you do that. If the new module loads quickly, then it may be related to the number of deleted and recreated assignments. If the new module still takes a long time to load, then we know that it's something else causing the delay.

An easier check would be to copy the course to another and then check the load speed. If it's faster in the new course, then it could indicate there was too much left-over crud in the old. If the old course had students in it, then it might also indicate that the problem was related to the number of students having completed those requirements (I don't think that's the issue, just throwing it out there).

If you do create a new module that fixes the issue, then be sure to check any prerequisites for other modules that might have been using the old one.

Some might think that the endeavor isn't worth the time -- just realize it's slow and move on. I wonder, though, if the delay is there for other users. That is, when a student comes in and views modules or if the instructor looks at module progress, does this slow them down? If so, then fixing it is beneficial. That would be a good check to do as well -- does the page load slowly in Canvas or just through the API request you're making? They do a good job of using the same code, so a problem in one place may be a problem in another. Sometimes they make an API call to get the data and in other places, like loading the modules page, they send all of the information in the HTML rather than making separate API calls. Sending it all as HTML may involve database optimizations not in the API.

The GraphQL misses a lot of things. Canvas made a decision to go to GraphQL, but only add things as they needed it rather than completely populating it. That means that if they haven't rewritten their code to use GraphQL, things might be missing. New Quizzes were going to have a GraphQL interface, but then they changed and went back to the REST. I find REST easier to understand and work with, but GraphQL faster. The modules haven't been reworked since switching to GraphQL, so the REST API is still the way to go for full functionality.