Trying to figure out module_item creation behavior (parallel requests)

[mike_young]

Hi all!

Long-time lurker, first-time poster. Many thanks to everyone who has shared their tricks for negotiating the Canvas REST API, especially @James with parallel requests and pagination strategies! Huge help. Big kudos. 😃

As it says in the subject, I'm trying to figure out some module_item creation behavior. Here's the story:

1. I am doing stacks of parallel cURL requests in PHP using curl_multi handles.
2. Specifically, I am building multiple module items at once. I am building my POST requests correctly, and everything works successfully, including positioning, EXCEPT... for completion_requirements.
3. The module item objects in the requests look like this in JSON format (real #s scrubbed, obv):
   
   {
   "module_item": {
   "title": "Whatever",
   "position": #,
   "indent": 0,
   "type": "Assignment",
   "module_id": #,
   "content_id": #,
   "completion_requirement": {
   "type": "min_score",
   "min_score": 80
   },
   "published": true,
   }
   }
   
   
4. The module item objects I get back from Canvas look like this (PHP print_r output):
   
   

   [id] => #
   [title] => Whatever
   [position] => #
   [indent] => 0
   [quiz_lti] =>
   [type] => Assignment
   [module_id] => #
   [html_url] => correct URL
   [content_id] => #
   [url] => correct URL
   [completion_requirement] => stdClass Object
   (
   [type] => min_score
   [min_score] => 80
   )

   [published] => 1
   [unpublishable] => 1

5. Nothing weird I can see except that "unpublishable" property, which I can't quite track down in the Canvas source code API behavior. The problem is that even though completion_requirement is returned correctly for every module item I create, it inconsistently ends up missing in the actual course when I check through the UI and through the API (by grabbing the new module item info).
6. As implied by the mysterious italics, this behavior is inconsistent. I've thought about patterns based on where the call is in the batch, and there is no seeming correlation between whether it's the last call of the batch, the first call, etc.

I suspect the issue is related to the parallel cURL calls because I know the Canvas REST API doesn't like them. Maybe @James or another kind parallel request guru with time on their hands might be able to shed some light?

Thank you all very much! 😃
Mike

[James]

@mike_young 

It's not that Canvas doesn't like some parallel requests, they're okay as long as you don't hit the API too hard and put some delay between them. One way to test the theory that it's the parallel nature that's messing things up (I suspect it's something else), is to make them one at time and see if it's still inconsistent applied. 

The reason I don't suspect the parallel nature is because the requests will fail if your x-rate-limit-remaining value goes to 0. Since you're getting the response back, then the request itself didn't fail. Now if some of your requests are coming back and others aren't, then it definitely could be the parallel request causing the issue.

One place that the parallel nature will bite you is if you're trying to set the position. You can really get things in a seemingly random order if you try setting position if you send a bunch of requests at one time and they finish out of order. For those, I always go to serial requests rather than parallel requests.

Let's look at what you might do if you really are hitting the throttling issue.

I use Node JS instead of PHP/cURL since asynchronous calls are supported out of the box. It also has a Bottleneck library I use to control the throttling. There are two aspects to consider: how many requests can you send at a time and how long do you delay between requests. With PHP/cURL, you can control the first but I don't see a native way to control the second. In my experience, setting the second is more important than setting the first.

Let's say you have 20 module items to set requirements for. Sending them all at once will almost certainly cause a throttling issue. But if you allow up to 20 at a time but put 100 ms between them, you will probably be okay. That's because the server isn't hit with all of them at once. In fact, with 100 ms between each one, you're limiting it to 10 requests per second. That's not the same thing as allowing 10 simultaneous requests.

The 100 ms delay could probably go lower. I use 20 ms for some of my parallel requests. It really depends on the requests, which is why I debug and dump the x-rate-limit-remaining after each request finishes to make sure it doesn't go to low. I try to keep it above 500 (it starts at 700).

The number of simultaneous requests is more problematic. It's that initial request that could burn up all 700 of the x-rate-limit-remaining in one shot. How low it should be? When browsers used to limit the number of connections, it was 5 or 6 connections and I didn't have throttling issues. It's when they opened that up that I started running into issues. That might have been with HTTP/2 connections allowing pipelining and though there were just five connections to the server, it was sending multiple requests through each connection.

PHP supports the curl_multi_setopt options that may help. I'm going from the descriptions, not looking up old code here.

• CURLMOPT_MAXCONNECTS sets the maximum number of simultaneously open connections. The default is four times the number of handles added with curl_multi_add_handle. I would consider keeping this low, say 5 or less.
• CURLMOPT_MAX_TOTAL_CONNECTIONS has a very similar description except the part about the default.
• CURLMOPT_PIPELINING allows you to send multiple requests through the same connection. This may run into a problem if enabled since you could send multiple requests through the same connection, therefore bypassing the effort to limit the number of connections.

There are other options that could be tweaked. Basically, I would like to keep the number of requests at any time to 5 or fewer to avoid that initial hit. If I could spread out the requests, I would do that instead. To accomplish that, turn off pipelining and set the maximum number of connections to 5.

There are a couple of alternatives to multicurl that might work. I'm not recommending either of these, just listing them as alternatives.

• rolling-curl was last updated 14 years ago. It doesn't appear to put space between the requests, but it does limit the number of requests at a time. In their example, they're hitting separate sites, so they don't have to worry about throttling. However, it's a single file.
• Guzzle is a whole framework. I tried Guzzle a couple of times, but eventually gave up as I prefer doing things the hard way. I don't see where it puts delays between requests.

The best solution would be if you could stagger when things are added to the multicurl request. Start it off with 1, then use usleep() to add a slight delay before adding the next request. The rolling-curl code may give some insight into that. Starting with line 304 in RollingCurl.php is the code to add a new request when one has finished.

If it's not a parallel issue, then we need to dig further into which ones are failing.

[mike_young]

Hi James,

Thank you so much for your help! Super informative! Did not know about CURLMOPT_MAXCONNECTS and had been taking care of that part manually, so I appreciate you pointing that out.

All requests are coming back successful and include the completion requirements, and the positioning is also correct. 🫠You're right that PHP/cURL/multi_handle doesn't allow you to set the delay between requests—good call. Many complaints out there on Stack Overflow about that. And yeah, I should just write all of this in Node JS, but I should also probably buy a new pair of shoes, etc. ¯\_(ツ)_/¯

I've actually already written this with staggering between batches of requests and usleep()—using your numbers from this post as a guide! 😃 But this behavior occurs when there are as few as three parallel requests, which is kind of wild to me? Maybe some race condition down the line with Canvas's module updating/module item construction?

In any event, I tested it through several times with serial requests: no issues. ¯\_(ツ)_/¯ Though I am needing currently to build handfuls of module items across hundreds of courses, I'm not going to need to do it very often, so I guess it's OK to run through one request at a time—but it would be fun to know why this mysterious failure (which, again, contradicts the returned results) with the parallel requests.

In any case, thank you again so much for the thorough help!! 😃 Much appreciated!

[James]

@mike_young 

I'm surprised the order was coming out correctly. Maybe it's because you're creating them while I typically update module items to reorder them?

Staggering batches of requests is still sending a batch of requests at the same time. 

Let's look at some made up numbers (because I don't have module items to add or the time to test it right now).

Assume that sending 3 requests at the same time drops the x-rate-limit-remaining by 400 and that it recovers 200 of that by the time all three requests finish. The first batch drops you from 700 to 300 and then back to 500. The next batch of 3 drops you to 100 and then it recovers to 300. The next batch drops you past 0 and things start failing.

You could fix that by increasing the time between batches. If you make it too long, you could end up spending more time pausing between requests than if you sequentially sent the requests in the first place, so that's no good.

Rather than usleep between batches, it would be better to use usleep between requests. The documentation states that the bucket fills faster than a request can finish. That's why serial requests never cause a problem. Since you're making a POST instead of GET, it likely takes longer, so you might want to pause 100 to 200 ms between requests.

I think that modification to the lines I suggested in the rolling-curl are the way to handle that, but I stopped developing with PHP years ago, the inability to make parallel requests being a large part of that. That was also an issue with PERL, which I used before PHP. Now I'm trying to standardize on JavaScript, figuring that it will be easier for someone else to take over.

The problem is that the numbers affecting x-rate-limit-remaining are dependent upon the request as well as the server load. If you're checking the x-rate-limit-remaining as you add a new request with curl_multi_add_handle, then you can base the time for the usleep on the response. It can be shorter if the x-rate-limit-remaining is high or longer if it starts to drop down.

I had originally taken that approach, but then found out that Bottleneck with Node JS doesn't adjust the timing for any existing requests, only for new ones. That meant that I had to figure out the number of concurrent requests and the time between them ahead of time. To assist with that, I print the x-rate-limit-remaining, x-runtime, status, and request url to the console as soon as the response is returned. The x-rate-limit-remaining is the important one, but the x-runtime gives me an idea of which calls are expensive so I make those slower. I take that output, put it into a spreadsheet, where I can then ask for the minimum x-rate-limit-remaining achieved. It also helps me spot any problems. For you, printing the HTTP status code along with that might be insightful.

I guess I could write some kind of middleware between my code and Bottleneck, but that thought never occurred to me until I was writing this response. It probably won't happen since I rarely need to get the information as fast as possible, just faster than one request at a time.

There is one more thing I can think of here (I'm withholding the right to think of other things later). I used to get into trouble when I was using my token to make automated requests. I set the timings so tight that the job would run as quickly as possible, but with little room for error. What would sometimes happen is that I would be using the web interface on Canvas and running a job in the background. Either one alone would be fine, but when they worked together, it pushed it over the limit. That's part of why I don't make the requests as quickly as possible, but the long term solution was to create a service account to make the backend requests. Then I was free to use Canvas as normal without my requests failing.

[mike_young]

Thanks @James ! All super helpful! I do have a few service account tokens that I use for different jobs after running into the same issue you're describing. 😃

Tracking my x-rate-limit-remaining through the response headers suggests that I'm doing fine on my rate limits, which aligns with the successful responses I'm getting with full objects (including correct completion_requirement). I looked into RollingCurl, and it's a cool implementation of curl_multi for sure! It follows the same logic as the wrapper I came up with (re: dynamically adding and removing links), but it's more elegant than mine, so I appreciate the nod! Sadly, it still doesn't overcome the under-the-hood issue of no way to delay between individual requests, only batches of requests (that's what would happen if we added a delay around line 304). =(

For the fun of it, I wrote a basic test version of the same script in Node JS, and it worked fine hmm. I will keep thinking!

[mike_young]

Ah scratch that: with more iterations of tests, the Node JS w/ Bottleneck script fails the same way. =(

[mike_young]

But with a Bottleneck minTime of >= ~400, it's consistently successful. 350 or lower and it gets dicey—despite 100% consistency in all requests (no matter the minTime) returning x-rate-limit-remainings in the high 600s or 700 and successful 200 codes with correct objects. What's more: lower minTimes, and the positioning gets weird too. ¯\_(ツ)_/¯