Process multiple cURL with PHP

[buddyhall]

Hi,

I'm new to programming, but made a page using PHP that lets us enter userIDs and add observers using the Canvas API and that worked great. I'm now trying to enter a list of people to be observed and have it process multiple URLs and that is where I'm running into problems. It only seems to process the final userID in the box. Any ideas what I'm doing wrong? This doesn't need to be run often and I'm not worried about performance so I don't really need the cURL commands to process simultaneously.

$observeeA = explode ("\n", $observee);
 $ch = curl_init();
 curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
 curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "PUT");
 curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
 foreach ($observeeA as $ob2){
   if ($idtype == "Canvas") {
     $url = $domain . "/api/v1/users/$observer/observees/" . $ob2; }
   else {
     $url = $domain . "/api/v1/users/sis_user_id:$observer/observees/sis_user_id:" . $ob2; }
   curl_setopt($ch, CURLOPT_URL, $url);
   $resp = curl_exec($ch);
   echo $ob2 . "<br>" . $url . "<br>" . $resp . "<br>";
   }
 curl_close($ch); 

The echo on line 13 outputs all the correct entries for the URL, but it never gets a response until the last one in the list.

[buddyhall · Accepted solution]

The variables in my array had an extra hidden return character in them. I inserted this line between 6 and 7 above and it works flawlessly now. I figured I would add this incase anyone else is trying to use arrays with multiple cURL calls in PHP.

$ob2 = trim($ob2);

[nschutz]

Hi Buddy,

Just saw your posting. I am new to canvas API but familiar with PHP. May I ask what PHP client you used to create your PHP page?

[buddyhall]

Hi  @nschutz ‌,

I just typed it up in a basic text editor called TextWrangler.

[James]

I stumbled across this because of  @nschutz s comment and even though it's not what the thread is about, the title mentioned multiple cURL, so I wanted to share some of what I've been working rather than asking a question.

I've been trying to get the curl_multi functions to work over the last couple of weeks to speed up the process. It's not quite the way I wanted it to be yet as I would like to be able to handle multiple gets at the same time but what I've been working on is handling the pagination in parallel when it's predictable.

When I load a list of 5721 courses 10 at a time but sequentially took 251 seconds with PHP and cURL. With cURL multi, it took 34 seconds with just 5 concurrent gets. When I bumped that up to 20 concurrent requests, it took 8 seconds. Bumping it up to 100 per page reduced that to 6 seconds. Bumping it up to 30 concurrent requests actually slowed things down, but there may be some issues with the way I'm adding more requests that could improve that. All of this is at home over DSL where it's possible that bandwidth was a limiting factor so it might be faster with a better connection.

Another part I don't have working yet, but have an idea of where to go, is monitoring the limiting factor and time something spends in the queue to dynamically change the number of requests. Hard coding it at 20 might be too many in some circumstances.

Normally, I'm like  @buddyhall  where I don't care how long something takes -- it's running as a background process through a cron job. But every now and then you get days like today where I had to shut down the normal process while someone else upgraded their system and then needed to run a full job instead of just an incremental to catch things up. That's normally ran in the middle of the night but now I needed it in the middle of the day and didn't want to wait.

And then I keep thinking how if I used Guzzle rather than cURL I might be able to simplify my code a lot, but it's already established and I don't want to take the time to rewrite it all right now.

[nschutz]

Hi James,

What platform do you use to run your PHP code with cURL? Do you run it on your localhost or FTP it somewhere?

[James]

 @nschutz ,

I have Linux servers setup, mostly with Ubuntu distributions, but PHP is going to be included with about all of them. I also have it installed on Windows 10 for my development machine.