Request-URI Too Long

[nschutz]

We need to replace a line that is included in all of our content pages. While testing out my PHP/cURL code that goes through each page of the course, gets the body of the page, finds and replaces the string within the body and sends the new body string to the PUT request, most of the pages worked and the old string was replaced with the new string. However, for some of the really long pages, I got this error below:

Request-URI Too Long

The requested URL's length exceeds the capacity limit for this server.

---

Apache Server at canvas.instructure.com Port 80

Is there any way to get around this 414 error? If not, what else can we do to mass update all content pages?

[robotcars · Accepted solution]

My first thought for trouble shooting would be to ensure that you are using HTTP Method PUT, sometimes POST, for sending data to the API and that you are passing data, and not using GET and appending the data to the URL query string. The body of some wiki pages can be quite lengthy, often over 8000 characters for longer content. I've noticed when I've improperly set the length of database columns for Canvas Data. URL's like to be around 2000 characters max with various exceptions.

PUT - Syllabus Body - Courses - Canvas LMS REST API Documentation 

PUT - Updating/Create a Page Pages - Canvas LMS REST API Documentation 

POST Creating a Page - Pages - Canvas LMS REST API Documentation 

[James · Accepted solution]

 @nschutz ,

carroll-ccsd  comments reminded me that there are two ways to send the payload with PUT. You can make it part of the Query String attached to the end of the URL or you can include it as part of the post like you would with a POST. With something long like an entire wiki page, you would definitely want to use the post approach and not as part of the URL. The URL itself is really short, most of it is in the CURLOPT_POSTFIELDS. The examples from the API that Robert linked to show the body of the cURL as the -d option rather than in the URL itself.

In my PHP, I have something like this for a PUT statement. I'm using JSON to encode the object. This is pieced together from several different blocks, so hopefully I got it all.

// $options is the JSON object to send
// $url is the URL
// $token is the API access token
$data_string = json_encode( $options );
$http_headers = array (
  'Authorization: Bearer ' . $token,
  'Content-Type: application/json',
  'Content-Length: ' . strlen( $data_string )
);
$ch = curl_init( $url );
curl_setopt_array( $ch, array (
  CURLOPT_HTTPHEADER => $http_headers,
  CURLOPT_RETURNTRANSFER => TRUE,
  CURLOPT_VERBOSE => FALSE,
  CURLOPT_HEADER => TRUE,
  CURLOPT_CUSTOMREQUEST => 'PUT',
  CURLOPT_POSTFIELDS => $data_string
) );
$response = curl_exec( $ch );‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

You might also need to add a header of 'Accept: application/json' and there is obviously some additonal logic that needs included, I was just trying to get the part about the PUT statement to work.

[robotcars · Accepted solution]

My first thought for trouble shooting would be to ensure that you are using HTTP Method PUT, sometimes POST, for sending data to the API and that you are passing data, and not using GET and appending the data to the URL query string. The body of some wiki pages can be quite lengthy, often over 8000 characters for longer content. I've noticed when I've improperly set the length of database columns for Canvas Data. URL's like to be around 2000 characters max with various exceptions.

PUT - Syllabus Body - Courses - Canvas LMS REST API Documentation 

PUT - Updating/Create a Page Pages - Canvas LMS REST API Documentation 

POST Creating a Page - Pages - Canvas LMS REST API Documentation 

[nschutz · Accepted solution]

Thanks so much, Robert! I was already using PUT request, however using the PHP/cURL code generated from Postman is the problem - it sends all data on the URL string by default. I added the following line, removed the param from the url and it now runs correctly. The $newPageBody was the data being too long sometimes if passed in on the URL. 

curl_setopt_array($curl, array(
CURLOPT_URL => $updatePageURL,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "PUT",
CURLOPT_POSTFIELDS => array(
'wiki_page[body]' => $newPageBody
),
CURLOPT_HTTPHEADER => array(
"Authorization: Bearer xxxxxx",
"cache-control: no-cache"
),
));

[robotcars]

Yep, that should have been an 'or' not an 'and' - appending the data to the query string. :smileygrin:

Happy you got it working.

I totally forgot Postman can generate code, that's a handy feature. Thanks for the reminder.

[nschutz]

I am sorry, you lost me on the " 'or' not and 'and' " part. Did I miss something?

[robotcars]

My first reply should have been...

and not using GET or appending the data to the URL query string

I was guessing that the code was doing one or the other, but you were using PUT correctly, it was the query string that was the issue.

Sorry for both confusions.

It's been a long week.

[James · Accepted solution]

 @nschutz ,

carroll-ccsd  comments reminded me that there are two ways to send the payload with PUT. You can make it part of the Query String attached to the end of the URL or you can include it as part of the post like you would with a POST. With something long like an entire wiki page, you would definitely want to use the post approach and not as part of the URL. The URL itself is really short, most of it is in the CURLOPT_POSTFIELDS. The examples from the API that Robert linked to show the body of the cURL as the -d option rather than in the URL itself.

In my PHP, I have something like this for a PUT statement. I'm using JSON to encode the object. This is pieced together from several different blocks, so hopefully I got it all.

// $options is the JSON object to send
// $url is the URL
// $token is the API access token
$data_string = json_encode( $options );
$http_headers = array (
  'Authorization: Bearer ' . $token,
  'Content-Type: application/json',
  'Content-Length: ' . strlen( $data_string )
);
$ch = curl_init( $url );
curl_setopt_array( $ch, array (
  CURLOPT_HTTPHEADER => $http_headers,
  CURLOPT_RETURNTRANSFER => TRUE,
  CURLOPT_VERBOSE => FALSE,
  CURLOPT_HEADER => TRUE,
  CURLOPT_CUSTOMREQUEST => 'PUT',
  CURLOPT_POSTFIELDS => $data_string
) );
$response = curl_exec( $ch );‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

You might also need to add a header of 'Accept: application/json' and there is obviously some additonal logic that needs included, I was just trying to get the part about the PUT statement to work.

[James]

At some point, it's going to look like I added a comment to an already solved problem. It wasn't solved when I wrote it, but it got hung up in the dreaded moderation.

At this point it doesn't add much and I could delete it, but it shows how to JSON encode the information, so I'm going to leave it there in case someone stumbles across this thread in the future.

I'm glad you two were able to get it worked out.

[nschutz]

James, If I unmark the correct answer will it speed up the moderation?

[robotcars]

Nan, unfortunately it doesn't. Moderation has nothing to do with the correct answer and is simply what happens when people post with the Syntax Highlighter option, making code in a post look nicer by adding line numbers, fixed width font, and semantic highlighting. It delays what would often be much quicker solutions or more rapid conversation. It's just a 'security'* feature of posting code to a public forum that requires human intervention. It a necessary evil, simply because not every contribution should be trusted, and not every user will vet code before executing it. It really hurts on weekend posts, because the moderators (who are awesome), often have to catch these during their off time.

* It's not really secure, because there is nothing preventing someone from pasting code into a post without the highlighter, just as you did in your original post.

[James]

Robert explained it well. There is some kind of mystery, voodoo logic behind it, but it seems very sporadic when it is applied. I've noticed if there are lots of external links in it, then it seems to get triggered more often. It sometimes seems that the more urgent or timely the response is, the more it gets held in moderation. It definitely feels like the more high-quality the response, with links detailed explanations and example code, the more likely it is to get moderated. As such, Robert has been hit by the moderation bug many times.

When moderation happens, we need to wait for someone to manually release the post. Depending on where the message is posted, different coaches have access to the moderation queue. For example, Kona can release some areas but not Canvas Developers. If she's on her phone instead of a computer, then the moderation queue doesn't show up for her.

[evanchitger]

Under Apache, the limit is a configurable value, LimitRequestLine. Change this value to something larger than its default of 8190 if you want to support a longer request URI. Extremely long URLs are usually a mistake. If you keep URLs under 2000 characters , they'll work in virtually any combination of client and server software. URI actually have a character limit depending on several things. Chrome limits url length of 2MB for practical reasons and to avoid causing denial-of-service problems in inter-process communication. On most platforms, Chrome's omnibox limits URL display to 32kB ( kMaxURLDisplayChars ) although a 1kB limit is used on VR platforms. IE - 2083 characters, Firefox - 2047 characters, Safari 80000 characters and Opera 190,000 characters.

To resolve the problem :

• By POST request: Convert query string to json object and sent to API request with POST
• By GET request: Max length of request is depend on sever side as well as client side. Most webserver have limit 8k which is configurable. On the client side the different browser has different limit. The browser IE and Safari limit to 2k, Opera 4k and Firefox 8k. means the max length for the GET request is 8k and min request length is 2k.

If exceed the request max length then the request truncated outside the limit by web server or browser without any warning. Some server truncated request data but the some server reject it because of data lose and they will return with response code 414.