How to get all data from Live API

[qnguyen]

Hi All,

I was trying to get User Page View from within Canvas of a particular student and the CSV only show 300 lines of data. I was told by Canvas to use Live API, I ran the   /v1/users/{user_id}/page_views but the data I got was even shorter, only 9 instances.

I know there is a way to get more than 300 instances but can't find that info any where.

Thanks in advance if anyone have any idea how to get this.

[James]

@qnguyen 

The Live API is just a user interface around the REST API to make it easy for people to test. What you're dealing with is the issue of pagination.

By default, most requests limit themselves to return 10 responses (perhaps you miscounted -- most software that displays it starts counting with 0 rather than 1). You can specify a query parameter of per_page=100 to get 100 at a time, but most requests won't support more than 100 at a time, even if you specify a bigger number.

There is a link header returned in the request that includes a link to the next page. It uses a bookmark for the page views (not the page=2 style used by other calls). You will need to make one request to get the link for the next request. You will need to take the link header and use that to request the next 100 entries and so forth.

The purpose of the Live API is to test and debug and pagination isn't really a part of that. You should use a programming language to make the API calls. Some people use Postman or another REST client, but you will still need to follow the headers to get additional pages.

[qnguyen]

James,

Thanks for your reply. Is there any posts by others mentioning how to work on these link header or Postman. I would like to use this Live API or Postman for this task. I used Postman and it only gave me that many responses.

As in this screenshot, I can't find a way to add or change that link header to create "per_page" parameter. Any additional info would be appreciated.

[jerry_nguyen]

@qnguyen 

You can set "per_page" as a parameter in Postman as follow

However, as James mentioned, this parameter will only return a maximum of 100 records, you will need to use a program to capture the Link header to request results for the next page (https://canvas.instructure.com/doc/api/file.pagination.html)

Thus to get the next page of results, you will need to get the next page bookmark parameter from the response header (of the previous request) and add it to the request params for the next request .

The Live API doesn't allow you to change any request headers or parameters.

Another way to get this data is through Canvas Data, you can run your own SQL query to get all the activities

[James]

@qnguyen 

Some people hear the word programming and they give up because they don't have those skills. I thought of a few ways that you can get this information without any programming. None of these are efficient and not something you would want to do on a regular basis, but it can be helpful in a pinch.

Live API

Under the Users API is the List user page views endpoint. It contains two parameters that may be more useful than forcing yourself to go back more than 300 pages. That is the start_time and end_time. What you could do is take the time from the last page view returned and use it as the start_time for your second request. Then take the time from the last page view returned and use it as the start_time for your third request.

The problem with the Live API is the limitation of 10 pages per request and that might cover as little as a few seconds of page views.

Postman

You can also use the start_time and end_time parameters in Postman. Very rarely do I want to know the last 300 page requests for a user. It's usually some specific time period where I suspect something is going on. I can use the start and end times to narrow in on that time. By using the approach above but with Postman and the per_page=100, you can avoid the Link header.

This is faster than the Live API because you can get 100 pages at a time. If you're going to use Postman, then using the link header is probably quicker than changing the start time.

Browser Developer Tools

In most browsers, I can press F12 to bring up the developer tools, but I'm going to use Chrome to illustrate.

By going to the Network tab and choosing Fetch/XHR, you can monitor the traffic that Canvas sends and then copy/paste it into a website that will convert JSON to CSV. You can then copy/paste that into Excel and quickly get more than 300 rows. This approach is good when you do want the most recent page views but you want to go back more than 300 rows.

I'm not advocating a site in particular, but a quick Google search found csvjson.com/json2csv that will allow you use tab as a separator and copy the contents to the clipboard.

Here is a video I made that illustrates the process.