Is there a way to return only certain fields through an API url request?

[salmonsm]

Hello,

I am trying to pull simple reports that go beyond Canvas' built-in reports- so I have been working with the API. I have a script I can use to pull what I need, but it's everything- I want to narrow it down to login_id, user_id, and name.

I see that I can deliberately include fields with include[]=fieldname. I don't suppose there's a way to use those additively- that is, only specify the fields I want?

This is the node.js script I am using:

const request = require('request');

let url = 'https://<my domain>.instructure.com:443/api/v1/courses/12/enrollments?type[]=StudentEnrollment&state[]=active&access_token=<my key>&page=5&per_page=100 rel="next"';
 
request(url, function (err, response, body) {
 if(err){
 console.log('error:', error);
 } else {
 let results = JSON.parse(body)
for (var i in results) {
 console.log(results[i]);
 }
 }
});‍‍‍‍‍‍‍‍‍‍‍‍‍‍

as you can see I am just manually paging through. If there is way to do this automatically, I would be very interested in that too.

My end goal is a report with the fields mentioned, in a format I could share with instructors.

Any help would be appreciated, thank you.

[robotcars]

Michael,

I don't think there is a way to ignore fields from the response unless they write the API to do so. You can specifically call the values or access them by element/key.

var json = '{"one":1,"two":2,"three":3,"four":4,"five":5,"six":6}';
results = JSON.parse(json);
console.log(results.one);
console.log(results.three);
console.log(results['four']);
var res = [results.two, results.five, results.six] 
console.log(res)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Automatic paging can be done with loops, here's an example on Lines 159 - 198. The basics to loop through pages, asking for all the records possible (100 per page), while the Link Header says there are more pages (next). Line 175, will show the easiest way to access the individual fields and forget about the rest of the response values, which is what you've already started with. You can also wrap that loop in a loop that works through an array of course id's. Looks good, keep going.

Check these out too:

request - npm - forms 

request - npm - http headers

request - npm - oauth signing

csv - npm 

https://www.npmjs.com/package/request#custom-http-headers

[salmonsm]

Thanks for the encouragement! I will definitely check out your links!

[stuart_ryan]

Hi  @salmonsm ,

I agree with carroll-ccsd, there is no way to filter the fields that are returned.

I am curious as to why you would like to do so, with the examples Robert has provided, you can simply filter out the data to only what you need.

Would love to hear more, also, have you considered Canvas Data for what you are attempting?

Cheers,
Stuart

[salmonsm]

Thank you for your reply.

Why am I doing this? To answer instructor questions like "can you provide a list of the first and last name and email address of everyone enrolled in course X?"...  this seemed like a straightforward way to at least start doing it. 

I don't have access to Canvas Data as it stands- at least it's not listed. If that is the best way to conduct ad hoc queries like this, I'll ask about it. Thanks for that tip!

As I can take the data gleaned from my script and massage it with search and replace,  it's processing that could take place in a script. That's where my thoughts went, anyway...

[James]

 @stuart_ryan , it wasn't my question, but let me try to address this question of yours:

I am curious as to why you would like to do so

Some of the APIs return a lot of information that is not needed. For example, fetching assignments returns the description (body) of the assignment and if it's a discussion, it returns it twice, once for the assignment and once for the discussion. Perhaps I'm only interested in the due dates and published status (say for my Adjust All Assignment Dates on One Page Google Sheet), but I still have to download all of the information about the assignment.

If you use equations in your discussions, then it seems to call some kind of microservice that can really slows things down. It was happening to the point that the API was timing out, and that with just 8 discussions. There was a thread here in the Community about it a couple of weeks ago about this API Timeout . 

If you could get the information about the assignment without the text of the assignment, then it would conceivably be quicker to download and wouldn't time out.

In my API libraries I write, I have been known to add another parameter to the API call that returns just the variables you want. I think part of that had to do with trying to cache the data downloaded and Google Sheets having a limit on how much you could cache. In other languages, it just might have been to reduce memory footprint and not save the full download.

GraphQL may be able to provide some of this functionality.

[stuart_ryan]

 @James ooooh very interesting, I hadn't thought about it like that. I can certainly see how having a filter could be beneficial. I guess I had become so used to doing the filtering post getting the results. I can see how there would be some good benefits in doing so before-hand.

I do remember the API timeout discussion, would be interesting to see if filtering would improve that (likely would depending on what data you wanted to pull back).

Love your work!

[James]

 @stuart_ryan , This issue might also be popping up here: Assignments Tab freezes when trying to scroll 

[stuart_ryan]

Hmm I see your point. Loading less could certainly help in that circumstance too!

[clawson]

Another answer to the 'Why' question is that we are building a solution in AWS to pull roster data and get charged for volume of inbound data. All we need is the Course Id and Course Name for our purposes but are required to pull down the full course object. We pay for retrieving data that we don't care about.

Despite the point about it being unreasonable to expect the browser to adapt to different responses, Blackboard provides the very same functionality that Michael, James and I would appreciate in Canvas.

learn/api/public/v1/users/userName:clawson/courses?fields=courseId,name

[stuart_ryan]

OOOh  @clawson , a very interesting point as well! That (at high volume) could significantly reduce the information transferred and cost. 

The more I am hearing about this, the more I am thinking it might be a great feature idea to be logged to the community. It certainly seems there are some great use cases for it!

[robotcars · Accepted solution]

There should be no need to process anything with PHP if you're using Node.

The API is working as desired, in that a REST API result is a record that sends a predefined set of values back to the requestor. If the API allows filtering the results thats extra. But by default a record contains fields, those fields are being returned. Anything else would be like telling your browser to exclude loading the header or footer from every web page you load... or maybe only from specific sites. But then your browser would have to learn how to take that request and Web sites would have to add that as a feature...

With the links I provided above, you should be able to produce a CSV file with the following headers and values.

course_id, login_id, user_id, name

[salmonsm]

I was live editing my response there. I did mention PHP but you replied right after I thought better of it...  But thank you, this reply really does provide the direction I need.