CANBASH: Canvas & BASH Scripting
- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
Canvas has a great GUI (graphical user interface). However, sometimes even a great GUI can't beat the speed and efficiency of a CLI (command line interface) for automating mundane tasks, gathering data, and leveraging that data in meaningful ways that allow you to do things you wouldn't be able to. Mostly less clicky-click copy-pasta.
Hi, my name is Kent Gruber and I am a Student and Course Designer at Eastern Michigan University. I'm not totally convinced that is my actual, official, proper title thing - so, forgive me if I am wrong about that. But, in any case. Most of my job at Eastern consists of (manually) migrating content from one LMS platform to the new Canvas LMS. So I do a lot of creating modules, module items, content pages, formatting stuff. You know, the typical build-a-course workshop kind of things.
After a while of doing the same sort of tasks all the time, I figured: "The has to be a better way of doing this." So I started poking around and found out the Canvas API was a thing. So, how do I combine the API things with things I know how to do from the command line? And that is what I found out.
In this blog post I will be using the OSX Bourne Again Shell (BASH) as my command line interface to interact with the Canvas API to create these little things called BASH Scripts. Doing this we can create, delete, modify, and request data from Canvas just like we would using the GUI and clicking around on buttons as we navigate through the web browser to see what we want to do.
Now, I feel like I should preface a little bit more before I continue: The way I am doing things may - and probably isn't - the best way to do what I am trying to accomplish. I'm sure the scripts could be much more eloquent or faster if I were creating these tools in another language. But, bash me for my BASH if you have to - or, if you don't know how to write shell scripts of any kind... it's not that hard. In fact I think it's a great place to start! Because you can great meaningful pieces of code relatively fast. So, that's at least one benefit I see. Also, I should say I am not a professional programmer, systems what-cha call it or anything of the sort. I'm familiar with these things and try to expand my knowledge the best I can. So, most importantly: somethings I may say might be totally wrong and I am open to corrections or any insight someone may have.
With that out of the way, what do?
Below is an example of a curl command to interact with the Canvas API to list all the modules in a sample course with a course ID of 1234 (Side note: most of the stuff I am using as my examples are listed in the Canvas API documentation with a lot of information you should go check out):
Following "$" are commands I am running in my terminal/shell that actually execute the commands.
$ curl -H 'Authorization: Bearer <token>' \
https://<canvas>/api/v1/courses/1234/modules
If you're not familiar with curl: Curl is a tool used in command lines or scripts to transfer data. And we can use curl to interact with canvas. While using curl commands as the basic framework to build these bash scripts we can do a lot of really powerful things.
But in order to use curl you'll need to generate yourself an access token in Canvas. Warning: access tokens are pretty much always sensitive data. It's sort'of like your login and password but in a long alpha-numeric string that is randomly generated by Canvas (or at least I assume randomness). Please check out this guide to creating an access token if you don't know how to already.
Now if you place your token where it says <token> and removing the greater-less-than symbols. Then you change the <canvas> part to your institutions canvas URL you could probably get a response like this with what you're asking from the API:
{"errors":[{"message":"The specified resource does not exist."}],"error_report_id":123456}
What the heck is that? It's a JSON response from Canvas basically saying "You're silly. That doesn't exist Here is a message telling you that and an error report ID for you." Now, if you were to give it to an actual course ID you may get a response like:
[{"id":86706,"name":"Module 1","position":1,"unlock_at":null,"require_sequential_progress":false,"publish_final_grade":false,"prerequisite_module_ids":[],"published":false,"items_count":0,"items_url":"https://<canvas>/api/v1/courses/1234/modules/86706/items"},{"id":86707,"name":"Module 2","position":2,"unlock_at":null,"require_sequential_progress":false,"publish_final_grade":false,"prerequisite_module_ids":[],"published":false,"items_count":0,"items_url":"https://<canvas>/api/v1/courses/1234/modules/86707/items"},{"id":86708,"name":"Module 3","position":3,"unlock_at":null,"require_sequential_progress":false,"publish_final_grade":false,"prerequisite_module_ids":[],"published":false,"items_count":0,"items_url":"https://<canvas>/api/v1/courses/1234/modules/86708/items"},{"id":86709,"name":"Module 4","position":4,"unlock_at":null,"require_sequential_progress":false,"publish_final_grade":false,"prerequisite_module_ids":[],"published":false,"items_count":0,"items_url":"https://<canvas>/api/v1/courses/1234/modules/86709/items"},{"id":86710,"name":"Module 5","position":5,"unlock_at":null,"require_sequential_progress":false,"publish_final_grade":false,"prerequisite_module_ids":[],"published":false,"items_count":0,"items_url":"https://<canvas>/api/v1/courses/1234/modules/86710/items"}]
Well, for one, that's a lot of crap that's kind of hard to read. Especially the raw response from the curl request in the command line itself even looks a little less pretty than that. As far as I know this is pretty typical of a JSON response - and that's actually a good thing for us to receive these messages in this format because we can use tools already out there to help make this look a little more reasonable to our mere human eyes to behold.
In this case I will be using a Python tool called "json.tool" that I believe comes with Python 2.6 and up. Since I am using OSX, Python comes already installed. We can check what version you're running by running this in your terminal:
$ python --version
And in my case I am using "Python 2.7.6" for this blog post.
Now I'm sure you don't want to have to copy and paste curl commands all the time when you're trying to work in all sorts of different courses. In a BASH Script there's something called "read" which will prompt the user and then store the response in a variable that can be passed along later in the script.
Below is a very simple bash script that prompts the user for the course ID they want to list the modules for and then calls it in the curl command that will then print the information on the screen as well as generate a .txt file that we can read or use later on in Canvas life.
I will number each line of the script and tell you what is going on for each line:
Our module script:
1. #!/bin/bash
2. #CANBASH: The demo's in the details."
3. #Prompt user for the course ID they want to work on.
4. read -p "[ ? ] COURSE ID: " COURSE
5. #hardcoded auth token value: a1b2c3d4e5
6. curl -o modules.txt -H 'Authorization: Bearer a1b2c3d4e5' \
https://<canvas>/api/v1/courses/$COURSE/modules
7. cat modules.txt | python -m json.tool >> modules.txt
8. cat modules.txt
What the script is doing:
1. Tell the computer it's a bash script -- the start of (almost) every bash script.
2. # Tells us that it is a comment line. I use these to help remember what the heck is going on in long bash scripts.
3. Actually says what's going to happen in line 4.
4. "read" will read what the user enters
5. A note about the auth token I am using
6. the actual curl command with a necessary "\" to work correctly.
7. This will do the actual the actual json "prettying" that will make it readable and store it back into modules.txt
8. cat is a command in bash that is used to read the contents of a file. This will print the content of modules.txt in this case.
Pretty cool, right? The response back from Canvas will be a JSON response that will probably look a little something like this:
[{"id":86706,"name":"Module 1","position":1,"unlock_at":null,"require_sequential_progress":false,"publish_final_grade":false,"prerequisite_module_ids":[],"published":true,"items_count":0,"items_url":"https://<canvas>/api/v1/courses/1234/modules/86706/items"},{"id":86707,"name":"Module 2","position":2,"unlock_at":null,"require_sequential_progress":false,"publish_final_grade":false,"prerequisite_module_ids":[],"published":true,"items_count":0,"items_url":"https://<canvas>/api/v1/courses/1234/modules/86707/items"},{"id":86708,"name":"Module 3","position":3,"unlock_at":null,"require_sequential_progress":false,"publish_final_grade":false,"prerequisite_module_ids":[],"published":true,"items_count":0,"items_url":"https://<canvas>/api/v1/courses/1234/modules/86708/items"},{"id":86709,"name":"Module 4","position":4,"unlock_at":null,"require_sequential_progress":false,"publish_final_grade":false,"prerequisite_module_ids":[],"published":false,"items_count":0,"items_url":"https://<canvas>/api/v1/courses/1234/modules/86709/items"},{"id":86710,"name":"Module 5","position":5,"unlock_at":null,"require_sequential_progress":false,"publish_final_grade":false,"prerequisite_module_ids":[],"published":false,"items_count":0,"items_url":"https://<canvas>/api/v1/courses/1234/modules/86710/items"}][
{
"id": 86706,
"items_count": 0,
"items_url": "https://<canvas>/api/v1/courses/1234/modules/86706/items",
"name": "Module 1",
"position": 1,
"prerequisite_module_ids": [],
"publish_final_grade": false,
"published": true,
"require_sequential_progress": false,
"unlock_at": null
},
{
"id": 86707,
"items_count": 0,
"items_url": "https://<canvas>/api/v1/courses/1234/modules/86707/items",
"name": "Module 2",
"position": 2,
"prerequisite_module_ids": [],
"publish_final_grade": false,
"published": true,
"require_sequential_progress": false,
"unlock_at": null
},
{
"id": 86708,
"items_count": 0,
"items_url": "https://<canvas>/api/v1/courses/1234/modules/86708/items",
"name": "Module 3",
"position": 3,
"prerequisite_module_ids": [],
"publish_final_grade": false,
"published": true,
"require_sequential_progress": false,
"unlock_at": null
},
{
"id": 86709,
"items_count": 0,
"items_url": "https://<canvas>/api/v1/courses/1234/modules/86709/items",
"name": "Module 4",
"position": 4,
"prerequisite_module_ids": [],
"publish_final_grade": false,
"published": false,
"require_sequential_progress": false,
"unlock_at": null
},
{
"id": 86710,
"items_count": 0,
"items_url": "https://<canvas>/api/v1/courses/1234/modules/86710/items",
"name": "Module 5",
"position": 5,
"prerequisite_module_ids": [],
"publish_final_grade": false,
"published": false,
"require_sequential_progress": false,
"unlock_at": null
}
]
All of that first line stuff is stuff that we probably don't want for our case. So, we can turn to a little piece of BASH magic called "Sed". Sed is a command line tool that parses and transform text. We can use sed to get rid of the first line. And this is where I've had trouble in the past to find a really clean solution to this. And it could simply be the case I don't know what I am doing as much as I wish I do. But, in order to get rid of the first line with sed I use echo to run a command and then store that back into modules.txt like so:
1. #!/bin/bash
2. #CANBASH: The demo's in the details."
3. #Prompt user for the course ID they want to work on.
4. read -p "[ ? ] COURSE ID: " COURSE
5. # hardcoded auth token value: a1b2c3d4e5
6. curl -o modules.txt -H 'Authorization: Bearer a1b2c3d4e5' \
https://<canvas>/api/v1/courses/$COURSE/modules
7. cat modules.txt | python -m json.tool >> modules.txt
8. echo "$(sed '1d' modules.txt)" > modules.txt
9. cat modules.txt
I know it may look a tad bit screwy -- but, as you'll find out if you ever decide to start bashing your head against BASH scripts then you will find out BASH is pretty finicky sometimes. All of the sometimes basically. And in some cases BASH on OSX will have differences than BASH on other operating systems. Thankfully there's lots of information and support online.
Either way, now we have a response that looks much nicer!
So if we want to run that, the JSON response willl be as follows:
{
"id": 86706,
"items_count": 0,
"items_url": "https://<canvas>/api/v1/courses/1234/modules/86706/items",
"name": "Module 1",
"position": 1,
"prerequisite_module_ids": [],
"publish_final_grade": false,
"published": true,
"require_sequential_progress": false,
"unlock_at": null
},
{
"id": 86707,
"items_count": 0,
"items_url": "https://<canvas>/api/v1/courses/1234/modules/86707/items",
"name": "Module 2",
"position": 2,
"prerequisite_module_ids": [],
"publish_final_grade": false,
"published": true,
"require_sequential_progress": false,
"unlock_at": null
},
{
"id": 86708,
"items_count": 0,
"items_url": "https://<canvas>/api/v1/courses/1234/modules/86708/items",
"name": "Module 3",
"position": 3,
"prerequisite_module_ids": [],
"publish_final_grade": false,
"published": true,
"require_sequential_progress": false,
"unlock_at": null
},
{
"id": 86709,
"items_count": 0,
"items_url": "https://<canvas>/api/v1/courses/1234/modules/86709/items",
"name": "Module 4",
"position": 4,
"prerequisite_module_ids": [],
"publish_final_grade": false,
"published": false,
"require_sequential_progress": false,
"unlock_at": null
},
{
"id": 86710,
"items_count": 0,
"items_url": "https://<canvas>/api/v1/courses/1234/modules/86710/items",
"name": "Module 5",
"position": 5,
"prerequisite_module_ids": [],
"publish_final_grade": false,
"published": false,
"require_sequential_progress": false,
"unlock_at": null
}
]
We can further use sed to get rid of any extra nonsense we may not want by adding the following line under our first sed command:
echo "$(sed 's|["{},]||g' modules.txt)" > modules.txt
Then our JSON response gets even prettier. And I do like them pretty. And in this case the last bracket of the json response is left in there. I've yet to come across something where this has been a problem - for some reason it being left in there can actually make my life easier. So, it's become my friend.
Sometimes I just want to grab one piece of information from a JSON response though. So, I turn to another command called grep. Grep is a command used to search files or, for our case, .txt documents by searching for a particular pattern. So we can even add a simple grep functionality to the script by adding this bit instead of just printing what is all in modules.txt:
1. #!/bin/bash
2. #CANBASH: The demo's in the details."
3. #Prompt user for the course ID they want to work on.
4. read -p "[ ? ] COURSE ID: " COURSE
5. # stupidly hardcoded auth token value: a1b2c3d4e5
6. curl -o modules.txt -H 'Authorization: Bearer a1b2c3d4e5' \
https://<canvas>/api/v1/courses/$COURSE/modules
7. cat modules.txt | python -m json.tool >> modules.txt
8. echo "$(sed '1d' modules.txt)" > modules.txt
9. echo -e "$(sed 's|["{},]||g' modules$COURSE.txt)\n" > modules$COURSE.txt
10. read -p "Enter the search pattern (email, id, name, sis):" pattern
11. if [ -f "modules.txt" ]
12. then
13. result=$(grep -i "$pattern" "modules.txt")
14. echo "$result"
15. fi
If you're like me, most of my job dosen't really involve looking stuff up. In fact, if you want to make the grep command above more powerful, there are pretty good ways to do just that. But, I don't do it all that often. Instead, I tend to create content on canvas. And just like I've figured out how to store content from the Canvas API and parsing out the JSON response, I can create content by feeding variables into the curl command via a list-and-a-loop, get a response, save it, do what I want, ect.
So, the loop and the list thing.
A really nice way I've found of feeding a list to the curl command is by creating .txt documents with the content of what I want to create on its own line. I assume you could just put it in a .lst now that I am writing this blog post. But, in this case I will be using a .txt document because I know that works. And I already hardcoded that in my most of my scripts because I can share those .txt documents really easily.
First we need to tell our bash script to read by each line in the .txt file by changing our IFS (Internal Field Separator) to $'\n' which will help read things on each line. Then we read (via the cat command) each line with a simple for loop as follows that is from a script to create modules from a list:
# prompt user for course and list (note that for list you only need to put the filename, not .txt extension)
read -p "[ ? ] COURSE ID: " COURSE
read -p "[ ? ] LIST: " LIST
# store working directory to make things cleanish
canPWD=${PWD}
# change internal IFS to read each line
IFS=$'\n'
# setup to read each line in
for i in $(cat $canPWD/$LIST.txt) ;
# for each line in list do the following
do
# print the working i value and then do the curl
echo "$i ..."
# the tack hashtag will show a prettier progress bar and tack o will output to a .txt
curl -# -o $i\_modules$COURSE.txt https://<canvas>/api/v1/courses/$COURSE/modules \
-X POST \
-d "module[name]=$i" \
-H 'Authorization: Bearer a1b2c3d4e5'
This can be a pretty powerful tool when creating contents and it can be used to do anything with the API including reporting, analytics, and other searching or creating functions within canvas. As long as the API supports it, and you have the right privileges ( I don't. ) than you can do really interesting stuff straight from the command line.
And, yes there are downsides. You probably already see one if you're a windows user or aren't familiar with working from the command line. To work with canvas from the command line the way I've been showing you, then you need to have BASH on your system. I'm sure there are ways to do all the things I'm doing in other shell environments, but I don't know exactly because I haven't tested them or have all that much experience with them either. There are ports of bash to windows, but I don't know how good any of them are or if you will potentially run into more problems than it is worth.
You may also find when you're first starting to work with canvas from the command line that it may be useful to keep canvas open up in one windows and your terminal next to it in the other. In fact, this is typically how I work because there defiantly is a place and time for a GUI. Then by building the tools as you need them can help you remember how they work and what there limitations are.
Another down-side is the secret that the Canvas API actually isn't all that perfect. For example, I can feed a long list of modules to canvas via the curl command. And if I don't put a sleep (a point where the script will stop for a second if set to 1) then I can actually get some odd canvas behavior where the same module will share the same position value wither another. The modules will display normally when listed in the course GUI. But, when calling it back via the API, I will get a shorter list because it will only return (I believe) the second module for each position value.
So, for example module 1 and module 2 will have the same position value set as 1 when, but when calling it back it will only list module 2 in the json response. By adding that sleep in there it is possible to work around that problem. But it sacrifices speed by a second per module creation. And from the command line speed is pretty much a priority kind of thing 'ya know?
This is actually the first I am talking about this position problem in canvas to canvas (rather indirectly). Trying to get in their bug bounty program. There was also a problem I found where I could actually request more from the json response than I should be allowed to. And while I'm not sure if this is a general permissions flaw, a token generation thing or an Oauth2 flaw more specifically. Or, even funnier, I could be totally wrong about the whole thing. I'm still waiting to hear back from Canvas about this actually as their engineers are apparently taking a look at the problem or the problem was passed onto them. Something like that. And since that is the case I guess I won't go into it too much.
That doesn't mean the API isn't really cool. It is. In fact it has taught me a lot about Canvas because I can learn more of how canvas works. And by scouring the documentation and figuring how to jigg BASH to do what I want it to I've learned a lot and I know I can still do a lot more. Because I'm sure there are more efficient ways to do what I am doing. I just don't know. But I'm going to continue to look into it and use bash scripting to do API testing when I have time.
And hopefully you get your own ideas if you ever want to build your own bash scripts! In fact, to get started, here is a simple script I use all the time to create modules from a list that you can use as a template to help create your own.
#!/bin/bash
#(c) Kent Gruber (ignore this because it's to amuse myself)
#CANBASH: Create modules from list
clear
echo "
__ __ __
/ /\ |\ ||__) /\ (_ |__|
\__/--\| \||__)/--\__)| | v1 - MODULES"
echo "[ + ] Create modules for given course..."
echo "[ * ] Optionally keeps the data generated (pretty) json format in a .txt document"
read -p "[ ? ] COURSE ID: " COURSE
read -p "[ ? ] LIST: " LIST
# hardcoded auth token:
#store working directory for fun
canPWD=${PWD}
# CREATE MODULES, PARSE DATA, STORE, SEARCH, LOG IT.
echo
echo "[ + ] Creating modules ... "
echo
IFS=$'\n'
for i in $(cat $canPWD/$LIST.txt) ;
do
echo "$i ..."
# DO THE CURLING
curl -# -o $i\_modules$COURSE.txt https://<canvas>/api/v1/courses/$COURSE/modules \
-X POST \
-d "module[name]=$i" \
-H 'Authorization: Bearer <token>'
cat $i\_modules$COURSE.txt | python -m json.tool >> modules$COURSE.txt
cat $i\_modules$COURSE.txt | python -m json.tool >> $i\_modules$COURSE.txt
#turn off for some flooding action. Will mess with position calls.
sleep 1
done
echo
read -p "[ ? ] Enter the search pattern (email, id, name...):" pattern
echo -e "$(sed '1d' modules$COURSE.txt)\n" > modules$COURSE.txt
echo -e "$(sed 's|["{},]||g' modules$COURSE.txt)\n" > modules$COURSE.txt
if [ -f "modules$COURSE.txt" ]
then
result=$(grep -i "$pattern" "modules$COURSE.txt")
echo "$result"
fi
echo "[ ? ] Keep generated files [Y,n]"
read input
if [[ $input == "Y" || $input == "y" ]]; then
clear
touch id.txt
grep -i "id:" modules$COURSE.txt | awk '{print $2}' >> id.txt
cat $LIST.txt > mods.txt
paste id.txt mods.txt > report.txt
echo "[ + ] Kept"
mkdir $COURSE
mv *.txt $COURSE
else
clear
for i in $(cat $canPWD/$LIST.txt) ; do rm $i\_modules$COURSE.txt ; done
rm modules$COURSE.txt
echo "[ - ] Removed"
fi
Lastly I have a small demo video showing a general workflow and me using this script to create modules in a sandbox course at my university.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
