Bulk or Batch Updating URLs included throughout Canvas

[rsauls]

I am looking to batch update 1000s of URLs throughout Canvas, but I am not finding a process for this type of update.  Any advice?  The URL's may have been added in Canvas in a variety of areas and in a variety of ways.  Needing to find and replace pattern within the URL- i.e wanting to change abc.com/example to xyz.com/example.

[robotcars · Accepted solution]

Hi,

I do not have any code to share on this at the moment, and each use case depends on what you have to target. I can provide some guidance for the most efficient way I've figured out how to do this. Since there is no 'content search' in Canvas, the best place to get a current state of key content and descriptions of content in Canvas is through Canvas Data.

If you have Wiki Pages to update, you can consume https://portal.inshosteddata.com/docs#wiki_page_dim into a Db. Do a query for your string, which would return you the course and the wiki page id.

In your code, you would grab that content, replace the necessary string/url, then update the page with the API and repeat for all items found. https://canvas.instructure.com/doc/api/pages.html#method.wiki_pages_api.update

This can be repeated for Quiz, Discussions, Assignments

[dgrobani · Accepted solution]

I've written a Python script to do this for Canvas pages. It gets each course in one or more subaccounts, gets each wiki page in the course, does a search and replace on the page content, and if the content changes, writes the page content back.

Unfortunately, you'd need to have experience running Python scripts against the Canvas API for it to help you.

[robotcars · Accepted solution]

Hi,

I do not have any code to share on this at the moment, and each use case depends on what you have to target. I can provide some guidance for the most efficient way I've figured out how to do this. Since there is no 'content search' in Canvas, the best place to get a current state of key content and descriptions of content in Canvas is through Canvas Data.

If you have Wiki Pages to update, you can consume https://portal.inshosteddata.com/docs#wiki_page_dim into a Db. Do a query for your string, which would return you the course and the wiki page id.

In your code, you would grab that content, replace the necessary string/url, then update the page with the API and repeat for all items found. https://canvas.instructure.com/doc/api/pages.html#method.wiki_pages_api.update

This can be repeated for Quiz, Discussions, Assignments

[dgrobani · Accepted solution]

I've written a Python script to do this for Canvas pages. It gets each course in one or more subaccounts, gets each wiki page in the course, does a search and replace on the page content, and if the content changes, writes the page content back.

Unfortunately, you'd need to have experience running Python scripts against the Canvas API for it to help you.

[rsauls]

Can you please share the script?

[dgrobani]

I have to do some work on it to make it shareable. I should be able to post it tomorrow.

[dgrobani · Accepted solution]

OK, I've gathered into one file all my helper functions scattered among multiple modules, expanded the options for specifying the desired accounts and terms, tested it a bunch to make sure it still works, and uploaded it to my GitHub repo: py3-canvaslms-api/page_text_replace.py at master · dgrobani/py3-canvaslms-api.

After the import lines, there are seven parameters to specify, hopefully self-explanatory.

A reminder that this searches only wiki pages, not discussions, etc.

Do let me know if you have any questions. And if you're able to use it, please let me know how it goes. In any event, good luck!

[Pete5484]

@dgrobani can I just extend a huge thankyou for your help and contribution to this topic, over 3 years ago!

About a month ago, we were informed that a site that our courses link to (extensively) would very shortly become unavailable. The scale/impact of this was enormous.

A CD2 query identified that there were more than 13,000 links spanning thousands of course pages (including still active course copies, not blueprints). Each of these links required updating.

We ended up with a team doing manual updates, while simultaneously working on a modified version of your python program. The latter ended up being part of a multi-stage process that involved using CD2 output to scrape the distinct links, it then searched for and generated replacement urls (which were subsequently scraped to confirm they were valid/correct).

We modified your program to read in the thousands of rows containing course_id, page_url, original_link, and replacement_link as a dataframe grouped by course_id and page_url. This allowed iterating over each row in the group to replace every link on the page before calling update_course_page_contents. The dataframe was then written back to a new file with extensive logging to record the status of each link and the overall page. There's been extensive testing, including testing rollback plans.

On production, the bulk link replacement completed in less than 13 minutes.

2024-09-14 02:29:02,249 - INFO - Finished: updated 7176 links in 744.24 seconds (0:12:24.238409)

While we still have work to do (e.g. legacy SCORM and item banks in New Quizzes that can only be updated in the UI sigh), your contribution provided us with a huge head start on bulk updating links in page content.

Thank you!