I don't know if this is directly responsive to your question @c_russell (please Accept as Solution if it is), but Canvas offers "Moderated grading" in SpeedGrader.

When creating an assignment, you can select moderated grading, which allows you to pick the number of graders and who are the graders.