[ARCHIVED] Trying to understand more about JIT provisioning, SIS CSV imports and "sticky" fields

[univsys]

I thought I'd ask the community if you know the answers to these questions at the bottom of this post.

We use both JIT provisioning and a user SIS CSV upload. We're considering only doing an hourly user upload, as we're told below that running with both can cause issues.

Our biggest issue is with name changes. It doesn't seem like the JIT provisioning is updating existing user account information, but rather is creating a whole new user without an SIS ID. Since the SIS ID is the "key" I'd expect it to update the login, first name, last name, email, etc when the user logs in with their new account that has the same SIS I as an older one.

I had submitted a ticket to canvas asking if adding the SIS ID to our JIT provisioning would help with these name changes. For some reason when we set this up originally, they couldn't figure out the SIS ID field in the claims. From this previous ticket, a tech responded with the following:

I believe it may help this case to understand how the SIS Import process works with regards to Users, then compare this to how your other systems are provisioning Users in other cases.

To begin, your understanding of 'sticky' fields is correct. It is important to note that the login_id field is also sticky, so if it is ever modified in the UI or through the API, a normal SIS Import cannot affect it. You can send an import with 'Override UI changes' to override these, and can add 'Clear UI changed state' to those types of imports to also 'clear' that sticky status so normal SIS Imports can affect those fields again.

The SIS ID ('user_id' in the files) is the identifier we use to distinguish a User account. If this value ever changes, you should consider that an entirely new User. If a User (or a login set on a User) is deleted with a SIS ID, that SIS ID is still 'in use' by that deleted User. Any file that contains that SIS ID again with an 'active' state will attempt to return them to an active state (assuming the login_id is available). As long as it is set on any User (active or deleted) it cannot be set for a different account. It is globally unique.

Meanwhile the login_id field is only globally unique for active Users. A deleted User could still have a specific login_id set without causing conflict on any active users, as long as you are not trying to restore that deleted User. Similarly, you could not apply a login_id on a User that is already in use by another active User. This is why you have the 'An existing Canvas user [..] has already claimed [..]' error, indicating the login_id is already claimed.

As an addendum to the above, you can 'claim' User accounts with a SIS Import as long as the login_id matches and they have no SIS ID. If an import attempts to send data and the SIS ID is not in use, it will first look for an active User with the requested login_id. If one exists, and they do not already have a SIS ID, it will apply the SIS ID to that User and 'claim' them. You've likely seen this process in action, based on your descriptions.

When you mention your ADFS system generating Users, I'm assuming you're referring to Users created using Just in Time Provisioning. We would typically not recommend using this when data is being provisioned with another method. I'm not entirely certain if JIT provisioned Users count as an API generation, or if their fields end up being sticky. If files are being sent every hour, I think I would rather Users need to wait an hour for the imports to provision them, rather than needing to clean up conflicts between these two systems.

If the JIT provisioning can create the Users with the correct SIS ID, the next SIS Import should recognize that new User with the SIS ID and attempt to make any necessary changes. If the file has a different name or login_id than what was set by JIT provisioning, it would attempt to update those. If JIT provisioning fields are sticky, they would not be modified, and the login_id would remain as what was set through that system. If JIT provisioning does NOT create sticky fields, it would modify the login_id and name to be whatever was in the file.

JIT provisioning will create these types of conflicts. If we assume the User already exists ('asmith', SIS ID 12345) and JIT provisioning creates a new User with a new login_id ('ajohnson'), these are now distinct Users. It wouldn't matter if the original User was active or deleted. When the next SIS Import ran to also make that login_id change (eg 'ajohnson', SIS ID 12345) it would fail with the 'An existing Canvas user [..] has already claimed [..]' error. The login_id would already be in place.

I am not entirely certain what the JIT provisioning system would do if you send a 'new' login_id and an existing SIS ID, but I would suspect it should attach to that User. Depending on whether JIT provisioning changes are considered API changes (and thus sticky), you still may or may not want this to occur.

I hope this answers the questions you have concerning how the SIS Import system works and how it likely interacts with JIT provisioning. If you have any other questions I may have missed, let us know and we'll be glad to assist you further.

----

We fixed the ADFS provisioning so that it now includes the SIS ID. However, we're still experiencing issues.

Anyone know the answers to the following?

• If a user account is JIT provisioned, are these considered API changes, and therefore "sticky"?

• If a user account is created through the SIS CSV process, the"sticky" fields mentioned here (https://canvas.instructure.com/doc/api/file.sis_csv.html) do not become "sticky" unless they are edited via the UI, correct?

• Lets say a user account is provisioned by the SIS CSV import, then - months later - their name changes, email address and username. If the user goes and logs into ADFS with their new login *before* a SIS CSV import comes in to update their information, the JIT provisioning will do which of the following?
  • It would recognize that the SIS ID (user_id) exists in the system already and update their user account with the updated ADFS properties (first name, last name, add a login, email, etc)
  • It would not find the login in the system, create a *new* user record, and then fail to associate the SIS ID (user_id) with the *new* account because it is already tied to another active login.

• In the paragraphs above, James mentioned the "override state changes" and "clear UI changed state" parameters. I think he was referring to the API params override_sis_stickiness and clear_sis_stickiness. We don't want to clear the stickiness of certain fields, like the first name or display name of the student. If they changed that in canvas, we want that to remain. Can we send in these un-stickify parameters and clear the stickiness of certain fields? Or does this clear the stickiness for the whole user record?

• I noticed that there is an integration_id secondary unique identifier for the user record in the CSV file format. Would throwing the SIS ID into that field in addition to the user_id help at all?

Thanks in advance!

Steve

[kona]

 @univsys , greetings! I'm not sure of the answer to this question, but I'm sharing it with the Canvas Developers‌ group in the Community to see if they can assist. You might consider joining this group so you have access to their resources and information. 

Kona

[ColinMurtaugh]

Hi Steve, 

This isn't really a direct answer to your questions, but we chose not to allow users to change their names or official email addresses in Canvas, partly due to some of the issues that you mention.  If changes can happen both in Canvas and upstream in the SIS, it becomes difficult (or maybe even impossible) to determine which data should win. We decided that our SIS should be the source of truth, and we direct our users to go there to make changes (and let them flow into Canvas via our regular feed process). We do let users set their preferred email address (for receiving notifications) directly in Canvas. 

We also chose not to use JIT user provisioning, but that was because we wanted to be sure that we could reliably create enrollments via SIS import, and enrollments can only be created if the referenced user account already exists. If we let users create their own accounts via JIT provisioning, then teachers wouldn't see an accurate list of the students in their class until all of the students had logged in. That seemed like a real source of confusion that we wanted to avoid.

I'm also interested in learning if and how other institutions are using JIT provisioning!

--Colin

[univsys]

Hi Colin,

Thanks for that info! I've since learned from our Canvas admins that we also don't allow students to change that information, only their personal email address. We also want to use our SIS as the system of truth, all name changes need to trickle down from there.

We're using both JIT provisioning and the SIS user file imports. If the student is "accepted" at our school (University), they get an account that day. I think that the idea revolves around there being an extra "hook" for the student to feel connected to the university. They can log into their email, browse classes and log into canvas. They're not enrolled in anything, but there are some open enrollment classes that they can see in Canvas. For instance, I think we have a course about "phishing" from IT that they can check out.

Our JIT provisioning also includes their SIS ID, so when the time comes for enrollments to come in for them - it finds them and enrolls them without a problem. We store the SIS ID in their ADFS account, and we map it to the SIS ID in the provisioning settings. You may be thinking about allowing users to create their own canvas accounts? We have canvas authentication turned off as well.

I got a reply back from Instructure support which basically said "We're not sure how using JIT AND SIS Imports can affect one other, and we don't have the test structure in place to test that ourselves, so you should give it a try and test it yourself". After thinking it over, the goal is to un-sticky these user fields. If doing an SIS file import with the clear_sis_stickiness flag will clear any stickiness - then it doesn't really matter what ADFS JIT is doing. We'll just clear it as often as we need to. They did report back that sending in that flag will only clear the stickiness of the fields that we are sending IN with the file. Which is what we want.

So our plan moving forward is to run a single file user.csv import hourly with those clear_sis_stickiness flags.

Our current issue is basically the following scenario:

1) User submits a name change

2) Change is made in SIS

3) Change is communicated to System Admins

4) Update is made in ADFS

5) User logs into their new account BEFORE an updated user.csv is uploaded, OR their user information was stickied and not updated in canvas

6) JIT provisioning occurs for the newer account username. An error occurs when trying to assign the SIS ID because it already exists with a previously imported user (their old account)

7) User cannot see their classes with their new account. This is because they're all associated with their old account which has the SIS ID number

😎 User submits ticket to IT

9) IT goes in and does an account "merge" in Canvas.

10) IT removes the SIS ID from the older login. Then deletes the older login. Then adds the SIS ID to the newer login.

11) User is all fixed!

During a name change, the name/username/email changes in advance of the ADFS update. If we upload a file hourly, its likely that it will be updated in Canvas before the ADFS change happens.

If the user tries to log in with their older account before the update occurs in ADFS, the worst that can happen is that a JIT provisioned user will be created without an SIS ID with their old credentials. Which much better than the other way around. Once their name change occurs in ADFS and they log in with their new account, things should be already switched over.

[jsavage2]

 @univsys ‌,

This is really the crux of it: 

I got a reply back from Instructure support which basically said "We're not sure how using JIT AND SIS Imports can affect one other."

When you use JIT, you're giving your authentication system, not Canvas, the power to decide what is a user, and what is a new user. And different auth systems might yield different results...you could probably convince a custom LDAP server to do what you want, for instance.

For ADFS, though, the primary identifier is the AD username. You're passing the SIS id, but that's an attribute, not an identifier. So the information flow goes something like this:

ADFS->Canvas: does user "sstratoli" exist?

Canvas->ADFS: "No"

ADFS->Canvas: "Then create a new user: sstraoli"

Canvas->ADFS: "OK"

ADFS->Canvas: "Good. Now give that user these attributes: first name: steve; last name: stratoli; sis_is: xxxxxxx"

Canvas->ADFS: "Sorry, Steve, I can't do that; user already exists for sis_id: xxxxxxx"

Without a specific directive to merge, Canvas don't know whether it's a name change or a legitimate data error, so the record gets left in an ambiguous state: The new record is created sans sis_id, and the old record is left untouched.

In an ideal world, the best thing would be to spawn a second process from the SIS to proactively change the ID in Canvas via API *before* (or as) is propagates to AD.

--j

[univsys]

Yep, that is what I suspect is happening. It makes sense because the SIS CSV import returns the same type of error.

1. User is created with SIS CSV import
2. User's login is changed.
3. User logs in with newer account. JIT provisions a new account without an SIS ID number.
4. SIS CSV import tries to import with the newer username and existing SIS ID tied to the older account. Can't, because it's already tied to the older user.

In this case, a merge is required, and the SIS ID has to be removed from the old login, delete the login, and then added to the new login.

Or, the user was originally provisioned with ADFS JIT, which might use the API (unconfirmed), which stickies the user fields, which means that in subsequent user imports the user data isn't updated / overwritten.

My running theory is that if the name change takes place, login info is updated, and a new SIS CSV file is uploaded *before* the user logs in and JIT provisions a new account - the existing user should have their user fields updated with the newer login, name, etc. Going to have to test this to confirm.

[ken_i_mayer]

I'm about to go into a meeting about a similar issue. We've set our Canvas instance to allow people to change their names in it. However, we are getting a few complaints from users who have formally changed their names in our SIS but the new names are not importing into Canvas. Our testing is showing the expected behavior, that the Canvas names are only sticky if they have been manually changed within Canvas. If the user has never manually changed the name in the Canvas GUI, the updated name in SIS ports over with the SIS import.

Is there any way to use the API to determine if a user has changed their name? In other words, is there any way to determine which fields in a user's profile are sticky? I'd like to walk into this meeting and say, "Your use case only affects X% of our users who have changed their names in Canvas."

[univsys]

Hi Ken,

So I've been doing some digging to chase down another issue, where the names in our AD don't match the official names in our SIS. We have a process for creating the accounts, but not really one for name changes.

There is a users report that you can run in Canvas. Settings > Reports Tab > Provisioning > Configure Button - select users.csv. That'll give you all of the users.

Then you can compare that list with your SIS.

We're running an hourly demographic import with just the users file to keep things in sync. The push is done through the API with the following flags:

import_type=instructure_csv&override_sis_stickiness=true&clear_sis_stickiness=true

which clears any stickiness on the user's account in the fields being imported. HTH.

[ken_i_mayer]

Thank you--this is very helpful!