Cannot send email notifications to students

[jcoehoorn]

I'm working on migrating from an old Moodle system to a shiny new (self-hosted) Canvas installation. I have a lot of things working, including importing courses, users, and enrollments from our SIS. One thing that is not working is the sending of any email notifications to these new users.

I want to be clear here: email works generally; I am able to send notifications to other admins. Additionally, I'm not only talking about the initial setup or invitation messages; this is any email notifications to students at all.

As an admin, I can log into my new Canvas account and use the Inbox to create messages for other admins. The other admin accounts will receive email notifications about those messages, and are able to reply using both e-mail and their Canvas Inbox. If I do the same thing for a test student user login, I can login as that user in a separate browser and the message will be there in the canvas inbox, but no email notification is ever sent. 

We have two primary accounts on this server (Site Admin, and one for "regular" institutional use), and I am listed as an admin on both of them.

I do see the correct email address for this user on their profile settings. It does show a star to indicate it is the default address, and there is no warning to confirm the address. I've been over the Notification settings, and I don't see anything wrong here. The correct email address also shows as the default address if I search for the test student user in the People section of that account as an admin.

I am also able to check the postfix mail logs on this server, and I can see the log entries there for the Canvas messages I sent to the other admins. But there are no postfix log entries, or even connection attempts, for the time periods where I've tried to send notices to the student.

Our admin accounts do have SIS_user_id values, but they were added later, so I'm not sure if the problem is because the users were first created by SIS, or because I'm missing some kind of role assignment, or if I'm missing something to mark the communication channel as confirmed. What could be going on here?

[kona]

 @jcoehoorn , greetings! I'm not sure of the answer to this question, but I'm going to share this with the Canvas Developers‌ group in the Community to see if they can help. You might consider joining this group to gain access to their resources and information. 

Kona

[jsavage2]

Hi  @jcoehoorn ‌,

Do you see any related entries in the postgres delayed_jobs table?

[jcoehoorn]

The table has 35 records, all with run_at times in the future.

I do see these records in the messages_* table for the correct week, where it shows the correct communication_channel_id. What's missing here is a value in the "from" column... but it's weird that other records are fine.

[jcoehoorn]

Update on this: I have a workstudy student, and I can see where a user record was imported for her from the SIS. This is a real user record with a real person behind it now, rather than one specifically setup for testing. We haven't announced this to students, but there's nothing exactly blocking them from reaching the new system, so I had her login. I can send her a message and she can see it in her canvas inbox, but she never gets an email notice. I also manually enrolled both her and myself as students in a test course. I get a course invitation message in my e-mail; she does not. 

Again checking the postfix logs, I can see the connection from canvas for my message, but no attempted connection at all for hers. I also don't see anything in delayed_jobs that might explain this.

For my next trick, I created a new Admin role,but with no special permissions, and added her as an admin in that role. This now shows on her user page as an Account enrollment rather than a Course enrollment. It didn't help, but at least I've narrowed down one option (admin or not).

We're hoping to start using this with a couple summer test courses with real students in about 2 1/2 weeks. It's gonna be a real pain if those instructors can't reliably message the students in their classes through the LMS. Pretty much everything else needed for a minimal launch is working.

I do have some raw access to the database (very limited: console psql over ssh only, where it's a pain to get results back to my workstation or browse the schema). I also happen to be very good with SQL. If I could even see what was different in the table data for her user, pseudonym, communication_channel, etc, records vs my own I could write a cron job to make the needed adjustments for all students automatically. Even better, that would likely point me to the right place to fix this in the normal interface. Unfortunately I can't find what could be missing 😕

Any ideas are still welcome.

[jcoehoorn]

More things I've tried:

• Changing the communication channel workflow_state and confirmation_sent_count database fields to 'unconfirmed' and 0, and then trying to re-send the confirmation message. The confirmation message shows in the messages table, but never sends via email as confirmed via postfix logs.
• Changing the e-mail address to something wrong via the admin interface, and then changing it back (same result)
• Changing the e-mail address to match MY address and trying to send to me via her record (same result, changed it back afterwards, also same result)
• Manually creating a new user in the Site Admin account with the same name, login id, and sis id. I was able to send to this user. Merge this user back with the original SIS-sourced user: can no longer send.
• Added my address as an additional e-mail address for the work study user, and sent a confirmation message. I never received the message.

[jcoehoorn]

More progress.

We have two top-level accounts:

• Site Admin. account_id = 2, parent_account_id = null
• Institutional account ("College"). account_id = 1, parent_account_id = null. This is where all our courses and users actually live.

I manually created a new person record in Site Admin via the + People button using a pretend name ("Fred") and a new gmail address. Canvas successfully sent a new user welcome message to Fred.

I manually created another new person record with the + People button, this time in the other top-level College account. I used a different pretend name ("George") and a different gmail address. Canvas did NOT send a new user welcome message to George.

It seems the issue is somewhere at the system account level. That at least is something to go on. 

[jcoehoorn]

The plot thickens. 

I create another manual user (Arnold) via +People in the College account, new email address. No welcome message is sent. I use pgcli to look in the raw database tables and find the pseudonym record for the new account:

id: 528, user_id: 519, account_id: 1

Now I issue an UPDATE sql statement to change the account_id from '1' to '2', which is the Site Admin account. Now I find the user in the normal Canvas UI and use the "Act as User" feature. From here I am able to re-send the welcome e-mail message... and it goes through.

So what could block pseudonyms for default account from sending messages?

[jcoehoorn]

I'm back to using the work study record for testing.

I can send messages to her if I change her pseudonym to use account_id = 2 instead of account_id=1. If I put things back how they where and add a new pseudonym I cannot send messages again. But if I then go into pgcli and use sql to swap the position values of the two pseudonym records, things work.

At this point, I think I could fix things by adding a cron job to run an sql update query every few minutes to simply change the account_id values. The query can be smart enough to not try to update a record if there's already a pseudonym for the target account... but I still want to do better. There must be a way to get the account_id=1 messages working.

I've been through the settings for both accounts, both in the user interface and looking directly and the database tables and what I can find in the /config folder, and I can't see anything different in this area. From what I've seen in the messages table, Canvas believes it has sent the e-mails. It has records in the messages table with a workflow_state of "sent" targeting the appropriate communications_channel, where the path_type is email. I've been scouring the file system looking for logs, but I haven't found anything yet. If I knew ruby+rails better I might be able to trace through the code, but I'm more of a C# guy; this is my first exposure to ruby, and when I try to follow the code I quickly get lost in all the scaffolding.

[jsavage2]

Hi Joel,

 It's probably time to take a step back, and look at the overall configuration. This is really starting to sound like an installation issue. What parts of the standard production installation instructions (https://github.com/instructure/canvas-lms/wiki/Production-Start) did you need to modify for your environment?

All users should reside in the root account, which should always be account 1 (IME). Somehow your root account seems have gotten id 2, which is strange, and you have apparently also have users being created in a sub account, which is stranger--in fact, it shouldn't be allowed by the user creation interfaces.

I can't think why it should prevent email being sent, specifically, but it's red flag that something is misconfigured.

That really sounds to me like there may have been a bug in one of the install scripts at the time you pulled them from GitHub (unless you overrode something for your environment that had unintended consequences). I haven't seen anything similar reported in the issue logs, so you may want to file a bug report.

Have you popped in to the freenode channel to see if anybody there has any insight?

[jcoehoorn]

We didn't install ourselves. The co-location host we used offers Canvas pre-installation as part of the setup service. Enough other people are happily using them I don't think it's an issue with their process... though a transient issue with a script pulled from github is always possible.

The users are not created in a sub-account. Both Site Admin and the college account have parent_account_id = null.  My understanding is there are supposed to be two root accounts. Aside from the account_id values potentially being backwards, what I see in my system looks correct. 

I agree, though, that something may have gone wrong with our install, therefore hitting a bug that's not otherwise often seen. And since I don't (yet) know enough about how things are supposed to be, the freenode channel is probably the first place to go, before filing a bug report. I'll check there later today.

[jsavage2]

I missed that you were referencing the Site Admin account. I thought it was a sandbox account you'd created for the admins. Yes that should all be correct, then.

What does your host say? If they offer it pre-installed it should work...

You might also try installing it locally somewhere, either pre-built or from source (Turnkey Linux offer a pre-configured VM that spins up pretty well on a laptop) and compare your settings to a known-good configuration to see if anything stands out.

[jcoehoorn]

I played with this over the weekend, and I have this SQL code I could choose to run if I need to (note: it looks like the editor pulled out my indentation, so it may be harder to follow the code):

INSERT INTO pseudonyms (

user_id,account_id,workflow_state,unique_id,crypted_password,password_salt,persistence_token
 ,single_access_token,perishable_token,login_count,failed_login_count,last_request_at
 ,last_login_at,current_login_at,last_login_ip,current_login_ip,reset_password_token,position,created_at
 ,updated_at,password_auto_generated,deleted_at,sis_batch_id,sis_user_id,sis_ssha,communication_channel_id
 ,sis_communication_channel_id,stuck_sis_fields,integration_id,authentication_provider_id
)
SELECT 
 user_id,2,p0.workflow_state,p0.unique_id,crypted_password,password_salt,persistence_token,
 single_access_token,perishable_token,0,0,null
 ,null,null,null,null,reset_password_token,position,created_at
 ,updated_at,password_auto_generated,deleted_at,sis_batch_id,sis_user_id,sis_ssha,communication_channel_id
 ,sis_communication_channel_id,stuck_sis_fields,integration_id,authentication_provider_id
FROM pseudonyms p0 
WHERE p0.account_id = 1
 AND NOT EXISTS
 (
 SELECT 1 
 FROM pseudonyms p1 
 WHERE p1.user_id = p0.user_id AND p1.id <> p0.id AND p1.account_id = 2
 );
-- Now we have two pseudonyms at position (usually) 1
-- Update the position for the "bad" account to come later than the one we just added
UPDATE pseudonyms p0
-- TODO: use a nested SELECT here to get the max current position, or find out if there is a sequence for assigning these
SET position = position + 1
WHERE account_id = 1 and position = 1 -- limit to position "1" for now
 and EXISTS
 (
 SELECT 1 
 FROM pseudonyms p1 
 WHERE p1.user_id = p0.user_id AND p1.id <> p0.id AND p1.account_id = 2 and p1.position = p0.position
 );
-- TODO: these should really be in a transaction, so they always run as an atomic pair‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This will "fix" the imported user records. I've tested it well enough to know users can still log in, access a course, and get notifications.

Again, this is my 2nd choice. I don't like this for several reasons, including that it may have side effects I haven't seen yet, that it's fragile in the case of an update adding a required column, that it causes regular faculty and students to be listed under People in the Site Admin account, which I'm concerned may eventually have security implications, and that it feels like I'm still leaving something broken in our system. 

In an effort to avoid needing to use this, I'm gonna open a new question. I feel like no one with any real canvas knowledge is paying attention here to what has become a longer thread. Opening a new question that goes right to what I've learned are closer to the real issue may be able to get some new and more-focused attention.