LTI 1.3 Token Request causes HTTP 400 Bad Request from Authlogic error

[Souvent22]

Our LTI 1.3 launch works fine. However, when our took attempts to get an access token, we're getting the following in the Canvas logs:

canvas_dev   | Started POST "/login/oauth2/token" for 127.0.0.1 at 2021-05-20 23:56:01 +0000
canvas_dev   |   SQL PluginSetting Load (0.5ms)  SELECT "plugin_settings".* FROM "public"."plugin_settings" WHERE "plugin_settings"."name" = 'sessions' ORDER BY "plugin_settings"."id" ASC LIMIT 1  [development:1 primary]
canvas_dev   | request throttling increment: [0,50.0,1621554961.65478,["client_identifier","ip:172.18.0.10"],["count",0.0],["last_touched",null]]
canvas_dev   | Redis (0.609ms) evalsha - [redis:6379]
canvas_dev   | Processing by Oauth2ProviderController#token as JSON
canvas_dev   |   Parameters: {"grant_type"=>"client_credentials", "client_assertion_type"=>"urn:ietf:params:oauth:client-assertion-type:jwt-bearer", "client_assertion"=>"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImUwMWRjNjQ5Mzk3NzA1N2FhMzQ0ZjY0MTg5YzBjZTNjIn0.eyJzdWIiOiIxMDAwMDAwMDAwMDAwNCIsImlzcyI6IjEwMDAwMDAwMDAwMDA0IiwiYXVkIjoiaHR0cHM6Ly9jYW52YXMuMnUubG9jYWxob3N0L2xvZ2luL29hdXRoMi90b2tlbiIsImp0aSI6InJwc3k2bzlhOGk1d2pibnF4MTJ6aGR3c3UiLCJpYXQiOjE2MjE1NTQ5NjEsImV4cCI6MTYyMTU1NTAyMX0.QHU4h8Tt5065BNfs0DfQycbkXycuop5eOFvSKZ_kkmRPlF34uajuYEaff43McwSPZzy7GXvoZXASS-GCOkhdiwnFhyTuTOtmh3NpExBYcZ_C1MHI_YX6tf3JRT_1CvwdZvcziXh_hH2dmlUOPQLxH9BQVDzFQHMY4u5NsnpLmmE0zIAVCZuztrkwa9LH8DKFYRZyafJ2woyFxm6PpG5OeYeZu92s6MmSKlbmtnK8GLLyWjuq2IO9As2U9QV-NBFlVB2qmzWjxKoxONme6uBmiBbca-sYYLIxzR944Y7ntyZPNMeNtFcMf52CdLfM4Edd7H5cg8iUasCsNcfTFVcrckVFggl5bv_OqWa5VB45JAutLDSSj-JrnT5DfgmOihVMZddDd8SVz915O8qsrnAV7gEi-kThnHy6WnZZFaAKWI6EVmClKMemMEBh_DY3wUY9grzZw-h5TBP85YxAlQBPVQEjZFU3Thp-qiEdQQdfPvZvSzUX8c14CvRi7pCCS_uh4mFVRe2hZukg4-Vm4axtRqC28SDIBla7CDqdf43jgrZ3GE8ySUOBp-C11BXMGriW9yxnC4LTu69WcDp84Dp1tRTkpJ9smKDNfHPmsJPKB7sgU7HPejhtmLT2eCZ1-zvci4j1kMn7-rFOoRE5aoVABm9BUdYL8HkhNXSJJAgUQ00", "scope"=>"https://purl.imsglobal.org/spec/lti-nrps/scope/contextmembership.readonly"}
canvas_dev   | [AUTH] Authlogic Validation Error: You did not provide any details for authentication.
canvas_dev   | [AUTH] Authlogic Failed Find
canvas_dev   | [AUTH] inital load: pseud -> , user -> 
canvas_dev   | [AUTH] final user: 
canvas_dev   |   SQL DeveloperKey Load (0.8ms)  SELECT "developer_keys".* FROM "public"."developer_keys" WHERE "developer_keys"."id" = 4 LIMIT 1  [development:1 secondary]
canvas_dev   | Completed 400 Bad Request in 205ms (Views: 0.1ms | ActiveRecord: 0.8ms)
canvas_dev   | 

So it's the /login/oauth2/token request. This tool works in Moodle and using the IMS Reference implementation. We can not figure out why Canvas is saying "You did not provide any details for authentication".  The scopes look fine, the JWT signature is ok, and the initial OAauth authentication runs fine. It's just attempting to get the access token fails and throws a 400 with this error from Canvas.

For readability, the token request looks like this:

{
  "grant_type": "client_credentials",
  "client_assertion_type": "urn:ietf:params:oauth:client-assertion-type:jwt-bearer",
  "client_assertion": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImUwMWRjNjQ5Mzk3NzA1N2FhMzQ0ZjY0MTg5YzBjZTNjIn0.eyJzdWIiOiIxMDAwMDAwMDAwMDAwNCIsImlzcyI6IjEwMDAwMDAwMDAwMDA0IiwiYXVkIjoiaHR0cHM6Ly9jYW52YXMuMnUubG9jYWxob3N0L2xvZ2luL29hdXRoMi90b2tlbiIsImp0aSI6InJwc3k2bzlhOGk1d2pibnF4MTJ6aGR3c3UiLCJpYXQiOjE2MjE1NTQ5NjEsImV4cCI6MTYyMTU1NTAyMX0.QHU4h8Tt5065BNfs0DfQycbkXycuop5eOFvSKZ_kkmRPlF34uajuYEaff43McwSPZzy7GXvoZXASS-GCOkhdiwnFhyTuTOtmh3NpExBYcZ_C1MHI_YX6tf3JRT_1CvwdZvcziXh_hH2dmlUOPQLxH9BQVDzFQHMY4u5NsnpLmmE0zIAVCZuztrkwa9LH8DKFYRZyafJ2woyFxm6PpG5OeYeZu92s6MmSKlbmtnK8GLLyWjuq2IO9As2U9QV-NBFlVB2qmzWjxKoxONme6uBmiBbca-sYYLIxzR944Y7ntyZPNMeNtFcMf52CdLfM4Edd7H5cg8iUasCsNcfTFVcrckVFggl5bv_OqWa5VB45JAutLDSSj-JrnT5DfgmOihVMZddDd8SVz915O8qsrnAV7gEi-kThnHy6WnZZFaAKWI6EVmClKMemMEBh_DY3wUY9grzZw-h5TBP85YxAlQBPVQEjZFU3Thp-qiEdQQdfPvZvSzUX8c14CvRi7pCCS_uh4mFVRe2hZukg4-Vm4axtRqC28SDIBla7CDqdf43jgrZ3GE8ySUOBp-C11BXMGriW9yxnC4LTu69WcDp84Dp1tRTkpJ9smKDNfHPmsJPKB7sgU7HPejhtmLT2eCZ1-zvci4j1kMn7-rFOoRE5aoVABm9BUdYL8HkhNXSJJAgUQ00",
  "scope": "https://purl.imsglobal.org/spec/lti-nrps/scope/contextmembership.readonly"
}

And the client_assertion JWT looks like this when decoded:

{
  "sub": "10000000000004",
  "iss": "10000000000004",
  "aud": "https://canvas.example.localhost/login/oauth2/token",
  "jti": "rpsy6o9a8i5wjbnqx12zhdwsu",
  "iat": 1621554961,
  "exp": 1621555021
}

Any thoughts?
FYI: The JWKS URLs of the tools is publicly available, and again works in Moodle and using the IMS Reference tool/tester.

[Souvent22]

This was caused because the permissions/scopes were not set on the Developer key. Once that was set, I was able to get the token. 

[Souvent22]

Ok, I'm actually stumped again. This has happened twice now with 2 different tool additions. This time, even when we enable all LTI Advnatage services on the key, we get the same error as mentioned above.