Long ID's in Javascript

[jtheaton]

What is the suggested way to deal with the very long ID's that Canvas creates in JavaScript?  For example, if I try to parse this response (from a list of courses), Javascript rounds the ID up to 60780000000000040, rather than the expected 60780000000000037. 

const str = '[{"id":60780000000000037,"name":"This is a test","account_id":60780000000000198}]'
console.log(JSON.parse(str)[0].id)

[robotcars]

See this, https://stackoverflow.com/questions/1379934/large-numbers-erroneously-rounded-in-javascript

[jtheaton]

Yes I saw that.  It only describes what the problem is.  Not how to solve it.  How do you parse such a large number from Canvas in Javascript.  I KNOW the issue is that JS is only 32 bit, but how does Canvas suggest dealing with the large numbers it generates.

The suggested solution seems to be to place quotes around the ID's in JSON.  I don't control what Canvas outputs, so that it not an option.  As an enhancement request (bug fix?), I would suggest that, if Canvas is going to generate such long ID's I would suggest adding a quote.  These are not really numbers, its not like you are doing any sort of math on them.  The way it is now, it makes Javascript development more difficult than it should be.

By the way, the above JSON comes from the course list at: https://canvas.instructure.com/api/v1/courses 

All of my course id's, student id's, etc are this large.

[robotcars]

I'm curious about what endpoint you found that returns the global id? Every API call I have used returns or accepts the local id. 60780000000000037 = 37 (local id)

I was trying to come up with a couple of quick fixes for you but stopped when toString() won't even convert the number to a string.

Overall it seems like a bug, see lines 9 and 463

switchman/shard_internal.rb at master · instructure/switchman · GitHub 

[James]

I wouldn't call it a bug.

See JavaScript data types and data structures - JavaScript | MDN  for restrictions on JavaScript numbers. Remember that Canvas uses Ruby for most things.

Canvas mentions the large IDs, and tells you what to do, in the very first page of the API documentation: Canvas LMS REST API Documentation 

[robotcars]

If the ruby source is set to create IDs up to 10 Trillion, wouldn't 60 Quadrillion be too much? 60,780,000,000,000,037

I'm actually very curious about this, because I want to write more shared code that can work for any instance. Since you're the math professor, please teach me. :smileygrin:

// 10 trillion - 10,000,000,000,000
typeof 10000000000000
// "number"

260000000123456 % 10000000000000
// 123456‍‍‍‍‍‍‍‍‍

607800000123456 % 10000000000000
// 7800000123456‍

60780000000000037 % 10000000000000
// 40‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Isn't it a bug if the ID's per SHARD can be more than the 10T defined by the code?

[James]

carroll-ccsd

If the ruby source is set to create IDs up to 10 Trillion, wouldn't 60 Quadrillion be too much? 60,780,000,000,000,037

You're misreading that. Each shard gets up to 10 trillion IDs. 

That means that the shard ID is multiplied by 10T to avoid conflicts and then the ID is added. If you're in shard 64, you would have 64*10T=640T. The example of Shards in the LTI Variable Substitutions uses 1234, to that would be 1234*10T=12Q 340T

According to Number.MAX_SAFE_INTEGER - JavaScript | MDN , you're safe up to about 9 quadrillion, so 60 quadrillion would experience the problem. The actual maximum safe number is 9,007,199,254,740,991. 

Basically, a Canvas ID is a string made up of digits instead of letters. If your system can't handle it as a number, then you have to treat it like a string.

We're on shard 1 and so all of our long IDs are between 10T <= ID < 20T and they fit without worrying about data loss since they're less than 9 quadrillion. Like you, all of mine come through without the extra padding and the included shard. But you're correct that I should probably watch out more for people in other Shards. It's one of those things that you don't think about because it's not your experience. Experiences drive beliefs and my experience was that all the Canvas IDs are short. Others may have trouble running some of the things I've written because I try to coerce them into numbers because I would rather work with numbers that strings (occupational hazard).

A conversation related to the problems that happened when the length of the keys occurred can be found here: Increased length of surrogate keys .

I hope that's enough explanation -- I've got to go pick up my daughter right now and have to stop writing.

[robotcars]

 @James ‌,

Thank you for the explanation and the extra link, really appreciated!

I have also been trying to force all IDs to numbers and this definitely creates portability problems.

Do we know if each shard is unique to a Canvas instance/institution or can multiple schools be on Shard 64?

[James]

I don't know about Shard 64 specifically, but I am about 99.999999% positive that multiple schools can be on the same shard. Otherwise, it would be extremely unlikely that we would be on shard 1 since we didn't come to Canvas until 2012. Our IDs are not sequential, which means that while course ID 1234 might be ours, 1235 might belong to someone else (and Canvas dutifully reports a 401 not authorized error when I try), yet given what Jeff is saying, we might be getting a much longer number if we weren't on shard 1.  It might (perhaps not) be easy for you to verify. Since you're able to use small IDs, what shard are you using? If it's 1 then we know that multiple institutions can share a shard 

Canvas is limited by the size of PostgreSQL's BIGINT to 9,223,372,036,854,775,807 (9 quintillion). That seems safe for a while, but it does limit them to only 922K shards. Maybe they are reserving the leading digits before the shard number for something else down the line?

What I can't answer right now is if each school is limited to a single shard.

[robotcars]

64 was just an example, we're on 26, and signed up in 2013. I'm going to run some queries next week to see if we're sharing, by trying to see if there are gaps in our IDs I can't access. I'd bet you're correct, and multiple institutions are sharing.

[robotcars]

 @James ,

Using Canvas Data, I tested for gaps in the ID sequence of 3 tables. It would appear we are sharing Shard 26~ with other institutions.

user_dim - 3881 gaps between IDs

The most obvious where the first few, testing out Canvas before buying in, then a trickle in of a few more staff. This gap was 98K, then 3K, and 1K, before a huge import of employees.

course_dim - 275 gaps

Indicating larger batch groups, but still gaps between that we didn't create.

submission_dim - 3,649,995 gaps

Finding (all the) gaps in an identity column (or any integer based column for that matter) using Sql...

The Earth is not flat and Canvas instances have neighbors.

[jtheaton]

Thanks, James...  I checked the API docs on large numbers, this fixed it for me:

"To force all ids to strings add the request header Accept: application/json+canvas-string-ids This will cause Canvas to return even integer IDs as strings, preventing problems with languages (particularly JavaScript) that can't properly process large integers."