@RubenAMartinezA
Not Possible As Is
You have an interesting case I haven't seen in the Community before. You are correct in that Canvas does not allow for this use-case. Your solution would work -- except for the obvious Canvas doesn't allow it.
This is specifically about New Quizzes and I haven't played much with them, but I did play a lot with Classic Quizzes and have written how to hack the formula questions there. Another user wrote a Chrome extension that added a lot of missing functionality to the creation of random numbers and formula questions.
None of our exploration led to ways that would allow you to change the tolerance on a per generated response basis. The tolerance is for the entire question and all generated responses that you have. You cannot set the tolerance based off the values.
That means that formula questions are not going to work for you in the way you are asking (see later for alternatives). If you want to see that feature developed, then you would need to do what @thovey said and file an idea request.
I don't expect much to happen if you do make a feature request, but it will provide an opportunity to get the idea out there (you asked this as a question, which can have answers). I've been using Canvas for 10 years now and I doubt that request see any development. As I mentioned, this is the first time I remember seeing a request for it and there have been much more popular requests for math and science functionality that have gone nowhere.
Multiple Non-Formula Questions
That leaves us with the question of what can be done right now?
The answer is to not use a formula question. Instead create a numeric question and set the tolerance appropriately. Repeat this process multiple times so that you have several questions -- as many as you would have with a formula question. Then put those into an item bank and randomly pull questions from the bank.
I haven't done this with New Quizzes, but I have with Classic Quizzes. I originally used Excel to generate a CSV file and then import the questions into Respondus. Then I would export them from Respondus into Canvas as a quiz. Eventually, I bypassed Respondus and would write a script that would generate the question and import it directly into Canvas using the Canvas API.
Beyond the absurd technical level needed to do that, the New Quizzes API isn't finalized or published. If you still have access to Classic Quizzes, you can import them there and then migrate them to New Quizzes.
However, it is likely going to be faster to simply create several versions of a numeric question within New Quizzes.
Another option is to change the way a question is asked. You could create a multiple choice question "which of these numbers is less than the product of x and y", but you would still need to manually supply the x and y and then create choices.
Modified Formula Question
I can think of one way to use a formula question, but you will need to change your question and what students give you as a response, so it may not be acceptable.
Is `z` smaller than the product of `x` and `y` ?
Enter 1 for yes and 0 for no.
The challenge comes with the formula to decide the answer. You will obviously need to use an if() function.
The if() statement has the syntax if(Boolean expression,result when true,result when false).
While the condition is Boolean, it isn't a simple true or false. It's truthiness is determined by 0 is false and any non-zero value is true. Further compounding the situation is that it only evaluates numbers, it does not allow for comparison operators such as z<x*y.
Since the student isn't typing in a number, only telling you whether the number you chose is smaller, you avoid the problem with 0 or negatives by making sure that z doesn't include them. You can also use z to control how many decimal places to give in the answer.
Assuming that you are looking for a strict less than, then one way to check if z is less than x*y is to see if z is the smaller number between z and x*y-1. If you want to allow less than or equal to, then drop the -1 from the x*y-1. Mathematically, we are checking to see if z=min(z,x*y-1). You should probably make sure that x and y cannot simultaneously be 1, although it would still work in that case.
The min(z,x*y-1) is easy to calculate since min is one of the functions available to you. The question becomes how do you check to see if z is equal to that without having access to an equal sign? Using a formula of if(z=min(z,x*y-1),1,0) returns an Error: unexpected equals at 5 (at least in Classic Quizzes, where I'm testing this). This 5 means position 5, which is where the equal sign is.
However, if two values are equal, there difference is 0. 0 is one of the allowed values in a Boolean check. That is, if z=min(z,x*y-1), then min(z,x*y-1)-z=0 and the Boolean expression is false. If z is not min(z,x*y-1), then the difference is not 0 and any value not 0 is treated as true. This means that a correct response is a false, so we need to make sure the answer for yes go with the false and the answer for no goes with the true.
if(min(z,x*y-1)-z,0,1) will give you 1 if z is less than x*y-1 and 0 if z is at least x*y.
The 1 in the x*y-1 should be set to the number of decimal places you give in z. If they are integers, then use 1 or any number less than 1 such as 0.5. If you are giving two decimal places, then use 0.01 or 0.005 to be safe). By safe, I mean less potential for the computer to incorrectly round a number.
You could also change the responses the students are to give: "Enter 1 for yes and 2 for no." In that case, change the 0 and 1 within the if() statement to those numbers: if(min(z,x*y-1)-z,2,1).
You could even really mess with the students and say "Enter `a` for yes and `b` for no." and then use if(min(z,x*y-1)-z,b,a) as the condition. I don't recommend that, but it shows some of the flexibility you have here.
Flexible Interval
Let's build on that last idea and allow for any open interval, not just positive numbers between 0 and x*y.
Here is a revised question.
Is `z` strictly between `x` and `y` ?
Enter 1 for yes and 0 for no.
You would need to make sure that x is strictly smaller than y when you created the random numbers.
We can use both max() and min() to check this. If z is the smaller of z and y and the larger of z and x, then it is between the interval. Checking a logical and statement can be challenging, but you can nest the max and min statements to get what you want. If the larger of x and the smaller of z and y is z, then we know that z is between x and y. If z is bigger than y, then the larger of x and y will be y, not z. If z is smaller than x, then the larger of x and z will be x, not z.
if(max(x+1,min(z,y-1))-z,0,1) checks for z in the open interval (x,y).
if(max(x,min(z,y))-z,0,1) checks for z in the closed interval [x,y].
