THANK YOU! I see now where I had gone wrong. I had already given some feedback for one of my assignments. The policy setting is working as Instructure has designed it, even if it can be confusing. Here is the scenario:
I am defining "annotation" as using the Canvas teacher app on an iPad to "write" directly onto the student's submitted work. On the other hand "submitting a comment" is putting a comment into a text field in the comments/feedback side pane associated with the submission. Finally, "grading" is entering a numerical score for the assignment.
I have several writing assignments this semester. All had "manual" posting enabled from my default grade book policy. For assignment 1, I was in a hurry, so I made some general comments in the side pain and submitted those as feedback. I sort of realized that students would not see the comments until I "posted" grades, so I did that step (even though I had not actually entered a grade). I then forgot about it because the assignment still needed to be graded, and I planned to go back and give more detailed feedback later.
When it came time to evaluate Assignment 2, I made extensive annotations on each student's submission. I also went back and finally fully annotated Assignment 1.
Students received a ridiculous number of Canvas notifications for Assignment 1 because that one had been previously been "posted". I would learn later they did not receive any notifications or see annotations for Assignment 2 at that time. But I could not see anything different on my end for Assignment 2 from Assignment 1. How odd.
It made so much more sense when I got to Assignment 3 and was able to test things on a student by student basis. I also went to enter actual scores for Assignments 1 and 2 and finally was able to see a difference for the two on my end.
Here is what was tripping me up: Just putting in annotations (again, by "writing" with a stylus/pad) on each submission will NOT trigger any alert to the instructor that these "handwritten" comments are hidden. It is also not possible to "post" such annotations alone. Posting is only an available option when a grade is entered OR a text comment has been submitted as feedback in the comments pane. Further, after either of those two things are done, the crossed out visibility symbol appears in Gradebook, and the submission's comments side pane shows "Hidden" in an orange box (until the instructor "posts" the grade/feedback).
Again, writing annotations on the submission alone does not trigger these visual cues that the feedback/grades are hidden. So I am sitting there very confused why both Assignment 1 and 2 looked like they are set up exactly the same way, when in fact, Assignment 2 was still Hidden, just not labeled as such. Interviewing students more closely revealed they were only get Canvas notifications for each written annotation for the Assignment that I had previously posted feedback. Once I started entering grades/comments into the side pane, then the hidden flag shows up in the side pane and the not-visible icon shows up in the grade book (until I post).
I guess my new workflow adaptation (because we all know Instructure won't fix anything) is to always submit a brief comment in the side pane whenever I am doing my written annotations. That then will trigger the visual cues needed to remind me that students can't see my annotations yet. Or if the cues are absent despite a submitted text comment from me, then I know that if I go back and annotate the submission then that poor kid is going to get a ton of notifications (and so I can consider the time of day/night).
Honestly, I can't even say this is the goofiest Canvas thing of all, but it might be nice and consistent if using Annotations also triggered the "it is still hidden" cues before posting if Canvas is going to also treat them as something that requires instant notifications after posting.