No, the settings are separate for the two options for students and groups. I recommend not allowing students to form their own groups especially for the situation you described, @lbach. That way there will not be any confusion about which group is submitting the assignment. It is possible to make a group assignment for a group set that is set up for self-enrollment.

I did some testing and discovered, however, that a student user can submit to that assignment even if they are not a group member! I've run into this problem with discussions, and I have to make sure that all my students are in groups before they can access the discussion. For self-enroll groups I have a no-submission assignment that I use as a module requirement, which allows me to prevent access to the group discussion. If a student does not join a group, they don't get the "join group" assignment marked as complete, and they can't access the remaining items in the module.

I'm going to mark my reply as a solution since it answers your question, but I hope other members of the Community share their experiences with working with groups. I'm especially interested to know how do people ensure that students are in groups before they can access a group assignment or discussion.