If you are having students upload files via assignments, there is an option to specify the file type to upload. Regretfully, there is no such limitation on a Quiz or in a Discussion. The best thing you can do there is communication of expectations. I wish I had a better answer for you, but students learning how to follow instructions is a valuable lesson for them.

--James