Great question, @JenniferJBland and welcome to the Community! I did some testing and found that the search does find videos that are embedded in assignments. Note that it searches neither the video captions nor the description of Studio videos (even if the media tabs are included with the video embed).
