No, we do that frequently. There are some browser settings that can cause that
kind of behavior. For example we have had people who had the "check for
certificate revocation" enabled and they would simply get a "page not found"
error when trying to access an SSL'd page.
There are security settings too that can also cause problems but these typically
give you a pop-up error message in my experience.
I presume you can access the aspx pages okay outside the web part environment?