You didn't say if you were using SharePoint 2003 or 2007. That would have been
a big help. If I knew that information I'd get into a more detailed response.
Short answer is that there is no way in 2003. You could maybe look through your
IIS logs to see who hit the user.aspx page in that site. I don't think you can
do it in 2007 either. There is some auditing, but I don't think it covers this,
just documents.