We usually recommend that individual users not be granted access to your
SharePoint sites. Rather, use Windows Groups instead. So, perhaps you have
granted some groups access and these users are members of these groups?
See who has access to the entire site collection (even in the portal) by
typing /_layouts/1033/userinfo.aspx in place of the default.aspx page. Like
this:
http://portal/_layouts/1033/userinfo.aspx
http://portal/sites/topsite/_layouts/1033/userinfo.aspx