You can't create a security group in a site  without at least giving them
some level of access  in the site.

However, there is another thing you can do that will basically
accomplish what you are looking for. Create a group, and give that group
access to the site. What you can then do is remove that group's access
to every list  and library  except for the one you want to give them
access to.

breaking permission inheritance OH MY!

Add them to the web part  and not the site. Then give them the URL to
the web part itself. They will automatically be a guest in the site  to
access the web part.

I don't think this is possible if you are using the All Authenticated
Users option under Manage anonymous access  screen.

You have to assign it to either Reader or Contributor, and while you can
strip out that group from all of the lists and libraries, the
authenticated user will still have access to the site, so anything on
the home page, be it maybe just a bunch of web parts that say you don't
have access. It doesn't seem like an ideal approach. Plus that means
you would have to create a custom site  group or assign your allowed site
users to the Contributors site group, since removing Reader from the
library permissions would effectively disallow your site users  from
reading the library  as well.

Is it possible to post the data on a site they can have access to or a
portal? I don't think you will be able to get a smooth solution or user
experience unless you settle for adding individual users to a site
group, cross site group or to the library itself.

Well, I seemed to have gotten this to work. Anonymous access  is set to no
and all authenticated  user access is also no.

In the list, I change permissions and add "nt authority\all authenticated
users" as a reader to the library. They have access to the files  in the
library but not the site  containing the library.

I had recalled something like that before. It seems that the trick was
remembering the "nt authority" thing.

I will try this again, but I believe that when I tried [domain]\all
authenticated users  it did not work.

Excellent! Glad you figured out a solution.