Giving a user access  rights to a list/library will automatically give  them
limited access rights to the site  containing the list/library. This is just
enough rights to traverse the site to get to the list/library without
actually having rights to anything else in the site. This is automatically
assigned. You don't need to give the user any rights at the site level,
just the library  level.

You can give item level security, but in an infopath  library you need access
to both the XML result file and the XSN form  template. In a regular
document library you only need access to the template to create a new file,
but not to read or edit the file. That isn't true for infopath.