We have been playing around with this for some time and have not yet
resolved the issue to our satisfaction.
What I can tell you is that this problem is caused by a combination of
your IIS security settings and the operating system of the workstation
trying to access SP.
We have not yet resolved the problem, but I can at least tell you what
we have discovered.
In the extranet white paper, you will see that Microsoft suggests that
you create separate virtual webs for each type of authentication method
that you require. In our scenario, we wanted to set SP up on an
Extranet and configured authentication in much the same way as you have.
We found that the login box appears if the authentication method on the
public and workspace virtual directories are set to basic, or if they
are set to integrated and you try to access SP from a Win 2K or Windows
9x workstation that is logged onto a different domain / workgroup. If
the authentication is set to integrated and you try to access from an
NT4 workstation that is logged onto a different domain, you just get an
access denied message. The fun really starts when you configure the SP
virtual directories as anonymous only, because you end up in a situation
where you cannot create / modify categories and documents even if you
are logged in as administrator on the SP server.
To get around this, we created a virtual web site with anonymous access
that was available only on the Intranet and another that was configured
for basic authentication that is available on the Extranet and the
Intranet. The problem here of course is that Intranet users cannot
subscribe and cannot check documents in and out. If we set this
authentication on the intranet site to integrated (we have two domains),
the users in the second domain running NT4 cannot access the site, and
users running Win2K or Win 9x get the login box. The obvious solution
here is to set up trusts between the two domains, but this will not
resolve the NT4 workstation issues.
I know that none of this helps to resolve your issue, but I thought I
would share our experience. We are still working on this and I am
confident that we will find a solution within the next week or so.