We just ran into this very issue on our intranet  site. There was a robots.txt
file that tightly controlled what processes could crawl the site. We needed to
add a rule to allow the SharePoint crawler.

You need to enter the full  string:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
And then the appropriate Allow directives.

We had been using the "MS Search 4.0 Robot" user agent string which does not
work (MS will be adding a KB article about this).

I wouldn't have thought  about the robots.txt file but
that makes  total sense. You wouldn't have one handy would you that is
already done?

Yes, we use that, but to disallow all robots except for the "select few".
That's why we needed a specifc rule to allow the SPS crawler.

Otherwise, using User-Agent: * with Allow: / should do the trick.

Here's what we have:

User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
Allow: /wwm/
Allow: /americas/
Allow: /consulting/
Disallow: /bin/

User-agent: *
Disallow: /


(there are a few rules for some other robots)

Our site  admins want to disallow all robots and enter explicit entries for
trusted bots.

I created the robots.txt file, but when I go to start  the full
update it still only lasts about 1 second.

Did you guys adjust the registry as outlined in this doc?

support.microsoft.com/default.aspx

No, we didn't need to do that (just updated our robots.txt file). I did add  a
Site Hit Frequency Rule in the SPS Central Administration to limit the request
to 1 page at a time (our intranet  servers were getting hit hard by the crawler).

That was the only thing we needed to do. What errors are you getting
specifically whe you look at the gatherer log?

I found in the log files the problem I think:

The address was excluded because its file extension is restricted in
the file type rules.

The site  I am trying to index, all of the pages end in .php. It
appears the indexer is excluding all of these pages (which in turn is
the entire site).

How do I turn this off and where?

Under Configure Search and Indexing > Include file types and add  php.

Also, it seems SharePoint caches the robots.txt. Log onto your index  server and
restart the search service: "Microsoft SharePointPS Search". Also, bring up IE
on the server and clear the temporary files and be sure to check for newer
versions of stored pages "every visit to the page" under settings.