We had the same revelation when we went with the large server farm  model – keep in mind that your users  are still accessing the Front End, just because you have 3+ server  handling the environment  doesn’t mean you’re going to see much increase in performance, you’ll just be able to handle more load.

…but here’s the kicker, once we went to our large farm, page loads were initially slow and our nightly export of all run of Sites (via s STSADM script), which normally took 4 hours took 11 hours. What the heck?! It turns out the next nights exports took only 6 hours, then the next nights it was back down to 4 hours. It just took a little while for all of the caches to be created/updated.

We have a single server  SharePoint deployment  that we are planning on taking to a 3 server farm  primarily because we want to do more indexing of non portal content. Should I expect to have slower performance  from the farm than the single  server installation or equal performance once the cache has been created and updated?

I would say equal or marginally better since your FE won’t be handling the index and search processes. As soon as a page loads, it’s cached and is fast the second time. We spent a few hours troubleshooting why our farm  was slower until we figured out it was just a caching thing. … and who would have guessed STSADM.exe utilizes web  caches, it must be working at the web level (web folder, etc)….

Equal or better performance...............

We have cleared up several of our performance  issues, but we are
currently in the state of denial that our buff prod servers should
be any slower than our single  box deployment  (or test  env
w/SQLServer). This is now personal, and curiosity is driving us :-)

Assuming the network and load  balancer aren't issues  (we're looking
into them seperately), what would be the elements that we should
examine to find these delays? (4 server  spread: web, web, SQL,
job/index)

Hard to say what is happening, although it could be factors outside of the farm  like network or that disk writes are going over the network instead of local (for example, I believe the job/index server  is backended by a database  on another server).

You might want to speed things up in general on the search indexing and search results front. One example is that Coveo Enterprise Search for SharePoint monitors the file and web  server for file changes instead of doing a complete recrawl. That alone makes a huge difference. We also do not use an RDBMS on the backend, but rather an inverted index—this speeds results significantly and allows for really fast begin/middle/end wildcard search, something SPS/WSS is unable to do. Among the other ~50 additional features and capabilities we provide include Boolean, spell suggest, cached copy, summary page, concepts page, Refine Search, Advanced Search, preferences, integrated search toolbar, hit highlighting, etc.