This is not a setting that is controlled by SharePoint. It's a
combination of IE settings on the Programs tab in IE>Tools>Options and
the application extension settings on the client OS. If you remove the
.xml extension from the OS, there is still a setting in the MIME types
in IE. The DOCICON.XML only controls which application will open  when
you click the new button in a library  and this only works for office
apps. Note the DLL that is referenced in the .DOC or .XLS entries as
opposed to the .XML entry.

I've tried just about everything I can think of... and I can not force
an .XML document  stored in SharePoint to open  with a 3rd party editor.

I've tried:

- Setting the Windows File Type associations to XMLMind

- Turned off MIME FEATURE_MIME_HANDLING and
FEATURE_MIME_SNIFFING in the Registry in HKLM and HKLU (the later not
having an entry)

- Deleting the HK_Classes_Root MIME Content Type - text/xml

I guess the only think I haven't don't is delete the registry entries
for infopath and word for xml  docs... but neither of those two programs
are opening the document when I click on it in sharepoint.

Does anyone know how to nuke XML relationships so that only the editor
you select actually opens a document from sharepoint? (It's got to be
an OS issue...)

For some reason when I put .PDF in the docicon.xml it would only display
the icon in search results. It would display a broken image for the new
button. Any idea why this would be?

Where did you physically locate the PDF graphic? And what relative path
did you use for its location? Also did you use an .ICO file  or a JPG or
GIF? The normal document  library display won't do .ICOs. I think the
search results pick up the .ICO image from the file itself rather than
from DOCICON.xml.

12/Template/Images/pdf16.gif

<ByExtension>

<Mapping Key="pdf" Value="pdf16.gif"/>

That all looks right. Have you done an IISreset since you uploaded the
pdf16.gif file  and did you make sure it was on all the web front ends?
Also, make sure that the gif is a 16x16 size. The doc libraries will
not resize the icon. Other than that those exact settings work for me.