Even if you find  a way to select  a different master page  I don't think
it will help. If you look at the coding of a page layout  there is no
masterpagefile declaration in the @Page tag at the top of the page.
That's because it gets the setting by inheriting from a PublishingPage
object. Inheriting from that object is required for page layout
functionality to work. The PublishingPage object class is hard coded to
use the custom master  (site master). Even if you override it I think
the default  setting in the class is the one it uses. I don't think
there is any way to get around that.

Does that mean there's no way to use a different master page  from
the custom master  page for some of the content pages?

The only possibility that I can think of would be to subclass the
PublishingPage object class, override the customweburl setting, and use
that class for the custom layout  page. I don't know if its possible or
not since I don't know if you can inherit from PublishingPage.

Otherwise, no you can't use a different masterpage for layout pages  in
the pages document library.