By default any pages not found in the _Layouts directory have their
MasterPage set to the token of ~masterUrl/default.master. This refers
them to a property of the SPWeb object stored in SharePoint called
MasterUrl which holds the path to the masterpage that will be used. In
a normal site this property is set to
_catalogs/masterpage/default.master, which points to the default.master
file in the masterpage gallery of the site. Until the page in the
gallery is customized it points to the default.master in the
Template/Global directory of the 12 hive.
You can use SPD to select a different .master file to be associated with
each layout page, but that is page specific. You can also use SPD to
edit the .master in the Site's master page gallery, which would apply to
every page in the site. You can also use the SharePoint Object model to
write a program that will modify the MasterUrl property of the SPWeb
object which will change the masterpage for the whole site without
unghosting objects to the database like SPD will. Finally, you could
edit the masterpagefile attribute of every page using SPD to change what
masterpage it uses.
You said you did this for the layout page. If you are using publishing
I suspect that means you changed the layout template from which new
pages are created. This will not change the page after it has been
created. It will change any new pages created. To change the already
created pages you would need either change the global setting (see
previous paragraph) or edit each page. I also think you have a
different problem with the setting you are using. If you are going to
hardcode the setting I think you want the address
'~site/_catalogs/masterpage/default.master'. I don't think the address
you are using will be evaluated as valid. If the site doesn't find the
specified masterpage SharePoint will fall back to the default.master in
the 12 hive. So your problem may be that you are either changing the
layout template in publishing or that your masterpage address is wrong.