I think you have to use the domain\group name format. If you are already using this format try it without the domain.
I have noticed that when I use security  groups in different domain (than mine, as a user), I have to use the domain\group name format. If the security group is in my domain then just the group name works fine.