1. When an AD group  and a sharepoint  group exists  in a site  with different
permission levels  with some users  are in both, which group dominates over the
other. The SharePoint group may have the AD group nested  or otherwise the same
users are added  as individuals.

Reply> Neither group dominates. The user gets the sum of all the permissions
given by the different permission  levels.

2. What if we have two or more SharePoint groups  with different permission
levels in the same site with some users are in both groups. That means which
permission level  would be most effective?

Reply> see answer above. They will get the sum of all the permissions applied.

3. If two or more AD groups ( with some users are in all) appear in a site with
different permission levels what would be the situation?

Reply> see the answer above. the key is that unlike AD SharePoint doesn't have
the ability to deny permissions based on membership (except at the web
application level when the deny takes precedence over everything), so
permissions are always additive.

Please correct me if I am wrong here.
The term sum you mean here is the most restricted right?
For example suppose there are two groups, one with limited access and the
other with read, the sum will be limited access right?

Actually, its just the opposite. By sum I mean addition of the individual
permissions. If you have read and contribute from two different groups  your
effective rights will be contribure permission  level, not read.

The situation I am having here is bit contradictory
to the sum
operation we talked about. You might have seen my earlier posting to say that
some of my users  were not able to see a page ( a library) with the correct
format. I thought it's some sort of CSS being not applied or a template issue.
After weeks of testing I found that there was a SharePoint group  with
Contribute( restricted) and the NT Authority/Authentication group had limited
access for the site. I upgraded the latter to Read and the users were able to
view the page. But I did this on a test server. Not on the production. That was
the point I started debating myself about permission  with different groups. This
could be another reason in which I am not sure what it is. Still working on it.

Did you figure out which CSS file was the problem? Most of the CSS is stored in
the 12 hive on the file system and can be affected by the NTFS rights provided
by specific AD groups. This has nothing to do with the SharePoint group
permissions. Custom CSS files that are created elsewhere and then copied to the
12 hive can often have this problem. Evidently in your case being part of the
SharePoint group  didn't provide access to the files, but everyone is a member of
NT Authority/Authenticated users, so increasing the rights there had the
required effect. You need to find the original files that couldn't be accessed
and check their NTFS rights. You can take a look at a tool called Fiddler to
find out what files weren't accessible.

Once I looked at the CSS file and didn't try to find NTFS rights since I was not
aware of it. I am sure that this CSS file is not a custom file. I will looked at
the Fiddler tool and let you know the outcome.