To use ADAM as for groups you need to implement a separate RoleProvider that
also points at ADAM. I did this a couple years ago and used AZMAN.msc to manage
the mapping of groups in ADAM for use in SharePoint. I don't have the config
files anymore so I can't post an example, but it wasn't all that difficult. I
used AZMAN to establish the group mappings and saved them as an XML file, then
pointed the rolemanager at the xml file. Take a look at the following article.

msdn.microsoft.com/en-us/library/ms998314.aspx

I just spent the last 6 hours trying to make AzMan work, and I'm
still back to where I was.
It's probably something simple or stupid that I'm missing, but I
cannot make MOSS use groups in ADAM or AzMan.
Learned some things, which is always good .

I'm going to step away from this for a day or two to give the brain a
break.

Fixed this finally.

Got ADAM groups to work inside of SharePoint
The final web.config components that seemed to work are:
<configuration>
<PeoplePickerWildcards>
<clear />
<add key="AspNetSqlMembershipProvider" value="%" />
<add key="ADAMRoles" value="*" />
<add key="ADAMMembership" value="*" />
</PeoplePickerWildcards>
</configuration>

<system.web>
<authorization>
<allow users="*" roles="Users" />
</authorization>
<membership defaultProvider="ADAMMembership">
<providers>
<add name="ADAMMembership"
type="Microsoft.Office.Server.Security.LDAPMembershipProvider,
Microsoft.Office.Server, Version=12.0.0.0, Culture=neutral,
PublicKeyToken=71E9BCE111E9429C"
server="<servername>"
port="<port>"
useSSL="false"
userDNAttribute="distinguishedName"
userNameAttribute="sAMAccountName"
userContainer="<ldap path in ADAM>"
userObjectClass="user"
userFilter="(|(ObjectCategory=group)(ObjectClass=user))"
scope="Subtree"
otherRequiredUserAttributes="sn,givenname,cn" />
</providers>
</membership>
<roleManager
enabled="true"
cacheRolesInCookie="true"
cookieName=".PeopleDCRole"
defaultProvider="ADAMRoles">
<providers>
<add
server="<server name>"
port="<port number>"
useSSL="false"
groupContainer="<ldap path in ADAM>"
groupNameAttribute="cn"
groupMemberAttribute="member"
userNameAttribute="sAMAccountName"
dnAttribute="distinguishedName"
groupFilter="(ObjectClass=group)"
scope="Subtree"
name="ADAMRoles"
type="Microsoft.Office.Server.Security.LDAPRoleProvider,
Microsoft.Office.Server, Version=12.0.0.0, Culture=neutral,
PublicKeyToken=71E9BCE111E9429C" />
</providers>
</roleManager>

Put the Membership & RoleProvder in the original web  application that
uses Integrated Auth.
In the extended Web app  that was created for Extranet Zone using FBA,
put everything in the web.config in the appropriate sections.
(Do this for all WFE servers)
Restart IIS
Logon to the original web app, go to Peoples & Groups
Click the Visitors group
Click the people picker icon (address book)
Type in the group name you have in ADAM
It should show up as: ADAMROLES:<group name>
Click OK & OK to save.

Logon to the FBA url with the ADAM account.