Do you have Manage Hierarchy? I believe that should do it, but if not
you're going to have to go Full Control.

Check the properties of the group in question. When you set up a new group
the default is to only allow the owner who created the group to add  members.
There is a radio button in the Group properties that expands this to anyone
with the property security authority. So I suspect its not that you don't
have enough permissions, its probably that you aren't the group owner.

The general rule is that a person needs to be a site owner or a site
administrator in order to manage users  on a site collection.

The Site Owners and Site Administrators have Full Control permission  on
the site collection they are assigned to.