Unless you plan to re-install it, yes. Just make sure if you installed
it using a Solution that you un-install it the same way. Also, just to
be clear, we are talking about Uninstall, not Deactivate. You need to
Deactivate it everywhwere and then uninstall  it. Then you can remove
the folder.

No. Uninstalling a Feature removes its information from the
configuration database.

To remove the folder from the 12 Hive, retract and remove the solution.

That was exactly what I was looking for.