Web parts  use a completely different storage mechanism to the
publishing features in SharePoint and therefore web  parts (and their
content) are not saved along with the version  history of a page.
Unfortunately the only way to get a web part  back is to restore  the
site from a backup taken when the web part existed.

The user checked in with a draft version. Anonymous users still see the
old web  part on the page. But it seems no way I can delete the user
checked-in draft version  even though I am administrator. The message I
got is always "You can not delete current version".

I am wondering why SharePoint does NOT allow removing current version.

As you've found, you cannot delete the current version. What you can to
is revert to a previous  version (making that version  current once
again), and then do whatever you want with the incorrect version (which
will no longer be current).

I know I can do that, but still I can not restore  the web  part the user
deleted. The old version  still has the webpart, when you restore the
version, the web part  does not come.

That is why I tried to seek a way to delete current version and expose
the old version instead of restore the old version.