You uninstall extensions in the Extensions Manager -- right click on the target extension for options -- but that only disables the extension (as you may have noted). You have to restart Firefox or Thunderbird for the extension to be gone from disk. That means shutting down all open Firefox windows. I haven't ever had to delete an extension by hand as you describe, unless it was an experimental extension of my own that I hadn't finished making yet.
If you choose to hot-up your car, it's up to you to keep it street-legal. Use some common sense strategies: Don't install extensions that are version 0.01 if stability is important to you -- only install extensions that have matured for a while (say at least version 0.5). Don't install two obviously incompatible extensions, like two extensions that fight over a single resource like the mouse. Install extensions that already have good reviews.
If you just love extensions, but tend to be a bit experimental with them, then consider creating a playpen. You can have a second install of Firefox next to your normal one (or a second profile in the one install you have) and you can lodge extensions in that other profile or other install while you work out if they suit you or not.
This was first published in February 2005