This is easy:
(asdf-install:uninstall [:library-name][how-to-install]) ASDF-INSTALL will ask you to confirm this and then it'll remove the library's source directory as well as the symbolic link to the system definition (if it exists).
Windows note: Due to the way systems are found on Windows ASDF-INSTALL will propose to delete an arbitrary version of your library if you've installed several of them. Make sure to read what it is about to remove before you confirm.