We're working on a method to do this in the UI but for now you can (carefully) edit the database. Run the commands as root on your control panel server.
To detach all websites from a particular backup server:
su - orchd
psql
delete from website_backup_servers where server_id = 'server id here';
You can get the server ID from your browser's address bar when managing the server in the UI.
To detach a single website:
su - orchd
psql
delete from website_backup_servers where website_id = 'website id here';
You can get the website ID from your browser's address bar when managing the website in the UI.