If you or any of your connections are case contacts, you can remove them by clicking the Remove button.
You can only remove a case contact to whom you're connected.