For me, I could not get rid of my last notifier for a while until I realized that simply deleting all of the alerts using it was not enough. All of the alert groups had to be deleted as well. At least that was the case for one of my clusters. Maybe I had a custom group added to that on.