Rancher hung in Removing state

After trying to upgrade a service rancher got wedged in some weird state. Right now it’s trying to delete a container that does not exist on the server anymore…

Inside of the Finishing Upgrade it shows two containers. One running successfully - the other in “Removing” state. However the container it’s trying to remove is not running on the host machine. What’s going on?

Seems like Rancher is in overall bad state. Can’t deactivate registries either?

I restarted the rancher/server container but it did not help resolve this or the other mentioned issues.

What can I do here? How do I force it to clean up the state?

Any ideas on how the container no longer exists? Was it deleted by some other way?

The container that it’s deleting should have been the old container of the service.

Do you have any logs from Rancher server?