Error: UPGRADE FAILED: the server has asked for the client to provide credentials (get configmaps)

Rancher
1

We have a couple of Rancher 2.3.5 clusters that have been running fine for quite some time.
Last week the authentication backend was changed from ADFS to AD.

Since this change we do experience problems with upgrades of workloads from before the change.
New deployments go without problems.

When an update is done the error message Error: UPGRADE FAILED: the server has asked for the client to provide credentials (get configmaps) appears. Given that the authentication backend has changed and that the error message is about authentication it seems that something is off with RBAC.

On the server we run helm 2.16.1 which should have fixes for similar issues.

Did we mis a step when changing the backend? Do we need to make modifications to the RBAC for Tiller? If so, which ones?

At the moment the only way to deploy the apps is to remove the app completely and then redeploy. Once this is done updates go fine again, but this not something we would like to do in our production environment.

Any help is appreciated

Thanks in advance,

Marco

2

Hi Marco,
We seem to be experiencing the same issue. Did you find a resolution that did not involve removing the app and deploying a new one?

Chris

3

Hi @chrisgelhaus Can you fix this issue?

4

Wee did find a workaround. What we did is that we asked all staff to remove their tokens using the GUI. Apparently the token of the user who deployed the app is no longer valid. You can see the owner in the API view.
Once the token is removed Rancher will assign a new owner to the app during the upgrade.

5

Hi @mverleun !
Thanks for the information I will give it a try. Is this safe to do when the App is stuck in the “Installing” state?

6

You do not have much choice do you? If I remember correctly we removed the tokens first and then ‘updated’ the app (to the same version).Otherwise it remains at installing.