Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
VMWare VMs Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)
Two node Rancher HA with external MySQL DB
Once Kubernetes finishes full setup, generate config file as user who created the ‘environment’/‘cluster’
Login with a user that’s not an owner but regular member of the environment
Generate kubeconfig file as that user
Compare kubeconfig files - they are identical effectively giving the regular user cluster-admin privileges.
Results:
Any user, be it an owner or member, get exactly the same kubeconfig file with the token which gives them cluster-admin privileges. This is highly insecure.
I’m on my phone but if true this has to be a comparatively minor UI bug that is only possible on logout/in in the same browser window without a full refresh in between because the old key is still loaded in client-side (your browser) memory.
The credential portion is an account API key, the secret part is only readable once on create so it couldn’t generate the same config twice if it wanted to. And the rest of the file is a template in the UI. There is no saved config on the backend to return to the wrong user.
Thanks for your swift reply. The fact is I tested this with two different users, both logged in on their own laptop using their default browser and using their personal AD login. All three kubeconfig files are identical, i.e. mine as cluster-admin.
I still believe this is impossible, and cannot reproduce. Left browser is Chrome, admin and owner of the env, right is Firefox, regular user and member of it. You can see a new API key is created each time (the get afterward is unrelated…)
I’m guessing what you’re seeing is they look the same from casual inspection but are not actually comparing the entire string.
The beginning and end are always going to be the same because the token is base64("Basic " + base64(accessKey + ":" + secretKey)). So the first several characters are the base64(“Basic “), and the last several are base64(”==”) because the keypair string is not a multiple of 3 characters long.
Thanks for taking the time to investigate. Wild guess, the users you tested with are local users, right? In my case Rancher is AD integrated so no local users. Don’t know if that would make a difference. Truth is that of course I only compared beginning and end of the string so I’ll have another look on Monday when I’m back in the office. This issue got my attention because one of the users, who’s supposed to be only namespace admin was able to, by accident, delete ClusterRole, Role, ResourceQuota and some more erroneous stuff that effectively forced me to rebuild the cluster to have it up and running as fast as possible.
I did test this scenario with Rancher 2 (alpha16) with local users of course and that works like a charm, even at username level in the kubeconfig file.
To be continued on Monday :-). Have a nice weekend and thanks again.
I did use local but it’s not going to matter. The config contains an API key, keys (and almost everything else) are tied to an account (DB entry). The auth provider you have on maps the person logging in to their account entry.
This is all a red herring rooted in describing the (perceived) cause instead of what the symptom actually was. You’ve now described the actual problem, “k8s rbac allowed someone to do something I don’t think they should have been able to”.
So the first thing to look at would be if RBAC is even enabled for the environment, and then start looking at what all the role bindings are (if they didn’t delete those too).
RBAC is enabled. I’m very ‘gung ho’ on security in our company especially since I’m the one who’ll have to fix issues. I thoroughly tested it by having one of my colleagues that’s not a cluster admin try to delete / modify quota on a namespace, modify / delete ClusterRoleBindings and RoleBindings. He was unable to.
I will double check on Monday and provide feedback. I sincerely hope it’s a red herring as you indicated.
I double checked the token in the kubeconfig file from different users and you are correct, it’s different in the middle. Guess I’ll have to get deeper in the RBAC setup to see what I did wrong.
Thanks for your assistance and feedback!
Kind regards,
Eric V.
PS: I closed the GitHub issue I created on Rancher
