Kubectl authentication error on fresh rancher 1.6 setup with kubernetes and access control

0 Single ubuntu 16 machine

  1. Fresh Rancher 1.6 setup as described here: http://rancher.com/docs/rancher/latest/en/installing-rancher/installing-server/basic-ssl-config/
  2. Access control enabled with local authentication
  3. Kubernetes environment created
  4. API keys created for account and for kubernetes environment (verified that these work via rancher API curl commands)
  5. kubectl at htps://mydomain/env/1a7/kubernetes/kubectl works and kubectl command can be executed
  6. kubernetes UI at htps://mydomain/r/projects/1a7/kubernetes-dashboard:9090 also works
  7. Installed kubectl on the same ubuntu instance and created a ~/.kube/config as instructed. Executing commands also works!!!

Now the problem. the generated ~/.kube/config file uses a ‘token’ to access kubernetes.
For automation purposes I want to run kubectl with a username\password or key pair that I generated in rancher. (see my step 4 above). But this does not work.

If I run “kubectl --username xxx --password get yyy all” I always get an error.
When xxx and yyy are a valid combination I get an authorization error "You must be logged in to the server (the server has asked for the client to provide credentials) "
When xxx and yyy are an invalid combination I get a json parse error

When disabling access control in rancher and running kubectl with blank username/password/token things start working again.

What am I missing here?

Going to answer my own question…

Initially I thought I could fix it by enabling kubernetes rbac in the environment template as I had not done this.
However after enabling this I only got a different error.
Instead of getting the ‘You must be logged in to the server’ message when running kubectl with username and password I now got a ‘anonymous user is not allowed go do xxx in the default namespace’

Again when using the token (instead of username\password) as suggested in the rancher CLI window I had no problems.
Then I stumbled on this post:

And when converting my username and password (actually an account key id and account secret in such a token) things worked!!!

This token approach was independent of enabling/disabling kubernetes rbac.

As a last test I generated another token. This time with an environment key and secret. (instead of an account key/secret).
Even though this keypair was for the same admin user that was owner of the environment, this time I was back to the ‘You must be logged in to the server’ message.

So,… progress but still questions…

  • Why does it work with a account token that is generated from a keyid and keysecret while it does not work when using the same keyid and secret as a username/password combination?
  • Why does it work for an account key/pair but not for an environment keypair for the same admin user/environment owner?