TechPreview v2.0.0-alpha16 single node setup /healthz 403 forbidden

Rancher 2.0 Tech Preview
1

Hello,
I just tried to set up a single node env with v2.0.0-alpha16

  • rancher server startup
  • add cluster
  • add node for etcd, worker node and management

After a few moments the UI shows the following error

[controlPlane] Failed to bring up Control Plane: Failed to verify healthcheck: service [kube-api] is not healthy response code: [403], response body: {“kind”:“Status”,“apiVersion”:“v1”,“metadata”:{},“status”:“Failure”,“message”:“forbidden: User “system:anonymous” cannot get path “/healthz””,“reason”:“Forbidden”,“details”:{},“code”:403}

Anybody experiences the same problem?

regards,
Christian

3 Likes
2

Can you share what you are doing exactly? Are you adding a cloud host or through docker run? What are the specs of the machine? What OS/kernel/Docker version are you using?

3

Of course,

I´m using Ubuntu 16.04 with 4.4.0-87-generic with docker 17.12.0-ce, but also tried 17.03 before.

I chose to create a rke cluster and select to setup nodes later. Then I add a custom node via docker run using for etcd, management as well as worker-node.

Just to clarify, rancher server and agent run on the same host.

regards

1 Like
4

Hello,

I have same problem. Same Ubuntu, same Docker version.

5

Someone created a issue for that and I appended some logs for the devs here https://github.com/rancher/rancher/issues/11368

1 Like
6

Hello,

The first time, disabled IPv6 solved my problem. But since, same. I tried to install and clean and re-install again rancher/preview. Always same error.

Now i am with alpha 24.

Error :
[controlPlane] Failed to bring up Control Plane: Failed to verify healthcheck: service [kube-apiserver] is not healthy response code: [403], response body: {“kind”:“Status”,“apiVersion”:“v1”,“metadata”:{},“status”:“Failure”,“message”:“forbidden: User “system:anonymous” cannot get path “/healthz””,“reason”:“Forbidden”,“details”:{},“code”:403}

7

will try it. thanks for sharing. will come with updates but i would appreciate a lot if you would come back with updates in case you find the solution! thanks a lot.

8

In my case ipv6 was always disabled and it didn´t work.

Does anybody tried now with beta (==alpha-24?)? Next week I can try it by myself. I´ll give a notice for result.

9

nope, not yet working.

10

First step is when re-using nodes, they need to be cleaned to make sure no old data is left behind: https://gist.github.com/superseb/2cf186726807a012af59a027cb41270d

After, ouput of docker logs --tail=all etcd and docker logs --tail=all kube-apiserver can help determining what is going wrong.

3 Likes
11

OMG, it works facepalm Thx for the tip.

12

I cleanedup something and advance to another issue

2018/05/04 19:52:03 [ERROR] cluster [c-skwp5] provisioning: [workerPlane] Failed to bring up Worker Plane: Failed to verify healthcheck: Failed to check https://localhost:10250/healthz for service [kubelet] on host [172.27.2.219]: Get https://localhost:10250/healthz: dial tcp 127.0.0.1:10250: getsockopt: connection refused

13

docker logs kubelet on host 172.27.2.219 will show why it’s not starting.