when I try to add a cluster to rancher rke2 it stays pending and in the logs of the cattle-cluster agent I see these errors
level=fatal msg="Certificate chain is not complete, please check if all needed intermediate certificates are included in the server certificate (in the correct order) and if the cacerts setting in Rancher either contains the correct CA certificate (in the case of using self signed certificates) or is empty (in the case of using a certificate signed by a recognized CA). Certificate information is displayed above. error: Get “https://xxxxxxxx ”: x509: certificate signed by unknown authority
1 Like
The error message seems to be pretty self explanatory, what is it you don’t understand ?
shlomi
June 23, 2022, 6:05am
3
Sound like your using self sign certs?
Rancher docs
I have executed on one of the nodes of the cluster that I want to import the kubectl insecure command to avoid the validation of the certificates when they are self-signed, but I still have the cluster pending and the cattle-cluster-agent pod gives an error. Any ideas?
There is a related problem and they point to a problem with version v2.6.5, do you advise me to uninstall and install a new version?
13:06
opened 08:37PM - 27 Dec 21 UTC
closed 11:30AM - 30 Apr 22 UTC
status/stale
**Rancher Server Setup**
- Rancher version:
- Stable/latest
- `docker run… -d --restart=unless-stopped -p 80:80 -p 443:443 --privileged rancher/rancher:stable`
- Installation option (Docker install/Helm Chart):
- If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc):
- Proxy/Cert Details:
- No proxy
**Information about the Cluster**
- Kubernetes version:
- Cluster Type (Local/Downstream):
- Downstream installed using `kubeadm` on Bare Metal
```
Client Version: v1.21.8
Server Version: v1.21.8
```
**Describe the bug**
On running `curl --insecure -sfL https://rancher.example.local/v3/import/tw42v2mdckhnm9p9s675vnk4rm7vtj7whwr6cgffk7lgqdrdfx8xsc_c-m-rj7pvd4w.yaml | kubectl apply -f -` the generic cluster should join the Rancher. However, it does not and keeps always in pending state.
There is no proxy server between the cluster and the rancher server. Also, if the rancher version is 2.5.4, it works perfectly fine.
**To Reproduce**
1. Install a Kubernetes Cluster on a Bare metal using kubeadm
2. Install a rancher server using docker version: `latest/stable/ 2.6`
3. Import the Kubernetes cluster to the rancher server
4. All the VMs (running `Debian buster`) are created on `Debian Bullseye` host running `KVM` and `libvirt` provisioned using terraform.
**Result**
**Expected Result**
Kubernetes cluster to be imported to the Rancher.
**Screenshots**
```
kubectl get all -n cattle-system
NAME READY STATUS RESTARTS AGE
pod/cattle-cluster-agent-84fb5bb984-r6n5t 1/1 Running 0 21m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cattle-cluster-agent ClusterIP 10.100.16.141 <none> 80/TCP,443/TCP 16h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/cattle-cluster-agent 1/1 1 1 16h
NAME DESIRED CURRENT READY AGE
replicaset.apps/cattle-cluster-agent-84fb5bb984 1 1 1 21m
```
Logs of the `cattle-cluster-agent`:
`kubectl logs cattle-cluster-agent-84fb5bb984-r6n5t -n cattle-system`
```
kubectl logs cattle-cluster-agent-84fb5bb984-r6n5t -n cattle-system
INFO: Environment: CATTLE_ADDRESS=10.244.0.14 CATTLE_CA_CHECKSUM=8464dde7dbb0f08913a9566c37a9de4c0052b687927b543c33dedadf81e7134e CATTLE_CLUSTER=true CATTLE_CLUSTER_AGENT_PORT=tcp://10.100.16.141:80 CATTLE_CLUSTER_AGENT_PORT_443_TCP=tcp://10.100.16.141:443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_ADDR=10.100.16.141 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PORT=443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_PORT_80_TCP=tcp://10.100.16.141:80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_ADDR=10.100.16.141 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PORT=80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_SERVICE_HOST=10.100.16.141 CATTLE_CLUSTER_AGENT_SERVICE_PORT=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTP=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTPS_INTERNAL=443 CATTLE_CLUSTER_REGISTRY= CATTLE_INGRESS_IP_DOMAIN=sslip.io CATTLE_INSTALL_UUID=4a9d48d4-5351-4a23-bec8-57cbec62edd7 CATTLE_INTERNAL_ADDRESS= CATTLE_IS_RKE=false CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-84fb5bb984-r6n5t CATTLE_SERVER=https://rancher.bitvijays.local CATTLE_SERVER_VERSION=v2.6.3
INFO: Using resolv.conf: search cattle-system.svc.cluster.local svc.cluster.local cluster.local bitvijays.local nameserver 10.96.0.10 options ndots:5
INFO: https://rancher.bitvijays.local/ping is accessible
INFO: rancher.bitvijays.local resolves to 192.168.1.208
INFO: Value from https://rancher.bitvijays.local/v3/settings/cacerts is an x509 certificate
time="2021-12-27T19:56:01Z" level=info msg="Listening on /tmp/log.sock"
time="2021-12-27T19:56:01Z" level=info msg="Rancher agent version v2.6.3 is starting"
time="2021-12-27T19:56:06Z" level=info msg="Connecting to wss://rancher.bitvijays.local/v3/connect/register with token starting with tw42v2mdckhnm9p9s675vnk4rm7"
time="2021-12-27T19:56:06Z" level=info msg="Connecting to proxy" url="wss://rancher.bitvijays.local/v3/connect/register"
time="2021-12-27T19:56:16Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp: i/o timeout"
time="2021-12-27T19:56:16Z" level=error msg="Remotedialer proxy error" error="dial tcp: i/o timeout"
time="2021-12-27T19:56:26Z" level=info msg="Connecting to wss://rancher.bitvijays.local/v3/connect/register with token starting with tw42v2mdckhnm9p9s675vnk4rm7"
time="2021-12-27T19:56:26Z" level=info msg="Connecting to proxy" url="wss://rancher.bitvijays.local/v3/connect/register"
time="2021-12-27T19:56:36Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp: i/o timeout"
time="2021-12-27T19:56:36Z" level=error msg="Remotedialer proxy error" error="dial tcp: i/o timeout"
```
Read https://github.com/rancher/docs/issues/2286, https://github.com/rancher/rancher/issues/24876 and thought that maybe if we specify `NO_PROXY` environment with the `ip_address` of the `rancher` server, it should work. However, It seems that currently, `rancher` image does not respect the `NO_PROXY` environment.
We set the environment variable using
```
- name: CATTLE_INGRESS_IP_DOMAIN
value: sslip.io
- name: NO_PROXY
value: 192.168.1.209,0.0.0.0
```
However, when checked using the logs, it seems it does not accepts the `NO_PROXY` environment variable.
```
kubectl logs cattle-cluster-agent-7988759d55-th2lj -n cattle-system -f
INFO: Environment: CATTLE_ADDRESS=10.244.0.21 CATTLE_CA_CHECKSUM=c0923901cefe032b6b642ba667979ac3d91eb6313fab985de7aa28b7b56fe96e CATTLE_CLUSTER=true CATTLE_CLUSTER_AGENT_PORT=tcp://10.100.16.141:80 CATTLE_CLUSTER_AGENT_PORT_443_TCP=tcp://10.100.16.141:443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_ADDR=10.100.16.141 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PORT=443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_PORT_80_TCP=tcp://10.100.16.141:80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_ADDR=10.100.16.141 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PORT=80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_SERVICE_HOST=10.100.16.141 CATTLE_CLUSTER_AGENT_SERVICE_PORT=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTP=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTPS_INTERNAL=443 CATTLE_CLUSTER_REGISTRY= CATTLE_INGRESS_IP_DOMAIN=sslip.io CATTLE_INSTALL_UUID=9ac14868-1948-4d3b-a080-8c684b465454 CATTLE_INTERNAL_ADDRESS= CATTLE_IS_RKE=false CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-7988759d55-th2lj CATTLE_SERVER=https://rancher.bitvijays.local CATTLE_SERVER_VERSION=v2.6.3
INFO: Using resolv.conf: search cattle-system.svc.cluster.local svc.cluster.local cluster.local bitvijays.local nameserver 10.96.0.10 options ndots:5
INFO: https://rancher.bitvijays.local/ping is accessible
INFO: rancher.bitvijays.local resolves to 192.168.1.209
INFO: Value from https://rancher.bitvijays.local/v3/settings/cacerts is an x509 certificate
```
**Additional context**
shlomi
June 24, 2022, 7:20am
6
What is the version of the rke cluster?
rke2 version v1.21.6+rke2r1
Is it possible that it has something to do with the tolerances that the cattle-node-agent has?Rancher Docs: Rancher Agents