I have a 2 node k3s cluster, the agent node is having issues and I’m seeing CIDRNotAvailable messages in the event log:
❯ kubectl get nodes node-01-01 -o go-template --template='{{range .spec.podCIDRs}}{{printf "%s\n" .}}{{end}}'
10.42.0.0/24
2001:470:c0b5:4::/64
❯ kubectl get nodes anbus -o go-template --template='{{range .spec.podCIDRs}}{{printf "%s\n" .}}{{end}}'
❯ kubectl get events | grep CIDRNotAvailable
4m10s Normal CIDRNotAvailable node/anbus Node anbus status is now: CIDRNotAvailable
I created the cluster with:
k3sup install --ip $IP01 --user mfreeman --k3s-extra-args '--disable traefik --disable servicelb --disable-cloud-controller --kube-proxy-arg proxy-mode=ipvs --cluster-cidr=10.42.0.0/16,2001:470:c0b5:4::/64 --service-cidr=10.43.0.0/16,2001:470:c0b5:5::/108 --disable-network-policy --flannel-backend=none'
Using Calico for CNI, MetalLB for load balancer.
❯ kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 82m
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 82m
kube-system metrics-server ClusterIP 10.43.40.38 <none> 443/TCP 82m
calico-system calico-typha ClusterIP 10.43.181.107 <none> 5473/TCP 78m
calico-system calico-kube-controllers-metrics ClusterIP None <none> 9094/TCP 78m
calico-apiserver calico-api ClusterIP 10.43.143.42 <none> 443/TCP 77m
metallb-system metallb-webhook-service ClusterIP 10.43.2.211 <none> 443/TCP 65m
default nginx LoadBalancer 10.43.247.146 10.43.0.3,2001:470:c0b5:5::1 80:32030/TCP 50m
❯ kubectl describe node anbus
Name: anbus
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=anbus
kubernetes.io/os=linux
Annotations: csi.volume.kubernetes.io/nodeid: {"csi.tigera.io":"anbus"}
k3s.io/node-args: ["agent"]
k3s.io/node-config-hash: 2BPHIFFMZ3HVLJARMEFKN6G6SIKDLWWAHM4GPQMDH5UAKAY5WYQQ====
k3s.io/node-env:
{"K3S_DATA_DIR":"/var/lib/rancher/k3s/data/b159f6e26663d8c92285e7bc4a6881d85bd8c81efc55eb2cf191c54100387fbb","K3S_TOKEN":"********","K3S_U...
node.alpha.kubernetes.io/ttl: 0
projectcalico.org/IPv4Address: 192.168.1.80/24
projectcalico.org/IPv6Address: fde7:d5a8:bddb:da4a:78b2:2b8a:c5b9:8839/64
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sat, 18 May 2024 20:50:32 -0500
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: anbus
AcquireTime: <unset>
RenewTime: Sat, 18 May 2024 22:12:38 -0500
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Sat, 18 May 2024 21:39:01 -0500 Sat, 18 May 2024 21:39:01 -0500 CalicoIsUp Calico is running on this node
MemoryPressure False Sat, 18 May 2024 22:07:53 -0500 Sat, 18 May 2024 20:56:56 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Sat, 18 May 2024 22:07:53 -0500 Sat, 18 May 2024 20:56:56 -0500 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Sat, 18 May 2024 22:07:53 -0500 Sat, 18 May 2024 20:56:56 -0500 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Sat, 18 May 2024 22:07:53 -0500 Sat, 18 May 2024 20:56:56 -0500 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 192.168.1.80
Hostname: anbus
Capacity:
cpu: 16
ephemeral-storage: 293165952Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 65497100Ki
pods: 110
Allocatable:
cpu: 16
ephemeral-storage: 285191837882
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 65497100Ki
pods: 110
System Info:
Machine ID: dc51060b2e144ede95bf797b25908473
System UUID: c1453500-1640-a57e-1dff-1831bf2648d9
Boot ID: e21dc610-835e-44b8-bd4d-192945b4acd4
Kernel Version: 6.5.0-35-generic
OS Image: Ubuntu 22.04.4 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.7.15-k3s1
Kubelet Version: v1.29.4+k3s1
Kube-Proxy Version: v1.29.4+k3s1
Non-terminated Pods: (10 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system local-path-provisioner-6c86858495-xp2ls 0 (0%) 0 (0%) 0 (0%) 0 (0%) 83m
calico-system csi-node-driver-bl7wj 0 (0%) 0 (0%) 0 (0%) 0 (0%) 79m
calico-system calico-typha-67c88d56f6-wgtnn 0 (0%) 0 (0%) 0 (0%) 0 (0%) 79m
calico-apiserver calico-apiserver-7c95c6697b-hvzw4 0 (0%) 0 (0%) 0 (0%) 0 (0%) 78m
kube-system coredns-6799fbcd5-ftb8z 100m (0%) 0 (0%) 70Mi (0%) 170Mi (0%) 83m
kube-system metrics-server-54fd9b65b-cqjq9 100m (0%) 0 (0%) 70Mi (0%) 0 (0%) 83m
tigera-operator tigera-operator-76c4974c85-nr9qg 0 (0%) 0 (0%) 0 (0%) 0 (0%) 80m
default nginx-d5bd76cff-jbkb8 100m (0%) 250m (1%) 128Mi (0%) 256Mi (0%) 46m
calico-system calico-node-8blgc 0 (0%) 0 (0%) 0 (0%) 0 (0%) 33m
default external-dns-6dc79c78c9-qffv2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 26m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 300m (1%) 250m (1%)
memory 268Mi (0%) 426Mi (0%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CIDRNotAvailable 4m55s (x24 over 72m) cidrAllocator Node anbus status is now: CIDRNotAvailable