Switching to containerd from docker causes intermittent pod networking connection refused

k3s, k3OS, and k3d
1

I need to move to containerd instead of docker. Currently k3s is started with:

ExecStart=/usr/local/bin/k3s server\
 --write-kubeconfig-mode 640 \
 --disable traefik --disable metrics-server --disable-cloud-controller \
 --docker \
 --cluster-cidr 10.242.0.0/16 --service-cidr 10.243.0.0/16 --cluster-dns 10.243.0.10 \
 --kube-controller-arg=terminated-pod-gc-threshold=100 \
 --kubelet-arg=image-gc-high-threshold=100 \
 --kubelet-arg=eviction-hard= \
 --kubelet-arg=cpu-cfs-quota=true \
 --kubelet-arg=volume-stats-agg-period=15m \
 --kubelet-arg=housekeeping-interval=30s \
 --node-name singlenodecluster \
 --flannel-iface tap0 \
 --kube-proxy-arg "proxy-mode=ipvs"

I am using 1.29.3. I have tried 1.29.5 and 1.30.x.

Since removing the --docker \ line I intermittently get networking issues - I have only seen this between pods. I have had connection refused and even dns failures. Looking at the logs it’s like networking stops working for 1 to 15 seconds. I get pod log message like:
Connection refused: AH00957: HTTP: attempt to connect to 10.243.88.156:8888 (*) failed Then it recovers after say 15 seconds of retries.

In general everything else is fine. No obvious changes in CPU, memory. Pods are healthy.

Things of note to explain why I have the above options:
–*-cidr: to avoid overlap with internal networks
–flannel-iface tap0: The idea of using a tap interface is to allow flannel, and thus k3s, to work if eth0/net0 is down for whatever reason and when there is no default gateway.
–kube-proxy-arg “proxy-mode=ipvs”: an IP alias is added for each service to the kube-ipvs0 interface which means routing to services works even when there’s no default gateway - as needed for true airgapped installations.

Any ideas?

2

I have noticed log entries like these seem to happen a lot of the time networking stops:

Jun 19 15:01:37 yorkie k3s: time="2024-06-19T15:01:37Z" level=info msg="COMPACT compactRev=11484 targetCompactRev=12070 currentRev=13070"
Jun 19 15:01:37 yorkie k3s: time="2024-06-19T15:01:37Z" level=info msg="COMPACT deleted 310 rows from 586 revisions in 11.367146ms - compacted to 12070/13070"
Jun 19 15:01:37 yorkie k3s: time="2024-06-19T15:01:37Z" level=info msg="COMPACT revision 12070 has already been compacted"