coeski
February 15, 2022, 4:01pm
1
Hi,
I currently have Rancher cluster 2.5.5 installed, and i have some issues related to one of my manager nodes hanging due to defunct PIDs and over time these build up until the OS runs out of file descriptors and the server falls over.
This appears to be a know issue that is fixed in Rancher v2.5.8
opened 06:54PM - 20 Nov 20 UTC
closed 07:26PM - 16 Apr 21 UTC
kind/enhancement
priority/2
internal
<!--
Please search for existing issues first, then read https://rancher.com/docs… /rancher/v2.x/en/contributing/#bugs-issues-or-questions to see what we expect in an issue
For security issues, please email security@rancher.com instead of posting a public issue in GitHub. You may (but are not required to) use the GPG key located on Keybase.
-->
**What kind of request is this (question/bug/enhancement/feature request):**
Bug
**Steps to reproduce (least amount of steps as possible):**
1. Set up a downstream RKE cluster with controlplane/worker/etcd roles
2. May need to let environment run for a few hours or days (or longer)
3. Run `ps -aef| grep git-remote-http | grep defunct`
**Result:**
```
root@ip-10-0-10-10:~# ps -aef| grep git-remote-http | grep defunct
root 691961 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 693565 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 701971 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 713516 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 718538 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 726678 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 728547 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 741905 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 748518 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 753543 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 761990 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 763578 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 771680 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 773302 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 776697 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 778313 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 781703 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 783296 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 786716 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 788309 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 791693 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 793317 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 796715 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 798323 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 803329 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 806739 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 808318 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 811774 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 813381 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 816810 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 818416 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 821823 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 823412 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 826827 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 828462 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 831853 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 833453 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 836843 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 841848 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 843445 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 846840 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 848460 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 851870 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 853489 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 856889 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 858516 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 861918 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 863487 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 868464 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 871850 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 873468 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 876856 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 878464 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 881852 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 883454 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 886899 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 888499 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 891887 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 893512 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 896904 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 898522 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 901929 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 903529 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 906939 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 911918 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 913517 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 916906 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 918523 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 921911 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 923521 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 926916 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 928517 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 931954 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 933534 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 936926 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 938508 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 941916 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 943524 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 946962 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 948577 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 952012 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 953619 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 957011 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 958599 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 962008 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 963620 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 967017 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 968633 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 972058 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 973662 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 977044 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 978652 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 982017 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 983625 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 987038 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 988639 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 992029 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 993636 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 997049 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 998635 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 1002027 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 1003623 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 1007068 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 1008661 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 1013683 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 1047331 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 1063943 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 1072376 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 1114015 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 1164008 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 1192443 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 1207442 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 1212456 1938477 0 Nov14 ? 00:00:00 [git-remote-http] <defunct>
root 1939739 1938477 0 Oct22 ? 00:00:00 [git-remote-http] <defunct>
root 1940142 1938477 0 Oct22 ? 00:00:00 [git-remote-http] <defunct>
root 1940285 1938477 0 Oct22 ? 00:00:00 [git-remote-http] <defunct>
root 1940319 1938477 0 Oct22 ? 00:00:00 [git-remote-http] <defunct>
root 1940684 1938477 0 Oct22 ? 00:00:00 [git-remote-http] <defunct>
root 1940750 1938477 0 Oct22 ? 00:00:00 [git-remote-http] <defunct>
root 1940763 1938477 0 Oct22 ? 00:00:00 [git-remote-http] <defunct>
root 1940793 1938477 0 Oct22 ? 00:00:00 [git-remote-http] <defunct>
root 1940822 1938477 0 Oct22 ? 00:00:00 [git-remote-http] <defunct>
root 1940828 1938477 0 Oct22 ? 00:00:00 [git-remote-http] <defunct>
root 1940914 1938477 0 Oct22 ? 00:00:00 [git-remote-http] <defunct>
root 1941101 1938477 0 Oct22 ? 00:00:00 [git-remote-http] <defunct>
root 1941272 1938477 0 Oct22 ? 00:00:00 [git-remote-http] <defunct>
root 1941642 1938477 0 Oct22 ? 00:00:00 [git-remote-http] <defunct>
root 2019753 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2023155 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2024758 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2029990 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2033161 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2034739 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2039756 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2043166 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2044772 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2048280 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2049884 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2053285 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2054868 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2058284 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2059877 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2063267 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2064879 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2068271 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2069877 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2073259 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2074850 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2078256 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2079849 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2083236 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2084824 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
root 2088252 1938477 0 Nov15 ? 00:00:00 [git-remote-http] <defunct>
```
**Other details that may be helpful:**
This PID matches this container:
```
# docker inspect 079 | grep -i "pid\|Name"
"Pid": 1938477,
"HostnamePath": "/var/lib/docker/containers/2b3fab56a2bd8d4b3be4a26d0eec54ab8f589e8cd4ab7fce7e031ea7a3fa09f2/hostname",
"Name": "/k8s_cluster-register_cattle-cluster-agent-5c97dd84f-h9ctr_cattle-system_c1b8ec65-40c3-4195-aa11-63d7e341c38d_0",
"Name": "no",
"PidMode": "",
"PidsLimit": null,
"Name": "overlay2"
"Hostname": "cattle-cluster-agent-5c97dd84f-h9ctr",
"Domainname": "",
"io.kubernetes.container.name": "cluster-register",
"io.kubernetes.pod.name": "cattle-cluster-agent-5c97dd84f-h9ctr",
"io.kubernetes.pod.namespace": "cattle-system",
```
Looks like there was a similar problem in v2.0 with another rancher workload - https://github.com/rancher/rancher/issues/13858
**Environment information**
- Rancher version (`rancher/rancher`/`rancher/server` image tag or shown bottom left in the UI):
v2.5.2
- Installation option (single install/HA):
HA
<!--
If the reported issue is regarding a created cluster, please provide requested info below
-->
**Cluster information**
- Cluster type (Hosted/Infrastructure Provider/Custom/Imported):
Custom
- Machine type (cloud/VM/metal) and specifications (CPU/memory):
t3a.medium on AWS EC2 - 2vCPU, 8GB RAM
- Kubernetes version (use `kubectl version`):
```
> kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:32:58Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
```
- Docker version (use `docker version`):
```
root@ip-10-0-10-10:~# docker version
Client:
Version: 19.03.8
API version: 1.40
Go version: go1.13.8
Git commit: afacb8b7f0
Built: Wed Oct 14 19:43:43 2020
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.13.8
Git commit: afacb8b7f0
Built: Wed Oct 14 16:41:21 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.3.3-0ubuntu2
GitCommit:
runc:
Version: spec: 1.0.1-dev
GitCommit:
docker-init:
Version: 0.18.0
GitCommit:
```
gz#13535
gz#16312
I’d like to know if anyone has any opinions on my upgrade path to 2.5.x?? i’d like to go for a stable new releases say .11?
Any help would be very much appreciated.
Regards