Hi Team,
We have setup a k8s environment with rancher. Currently, we notice that the k8s-agent and k8s-ctld containers restarted automatically and periodically by rancher, about twice a day.
Is this normal operations? If not, how can I look into this issue and fix it?
Rancher: 1.6.12
K8s: 1.8
Docker: 17.03.2-ce
OS: Ubuntu 16.04.3
The log found in k8s-agent container, the error message is nil
time=“2018-01-01T17:12:46Z” level=info msg=“Received event: [ADDED] for Service: default/xxx, Handling event.”
time=“2018-01-02T05:12:59Z” level=error msg=“Rancher stream listener exited with error: %!s()”
time=“2018-01-02T05:12:59Z” level=info msg=Exiting.
MD=169.254.169.250
curl -s -f http://169.254.169.250/2015-12-19/stacks/Kubernetes/services/kubernetes/uuid
/usr/bin/update-rancher-ssl
No useful message found in k8s-ctld log
These 2 containers are restarted twice a day around 11:11 UTC and 23:11 UTC, here are the system logs about the container restarted,
Dec 31 11:11:49 kernel: [4055061.484900] docker0: port 6(vethr730bb3e96c) entered disabled state
Dec 31 11:11:49 kernel: [4055061.489008] device vethr730bb3e96c left promiscuous mode
Dec 31 11:11:49 kernel: [4055061.489013] docker0: port 6(vethr730bb3e96c) entered disabled state
Dec 31 11:11:50 kernel: [4055062.438745] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Dec 31 11:11:50 kernel: [4055062.467757] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Dec 31 11:11:50 kernel: [4055062.468055] device vethr730bb3e96c entered promiscuous mode
Dec 31 11:11:50 kernel: [4055062.468088] docker0: port 6(vethr730bb3e96c) entered forwarding state
Dec 31 11:11:50 kernel: [4055062.468101] docker0: port 6(vethr730bb3e96c) entered forwarding state
Dec 31 11:12:05 kernel: [4055077.498599] docker0: port 6(vethr730bb3e96c) entered forwarding state
Dec 31 23:12:07 kernel: [4098278.742598] docker0: port 6(vethr730bb3e96c) entered disabled state
Dec 31 23:12:07 kernel: [4098278.746990] device vethr730bb3e96c left promiscuous mode
Dec 31 23:12:07 kernel: [4098278.746996] docker0: port 6(vethr730bb3e96c) entered disabled state
Dec 31 23:12:08 kernel: [4098279.726146] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Dec 31 23:12:08 kernel: [4098279.757667] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Dec 31 23:12:08 kernel: [4098279.758037] device vethr730bb3e96c entered promiscuous mode
Dec 31 23:12:08 kernel: [4098279.758077] docker0: port 6(vethr730bb3e96c) entered forwarding state
Dec 31 23:12:08 kernel: [4098279.758091] docker0: port 6(vethr730bb3e96c) entered forwarding state
Dec 31 23:12:23 kernel: [4098294.780291] docker0: port 6(vethr730bb3e96c) entered forwarding state
Jan 1 11:12:25 kernel: [4141495.911471] docker0: port 6(vethr730bb3e96c) entered disabled state
Jan 1 11:12:25 kernel: [4141495.915088] device vethr730bb3e96c left promiscuous mode
Jan 1 11:12:25 kernel: [4141495.915092] docker0: port 6(vethr730bb3e96c) entered disabled state
Jan 1 11:12:43 kernel: [4141513.180568] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jan 1 11:12:43 kernel: [4141513.205615] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jan 1 11:12:43 kernel: [4141513.205933] device vethr730bb3e96c entered promiscuous mode
Jan 1 11:12:43 kernel: [4141513.205987] docker0: port 6(vethr730bb3e96c) entered forwarding state
Jan 1 11:12:43 kernel: [4141513.206001] docker0: port 6(vethr730bb3e96c) entered forwarding state
Jan 1 11:12:58 kernel: [4141528.216448] docker0: port 6(vethr730bb3e96c) entered forwarding state
Jan 1 23:12:59 kernel: [4184728.666132] docker0: port 6(vethr730bb3e96c) entered disabled state
Jan 1 23:12:59 kernel: [4184728.670135] device vethr730bb3e96c left promiscuous mode
Jan 1 23:12:59 kernel: [4184728.670142] docker0: port 6(vethr730bb3e96c) entered disabled state
Jan 1 23:13:00 kernel: [4184729.587861] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jan 1 23:13:00 kernel: [4184729.601042] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jan 1 23:13:00 kernel: [4184729.601351] device vethr730bb3e96c entered promiscuous mode
Jan 1 23:13:00 kernel: [4184729.601391] docker0: port 6(vethr730bb3e96c) entered forwarding state
Jan 1 23:13:00 kernel: [4184729.601405] docker0: port 6(vethr730bb3e96c) entered forwarding state
Jan 1 23:13:15 kernel: [4184744.631996] docker0: port 6(vethr730bb3e96c) entered forwarding state
Thanks very much for your great help