No container stats and cannot execute shell/view logs on some hosts

rancher 1.1.0-dev1
docker 1.10.3

Running into some issues on some hosts. 2 hosts are fine (running 1.10.3 and 1.7.0). 2 other hosts that I just brought up (1.10.3) have issues.

If I click on a container on a problematic host, the CPU/Memory/Network/Storage graphs displays “Connecting…” and occasionally flashes the graph axis for less than a second. Execute shell will flash the shell window up for less than a second before it goes away. View Logs will bring up the window but it says “Disconnected” on the bottom.

Any ideas?

What type of OS is the host running??

Server on Debian Jessie. Host on that same machine and other Debian Jessie is fine.

Problematic hosts are running CentOS 7.2 and Debian Jessie.

FYI - we are experiencing this issue too. We’re running REHL 7.2.

I was exeriencing this as well. So I upgraded Docker and Rancher, and verified the time on each server was close, and this seemed to help. I think there are other similar “problems” reported on this topc.

Ah, that seems to be it. The server time was a bit off (498 s) and the two new machines weren’t able to be connected to. One of the older ones that did work was 234 s from the server, so there must be a threshold where it fails. Time to set up ntp on these test boxes.

Yes, the server generates a JWT (token) which contains the host/container/etc the request is for, then the client gives that back to what ends up being the target host to verify, and they are good for 5 minutes.