No graphs show for hardware hosts

AlexRRR · December 14, 2015, 9:08am

Hello Everyone,

First of all, thanks a lot, I am really liking rancher so far and I think is has a great future. Right now I am running into the following problem: I cannot see any data in the CPU, Memory, Storage or Network graphs under infrastructure. From the dev-tools console I see the following message

WebSocket connection to 'ws://docker-dev-hw-inx04.intranet.local.com:8080/v1/hoststats?token=eyJhbGciOiJSUzI1NiJ9.eyJleHAiOjE0NTAwODM5MjksInN1YiI6ImNhdHRsZSIsInJlc291cmNlSWQiOiIxaDEiLCJpc3MiOiJodHRwOlwvXC9jYXR0bGUuaW8iLCJob3N0VXVpZCI6ImQ0NTQwNjc4LWY4MzUtNDEwNi04NTVjLWJjZTcyNjAyNjZkMyIsImlhdCI6MTQ1MDA4MzYyOSwia2lkIjoiZGVmYXVsdCJ9.JqyLeHuKuyCqQAMmnhF0Y1mXNiF4s8nYcdzlVkHXaTvchWbuUiTxUkA_1GSzBUogXV_r8RK1HMXI0hIk5GgDRNKzZtWlbYbvtl9qUkMbxRUbLDl7fHJ___G2RI40SGqleILqPnAsTTXmnYIyotxDQerp420eXGR7sDPmESOpHuHuoVB6Sp1Kw8F_M5UI8tcio9dobIZrrzovZPhzvQxhhh_dqGXYOt3kZRj6yYSIr5y2XvoxFE4-RNJA22GM59dfSwav7fpMGPuRD59HBTLhLG8FAoXeU75kCoAUUBpKwpOaOYo_eZ1kvVcMKEMZJ6MbILT_CAKY0wRSSoHh7q2mBA' failed: HTTP Authentication failed; no valid credentials available

I am not sure why this is happening, everything else works like a charm, but this is confusing. Is cAdvisor or something else needed on the host for these graphs to work?

Any tips or ideas are welcome!

Best regards,

Alejandro

demarant · December 14, 2015, 11:16am

@AlexRRR
I had same issue, and it was due to our forward proxy (squid) at our work that was blocking the websockets protocol when not running Rancher on secure https.

To make the graphs work I enabled SSL on Rancher server and the charts worked, plus also the execute shell, view logs and many other features in Rancher UI that uses websockets. Make also sure you are on a stable modern browser.

You can check your browser support websockets on this websockets echo stie, if you try to connect to ws://echo.websocket.org and it fails, it means you are behind a proxy that is preventing ws protocol somehow.

The solution for me was as mentioned to enable SSL on rancher server. I hope this works for you.

cjellick · December 14, 2015, 4:40pm

@AlexRRR as @demarant said, I suspect some type of proxy between your browser and your rancher server that does not like/allow websocket tracffict. Any chance there is such a device (perhaps a corporate proxy) causing this?

AlexRRR · December 14, 2015, 6:57pm

@demarant @cjellick Thanks for the feedback on this.

There is nothing between the hosts and my browser in this case, other webSocket connections work (execute shell, view logs). It is the container stats and the host stats what seems to be broken.

My browser is the latest available Chrome for mac Version 47.0.2526.80 (64-bit).

The error appear exactly the moment when I click the hostname in the “infrastructure” section of
Rancher.
Again with some more context:

ui-201f54a90666244b49099ee70c63069d.js:60 Socket connect 3 to ws://docker-dev-hw-inx04:8080/v1/hoststats...
ui-201f54a90666244b49099ee70c63069d.js:60 Socket 2 opened
ui-201f54a90666244b49099ee70c63069d.js:60 WebSocket connection to 'ws://docker-dev-hw-inx04:8080/v1/hoststats?token=eyJhbGciOiJSUzI1NiJ9.eyJleHAiOjE0NTAxMTg5OTYsInN1YiI6ImNhdHRsZSIsInJlc291cmNlSWQiOiIxaDEiLCJpc3MiOiJodHRwOlwvXC9jYXR0bGUuaW8iLCJob3N0VXVpZCI6ImQ0NTQwNjc4LWY4MzUtNDEwNi04NTVjLWJjZTcyNjAyNjZkMyIsImlhdCI6MTQ1MDExODY5Niwia2lkIjoiZGVmYXVsdCJ9.W6e8k9AMIIJQiigdw8wt7UeBqFVYaALE798agQSdyMa0thS8f-bgh780fOCH8Jd_kIVjjWylTNj-kOschFGWDgF5jmO7Gcm13k246m5Z_mOGRI0KfF6PvvTtkWJqcPNPULCtQdnFed4siGGhH1FAjJ9PzqXFyX8Dwxm9HI8OeLCO0bx2Wuv4ZlzQdFVdCrGF6uXbs34apCfm_TmP7Bbp4hlMnFgJ4FZr9UokwgtuZNF8kzx8FNP33k_jjzrAN_vsT9yVvgUY7Au8gTIc_FXapgEDz7TMz9grceqV8ITqUv52JmZSrI25e20KX9s3L_-wjt4vVxIyd7GiymP_ONq-vA' failed: HTTP Authentication failed; no valid credentials available
ui-201f54a90666244b49099ee70c63069d.js:60 Socket 3 error
ui-201f54a90666244b49099ee70c63069d.js:60 Socket 3 disconnect``` 

Could it be the wrong URI used? needs a FQDN versus a Hostname? or maybe I added the hosts incorrectly (although I would expect other things to be broken in that case).

I will keep digging, Thank you very much again!

Cheers,

Alejandro

denise · December 14, 2015, 7:23pm

What version of Rancher are you using and what is the OS of your host?

AlexRRR · December 14, 2015, 7:58pm

Hi @denise,

I am using Debian GNU/Linux 8.2 (jessie) with:

Rancher v0.47.0
Cattle v0.115.0
User Interface v0.68.0
Rancher Compose v0.5.3

It is all rather new I guess.

Thanks!

Alejandro

PS: I had to add some parameters to GRUB (GRUB_CMDLINE_LINUX_DEFAULT="cgroup_enable=memory swapaccount=1") so the memory stats would show
correctly on cAdvisor.

cjellick · December 15, 2015, 1:59pm

@AlexRRR do you see anything that would be seemingly related in the rancher-agent container logs on host that you are trying to connect to?

@vincent any thoughts from the UI side on this?

covalence · January 15, 2016, 3:44am

@AlexRRR - I ran into a similar issue and the fix for me was to delete the rancher cookie in my browser and start with a fresh new login session. Maybe give that a shot?

AlexRRR · January 29, 2016, 1:20pm

Thank you all for replies, and sorry for the delay. I managed to see something in the agent/server log that pops up whenever I want to see the graphs in that specific HW host.

BTW I’ve updated to the latest Rancher version

Rancher v0.56.0
Cattle v0.141.0
User Interface v0.85.0
Rancher Compose v0.7.1

SERVER LOG:

2016/01/29 13:42:12 http: proxy error: net/http: request canceled
time="2016-01-29T13:51:08Z" level=error msg="Error parsing token." error="token is expired" 
time="2016-01-29T13:58:25Z" level=info msg="Invalid HostUuid." hostUuid=d58807ad-4684-4231-82eb-d1d1fc750441 
time="2016-01-29T13:58:25Z" level=error msg="could not find host uuid" 
time="2016-01-29T13:58:26Z" level=info msg="Registering backend for host [d58807ad-4684-4231-82eb-d1d1fc750441]"
```

AGENT LOG:
```

panic: runtime error: index out of range

goroutine 54 [running]:
github.com/rancherio/host-api/stats.convertToAggregatedStats(0x0, 0x0, 0xc2080dc420, 0x8d3e90, 0x9, 0xc2080fbc00, 0x0, 0x0, 0xbd0902000, 0x0, ...)
	/source/gopath/src/github.com/rancherio/host-api/stats/aggregated_stats.go:21 +0x590
github.com/rancherio/host-api/stats.writeAggregatedStats(0x0, 0x0, 0xc2080dc420, 0x8d3e90, 0x9, 0xc2080fbc00, 0x0, 0x0, 0xbd0902000, 0x7f5725bf9e70, ...)
	/source/gopath/src/github.com/rancherio/host-api/stats/aggregated_stats.go:70 +0xb6
github.com/rancherio/host-api/stats.(*ContainerStatsHandler).Handle(0xb2a6d0, 0xc208239000, 0x24, 0xc208239029, 0xf75, 0xc2080646c0, 0xc2080d9e00)
	/source/gopath/src/github.com/rancherio/host-api/stats/container_stats.go:125 +0x14b5
created by github.com/rancherio/websocket-proxy/backend.connectToProxyWS
	/source/Godeps/_workspace/src/github.com/rancherio/websocket-proxy/backend/backend.go:72 +0x5c5

goroutine 1 [IO wait]:
net.(*pollDesc).Wait(0xc2080f2530, 0x72, 0x0, 0x0)
	/usr/src/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2080f2530, 0x0, 0x0)
	/usr/src/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc2080f24d0, 0xc208238000, 0x1000, 0x1000, 0x0, 0x7f5725befbf0, 0xc208206118)
	/usr/src/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20802cae8, 0xc208238000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/src/go/src/net/net.go:121 +0xdc
bufio.(*Reader).fill(0xc2080d9da0)
	/usr/src/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Read(0xc2080d9da0, 0xc208206110, 0x2, 0x8, 0x413cd0, 0x0, 0x0)
	/usr/src/go/src/bufio/bufio.go:174 +0x26c
github.com/gorilla/websocket.(*Conn).readFull(0xc2080ca870, 0xc208206110, 0x2, 0x8, 0x0, 0x0)
	/source/Godeps/_workspace/src/github.com/gorilla/websocket/conn.go:542 +0xab
github.com/gorilla/websocket.(*Conn).advanceFrame(0xc2080ca870, 0x3d924a08, 0x0, 0x0)
	/source/Godeps/_workspace/src/github.com/gorilla/websocket/conn.go:566 +0x15c
github.com/gorilla/websocket.(*Conn).NextReader(0xc2080ca870, 0xc2080646e0, 0x0, 0x0, 0x0, 0x0)
	/source/Godeps/_workspace/src/github.com/gorilla/websocket/conn.go:706 +0x7e
github.com/gorilla/websocket.(*Conn).ReadMessage(0xc2080ca870, 0xc208239000, 0x0, 0x0, 0x0, 0x0, 0x0)
	/source/Godeps/_workspace/src/github.com/gorilla/websocket/conn.go:769 +0x6a
github.com/rancherio/websocket-proxy/backend.connectToProxyWS(0xc2080ca870, 0xc208234cf0)
	/source/Godeps/_workspace/src/github.com/rancherio/websocket-proxy/backend/backend.go:53 +0x168
github.com/rancherio/websocket-proxy/backend.ConnectToProxy(0xc208040500, 0x275, 0xc208234cf0)
	/source/Godeps/_workspace/src/github.com/rancherio/websocket-proxy/backend/backend.go:32 +0x45d
main.main()
	/source/main.go:93 +0xfac

goroutine 5 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0xb21b80)
	/source/Godeps/_workspace/src/github.com/golang/glog/glog.go:879 +0x78
created by github.com/golang/glog.init·1
	/source/Godeps/_workspace/src/github.com/golang/glog/glog.go:410 +0x2a7

goroutine 17 [syscall, 1 minutes, locked to thread]:
runtime.goexit()
	/usr/src/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 7 [chan receive]:
github.com/rancherio/go-machine-service/locks.locker()
	/source/Godeps/_workspace/src/github.com/rancherio/go-machine-service/locks/locks.go:61 +0x6c
created by github.com/rancherio/go-machine-service/locks.init·1
	/source/Godeps/_workspace/src/github.com/rancherio/go-machine-service/locks/locks.go:7 +0x5c

goroutine 16 [chan receive]:
github.com/rancherio/host-api/events.(*EventRouter).routeEvents(0xc2080bbda0)
	/source/gopath/src/github.com/rancherio/host-api/events/event_router.go:62 +0x56
created by github.com/rancherio/host-api/events.(*EventRouter).Start
	/source/gopath/src/github.com/rancherio/host-api/events/event_router.go:43 +0x107

goroutine 20 [syscall]:
syscall.Syscall6(0xe8, 0xf, 0xc20810bcb4, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0xb22480, 0x41894f, 0xc2080a17a0)
	/usr/src/go/src/syscall/asm_linux_amd64.s:46 +0x5
syscall.EpollWait(0xf, 0xc20810bcb4, 0x7, 0x7, 0xffffffffffffffff, 0x1, 0x0, 0x0)
	/usr/src/go/src/syscall/zsyscall_linux_amd64.go:376 +0x93
gopkg.in/fsnotify%2ev1.(*fdPoller).wait(0xc2080d11a0, 0xc208064b00, 0x0, 0x0)
	/source/Godeps/_workspace/src/gopkg.in/fsnotify.v1/inotify_poller.go:85 +0xc7
gopkg.in/fsnotify%2ev1.(*Watcher).readEvents(0xc20802ebe0)
	/source/Godeps/_workspace/src/gopkg.in/fsnotify.v1/inotify.go:179 +0x16f
created by gopkg.in/fsnotify%2ev1.NewWatcher
	/source/Godeps/_workspace/src/gopkg.in/fsnotify.v1/inotify.go:58 +0x32b

goroutine 24 [IO wait, 1 minutes]:
net.(*pollDesc).Wait(0xc20809ea00, 0x72, 0x0, 0x0)
	/usr/src/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20809ea00, 0x0, 0x0)
	/usr/src/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20809e9a0, 0xc2080db000, 0x1000, 0x1000, 0x0, 0x7f5725befbf0, 0xc2080c17e8)
	/usr/src/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20802c440, 0xc2080db000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/src/go/src/net/net.go:121 +0xdc
bufio.(*Reader).fill(0xc208064900)
	/usr/src/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadSlice(0xc208064900, 0x41b20a, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/src/go/src/bufio/bufio.go:295 +0x257
net/http/internal.readLine(0xc208064900, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/src/go/src/net/http/internal/chunked.go:110 +0x5a
net/http/internal.(*chunkedReader).beginChunk(0xc2080dc510)
	/usr/src/go/src/net/http/internal/chunked.go:47 +0x46
net/http/internal.(*chunkedReader).Read(0xc2080dc510, 0xc208092200, 0x200, 0x200, 0x0, 0x0, 0x0)
	/usr/src/go/src/net/http/internal/chunked.go:77 +0xbb
net/http.(*body).readLocked(0xc2080e02c0, 0xc208092200, 0x200, 0x200, 0x0, 0x0, 0x0)
	/usr/src/go/src/net/http/transfer.go:584 +0x7a
net/http.(*body).Read(0xc2080e02c0, 0xc208092200, 0x200, 0x200, 0x0, 0x0, 0x0)
	/usr/src/go/src/net/http/transfer.go:579 +0x115
encoding/json.(*Decoder).readValue(0xc2080e4000, 0xc207ff1fc3, 0x0, 0x0)
	/usr/src/go/src/encoding/json/stream.go:124 +0x5e1
encoding/json.(*Decoder).Decode(0xc2080e4000, 0x710ba0, 0xc2080e03c0, 0x0, 0x0)
	/usr/src/go/src/encoding/json/stream.go:44 +0x7b
github.com/fsouza/go-dockerclient.func·006(0xc20806e360, 0xc208076fd0)
	/source/Godeps/_workspace/src/github.com/fsouza/go-dockerclient/event.go:289 +0x1b7
created by github.com/fsouza/go-dockerclient.(*Client).eventHijack
	/source/Godeps/_workspace/src/github.com/fsouza/go-dockerclient/event.go:307 +0x625

goroutine 59 [select]:
net/http.(*persistConn).writeLoop(0xc208076210)
	/usr/src/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
	/usr/src/go/src/net/http/transport.go:661 +0xcbc

goroutine 18 [select]:
github.com/fsouza/go-dockerclient.(*eventMonitoringState).monitorEvents(0xc20809e1c0, 0xc20805a700)
	/source/Godeps/_workspace/src/github.com/fsouza/go-dockerclient/event.go:175 +0x41c
created by github.com/fsouza/go-dockerclient.(*eventMonitoringState).enableEventMonitoring
	/source/Godeps/_workspace/src/github.com/fsouza/go-dockerclient/event.go:148 +0x16f

goroutine 19 [select, 1 minutes]:
github.com/rancherio/host-api/events.watchInternalFn(0xc208064240, 0x7ffdef6a5c4e, 0x21, 0x1dcd65000, 0x2540be400, 0x9bd6b8, 0x0, 0x0, 0x0)
	/source/gopath/src/github.com/rancherio/host-api/events/rancher_state_watcher.go:140 +0xaba
github.com/rancherio/host-api/events.(*rancherStateWatcher).watch(0xc20802ea00)
	/source/gopath/src/github.com/rancherio/host-api/events/rancher_state_watcher.go:57 +0x1d9
created by github.com/rancherio/host-api/events.(*DockerEventsProcessor).Process
	/source/gopath/src/github.com/rancherio/host-api/events/entry.go:49 +0x331

goroutine 21 [select]:
github.com/rancherio/host-api/events.func·003()
	/source/gopath/src/github.com/rancherio/host-api/events/rancher_state_watcher.go:103 +0x678
created by github.com/rancherio/host-api/events.watchInternalFn
	/source/gopath/src/github.com/rancherio/host-api/events/rancher_state_watcher.go:124 +0x614

goroutine 22 [select]:
github.com/rancherio/host-api/events.func·004()
	/source/gopath/src/github.com/rancherio/host-api/events/rancher_state_watcher.go:172 +0x22d
created by github.com/rancherio/host-api/events.initHealthCheck
	/source/gopath/src/github.com/rancherio/host-api/events/rancher_state_watcher.go:181 +0x233

goroutine 23 [select]:
github.com/rancherio/host-api/events.monitorHealth(0xc208064cc0, 0xc20809ec40, 0x2540be400)
	/source/gopath/src/github.com/rancherio/host-api/events/rancher_state_watcher.go:189 +0x2a7
created by github.com/rancherio/host-api/events.watchInternalFn
	/source/gopath/src/github.com/rancherio/host-api/events/rancher_state_watcher.go:138 +0x8fa

goroutine 56 [semacquire]:
sync.(*Cond).Wait(0xc2080f0030)
	/usr/src/go/src/sync/cond.go:62 +0x9e
io.(*pipe).read(0xc2080f0000, 0xc208143000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/src/go/src/io/pipe.go:52 +0x303
io.(*PipeReader).Read(0xc20802c048, 0xc208143000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/src/go/src/io/pipe.go:134 +0x5b
bufio.(*Scanner).Scan(0xc20805a080, 0xc208143000)
	/usr/src/go/src/bufio/scan.go:180 +0x688
github.com/rancherio/host-api/stats.func·003(0xc20802c048)
	/source/gopath/src/github.com/rancherio/host-api/stats/container_stats.go:80 +0x1af
created by github.com/rancherio/host-api/stats.(*ContainerStatsHandler).Handle
	/source/gopath/src/github.com/rancherio/host-api/stats/container_stats.go:92 +0xd3e

goroutine 53 [runnable]:
github.com/rancherio/websocket-proxy/backend.func·001()
	/source/Godeps/_workspace/src/github.com/rancherio/websocket-proxy/backend/backend.go:42 +0x6d
created by github.com/rancherio/websocket-proxy/backend.connectToProxyWS
	/source/Godeps/_workspace/src/github.com/rancherio/websocket-proxy/backend/backend.go:49 +0x152

goroutine 55 [chan receive]:
github.com/rancherio/host-api/stats.func·002(0xc20802c058)
	/source/gopath/src/github.com/rancherio/host-api/stats/container_stats.go:70 +0x5c
created by github.com/rancherio/host-api/stats.(*ContainerStatsHandler).Handle
	/source/gopath/src/github.com/rancherio/host-api/stats/container_stats.go:76 +0xca5

goroutine 46 [IO wait]:
net.(*pollDesc).Wait(0xc20809f480, 0x72, 0x0, 0x0)
	/usr/src/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20809f480, 0x0, 0x0)
	/usr/src/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20809f420, 0xc2080d3000, 0x1000, 0x1000, 0x0, 0x7f5725befbf0, 0xc208236228)
	/usr/src/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20802c9d8, 0xc2080d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/src/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7f5725bf3de8, 0xc20802c9d8, 0xc208076e18, 0xc2080d3000, 0x1000, 0x1000, 0x79f3a0, 0x0, 0x0)
	/usr/src/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc2080ee720, 0xc2080d3000, 0x1000, 0x1000, 0xc208010000, 0x0, 0x0)
	<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc2080d8fc0)
	/usr/src/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc2080d8fc0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/src/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc208076dc0)
	/usr/src/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
	/usr/src/go/src/net/http/transport.go:660 +0xc9f

goroutine 58 [runnable]:
net/http.(*persistConn).readLoop(0xc208076210)
	/usr/src/go/src/net/http/transport.go:928 +0x9ce
created by net/http.(*Transport).dialConn
	/usr/src/go/src/net/http/transport.go:660 +0xc9f

goroutine 47 [select]:
net/http.(*persistConn).writeLoop(0xc208076dc0)
	/usr/src/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
	/usr/src/go/src/net/http/transport.go:661 +0xcbc
```

Best regards, 

Alejandro

sfarges · February 10, 2016, 3:34pm

Issue is also discussed here : https://github.com/rancher/rancher/issues/3384

Topic		Replies	Views
Container CPU graphs not showing Rancher 1.x	3	1491	March 31, 2016
Container CPU graph is empty in 1.6.16 Rancher 1.x	1	728	June 12, 2018
No container stats and cannot execute shell/view logs on some hosts Rancher 1.x	6	2404	May 6, 2016
Websocket connections ha cluster: HTTP Auth failed Rancher 1.x	0	1071	September 23, 2016
Stats for host/containers always show as "Connecting..." Rancher 1.x	7	2816	April 27, 2018

No graphs show for hardware hosts

Related Topics