Goroutines metrics for Rancher pod?

jhunty · May 4, 2020, 6:25am

Hi

How do i get the “number of goroutines” metric for the Rancher pods?

The same metric is exposed e.g. by the node_exporter as go_goroutines. But i assume this metric is just for the node_exporter pod. not for Rancher itself.

I’m trying to verify the same that was found here:

The OOM killer is regularly killing our Rancher pods and just trying to work out what is going on…

ukinau · May 7, 2020, 12:14am

Hi I’m author of above blog, thanks for reading.

Let me answer for your question.
At default, rancher would not expose metrics.
But if you specify CATTLE_PROMETHEUS_METRICS=true in environment variable for rancher-server container, you will be able to access https:///metrics endpoint

github.com

rancher/rancher/blob/master/app/app.go#L181


		return nil, err
	}


	auditLogWriter := audit.NewLogWriter(cfg.AuditLogPath, cfg.AuditLevel, cfg.AuditLogMaxage, cfg.AuditLogMaxbackup, cfg.AuditLogMaxsize)


	authMiddleware, handler, err := server.Start(ctx, localClusterEnabled(*cfg), scaledContext, clusterManager, auditLogWriter, rbac.NewAccessControlHandler())
	if err != nil {
		return nil, err
	}


	if os.Getenv("CATTLE_PROMETHEUS_METRICS") == "true" {
		metrics.Register(ctx, scaledContext)
	}


	if auditLogWriter != nil {
		go func() {
			<-ctx.Done()
			auditLogWriter.Output.Close()
		}()
	}

One thing you need to be aware of is that this functionality is newly introduced from 2.4 (if i remember correctly). That’s why your rancher-server may not support this yet if you are using earlier than 2.4

The reason why above blog could get metrics from rancher-server even earlier than 2.4 is we build rancher-server binary with several our custom patch.

Topic		Replies	Views
Monitoring - no metrics for pods in 1.24 Rancher	1	1163	December 23, 2022
Configure Rancher to send cluster metrics to OpenNMS	1	334	December 8, 2021
Monitoring with node-exporter RancherOS	1	3373	July 7, 2020
I dont understand Prometheus in Rancher at all Rancher	7	6086	June 16, 2019
Issues with metrics feeding in to HPA/AutoScaling Rancher 2.0 Tech Preview	0	1534	April 18, 2018

Goroutines metrics for Rancher pod?

Related topics