Goroutines metrics for Rancher pod?

jhunty · May 4, 2020, 6:25am

Hi

How do i get the “number of goroutines” metric for the Rancher pods?

The same metric is exposed e.g. by the node_exporter as go_goroutines. But i assume this metric is just for the node_exporter pod. not for Rancher itself.

I’m trying to verify the same that was found here:

The OOM killer is regularly killing our Rancher pods and just trying to work out what is going on…

ukinau · May 7, 2020, 12:14am

Hi I’m author of above blog, thanks for reading.

Let me answer for your question.
At default, rancher would not expose metrics.
But if you specify CATTLE_PROMETHEUS_METRICS=true in environment variable for rancher-server container, you will be able to access https:///metrics endpoint

github.com

rancher/rancher/blob/master/app/app.go#L181


		return nil, err
	}


	auditLogWriter := audit.NewLogWriter(cfg.AuditLogPath, cfg.AuditLevel, cfg.AuditLogMaxage, cfg.AuditLogMaxbackup, cfg.AuditLogMaxsize)


	authMiddleware, handler, err := server.Start(ctx, localClusterEnabled(*cfg), scaledContext, clusterManager, auditLogWriter, rbac.NewAccessControlHandler())
	if err != nil {
		return nil, err
	}


	if os.Getenv("CATTLE_PROMETHEUS_METRICS") == "true" {
		metrics.Register(ctx, scaledContext)
	}


	if auditLogWriter != nil {
		go func() {
			<-ctx.Done()
			auditLogWriter.Output.Close()
		}()
	}

One thing you need to be aware of is that this functionality is newly introduced from 2.4 (if i remember correctly). That’s why your rancher-server may not support this yet if you are using earlier than 2.4

The reason why above blog could get metrics from rancher-server even earlier than 2.4 is we build rancher-server binary with several our custom patch.

Topic		Replies	Views
Configure Rancher to send cluster metrics to OpenNMS	1	318	December 8, 2021
I dont understand Prometheus in Rancher at all Rancher 2.x	7	5874	June 16, 2019
Rancher 2.6 with Windows Container - Metrics + Monitoring	2	549	March 2, 2022
Apache Camel-k metrics in rancher embedded prometheus Rancher 2.x	0	434	November 25, 2021
Enable Fluentbit / Fluentd Metrics Rancher 2.x	3	133	February 14, 2024

Goroutines metrics for Rancher pod?

Related Topics