after installing the new rancher monitoring I am seeing errors in the push proxy for the controller-manger, and schedler, when I look at the logs I see the following
level=info ts=2020-11-05T19:46:23.592Z caller=coordinator.go:137 msg=WaitForScrapeInstruction fqdn=10.136.2.56
level=info ts=2020-11-05T19:46:23.593Z caller=main.go:136 msg="Got /push" scrape_id=296c58a4-8fac-47b4-93d6-af84bd800bc6
level=info ts=2020-11-05T19:46:23.593Z caller=coordinator.go:169 msg=ScrapeResult scrape_id=296c58a4-8fac-47b4-93d6-af84bd800bc6
level=info ts=2020-11-05T19:46:23.677Z caller=coordinator.go:116 msg=DoScrape scrape_id=301865dd-0a42-41a8-b49b-ba4a6494058b url=http://10.136.2.58:10252/metrics
level=info ts=2020-11-05T19:46:23.677Z caller=main.go:154 msg="Responded to /poll" url=http://10.136.2.58:10252/metrics scrape_id=301865dd-0a42-41a8-b49b-ba4a6494058b
level=info ts=2020-11-05T19:46:23.679Z caller=coordinator.go:137 msg=WaitForScrapeInstruction fqdn=10.136.2.58
level=info ts=2020-11-05T19:46:23.686Z caller=main.go:136 msg="Got /push" scrape_id=301865dd-0a42-41a8-b49b-ba4a6494058b
level=info ts=2020-11-05T19:46:23.686Z caller=coordinator.go:169 msg=ScrapeResult scrape_id=301865dd-0a42-41a8-b49b-ba4a6494058b
level=info ts=2020-11-05T19:46:25.075Z caller=coordinator.go:137 msg=WaitForScrapeInstruction fqdn=10.136.2.58
level=info ts=2020-11-05T19:46:25.075Z caller=main.go:149 msg="Error WaitForScrapeInstruction:" err="request is expired"
level=info ts=2020-11-05T19:46:25.645Z caller=coordinator.go:137 msg=WaitForScrapeInstruction fqdn=10.136.2.56
level=info ts=2020-11-05T19:46:25.645Z caller=main.go:149 msg="Error WaitForScrapeInstruction:" err="request is expired"
level=info ts=2020-11-05T19:46:25.694Z caller=coordinator.go:137 msg=WaitForScrapeInstruction fqdn=10.136.2.57
level=info ts=2020-11-05T19:46:25.694Z caller=main.go:149 msg="Error WaitForScrapeInstruction:" err="request is expired"
level=info ts=2020-11-05T19:46:26.164Z caller=coordinator.go:137 msg=WaitForScrapeInstruction fqdn=10.136.2.58
level=info ts=2020-11-05T19:46:26.164Z caller=main.go:149 msg="Error WaitForScrapeInstruction:" err="request is expired"
level=info ts=2020-11-05T19:46:26.246Z caller=coordinator.go:137 msg=WaitForScrapeInstruction fqdn=10.136.2.56
level=info ts=2020-11-05T19:46:26.246Z caller=main.go:149 msg="Error WaitForScrapeInstruction:" err="request is expired"
level=info ts=2020-11-05T19:46:26.407Z caller=coordinator.go:221 msg="GC of clients completed" deleted=0 remaining=3
so it seems like sometimes it’s getting the metrics, and sometimes it’s not, any ideas on how I can solve this?