WARN: Failure to send result to graphite server

Archive Rancher 1.x
1

Since upgrading to Rancher 1.2.2 continually getting a WARN over and over again in master container logs,

Infra: Amazon EC2
Rancher: v1.2.2
OS: Amazon Linux 2016.09 for agents and master
Docker: v1.11.2 for agents and master
Database: MySQL 5.7 in AWS RDS
Orchestration: Cattle

Full exception

2016-12-29 10:38:06,804 WARN  [:] [] [] [] [xtrans-export-1] [o.j.embedded.output.GraphiteWriter  ] Failure to send result to graphite server 'HostAndPort{host='10.9.2.106', port=2003}' with null java.net.SocketTimeoutException: connect timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_72]
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_72]
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_72]
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_72]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_72]
        at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_72]
        at org.jmxtrans.embedded.util.pool.SocketWriterPoolFactory.makeObject(SocketWriterPoolFactory.java:59) ~[embedded-jmxtrans-1.0.9.jar:na]
        at org.jmxtrans.embedded.util.pool.SocketWriterPoolFactory.makeObject(SocketWriterPoolFactory.java:40) ~[embedded-jmxtrans-1.0.9.jar:na]
        at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220) ~[commons-pool-1.6.jar:1.6]
        at org.jmxtrans.embedded.output.GraphiteWriter.write(GraphiteWriter.java:134) ~[embedded-jmxtrans-1.0.9.jar:na]
        at org.jmxtrans.embedded.Query.exportCollectedMetrics(Query.java:208) [embedded-jmxtrans-1.0.9.jar:na]
        at org.jmxtrans.embedded.EmbeddedJmxTrans$2.run(EmbeddedJmxTrans.java:152) [embedded-jmxtrans-1.0.9.jar:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_72]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_72]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
2

Graphite isn’t on by default so someone must have turned it on… you can go to the API -> /v2-beta/settings/graphite.host and delete the setting or update it to an IP that is actually running Graphite.

3

Thank you @vincent, deleting that setting and then removing and staring a new master (restarting the existing container) stopped that warning from appearing.