Been having a problem with our rancher server, its been posted here but without good logs.
Memory for a Java process surges incredibly high leading to a massive slowdown in performance on rancher server, the only fix for now is to manually kill the process, and restart the docker container.
This is on an EC2 instance WITHOUT an external DB.
The relevant error that is getting thrown is here:
time="2017-01-18 13:53:04" level=info msg="Telemetry Client v0.4.0"
time="2017-01-18 13:53:19" level=info msg="Listening on 0.0.0.0:8114"
time="2017-01-18 13:53:28" level=fatal msg="listen tcp 0.0.0.0:8114: bind: address already in use"
2017-01-18 13:53:38,758 ERROR [:] [] [] [] [igurationSource] [c.n.config.AbstractPollingScheduler ] Error getting result from polling source org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114) ~[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) ~[commons-dbcp-1.4.jar:1.4]
at com.netflix.config.sources.JDBCConfigurationSource.getConnection(JDBCConfigurationSource.java:143) ~[archaius-core-0.5.14.jar:na]
at com.netflix.config.sources.JDBCConfigurationSource.load(JDBCConfigurationSource.java:105) ~[archaius-core-0.5.14.jar:na]
at com.netflix.config.sources.JDBCConfigurationSource.poll(JDBCConfigurationSource.java:88) ~[archaius-core-0.5.14.jar:na]
at io.cattle.platform.archaius.sources.LazyJDBCSource.poll(LazyJDBCSource.java:31) ~[cattle-framework-archaius-0.5.0-SNAPSHOT.jar:na]
at com.netflix.config.AbstractPollingScheduler$1.run(AbstractPollingScheduler.java:163) ~[archaius-core-0.5.14.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_72]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174) ~[commons-pool-1.6.jar:1.6]
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106) ~[commons-dbcp-1.4.jar:1.4]
... 13 common frames omitted
This just loops forever and keeps on using more and more memory.