Java Memory Surge

Been having a problem with our rancher server, its been posted here but without good logs.

Memory for a Java process surges incredibly high leading to a massive slowdown in performance on rancher server, the only fix for now is to manually kill the process, and restart the docker container.

This is on an EC2 instance WITHOUT an external DB.

The relevant error that is getting thrown is here:

time="2017-01-18 13:53:04" level=info msg="Telemetry Client v0.4.0"
time="2017-01-18 13:53:19" level=info msg="Listening on 0.0.0.0:8114"
time="2017-01-18 13:53:28" level=fatal msg="listen tcp 0.0.0.0:8114: bind: address already in use"
2017-01-18 13:53:38,758 ERROR [:] [] [] [] [igurationSource] [c.n.config.AbstractPollingScheduler ] Error getting result from polling source org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
	at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114) ~[commons-dbcp-1.4.jar:1.4]
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) ~[commons-dbcp-1.4.jar:1.4]
	at com.netflix.config.sources.JDBCConfigurationSource.getConnection(JDBCConfigurationSource.java:143) ~[archaius-core-0.5.14.jar:na]
	at com.netflix.config.sources.JDBCConfigurationSource.load(JDBCConfigurationSource.java:105) ~[archaius-core-0.5.14.jar:na]
	at com.netflix.config.sources.JDBCConfigurationSource.poll(JDBCConfigurationSource.java:88) ~[archaius-core-0.5.14.jar:na]
	at io.cattle.platform.archaius.sources.LazyJDBCSource.poll(LazyJDBCSource.java:31) ~[cattle-framework-archaius-0.5.0-SNAPSHOT.jar:na]
	at com.netflix.config.AbstractPollingScheduler$1.run(AbstractPollingScheduler.java:163) ~[archaius-core-0.5.14.jar:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_72]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_72]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
	at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174) ~[commons-pool-1.6.jar:1.6]
	at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106) ~[commons-dbcp-1.4.jar:1.4]
	... 13 common frames omitted

This just loops forever and keeps on using more and more memory.

You can increase to 4GB on a 32 bit system. If you’re on a 64 bit system you can go higher. No need to worry if you’ve chosen incorrectly, if you ask for 5g on a 32 bit system java will complain about an invalid value and quit.

Eldo