Hi there -
Having something of an issue with Rancher. The Host is a 1GB host on digital ocean and rancher is the only thing running on it, memory usage is at 5% but the JVM is throwing an out of memory exception, the server will not accept connections and the reverse proxy returns a 503. If I hard reboot the server (eg. reboot now) the machine will come back up and rancher will be “game on”. But it’s annoying to have to go in and reboot the server every 3-4 days.
When I try to restart just the container, it locks and will not restart the container. The only way to recover is to reboot the host.
Any advice would be most welcome. I’m running rancher:latest as of 6 weeks ago.
Thanks in advance!
Michael
VM Arguments:
jvm_args: -Xms128m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/cattle/logs -Dlogback.bootstrap.level=WARN
java_command: io.cattle.platform.launcher.Main
Launcher Type: SUN_STANDARD
Environment Variables:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Signal Handlers:
SIGSEGV: [libjvm.so+0x955bc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x955bc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x7caff0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGXFSZ: [libjvm.so+0x7caff0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x7caff0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x7cae90], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: [libjvm.so+0x7cb070], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGINT: [libjvm.so+0x7cb070], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGTERM: [libjvm.so+0x7cb070], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGQUIT: [libjvm.so+0x7cb070], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGTRAP: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
--------------- S Y S T E M ---------------
OS:Ubuntu 14.04 (trusty)
uname:Linux 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64
libc:glibc 2.19 NPTL 2.19
rlimit: STACK 8192k, CORE 0k, NPROC 524288, NOFILE 1048576, AS infinity
load average:0.30 0.33 0.37
/proc/meminfo:
MemTotal: 1017856 kB
MemFree: 60196 kB
Buffers: 476 kB
Cached: 8652 kB
SwapCached: 0 kB
Active: 903748 kB
Inactive: 5648 kB
Active(anon): 900652 kB
Inactive(anon): 264 kB
Active(file): 3096 kB
Inactive(file): 5384 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 12 kB
Writeback: 0 kB
AnonPages: 900320 kB
Mapped: 4820 kB
Shmem: 596 kB
Slab: 22764 kB
SReclaimable: 10240 kB
SUnreclaim: 12524 kB
KernelStack: 2288 kB
PageTables: 5012 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 508928 kB
Committed_AS: 1349880 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 8764 kB
VmallocChunk: 34359721724 kB
HardwareCorrupted: 0 kB
AnonHugePages: 804864 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 38904 kB
DirectMap2M: 1009664 kB
DirectMap1G: 0 kB
CPU:total 1 (1 cores per cpu, 1 threads per core) family 6 model 45 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, tsc
/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel® Xeon® CPU E5-2630 0 @ 2.30GHz
stepping : 7
microcode : 0x1
cpu MHz : 2299.998
cache size : 15360 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm arat xsaveopt vnmi ept tsc_adjust
bogomips : 4599.99
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
Memory: 4k page, physical 1017856k(62492k free), swap 0k(0k free)
vm_info: OpenJDK 64-Bit Server VM (24.95-b01) for linux-amd64 JRE (1.7.0_95-b00), built on Jan 21 2016 03:31:40 by “buildd” with gcc 4.8.4
time: Tue Oct 4 21:19:43 2016
elapsed time: 55674 seconds
time=“2016-10-04T21:19:44Z” level=info msg=“Exiting rancher-compose-executor” version=v0.7.3