Rancher 1.6.18 - Cattle - HA - Self-Hosted - Dashboard keeps crashing!

mbuckman · August 20, 2018, 11:52pm

Losing primary Rancher server every day or two, sometimes reboot brings it backup within a few minutes, sometimes does not. Other HA servers only show login then redirect to primary which is not responding and spitting out so m,any errors. Recently upgraded, Rancher was rock solid prior.

During the loss of dashboard at least containers remain running, although inaccessible unless I ssh and manually manage via Docker CLI.

Please help1

Sample of Errors:

2018-08-02 22:51:00,577 ERROR [1f3d65e0-fd50-4dab-b11b-cfb8e1f9db6b:23136] [service:99] [service.update] [] [ecutorService-7] [c.p.e.p.i.DefaultProcessInstanceImpl] Container should have been running but is in stopped state. Check logs for more information. 
2018-08-02 22:51:00,589 ERROR [:] [] [] [] [ecutorService-7] [.e.s.i.ProcessInstanceDispatcherImpl] Container should have been running but is in stopped state. Check logs for more information. 
2018-08-02 22:51:15,574 ERROR [deeb0b8f-1108-4f99-8ead-f4c7b954c064:23146] [service:99] [service.update] [] [ecutorService-7] [c.p.e.p.i.DefaultProcessInstanceImpl] Container should have been running but is in stopped state. Check logs for more information. 
2018-08-02 22:51:15,588 ERROR [:] [] [] [] [ecutorService-7] [.e.s.i.ProcessInstanceDispatcherImpl] Container should have been running but is in stopped state. Check logs for more information. 
2018-08-02 23:03:59,765 WARN  [:] [] [] [] [p1020391880-156] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1353] 
2018-08-02 23:03:59,783 WARN  [:] [] [] [] [p1020391880-157] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [557] 
2018/08/02 23:03:59 http: proxy error: net/http: request canceled
2018-08-02 23:03:59,867 WARN  [:] [] [] [] [p1020391880-206] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1746] 
2018-08-02 23:03:59,997 WARN  [:] [] [] [] [p1020391880-209] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1734] 
2018-08-02 23:04:00,097 WARN  [:] [] [] [] [p1020391880-174] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1726] 
2018-08-02 23:04:00,199 WARN  [:] [] [] [] [p1020391880-158] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1738] 
2018-08-02 23:04:00,254 WARN  [:] [] [] [] [tp1020391880-19] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1701] 
2018-08-02 23:04:00,286 WARN  [:] [] [] [] [p1020391880-229] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1772] 
2018-08-02 23:04:00,450 WARN  [:] [] [] [] [p1020391880-138] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1663] 
2018-08-02 23:04:00,499 WARN  [:] [] [] [] [p1020391880-132] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1800] 
2018-08-02 23:04:00,830 WARN  [:] [] [] [] [p1020391880-167] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1732] 
2018-08-02 23:04:00,875 WARN  [:] [] [] [] [tp1020391880-21] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1660] 
2018-08-02 23:04:01,193 WARN  [:] [] [] [] [p1020391880-182] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1749] 
2018-08-02 23:04:01,279 WARN  [:] [] [] [] [p1020391880-140] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1714] 
2018-08-02 23:04:01,286 WARN  [:] [] [] [] [p1020391880-136] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [592] 
2018-08-02 23:04:01,718 WARN  [:] [] [] [] [p1020391880-176] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1762] 
2018-08-02 23:04:02,227 WARN  [:] [] [] [] [p1020391880-228] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1699] 
2018-08-02 23:04:02,869 WARN  [:] [] [] [] [p1020391880-162] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1797] 
2018-08-02 23:04:02,896 WARN  [:] [] [] [] [p1020391880-180] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1481] 
2018-08-02 23:04:03,209 WARN  [:] [] [] [] [p1020391880-231] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1710] 
2018-08-02 23:04:03,235 WARN  [:] [] [] [] [p1020391880-131] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1748] 
2018-08-02 23:04:04,440 WARN  [:] [] [] [] [.IO.thread-in-0] [c.hazelcast.nio.tcp.TcpIpConnection ] [10.5.0.84]:9345 [09174270-ee63-403c-b184-0ff2a1bb0c4f] [3.8] Connection[id=1, /172.17.0.2:41051->/10.5.0.25:9345, endpoint=[10.5.0.25]:9345, alive=false, type=MEMBER] closed. Reason: Exception in NonBlockingSocketReader java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:197)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at com.hazelcast.nio.tcp.DefaultSocketChannelWrapper.read(DefaultSocketChannelWrapper.java:61)
	at com.hazelcast.internal.networking.nonblocking.NonBlockingSocketReader.handle(NonBlockingSocketReader.java:150)
	at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.handleSelectionKey(NonBlockingIOThread.java:350)
	at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.handleSelectionKeys(NonBlockingIOThread.java:335)
	at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.selectLoop(NonBlockingIOThread.java:253)
	at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.run(NonBlockingIOThread.java:206)

2018-08-02 23:04:05,972 WARN  [:] [] [] [] [.IO.thread-in-1] [c.hazelcast.nio.tcp.TcpIpConnection ] [10.5.0.84]:9345 [09174270-ee63-403c-b184-0ff2a1bb0c4f] [3.8] Connection[id=2, /172.17.0.2:57487->/10.5.0.99:9345, endpoint=[10.5.0.99]:9345, alive=false, type=MEMBER] closed. Reason: Exception in NonBlockingSocketReader java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:197)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at com.hazelcast.nio.tcp.DefaultSocketChannelWrapper.read(DefaultSocketChannelWrapper.java:61)
	at com.hazelcast.internal.networking.nonblocking.NonBlockingSocketReader.handle(NonBlockingSocketReader.java:150)
	at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.handleSelectionKey(NonBlockingIOThread.java:350)
	at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.handleSelectionKeys(NonBlockingIOThread.java:335)
	at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.selectLoop(NonBlockingIOThread.java:253)
	at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.run(NonBlockingIOThread.java:206)

2018/08/02 23:04:10 http: proxy error: net/http: request canceled
2018-08-02 23:04:59,145 ERROR [f70ee122-bc76-4377-9918-bb53838686c2:23982] [containerEvent:136357->instance:5099] [containerevent.create->(ContainerEventCreate)->instance.stop] [] [ecutorService-7] [c.p.e.p.i.DefaultProcessInstanceImpl] Unknown exception org.jooq.exception.DataAccessException: SQL [select `instance`.`id`, `instance`.`name`, `instance`.`account_id`, `instance`.`kind`, `instance`.`uuid`, `instance`.`description`, `instance`.`state`, `instance`.`created`, `instance`.`removed`, `instance`.`remove_time`, `instance`.`data`, `instance`.`allocation_state`, `instance`.`compute`, `instance`.`memory_mb`, `instance`.`image_id`, `instance`.`hostname`, `instance`.`zone_id`, `instance`.`instance_triggered_stop`, `instance`.`agent_id`, `instance`.`domain`, `instance`.`first_running`, `instance`.`token`, `instance`.`userdata`, `instance`.`registry_credential_id`, `instance`.`external_id`, `instance`.`native_container`, `instance`.`network_container_id`, `instance`.`health_state`, `instance`.`start_count`, `instance`.`create_index`, `instance`.`deployment_unit_uuid`, `instance`.`version`, `instance`.`health_updated`, `instance`.`service_index_id`, `instance`.`dns_internal`, `instance`.`dns_search_internal`, `instance`.`memory_reservation`, `instance`.`milli_cpu_reservation`, `instance`.`system`, `instance`.`service_id`, `instance`.`environment_id` from `instance` where `instance`.`id` = ? for update]; (conn:31742) Lock wait timeout exceeded; try restarting transaction
Query is: select `instance`.`id`, `instance`.`name`, `instance`.`account_id`, `instance`.`kind`, `instance`.`uuid`, `instance`.`description`, `instance`.`state`, `instance`.`created`, `instance`.`removed`, `instance`.`remove_time`, `instance`.`data`, `instance`.`allocation_state`, `instance`.`compute`, `instance`.`memory_mb`, `instance`.`image_id`, `instance`.`hostname`, `instance`.`zone_id`, `instance`.`instance_triggered_stop`, `instance`.`agent_id`, `instance`.`domain`, `instance`.`first_running`, `instance`.`token`, `instance`.`userdata`, `instance`.`registry_credential_id`, `instance`.`external_id`, `instance`.`native_container`, `instance`.`network_container_id`, `instance`.`health_state`, `instance`.`start_count`, `instance`.`create_index`, `instance`.`deployment_unit_uuid`, `instance`.`version`, `instance`.`health_updated`, `instance`.`service_index_id`, `instance`.`dns_internal`, `instance`.`dns_search_internal`, `instance`.`memory_reservation`, `instance`.`milli_cpu_reservation`, `instance`.`system`, `inst...
	at org.jooq.impl.Utils.translate(Utils.java:1287)
	at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:495)
	at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:326)
	at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:351)
	at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:341)
	at org.jooq.impl.AbstractResultQuery.fetchOne(AbstractResultQuery.java:470)
	at org.jooq.impl.UpdatableRecordImpl.checkIfChanged(UpdatableRecordImpl.java:536)
	at org.jooq.impl.UpdatableRecordImpl.storeUpdate0(UpdatableRecordImpl.java:291)
	at org.jooq.impl.UpdatableRecordImpl.access$200(UpdatableRecordImpl.java:90)
	at org.jooq.impl.UpdatableRecordImpl$3.operate(UpdatableRecordImpl.java:260)
	at org.jooq.impl.RecordDelegate.operate(RecordDelegate.java:123)
	at org.jooq.impl.UpdatableRecordImpl.storeUpdate(UpdatableRecordImpl.java:255)
	at org.jooq.impl.UpdatableRecordImpl.update(UpdatableRecordImpl.java:149)
	at io.cattle.platform.object.impl.JooqObjectManager.persistRecord(JooqObjectManager.java:228)
	at io.cattle.platform.object.impl.JooqObjectManager.setFieldsInternal(JooqObjectManager.java:133)
	at io.cattle.platform.object.impl.JooqObjectManager$3.execute(JooqObjectManager.java:121)
	at io.cattle.platform.engine.idempotent.Idempotent.change(Idempotent.java:88)
	at io.cattle.platform.object.impl.JooqObjectManager.setFields(JooqObjectManager.java:118)
	at io.cattle.platform.object.impl.JooqObjectManager.setFields(JooqObjectManager.java:113)
	at io.cattle.platform.object.impl.AbstractObjectManager.setFields(AbstractObjectManager.java:139)
	at io.cattle.platform.process.common.generic.GenericResourceProcessState.setState(GenericResourceProcessState.java:108)
	at io.cattle.platform.engine.process.impl.AbstractStatesBasedProcessState.setTransitioning(AbstractStatesBasedProcessState.java:27)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.setTransitioning(DefaultProcessInstanceImpl.java:569)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:329)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:245)
	at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7)
	at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40)
	at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:242)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:184)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:157)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:116)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:113)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40)
	at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:113)
	at io.cattle.platform.engine.manager.impl.DefaultProcessManager.scheduleProcessInstance(DefaultProcessManager.java:69)
	at io.cattle.platform.object.process.impl.DefaultObjectProcessManager.scheduleProcessInstance(DefaultObjectProcessManager.java:50)
	at io.cattle.platform.object.process.impl.DefaultObjectProcessManager.scheduleProcessInstance(DefaultObjectProcessManager.java:55)
	at io.cattle.platform.process.containerevent.ContainerEventCreate$1.doWithLock(ContainerEventCreate.java:181)
	at io.cattle.platform.process.containerevent.ContainerEventCreate$1.doWithLock(ContainerEventCreate.java:144)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40)
	at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37)
	at io.cattle.platform.process.containerevent.ContainerEventCreate.handleInternal(ContainerEventCreate.java:144)
	at io.cattle.platform.process.containerevent.ContainerEventCreate.handle(ContainerEventCreate.java:118)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:470)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:421)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:415)
	at io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandlers(DefaultProcessInstanceImpl.java:415)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runLogic(DefaultProcessInstanceImpl.java:517)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:345)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:245)
	at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7)
	at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40)
	at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:242)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:184)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:157)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:116)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:113)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40)
	at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37)
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:113)
	at io.cattle.platform.engine.server.impl.ProcessInstanceDispatcherImpl.processExecuteWithLock(ProcessInstanceDispatcherImpl.java:136)
	at io.cattle.platform.engine.server.impl.ProcessInstanceDispatcherImpl$1.doWithLock(ProcessInstanceDispatcherImpl.java:123)
	at io.cattle.platform.engine.server.impl.ProcessInstanceDispatcherImpl$1.doWithLock(ProcessInstanceDispatcherImpl.java:120)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl$4.doWithLock(AbstractLockManagerImpl.java:50)
	at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.tryLock(AbstractLockManagerImpl.java:25)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.tryLock(AbstractLockManagerImpl.java:47)
	at io.cattle.platform.engine.server.impl.ProcessInstanceDispatcherImpl.execute(ProcessInstanceDispatcherImpl.java:120)
	at io.cattle.platform.engine.server.ProcessInstanceReference.doRun(ProcessInstanceReference.java:59)
	at org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:15)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: (conn:31742) Lock wait timeout exceeded; try restarting transaction
Query is: select `instance`.`id`, `instance`.`name`, `instance`.`account_id`, `instance`.`kind`, `instance`.`uuid`, `instance`.`description`, `instance`.`state`, `instance`.`created`, `instance`.`removed`, `instance`.`remove_time`, `instance`.`data`, `instance`.`allocation_state`, `instance`.`compute`, `instance`.`memory_mb`, `instance`.`image_id`, `instance`.`hostname`, `instance`.`zone_id`, `instance`.`instance_triggered_stop`, `instance`.`agent_id`, `instance`.`domain`, `instance`.`first_running`, `instance`.`token`, `instance`.`userdata`, `instance`.`registry_credential_id`, `instance`.`external_id`, `instance`.`native_container`, `instance`.`network_container_id`, `instance`.`health_state`, `instance`.`start_count`, `instance`.`create_index`, `instance`.`deployment_unit_uuid`, `instance`.`version`, `instance`.`health_updated`, `instance`.`service_index_id`, `instance`.`dns_internal`, `instance`.`dns_search_internal`, `instance`.`memory_reservation`, `instance`.`milli_cpu_reservation`, `instance`.`system`, `inst...
	at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:150)
	at org.mariadb.jdbc.internal.util.ExceptionMapper.getException(ExceptionMapper.java:101)
	at org.mariadb.jdbc.internal.util.ExceptionMapper.throwAndLogException(ExceptionMapper.java:77)
	at org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:224)
	at org.mariadb.jdbc.MariaDbServerPreparedStatement.executeInternal(MariaDbServerPreparedStatement.java:411)
	at org.mariadb.jdbc.MariaDbServerPreparedStatement.execute(MariaDbServerPreparedStatement.java:361)
	at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
	at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
	at org.jooq.tools.jdbc.DefaultPreparedStatement.execute(DefaultPreparedStatement.java:194)
	at org.jooq.impl.AbstractResultQuery.execute(AbstractResultQuery.java:261)
	at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:322)
	... 91 common frames omitted
Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Lock wait timeout exceeded; try restarting transaction
Query is: select `instance`.`id`, `instance`.`name`, `instance`.`account_id`, `instance`.`kind`, `instance`.`uuid`, `instance`.`description`, `instance`.`state`, `instance`.`created`, `instance`.`removed`, `instance`.`remove_time`, `instance`.`data`, `instance`.`allocation_state`, `instance`.`compute`, `instance`.`memory_mb`, `instance`.`image_id`, `instance`.`hostname`, `instance`.`zone_id`, `instance`.`instance_triggered_stop`, `instance`.`agent_id`, `instance`.`domain`, `instance`.`first_running`, `instance`.`token`, `instance`.`userdata`, `instance`.`registry_credential_id`, `instance`.`external_id`, `instance`.`native_container`, `instance`.`network_container_id`, `instance`.`health_state`, `instance`.`start_count`, `instance`.`create_index`, `instance`.`deployment_unit_uuid`, `instance`.`version`, `instance`.`health_updated`, `instance`.`service_index_id`, `instance`.`dns_internal`, `instance`.`dns_search_internal`, `instance`.`memory_reservation`, `instance`.`milli_cpu_reservation`, `instance`.`system`, `inst...
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:1115)
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1047)
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1002)
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executePreparedQuery(AbstractQueryProtocol.java:588)
	at org.mariadb.jdbc.MariaDbServerPreparedStatement.executeInternal(MariaDbServerPreparedStatement.java:399)
	... 97 common frames omitted

mbuckman · August 20, 2018, 11:56pm

More Error logs:

2018/07/21 21:41:44 http: proxy error: net/http: request canceled
2018/07/21 23:17:59 http: TLS handshake error from 10.254.0.2:18586: EOF
2018/07/21 23:18:30 http: TLS handshake error from 10.254.0.2:62345: EOF
2018/07/21 23:18:30 http: TLS handshake error from 10.254.0.2:43385: EOF
2018/07/21 23:18:30 http: TLS handshake error from 10.254.0.2:53400: EOF
2018/07/22 00:18:17 http: TLS handshake error from 10.254.0.2:58678: EOF
2018/07/22 01:18:35 http: TLS handshake error from 10.254.0.2:1084: EOF
2018/07/22 02:18:53 http: TLS handshake error from 10.254.0.2:30506: EOF
2018/07/22 03:19:09 http: TLS handshake error from 10.254.0.2:37917: EOF
2018/07/22 04:19:26 http: TLS handshake error from 10.254.0.2:9434: EOF
2018/07/22 05:19:44 http: TLS handshake error from 10.254.0.2:7593: EOF
2018/07/22 06:20:01 http: TLS handshake error from 10.254.0.2:36184: EOF
2018/07/22 07:20:19 http: TLS handshake error from 10.254.0.2:6625: EOF
2018/07/22 08:20:37 http: TLS handshake error from 10.254.0.2:12893: EOF
2018/07/22 09:20:55 http: TLS handshake error from 10.254.0.2:48155: EOF
2018/07/22 10:21:11 http: TLS handshake error from 10.254.0.2:38140: EOF
2018/07/22 11:21:29 http: TLS handshake error from 10.254.0.2:47810: EOF
2018/07/22 12:21:46 http: TLS handshake error from 10.254.0.2:25714: EOF
2018/07/22 13:22:04 http: TLS handshake error from 10.254.0.2:27040: EOF
2018/07/23 17:18:20 http: TLS handshake error from 10.254.0.2:13748: EOF
2018/07/23 17:18:20 http: TLS handshake error from 10.254.0.2:11961: EOF
2018/07/23 18:18:24 http: TLS handshake error from 10.254.0.2:44002: EOF
2018/07/23 19:18:28 http: TLS handshake error from 10.254.0.2:7132: EOF
2018/07/23 20:18:45 http: TLS handshake error from 10.254.0.2:10319: EOF
2018/07/23 20:30:40 http: TLS handshake error from 10.254.0.2:19749: EOF
2018/07/23 21:08:42 http: TLS handshake error from 10.254.0.2:63895: EOF
2018/07/23 21:50:51 http: TLS handshake error from 10.254.0.2:3624: EOF
2018/07/23 22:17:01 http: TLS handshake error from 10.254.0.2:49280: EOF
2018/07/23 23:17:19 http: TLS handshake error from 10.254.0.2:29548: EOF
2018/07/23 23:50:58 http: TLS handshake error from 10.254.0.2:2298: EOF
2018/07/23 23:51:27 http: TLS handshake error from 10.254.0.2:52950: EOF
2018/07/24 00:51:46 http: TLS handshake error from 10.254.0.2:57549: EOF
2018/07/24 01:52:04 http: TLS handshake error from 10.254.0.2:4244: EOF
2018/07/24 02:52:21 http: TLS handshake error from 10.254.0.2:32967: EOF
2018/07/24 03:52:38 http: TLS handshake error from 10.254.0.2:29663: EOF
2018/07/24 04:52:56 http: TLS handshake error from 10.254.0.2:37678: EOF
2018/07/24 05:53:14 http: TLS handshake error from 10.254.0.2:11851: EOF
2018/07/24 06:53:31 http: TLS handshake error from 10.254.0.2:46208: EOF
2018/07/24 07:53:48 http: TLS handshake error from 10.254.0.2:7674: EOF
2018/07/24 08:54:05 http: TLS handshake error from 10.254.0.2:31903: EOF
2018/07/24 21:28:11 http: TLS handshake error from 10.254.0.2:39159: EOF
2018/07/24 21:28:43 http: TLS handshake error from 10.254.0.2:49953: EOF
time="2018-07-25T04:27:13Z" level=error msg="Failed to perform catalog refresh: [Catalog refresh failed for library (https://git.rancher.io/rancher-catalog.git): Update failed: exit status 128]" 
2018/07/25 09:46:35 http: proxy error: net/http: request canceled
2018/07/25 19:43:35 http: TLS handshake error from 10.254.0.2:12788: EOF
2018/07/25 19:43:35 http: TLS handshake error from 10.254.0.2:28918: EOF
2018/07/25 20:43:54 http: TLS handshake error from 10.254.0.2:60775: EOF
2018/07/25 21:44:12 http: TLS handshake error from 10.254.0.2:38593: EOF
2018/07/25 22:44:30 http: TLS handshake error from 10.254.0.2:36668: EOF
2018/07/25 23:44:48 http: TLS handshake error from 10.254.0.2:57481: EOF
2018/07/26 00:45:06 http: TLS handshake error from 10.254.0.2:34889: EOF
2018/07/26 01:45:24 http: TLS handshake error from 10.254.0.2:40544: EOF
2018/07/26 02:45:42 http: TLS handshake error from 10.254.0.2:48810: EOF
2018/07/26 03:46:00 http: TLS handshake error from 10.254.0.2:17989: EOF
2018/07/26 04:46:18 http: TLS handshake error from 10.254.0.2:16311: EOF
2018/07/26 05:46:36 http: TLS handshake error from 10.254.0.2:41756: EOF
2018/07/26 06:46:54 http: TLS handshake error from 10.254.0.2:6779: EOF
2018/07/26 07:47:11 http: TLS handshake error from 10.254.0.2:14670: EOF
2018/07/26 08:47:29 http: TLS handshake error from 10.254.0.2:63075: EOF
2018/07/26 09:47:46 http: TLS handshake error from 10.254.0.2:18883: EOF
2018/07/26 10:48:03 http: TLS handshake error from 10.254.0.2:54471: EOF
2018/07/26 11:48:21 http: TLS handshake error from 10.254.0.2:45806: EOF
2018/07/26 14:55:20 http: TLS handshake error from 10.254.0.2:63537: EOF
2018/07/26 19:30:20 http: TLS handshake error from 10.254.0.2:1818: EOF
2018/07/26 19:30:20 http: TLS handshake error from 10.254.0.2:46791: EOF
2018/07/26 19:30:20 http: TLS handshake error from 10.254.0.2:22257: EOF
2018/07/26 19:30:20 http: TLS handshake error from 10.254.0.2:13980: EOF
2018/07/26 19:30:20 http: TLS handshake error from 10.254.0.2:29626: EOF
2018/07/26 19:30:20 http: TLS handshake error from 10.254.0.2:52499: EOF
2018/07/26 20:15:03 http: TLS handshake error from 10.254.0.2:54290: EOF
2018/07/26 21:15:21 http: TLS handshake error from 10.254.0.2:11907: EOF
2018/07/26 21:32:05 http: TLS handshake error from 10.254.0.2:1588: EOF
2018/07/26 22:15:39 http: TLS handshake error from 10.254.0.2:8499: EOF
2018/07/26 23:15:57 http: TLS handshake error from 10.254.0.2:46991: EOF
2018/07/27 00:16:15 http: TLS handshake error from 10.254.0.2:12416: EOF
2018/07/27 01:16:33 http: TLS handshake error from 10.254.0.2:19084: EOF
2018/07/27 02:16:51 http: TLS handshake error from 10.254.0.2:59461: EOF
2018/07/27 03:17:08 http: TLS handshake error from 10.254.0.2:59385: EOF
2018/07/27 04:17:25 http: TLS handshake error from 10.254.0.2:52332: EOF
2018/07/27 05:17:40 http: TLS handshake error from 10.254.0.2:38362: EOF
2018/07/27 06:17:57 http: TLS handshake error from 10.254.0.2:46378: EOF
2018/07/27 07:18:13 http: TLS handshake error from 10.254.0.2:20656: EOF
2018/07/27 08:18:31 http: TLS handshake error from 10.254.0.2:24224: EOF
2018/07/27 09:18:49 http: TLS handshake error from 10.254.0.2:26876: EOF
2018/07/27 10:19:06 http: TLS handshake error from 10.254.0.2:7956: EOF
2018/07/27 11:19:23 http: TLS handshake error from 10.254.0.2:2430: EOF
2018/07/27 18:05:51 http: TLS handshake error from 10.254.0.2:27555: EOF
2018/07/27 18:05:51 http: TLS handshake error from 10.254.0.2:64489: EOF
2018/07/27 18:05:51 http: TLS handshake error from 10.254.0.2:28909: EOF
2018/07/27 18:05:51 http: TLS handshake error from 10.254.0.2:39462: EOF
2018/07/27 18:05:51 http: TLS handshake error from 10.254.0.2:62744: EOF
2018/07/27 18:05:51 http: TLS handshake error from 10.254.0.2:29551: EOF
2018/07/27 19:06:10 http: TLS handshake error from 10.254.0.2:21312: EOF
2018/07/27 20:06:27 http: TLS handshake error from 10.254.0.2:4233: EOF
2018/07/27 21:06:45 http: TLS handshake error from 10.254.0.2:63535: EOF
2018/07/27 22:07:21 http: TLS handshake error from 10.254.0.2:9661: EOF
2018/07/27 23:07:39 http: TLS handshake error from 10.254.0.2:40453: EOF
FATAL: Exiting due to failed cluster check-in
2018/07/27 23:13:30 http: proxy error: read tcp 127.0.0.1:56980->127.0.0.1:8081: read: connection reset by peer
2018/07/27 23:13:30 http: proxy error: read tcp 127.0.0.1:56978->127.0.0.1:8081: read: connection reset by peer
2018/07/27 23:13:30 http: proxy error: EOF
2018/07/27 23:13:30 http: proxy error: EOF
2018/07/27 23:13:30 http: proxy error: EOF
2018/07/27 23:13:30 http: proxy error: EOF
2018/07/27 23:13:30 http: proxy error: EOF
2018/07/27 23:13:30 http: proxy error: EOF
2018/07/27 23:13:30 http: proxy error: EOF
2018/07/27 23:13:30 http: proxy error: EOF
2018/07/27 23:13:30 http: proxy error: read tcp 127.0.0.1:56906->127.0.0.1:8081: read: connection reset by peer
2018/07/27 23:13:30 http: proxy error: EOF
2018/07/27 23:13:30 http: proxy error: EOF
2018/07/27 23:13:30 http: proxy error: EOF
2018/07/27 23:13:30 http: proxy error: EOF
2018/07/27 23:13:30 http: proxy error: EOF
2018/07/27 23:13:30 http: proxy error: EOF
2018/07/27 23:13:30 http: proxy error: EOF
2018/07/27 23:13:30 http: proxy error: EOF
2018/07/27 23:13:30 http: proxy error: EOF
2018/07/27 23:13:30 http: proxy error: read tcp 127.0.0.1:56982->127.0.0.1:8081: read: connection reset by peer
2018/07/27 23:13:30 http: proxy error: read tcp 127.0.0.1:56986->127.0.0.1:8081: read: connection reset by peer
2018/07/27 23:13:30 http: proxy error: read tcp 127.0.0.1:56984->127.0.0.1:8081: read: connection reset by peer
2018/07/27 23:13:30 http: proxy error: EOF
2018/07/27 23:13:30 http: proxy error: dial tcp 127.0.0.1:8081: getsockopt: connection refused
2018/07/27 23:13:30 http: proxy error: dial tcp 127.0.0.1:8081: getsockopt: connection refused
2018/07/27 23:13:30 http: proxy error: dial tcp 127.0.0.1:8081: getsockopt: connection refused
2018/07/27 23:13:30 http: proxy error: dial tcp 127.0.0.1:8081: getsockopt: connection refused
2018/07/27 23:13:30 http: proxy error: dial tcp 127.0.0.1:8081: getsockopt: connection refused
2018/07/27 23:13:30 http: proxy error: dial tcp 127.0.0.1:8081: getsockopt: connection refused
2018/07/27 23:13:30 http: proxy error: dial tcp 127.0.0.1:8081: getsockopt: connection refused
2018/07/27 23:13:30 http: proxy error: dial tcp 127.0.0.1:8081: getsockopt: connection refused
2018/07/27 23:13:30 http: proxy error: dial tcp 127.0.0.1:8081: getsockopt: connection refused
2018/07/27 23:13:30 http: proxy error: dial tcp 127.0.0.1:8081: getsockopt: connection refused
2018/07/27 23:13:30 http: proxy error: dial tcp 127.0.0.1:8081: getsockopt: connection refused
2018/07/27 23:13:30 http: proxy error: dial tcp 127.0.0.1:8081: getsockopt: connection refused
2018/07/27 23:13:30 http: proxy error: dial tcp 127.0.0.1:8081: getsockopt: connection refused
2018/07/27 23:13:30 http: proxy error: dial tcp 127.0.0.1:8081: getsockopt: connection refused
2018/07/27 23:13:30 http: proxy error: dial tcp 127.0.0.1:8081: getsockopt: connection refused
time="2018-07-27T23:13:30Z" level=fatal msg="Parent process went away. Shutting down." 
time="2018-07-27T23:14:06Z" level=info msg="Downloading key from http://localhost:8081/v1/scripts/api.crt" 
time="2018-07-27T23:14:06Z" level=fatal msg="Error getting config." error="Invalid key content" 
time="2018-07-27T23:14:08Z" level=info msg="Downloading key from http://localhost:8081/v1/scripts/api.crt" 
time="2018-07-27T23:14:08Z" level=fatal msg="Error getting config." error="Invalid key content" 
time="2018-07-27T23:14:10Z" level=info msg="Downloading key from http://localhost:8081/v1/scripts/api.crt" 
time="2018-07-27T23:14:10Z" level=fatal msg="Error getting config." error="Invalid key content" 
time="2018-07-27T23:14:12Z" level=info msg="Downloading key from http://localhost:8081/v1/scripts/api.crt" 
time="2018-07-27T23:14:12Z" level=fatal msg="Error getting config." error="Invalid key content" 
time="2018-07-27T23:14:13Z" level=warning msg="Couldn't load install uuid: Get http://localhost:8080/v2-beta: dial tcp 127.0.0.1:8080: getsockopt: connection refused" 
time="2018-07-27T23:14:13Z" level=info msg="Starting Catalog Service (port 8088, refresh interval 300 seconds)" 
time="2018-07-27T23:14:13Z" level=fatal msg="Failed to configure cattle client: Get http://localhost:8080/v2-beta: dial tcp 127.0.0.1:8080: connect: connection refused" 
time="2018-07-27T23:14:14Z" level=info msg="Downloading key from http://localhost:8081/v1/scripts/api.crt" 
time="2018-07-27T23:14:15Z" level=info msg="Starting websocket proxy. Listening on [:8080], Proxying to cattle API at [localhost:8081], Monitoring parent pid [10]." 
time="2018-07-27T23:14:15Z" level=info msg="Configured http API filter" 
time="2018-07-27T23:14:15Z" level=info msg="Master config file: master.conf" 
time="2018-07-27T23:14:15Z" level=info msg="Master address: 10.5.0.25:8080" 
time="2018-07-27T23:14:15Z" level=info msg="Configured authTokenValidator API filter" 
time="2018-07-27T23:14:15Z" level=info msg="Downloading certificate from http://localhost:8081/v1/credentials/1c1/certificate" 
time="2018-07-27T23:14:16Z" level=info msg="Webhook service listening on 8085" 
2018/07/27 23:14:22 http: TLS handshake error from 10.254.0.2:53346: EOF
time="2018-07-27T23:14:24Z" level=info msg="Starting Rancher Auth service" 
time="2018-07-27T23:14:24Z" level=info msg="Config stored" 
time="2018-07-27T23:14:25Z" level=info msg="Auth provider configured githubconfig" 
time="2018-07-27T23:14:25Z" level=info msg="Listening on :8090" 
2018/07/27 23:14:33 http: TLS handshake error from 10.254.0.2:17981: EOF
2018/07/27 23:14:46 http: TLS handshake error from 10.254.0.2:24937: EOF
2018/07/27 23:14:58 http: TLS handshake error from 10.254.0.2:31104: EOF
2018/07/27 23:15:12 http: TLS handshake error from 10.254.0.2:19507: EOF
2018/07/27 23:15:27 http: TLS handshake error from 10.254.0.2:15159: EOF
2018/07/27 23:15:43 http: TLS handshake error from 10.254.0.2:51276: EOF
2018/07/27 23:16:00 http: TLS handshake error from 10.254.0.2:47312: EOF
2018/07/27 23:16:18 http: TLS handshake error from 10.254.0.2:2431: EOF
2018/07/27 23:16:37 http: TLS handshake error from 10.254.0.2:47575: EOF
2018/07/27 23:16:57 http: TLS handshake error from 10.254.0.2:65334: EOF
2018/07/27 23:17:18 http: TLS handshake error from 10.254.0.2:33196: EOF
2018/07/27 23:17:40 http: TLS handshake error from 10.254.0.2:14317: EOF
2018/07/27 23:17:42 http: TLS handshake error from 10.254.0.2:47099: EOF
2018/07/27 23:17:43 http: TLS handshake error from 10.254.0.2:13873: EOF
2018/07/27 23:18:03 http: TLS handshake error from 10.254.0.2:35299: EOF
2018/07/28 05:07:43 http: proxy error: net/http: request canceled
2018/07/29 01:59:52 http: proxy error: net/http: request canceled
2018/07/29 01:59:52 http: proxy error: net/http: request canceled
2018/07/29 01:59:52 http: proxy error: net/http: request canceled
2018/07/29 14:21:20 http: TLS handshake error from 10.254.0.2:26663: EOF
2018/07/29 15:21:37 http: TLS handshake error from 10.254.0.2:50368: EOF
2018/07/29 16:21:54 http: TLS handshake error from 10.254.0.2:64608: EOF
2018/07/29 17:22:11 http: TLS handshake error from 10.254.0.2:19104: EOF
2018/07/29 18:22:28 http: TLS handshake error from 10.254.0.2:9543: EOF
2018/07/29 19:22:46 http: TLS handshake error from 10.254.0.2:12538: EOF
2018/07/29 20:23:04 http: TLS handshake error from 10.254.0.2:33218: EOF
2018/07/29 20:30:02 http: TLS handshake error from 10.254.0.2:38760: EOF
2018/07/29 20:30:02 http: TLS handshake error from 10.254.0.2:15283: EOF
2018/07/29 20:30:02 http: TLS handshake error from 10.254.0.2:2935: EOF
2018/07/29 20:50:12 http: proxy error: net/http: request canceled
2018/07/29 20:50:12 http: proxy error: net/http: request canceled
2018/07/29 20:50:12 http: proxy error: net/http: request canceled
2018/07/29 21:23:22 http: TLS handshake error from 10.254.0.2:9917: EOF
2018/07/29 21:30:23 http: TLS handshake error from 10.254.0.2:30440: EOF
2018/07/29 22:23:40 http: TLS handshake error from 10.254.0.2:34699: EOF
2018/07/29 22:30:40 http: TLS handshake error from 10.254.0.2:17012: EOF
2018/07/29 23:13:17 http: TLS handshake error from 10.254.0.2:45620: EOF
2018/07/29 23:13:17 http: TLS handshake error from 10.254.0.2:12395: EOF
2018/07/29 23:23:58 http: TLS handshake error from 10.254.0.2:26925: EOF
2018/07/30 00:13:37 http: TLS handshake error from 10.254.0.2:27009: EOF
2018/07/30 00:24:15 http: TLS handshake error from 10.254.0.2:55269: EOF
2018/07/30 01:13:49 http: TLS handshake error from 10.254.0.2:45865: EOF
2018/07/30 01:24:32 http: TLS handshake error from 10.254.0.2:25310: EOF
2018/07/30 02:14:05 http: TLS handshake error from 10.254.0.2:26236: EOF
2018/07/30 02:24:50 http: TLS handshake error from 10.254.0.2:41655: EOF
2018/07/30 03:14:23 http: TLS handshake error from 10.254.0.2:4955: EOF
2018/07/30 03:25:07 http: TLS handshake error from 10.254.0.2:62209: EOF
2018/07/30 04:14:37 http: TLS handshake error from 10.254.0.2:54873: EOF
2018/07/30 04:25:25 http: TLS handshake error from 10.254.0.2:47640: EOF
2018/07/30 05:14:52 http: TLS handshake error from 10.254.0.2:15587: EOF
2018/07/30 05:25:42 http: TLS handshake error from 10.254.0.2:33241: EOF
2018/07/30 06:15:07 http: TLS handshake error from 10.254.0.2:36456: EOF
FATAL: Exiting due to failed cluster check-in
2018/07/30 07:13:47 http: proxy error: EOF
2018/07/30 07:13:47 http: proxy error: read tcp 127.0.0.1:46138->127.0.0.1:8081: read: connection reset by peer
2018/07/30 07:13:47 http: proxy error: EOF
2018/07/30 07:13:47 http: proxy error: read tcp 127.0.0.1:46160->127.0.0.1:8081: read: connection reset by peer
time="2018-07-30T07:13:47Z" level=fatal msg="Parent process went away. Shutting down." 
time="2018-07-30T07:14:22Z" level=info msg="Downloading key from http://localhost:8081/v1/scripts/api.crt" 
time="2018-07-30T07:14:22Z" level=fatal msg="Error getting config." error="Invalid key content" 
time="2018-07-30T07:14:24Z" level=info msg="Downloading key from http://localhost:8081/v1/scripts/api.crt" 
time="2018-07-30T07:14:24Z" level=fatal msg="Error getting config." error="Invalid key content" 
time="2018-07-30T07:14:26Z" level=info msg="Downloading key from http://localhost:8081/v1/scripts/api.crt" 
time="2018-07-30T07:14:26Z" level=fatal msg="Error getting config." error="Invalid key content" 
time="2018-07-30T07:14:28Z" level=info msg="Downloading key from http://localhost:8081/v1/scripts/api.crt" 
time="2018-07-30T07:14:28Z" level=fatal msg="Error getting config." error="Invalid key content" 
time="2018-07-30T07:14:28Z" level=warning msg="Couldn't load install uuid: Get http://localhost:8080/v2-beta: dial tcp 127.0.0.1:8080: getsockopt: connection refused" 
time="2018-07-30T07:14:28Z" level=info msg="Starting Catalog Service (port 8088, refresh interval 300 seconds)" 
time="2018-07-30T07:14:28Z" level=fatal msg="Failed to configure cattle client: Get http://localhost:8080/v2-beta: dial tcp 127.0.0.1:8080: connect: connection refused" 
time="2018-07-30T07:14:30Z" level=info msg="Downloading key from http://localhost:8081/v1/scripts/api.crt" 
time="2018-07-30T07:14:30Z" level=info msg="Starting websocket proxy. Listening on [:8080], Proxying to cattle API at [localhost:8081], Monitoring parent pid [10]." 
time="2018-07-30T07:14:30Z" level=info msg="Configured http API filter" 
time="2018-07-30T07:14:30Z" level=info msg="Configured authTokenValidator API filter" 
time="2018-07-30T07:14:30Z" level=info msg="Master config file: master.conf" 
time="2018-07-30T07:14:30Z" level=info msg="Master address: 10.5.0.25:8080" 
time="2018-07-30T07:14:30Z" level=info msg="Downloading certificate from http://localhost:8081/v1/credentials/1c1/certificate" 
time="2018-07-30T07:14:31Z" level=info msg="Webhook service listening on 8085" 
2018/07/30 07:14:33 http: TLS handshake error from 10.254.0.2:21075: EOF
time="2018-07-30T07:14:35Z" level=info msg="Starting Rancher Auth service" 
time="2018-07-30T07:14:36Z" level=info msg="Config stored" 
2018/07/30 07:14:38 http: TLS handshake error from 10.254.0.2:40290: EOF
time="2018-07-30T07:14:38Z" level=info msg="Auth provider configured githubconfig" 
time="2018-07-30T07:14:39Z" level=info msg="Listening on :8090" 
2018/07/30 07:14:45 http: TLS handshake error from 10.254.0.2:1580: EOF
2018/07/30 07:14:52 http: TLS handshake error from 10.254.0.2:46130: EOF
2018/07/30 07:15:00 http: TLS handshake error from 10.254.0.2:48609: EOF
2018/07/30 07:15:09 http: TLS handshake error from 10.254.0.2:28831: EOF
2018/07/30 07:15:19 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n\n"; err=<nil>
2018/07/30 07:15:22 http: TLS handshake error from 10.254.0.2:39179: EOF
2018/07/30 07:15:22 http: TLS handshake error from 10.254.0.2:11223: EOF
2018/07/30 07:15:23 http: TLS handshake error from 10.254.0.2:27230: EOF
2018/07/30 07:15:33 http: TLS handshake error from 10.254.0.2:9183: EOF
2018/07/30 08:41:52 http: proxy error: net/http: request canceled
2018/07/30 08:41:53 http: proxy error: net/http: request canceled
2018/07/30 17:37:48 http: TLS handshake error from 10.254.0.2:38333: EOF
2018/07/30 17:37:48 http: TLS handshake error from 10.254.0.2:29912: EOF
2018/07/30 18:39:37 http: TLS handshake error from 10.254.0.2:18684: EOF
2018/07/30 19:39:53 http: TLS handshake error from 10.254.0.2:43834: EOF
2018/07/30 20:40:11 http: TLS handshake error from 10.254.0.2:30132: EOF
2018/07/30 21:40:26 http: TLS handshake error from 10.254.0.2:45604: EOF
2018/07/30 22:40:42 http: TLS handshake error from 10.254.0.2:58339: EOF
2018/07/30 23:40:59 http: TLS handshake error from 10.254.0.2:61258: EOF
time="2018-08-01T18:54:31Z" level=error msg="Failed to perform catalog refresh: [Catalog refresh failed for library (https://git.rancher.io/rancher-catalog.git): Update failed: exit status 128]" 
time="2018-08-01T18:59:33Z" level=error msg="Failed to perform catalog refresh: [Catalog refresh failed for library (https://git.rancher.io/rancher-catalog.git): Update failed: exit status 128]" 
2018/08/02 20:00:47 http: TLS handshake error from 10.254.0.2:25073: EOF
2018/08/02 20:25:12 http: proxy error: net/http: request canceled
2018/08/02 20:45:31 http: TLS handshake error from 10.254.0.2:50710: EOF
2018/08/02 20:45:31 http: TLS handshake error from 10.254.0.2:59859: EOF
2018/08/02 21:45:33 http: TLS handshake error from 10.254.0.2:18548: EOF
FATAL: Exiting due to failed cluster check-in
2018/08/02 22:41:45 http: proxy error: net/http: request canceled
2018/08/02 22:41:45 http: proxy error: EOF
2018/08/02 22:41:45 http: proxy error: read tcp 127.0.0.1:46424->127.0.0.1:8081: read: connection reset by peer
2018/08/02 22:41:45 http: proxy error: read tcp 127.0.0.1:46398->127.0.0.1:8081: read: connection reset by peer
2018/08/02 22:41:45 http: proxy error: EOF
2018/08/02 22:41:45 http: proxy error: EOF
2018/08/02 22:41:45 http: proxy error: EOF
2018/08/02 22:41:45 http: proxy error: EOF
2018/08/02 22:41:45 http: proxy error: EOF
2018/08/02 22:41:45 http: proxy error: read tcp 127.0.0.1:46458->127.0.0.1:8081: read: connection reset by peer
2018/08/02 22:41:45 http: proxy error: read tcp 127.0.0.1:46456->127.0.0.1:8081: read: connection reset by peer
2018/08/02 22:41:45 http: proxy error: read tcp 127.0.0.1:46454->127.0.0.1:8081: read: connection reset by peer
2018/08/02 22:41:45 http: proxy error: read tcp 127.0.0.1:46474->127.0.0.1:8081: read: connection reset by peer
2018/08/02 22:41:45 http: proxy error: read tcp 127.0.0.1:46452->127.0.0.1:8081: read: connection reset by peer
2018/08/02 22:41:45 http: proxy error: read tcp 127.0.0.1:46448->127.0.0.1:8081: read: connection reset by peer
2018/08/02 22:41:45 http: proxy error: read tcp 127.0.0.1:46450->127.0.0.1:8081: read: connection reset by peer
2018/08/02 22:41:45 http: proxy error: read tcp 127.0.0.1:46442->127.0.0.1:8081: read: connection reset by peer
2018/08/02 22:41:45 http: proxy error: read tcp 127.0.0.1:46440->127.0.0.1:8081: read: connection reset by peer
2018/08/02 22:41:45 http: proxy error: read tcp 127.0.0.1:46430->127.0.0.1:8081: read: connection reset by peer
2018/08/02 22:41:45 http: proxy error: read tcp 127.0.0.1:46428->127.0.0.1:8081: read: connection reset by peer
2018/08/02 22:41:45 http: proxy error: read tcp 127.0.0.1:46426->127.0.0.1:8081: read: connection reset by peer
2018/08/02 22:41:45 http: proxy error: read tcp 127.0.0.1:46446->127.0.0.1:8081: read: connection reset by peer
2018/08/02 22:41:45 http: proxy error: read tcp 127.0.0.1:46466->127.0.0.1:8081: read: connection reset by peer
2018/08/02 22:41:45 http: proxy error: dial tcp 127.0.0.1:8081: getsockopt: connection refused
time="2018-08-02T22:41:45Z" level=fatal msg="Parent process went away. Shutting down." 
time="2018-08-02T22:42:22Z" level=info msg="Downloading key from http://localhost:8081/v1/scripts/api.crt" 
time="2018-08-02T22:42:22Z" level=fatal msg="Error getting config." error="Invalid key content" 
time="2018-08-02T22:42:24Z" level=info msg="Downloading key from http://localhost:8081/v1/scripts/api.crt" 
time="2018-08-02T22:42:24Z" level=fatal msg="Error getting config." error="Invalid key content" 
time="2018-08-02T22:42:26Z" level=info msg="Downloading key from http://localhost:8081/v1/scripts/api.crt" 
time="2018-08-02T22:42:26Z" level=fatal msg="Error getting config." error="Invalid key content" 
time="2018-08-02T22:42:28Z" level=info msg="Downloading key from http://localhost:8081/v1/scripts/api.crt" 
time="2018-08-02T22:42:28Z" level=fatal msg="Error getting config." error="Invalid key content" 
time="2018-08-02T22:42:28Z" level=fatal msg="Failed to configure cattle client: Get http://localhost:8080/v2-beta: dial tcp 127.0.0.1:8080: connect: connection refused" 
time="2018-08-02T22:42:28Z" level=warning msg="Couldn't load install uuid: Get http://localhost:8080/v2-beta: dial tcp 127.0.0.1:8080: getsockopt: connection refused" 
time="2018-08-02T22:42:28Z" level=info msg="Starting Catalog Service (port 8088, refresh interval 300 seconds)" 
time="2018-08-02T22:42:30Z" level=info msg="Downloading key from http://localhost:8081/v1/scripts/api.crt" 
time="2018-08-02T22:42:30Z" level=info msg="Starting websocket proxy. Listening on [:8080], Proxying to cattle API at [localhost:8081], Monitoring parent pid [10]." 
time="2018-08-02T22:42:30Z" level=info msg="Configured http API filter" 
time="2018-08-02T22:42:30Z" level=info msg="Configured authTokenValidator API filter" 
time="2018-08-02T22:42:30Z" level=info msg="Master config file: master.conf" 
time="2018-08-02T22:42:30Z" level=info msg="Master address: 10.5.0.25:8080" 
time="2018-08-02T22:42:30Z" level=info msg="Downloading certificate from http://localhost:8081/v1/credentials/1c1/certificate" 
time="2018-08-02T22:42:31Z" level=info msg="Webhook service listening on 8085" 
2018/08/02 22:42:33 http: TLS handshake error from 10.254.0.2:53197: EOF
2018/08/02 22:42:36 http: TLS handshake error from 10.254.0.2:1025: EOF
2018/08/02 22:42:41 http: TLS handshake error from 10.254.0.2:36676: EOF
time="2018-08-02T22:42:42Z" level=info msg="Starting Rancher Auth service" 
time="2018-08-02T22:42:42Z" level=info msg="Config stored" 
time="2018-08-02T22:42:44Z" level=info msg="Auth provider configured githubconfig" 
time="2018-08-02T22:42:44Z" level=info msg="Listening on :8090" 
2018/08/02 22:42:48 http: TLS handshake error from 10.254.0.2:12541: EOF
2018/08/02 22:42:56 http: TLS handshake error from 10.254.0.2:25565: EOF
2018/08/02 22:43:07 http: TLS handshake error from 10.254.0.2:55814: EOF
2018/08/02 22:43:17 http: TLS handshake error from 10.254.0.2:64755: EOF
2018/08/02 22:43:19 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n\n"; err=<nil>
2018/08/02 22:43:26 http: TLS handshake error from 10.254.0.2:34175: EOF
2018/08/02 22:43:37 http: TLS handshake error from 10.254.0.2:31578: EOF
2018/08/02 22:43:49 http: TLS handshake error from 10.254.0.2:50068: EOF
2018/08/02 22:44:02 http: TLS handshake error from 10.254.0.2:62279: EOF
2018/08/02 22:44:17 http: TLS handshake error from 10.254.0.2:3621: EOF
2018/08/02 22:44:33 http: TLS handshake error from 10.254.0.2:8843: EOF
2018/08/02 22:44:49 http: TLS handshake error from 10.254.0.2:44531: EOF
2018/08/02 22:45:10 http: TLS handshake error from 10.254.0.2:52400: EOF
2018/08/02 22:45:30 http: TLS handshake error from 10.254.0.2:18304: EOF
2018/08/02 22:45:52 http: TLS handshake error from 10.254.0.2:11316: EOF
2018/08/02 22:46:15 http: TLS handshake error from 10.254.0.2:61118: EOF
2018/08/02 22:46:35 http: TLS handshake error from 10.254.0.2:29104: EOF
2018/08/02 22:46:57 http: TLS handshake error from 10.254.0.2:36186: EOF
2018/08/02 22:47:20 http: TLS handshake error from 10.254.0.2:24073: EOF
2018/08/02 22:47:44 http: TLS handshake error from 10.254.0.2:51560: EOF
2018/08/02 22:48:10 http: TLS handshake error from 10.254.0.2:33494: EOF
2018/08/02 22:48:36 http: TLS handshake error from 10.254.0.2:29866: EOF
2018/08/02 22:48:53 http: TLS handshake error from 10.254.0.2:8502: EOF
2018/08/02 22:48:54 http: TLS handshake error from 10.254.0.2:23396: EOF
2018/08/02 22:49:02 http: TLS handshake error from 10.254.0.2:1687: EOF
2018/08/02 23:03:59 http: proxy error: net/http: request canceled
2018/08/02 23:04:10 http: proxy error: net/http: request canceled
time="2018-08-03T00:57:47Z" level=error msg="Failed to perform catalog refresh: [Catalog refresh failed for community (https://git.rancher.io/community-catalog.git): Remote commit check failed: Repo [https://git.rancher.io/community-catalog.git] is not accessible: Get https://git.rancher.io/repos/community-catalog/commits/master: net/http: request canceled (Client.Timeout exceeded while awaiting headers)]" 
2018/08/03 02:26:13 http: proxy error: read tcp 127.0.0.1:45132->127.0.0.1:8081: read: connection reset by peer
2018/08/03 04:17:57 http: proxy error: net/http: request canceled
2018/08/03 04:18:00 http: proxy error: net/http: request canceled
2018/08/03 04:18:00 http: proxy error: net/http: request canceled
2018/08/03 05:33:35 http: proxy error: net/http: request canceled
2018/08/03 06:18:26 http: proxy error: net/http: request canceled
time="2018-08-03T08:32:49Z" level=error msg="Multiple errors encountered performing catalog refresh" 
time="2018-08-03T08:32:49Z" level=error msg="Catalog refresh failed for community (https://git.rancher.io/community-catalog.git): Remote commit check failed: Repo [https://git.rancher.io/community-catalog.git] is not accessible: Get https://git.rancher.io/repos/community-catalog/commits/master: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" 
time="2018-08-03T08:32:49Z" level=error msg="Catalog refresh failed for library (https://git.rancher.io/rancher-catalog.git): Remote commit check failed: Repo [https://git.rancher.io/rancher-catalog.git] is not accessible: Get https://git.rancher.io/repos/rancher-catalog/commits/v1.6-release: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" 
2018/08/03 08:33:52 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n\n"; err=<nil>
2018/08/03 09:58:56 http: proxy error: net/http: request canceled
2018/08/03 10:51:52 http: proxy error: net/http: request canceled
2018/08/03 11:17:05 http: proxy error: net/http: request canceled
2018/08/03 13:36:11 http: TLS handshake error from 10.254.0.2:11211: EOF
2018/08/03 13:36:14 http: TLS handshake error from 10.254.0.2:3944: EOF
2018/08/03 13:36:14 http: TLS handshake error from 10.254.0.2:44243: EOF
2018/08/03 14:36:37 http: TLS handshake error from 10.254.0.2:21295: EOF
2018/08/03 15:36:52 http: TLS handshake error from 10.254.0.2:24449: EOF
2018/08/03 16:37:06 http: TLS handshake error from 10.254.0.2:9465: EOF
2018/08/03 17:47:20 http: TLS handshake error from 10.254.0.2:36573: EOF
2018/08/03 18:47:27 http: TLS handshake error from 10.254.0.2:50910: EOF
2018/08/03 19:47:37 http: TLS handshake error from 10.254.0.2:64503: EOF
2018/08/03 20:47:51 http: TLS handshake error from 10.254.0.2:29008: EOF
2018/08/03 21:48:03 http: TLS handshake error from 10.254.0.2:35817: EOF
2018/08/03 22:48:16 http: TLS handshake error from 10.254.0.2:10007: EOF
2018/08/03 23:48:17 http: TLS handshake error from 10.254.0.2:45416: EOF
2018/08/04 00:48:44 http: TLS handshake error from 10.254.0.2:39671: EOF
2018/08/04 00:48:44 http: TLS handshake error from 10.254.0.2:62206: EOF
2018/08/04 01:48:58 http: TLS handshake error from 10.254.0.2:7714: EOF
2018/08/04 02:49:12 http: TLS handshake error from 10.254.0.2:36703: EOF
2018/08/04 03:49:14 http: TLS handshake error from 10.254.0.2:61446: EOF
2018/08/04 04:49:41 http: TLS handshake error from 10.254.0.2:34797: EOF
2018/08/06 20:22:34 http: TLS handshake error from 10.254.0.2:4046: EOF
2018/08/06 20:22:36 http: TLS handshake error from 10.254.0.2:10924: EOF
2018/08/06 20:22:36 http: TLS handshake error from 10.254.0.2:21533: EOF
2018/08/06 21:22:51 http: TLS handshake error from 10.254.0.2:27857: EOF
2018/08/06 22:23:07 http: TLS handshake error from 10.254.0.2:8261: EOF
2018/08/06 23:23:24 http: TLS handshake error from 10.254.0.2:27504: EOF
2018/08/07 00:23:40 http: TLS handshake error from 10.254.0.2:33136: EOF
2018/08/07 01:23:55 http: TLS handshake error from 10.254.0.2:62258: EOF
2018/08/07 02:24:11 http: TLS handshake error from 10.254.0.2:10572: EOF
2018/08/07 03:24:25 http: TLS handshake error from 10.254.0.2:55733: EOF
2018/08/07 04:24:37 http: TLS handshake error from 10.254.0.2:2830: EOF
2018/08/07 05:24:49 http: TLS handshake error from 10.254.0.2:44161: EOF
2018/08/07 06:25:03 http: TLS handshake error from 10.254.0.2:34496: EOF
2018/08/07 07:25:11 http: TLS handshake error from 10.254.0.2:5138: EOF
2018/08/07 08:25:36 http: TLS handshake error from 10.254.0.2:12927: EOF
2018/08/07 09:25:51 http: TLS handshake error from 10.254.0.2:43841: EOF
2018/08/07 10:26:01 http: TLS handshake error from 10.254.0.2:29841: EOF
2018/08/07 11:26:24 http: TLS handshake error from 10.254.0.2:57346: EOF
2018/08/07 12:26:24 http: TLS handshake error from 10.254.0.2:63315: EOF
2018/08/07 15:15:51 http: TLS handshake error from 10.254.0.2:59062: EOF
2018/08/07 16:16:05 http: TLS handshake error from 10.254.0.2:32115: EOF
time="2018-08-07T16:27:33Z" level=error msg="Failed to perform catalog refresh: [Catalog refresh failed for library (https://git.rancher.io/rancher-catalog.git): Update failed: exit status 128]" 
2018/08/07 17:16:17 http: TLS handshake error from 10.254.0.2:29351: EOF
2018/08/07 18:16:30 http: TLS handshake error from 10.254.0.2:48147: EOF
2018/08/07 19:16:42 http: TLS handshake error from 10.254.0.2:9881: EOF
2018/08/07 20:05:05 http: proxy error: net/http: request canceled
2018/08/07 20:16:54 http: TLS handshake error from 10.254.0.2:42014: EOF
2018/08/07 21:17:06 http: TLS handshake error from 10.254.0.2:20921: EOF
2018/08/07 22:17:18 http: TLS handshake error from 10.254.0.2:26437: EOF
2018/08/07 23:17:32 http: TLS handshake error from 10.254.0.2:56711: EOF
2018/08/08 00:17:42 http: TLS handshake error from 10.254.0.2:49724: EOF
2018/08/08 01:17:53 http: TLS handshake error from 10.254.0.2:11410: EOF
2018/08/08 02:18:05 http: TLS handshake error from 10.254.0.2:50631: EOF
2018/08/08 03:18:16 http: TLS handshake error from 10.254.0.2:27755: EOF
2018/08/08 04:18:31 http: TLS handshake error from 10.254.0.2:50565: EOF
2018/08/08 05:18:46 http: TLS handshake error from 10.254.0.2:27864: EOF
2018/08/08 06:18:57 http: TLS handshake error from 10.254.0.2:12581: EOF
time="2018-08-08T06:42:39Z" level=error msg="Failed to perform catalog refresh: [Catalog refresh failed for community (https://git.rancher.io/community-catalog.git): Remote commit check failed: Repo [https://git.rancher.io/community-catalog.git] is not accessible: Get https://git.rancher.io/repos/community-catalog/commits/master: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)]" 
2018/08/08 06:43:29 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n\n"; err=<nil>
2018/08/08 07:19:07 http: TLS handshake error from 10.254.0.2:52752: EOF
time="2018-08-08T14:00:17Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:17Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:17Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:17Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:19Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:19Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:21Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:21Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:21Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:21Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:21Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:21Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:21Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:21Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:21Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:21Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:22Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:22Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:22Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:22Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:22Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:22Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:24Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:24Z" level=error msg="Error dialing websocket backend." error="dial tcp 10.5.0.25:8080: getsockopt: connection refused" 
time="2018-08-08T14:00:25Z" level=info msg="Master address: 10.5.0.99:8080" 
2018/08/09 13:37:46 http: TLS handshake error from 10.254.0.2:12280: EOF
2018/08/09 14:38:08 http: TLS handshake error from 10.254.0.2:32865: EOF
2018/08/09 14:44:57 http: proxy error: net/http: request canceled
2018/08/09 14:44:57 http: proxy error: net/http: request canceled
[mysql] 2018/08/09 14:47:22 packets.go:66: read tcp 172.17.0.2:53386->10.5.0.87:3306: read: connection reset by peer
[mysql] 2018/08/09 14:47:22 packets.go:412: busy buffer
[mysql] 2018/08/09 14:47:22 connection.go:312: invalid connection
2018/08/09 14:53:04 http: proxy error: net/http: request canceled
2018/08/09 15:15:41 http: TLS handshake error from 10.254.0.2:11610: EOF
2018/08/09 15:15:41 http: TLS handshake error from 10.254.0.2:63121: EOF
2018/08/09 15:40:06 http: TLS handshake error from 10.254.0.2:20498: EOF
2018/08/09 16:16:15 http: TLS handshake error from 10.254.0.2:2474: EOF
2018/08/09 16:48:12 http: TLS handshake error from 10.254.0.2:20005: EOF
2018/08/09 17:16:33 http: TLS handshake error from 10.254.0.2:45370: EOF
2018/08/09 17:18:59 http: proxy error: net/http: request canceled
2018/08/09 17:18:59 http: proxy error: net/http: request canceled
2018/08/09 17:48:24 http: TLS handshake error from 10.254.0.2:54619: EOF
2018/08/09 18:16:51 http: TLS handshake error from 10.254.0.2:54033: EOF
2018/08/09 18:49:10 http: TLS handshake error from 10.254.0.2:57989: read tcp 172.17.0.2:8080->10.254.0.2:57989: read: connection reset by peer
2018/08/09 18:51:02 http: TLS handshake error from 10.254.0.2:2976: EOF
error from daemon in stream: Error grabbing logs: invalid character '\x00' looking for beginning of value

mbuckman · August 21, 2018, 12:11am

This one is repeated numerous times:

error from daemon in stream: Error grabbing logs: invalid character '\x00' looking for beginning of value

Topic		Replies	Views
V1.6.28 Rancher server crashes after 2 weeks (usually on sunday) Rancher 1.x	0	944	November 5, 2019
Auto restart sometimes not working Rancher 1.x	3	2509	March 25, 2019
Rancher-server crashed, causes docker to hang Rancher	2	1833	July 24, 2019
Rancher HA Management Stack not coming up Rancher 1.x	20	2979	April 30, 2016
Scheduler in Environments crashed after RancherServer Crash	2	907	November 16, 2017

Rancher 1.6.18 - Cattle - HA - Self-Hosted - Dashboard keeps crashing!

Related topics