Losing primary Rancher server every day or two, sometimes reboot brings it backup within a few minutes, sometimes does not. Other HA servers only show login then redirect to primary which is not responding and spitting out so m,any errors. Recently upgraded, Rancher was rock solid prior.
During the loss of dashboard at least containers remain running, although inaccessible unless I ssh and manually manage via Docker CLI.
Please help1
Sample of Errors:
2018-08-02 22:51:00,577 ERROR [1f3d65e0-fd50-4dab-b11b-cfb8e1f9db6b:23136] [service:99] [service.update] [] [ecutorService-7] [c.p.e.p.i.DefaultProcessInstanceImpl] Container should have been running but is in stopped state. Check logs for more information.
2018-08-02 22:51:00,589 ERROR [:] [] [] [] [ecutorService-7] [.e.s.i.ProcessInstanceDispatcherImpl] Container should have been running but is in stopped state. Check logs for more information.
2018-08-02 22:51:15,574 ERROR [deeb0b8f-1108-4f99-8ead-f4c7b954c064:23146] [service:99] [service.update] [] [ecutorService-7] [c.p.e.p.i.DefaultProcessInstanceImpl] Container should have been running but is in stopped state. Check logs for more information.
2018-08-02 22:51:15,588 ERROR [:] [] [] [] [ecutorService-7] [.e.s.i.ProcessInstanceDispatcherImpl] Container should have been running but is in stopped state. Check logs for more information.
2018-08-02 23:03:59,765 WARN [:] [] [] [] [p1020391880-156] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1353]
2018-08-02 23:03:59,783 WARN [:] [] [] [] [p1020391880-157] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [557]
2018/08/02 23:03:59 http: proxy error: net/http: request canceled
2018-08-02 23:03:59,867 WARN [:] [] [] [] [p1020391880-206] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1746]
2018-08-02 23:03:59,997 WARN [:] [] [] [] [p1020391880-209] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1734]
2018-08-02 23:04:00,097 WARN [:] [] [] [] [p1020391880-174] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1726]
2018-08-02 23:04:00,199 WARN [:] [] [] [] [p1020391880-158] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1738]
2018-08-02 23:04:00,254 WARN [:] [] [] [] [tp1020391880-19] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1701]
2018-08-02 23:04:00,286 WARN [:] [] [] [] [p1020391880-229] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1772]
2018-08-02 23:04:00,450 WARN [:] [] [] [] [p1020391880-138] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1663]
2018-08-02 23:04:00,499 WARN [:] [] [] [] [p1020391880-132] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1800]
2018-08-02 23:04:00,830 WARN [:] [] [] [] [p1020391880-167] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1732]
2018-08-02 23:04:00,875 WARN [:] [] [] [] [tp1020391880-21] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1660]
2018-08-02 23:04:01,193 WARN [:] [] [] [] [p1020391880-182] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1749]
2018-08-02 23:04:01,279 WARN [:] [] [] [] [p1020391880-140] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1714]
2018-08-02 23:04:01,286 WARN [:] [] [] [] [p1020391880-136] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [592]
2018-08-02 23:04:01,718 WARN [:] [] [] [] [p1020391880-176] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1762]
2018-08-02 23:04:02,227 WARN [:] [] [] [] [p1020391880-228] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1699]
2018-08-02 23:04:02,869 WARN [:] [] [] [] [p1020391880-162] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1797]
2018-08-02 23:04:02,896 WARN [:] [] [] [] [p1020391880-180] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1481]
2018-08-02 23:04:03,209 WARN [:] [] [] [] [p1020391880-231] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1710]
2018-08-02 23:04:03,235 WARN [:] [] [] [] [p1020391880-131] [i.c.p.a.p.s.j.WebSocketMessageWriter] Unexpected websocket error for agent [1748]
2018-08-02 23:04:04,440 WARN [:] [] [] [] [.IO.thread-in-0] [c.hazelcast.nio.tcp.TcpIpConnection ] [10.5.0.84]:9345 [09174270-ee63-403c-b184-0ff2a1bb0c4f] [3.8] Connection[id=1, /172.17.0.2:41051->/10.5.0.25:9345, endpoint=[10.5.0.25]:9345, alive=false, type=MEMBER] closed. Reason: Exception in NonBlockingSocketReader java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at com.hazelcast.nio.tcp.DefaultSocketChannelWrapper.read(DefaultSocketChannelWrapper.java:61)
at com.hazelcast.internal.networking.nonblocking.NonBlockingSocketReader.handle(NonBlockingSocketReader.java:150)
at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.handleSelectionKey(NonBlockingIOThread.java:350)
at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.handleSelectionKeys(NonBlockingIOThread.java:335)
at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.selectLoop(NonBlockingIOThread.java:253)
at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.run(NonBlockingIOThread.java:206)
2018-08-02 23:04:05,972 WARN [:] [] [] [] [.IO.thread-in-1] [c.hazelcast.nio.tcp.TcpIpConnection ] [10.5.0.84]:9345 [09174270-ee63-403c-b184-0ff2a1bb0c4f] [3.8] Connection[id=2, /172.17.0.2:57487->/10.5.0.99:9345, endpoint=[10.5.0.99]:9345, alive=false, type=MEMBER] closed. Reason: Exception in NonBlockingSocketReader java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at com.hazelcast.nio.tcp.DefaultSocketChannelWrapper.read(DefaultSocketChannelWrapper.java:61)
at com.hazelcast.internal.networking.nonblocking.NonBlockingSocketReader.handle(NonBlockingSocketReader.java:150)
at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.handleSelectionKey(NonBlockingIOThread.java:350)
at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.handleSelectionKeys(NonBlockingIOThread.java:335)
at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.selectLoop(NonBlockingIOThread.java:253)
at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.run(NonBlockingIOThread.java:206)
2018/08/02 23:04:10 http: proxy error: net/http: request canceled
2018-08-02 23:04:59,145 ERROR [f70ee122-bc76-4377-9918-bb53838686c2:23982] [containerEvent:136357->instance:5099] [containerevent.create->(ContainerEventCreate)->instance.stop] [] [ecutorService-7] [c.p.e.p.i.DefaultProcessInstanceImpl] Unknown exception org.jooq.exception.DataAccessException: SQL [select `instance`.`id`, `instance`.`name`, `instance`.`account_id`, `instance`.`kind`, `instance`.`uuid`, `instance`.`description`, `instance`.`state`, `instance`.`created`, `instance`.`removed`, `instance`.`remove_time`, `instance`.`data`, `instance`.`allocation_state`, `instance`.`compute`, `instance`.`memory_mb`, `instance`.`image_id`, `instance`.`hostname`, `instance`.`zone_id`, `instance`.`instance_triggered_stop`, `instance`.`agent_id`, `instance`.`domain`, `instance`.`first_running`, `instance`.`token`, `instance`.`userdata`, `instance`.`registry_credential_id`, `instance`.`external_id`, `instance`.`native_container`, `instance`.`network_container_id`, `instance`.`health_state`, `instance`.`start_count`, `instance`.`create_index`, `instance`.`deployment_unit_uuid`, `instance`.`version`, `instance`.`health_updated`, `instance`.`service_index_id`, `instance`.`dns_internal`, `instance`.`dns_search_internal`, `instance`.`memory_reservation`, `instance`.`milli_cpu_reservation`, `instance`.`system`, `instance`.`service_id`, `instance`.`environment_id` from `instance` where `instance`.`id` = ? for update]; (conn:31742) Lock wait timeout exceeded; try restarting transaction
Query is: select `instance`.`id`, `instance`.`name`, `instance`.`account_id`, `instance`.`kind`, `instance`.`uuid`, `instance`.`description`, `instance`.`state`, `instance`.`created`, `instance`.`removed`, `instance`.`remove_time`, `instance`.`data`, `instance`.`allocation_state`, `instance`.`compute`, `instance`.`memory_mb`, `instance`.`image_id`, `instance`.`hostname`, `instance`.`zone_id`, `instance`.`instance_triggered_stop`, `instance`.`agent_id`, `instance`.`domain`, `instance`.`first_running`, `instance`.`token`, `instance`.`userdata`, `instance`.`registry_credential_id`, `instance`.`external_id`, `instance`.`native_container`, `instance`.`network_container_id`, `instance`.`health_state`, `instance`.`start_count`, `instance`.`create_index`, `instance`.`deployment_unit_uuid`, `instance`.`version`, `instance`.`health_updated`, `instance`.`service_index_id`, `instance`.`dns_internal`, `instance`.`dns_search_internal`, `instance`.`memory_reservation`, `instance`.`milli_cpu_reservation`, `instance`.`system`, `inst...
at org.jooq.impl.Utils.translate(Utils.java:1287)
at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:495)
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:326)
at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:351)
at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:341)
at org.jooq.impl.AbstractResultQuery.fetchOne(AbstractResultQuery.java:470)
at org.jooq.impl.UpdatableRecordImpl.checkIfChanged(UpdatableRecordImpl.java:536)
at org.jooq.impl.UpdatableRecordImpl.storeUpdate0(UpdatableRecordImpl.java:291)
at org.jooq.impl.UpdatableRecordImpl.access$200(UpdatableRecordImpl.java:90)
at org.jooq.impl.UpdatableRecordImpl$3.operate(UpdatableRecordImpl.java:260)
at org.jooq.impl.RecordDelegate.operate(RecordDelegate.java:123)
at org.jooq.impl.UpdatableRecordImpl.storeUpdate(UpdatableRecordImpl.java:255)
at org.jooq.impl.UpdatableRecordImpl.update(UpdatableRecordImpl.java:149)
at io.cattle.platform.object.impl.JooqObjectManager.persistRecord(JooqObjectManager.java:228)
at io.cattle.platform.object.impl.JooqObjectManager.setFieldsInternal(JooqObjectManager.java:133)
at io.cattle.platform.object.impl.JooqObjectManager$3.execute(JooqObjectManager.java:121)
at io.cattle.platform.engine.idempotent.Idempotent.change(Idempotent.java:88)
at io.cattle.platform.object.impl.JooqObjectManager.setFields(JooqObjectManager.java:118)
at io.cattle.platform.object.impl.JooqObjectManager.setFields(JooqObjectManager.java:113)
at io.cattle.platform.object.impl.AbstractObjectManager.setFields(AbstractObjectManager.java:139)
at io.cattle.platform.process.common.generic.GenericResourceProcessState.setState(GenericResourceProcessState.java:108)
at io.cattle.platform.engine.process.impl.AbstractStatesBasedProcessState.setTransitioning(AbstractStatesBasedProcessState.java:27)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.setTransitioning(DefaultProcessInstanceImpl.java:569)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:329)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:245)
at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7)
at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3)
at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40)
at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33)
at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13)
at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:242)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:184)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:157)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:116)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:113)
at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40)
at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33)
at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13)
at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:113)
at io.cattle.platform.engine.manager.impl.DefaultProcessManager.scheduleProcessInstance(DefaultProcessManager.java:69)
at io.cattle.platform.object.process.impl.DefaultObjectProcessManager.scheduleProcessInstance(DefaultObjectProcessManager.java:50)
at io.cattle.platform.object.process.impl.DefaultObjectProcessManager.scheduleProcessInstance(DefaultObjectProcessManager.java:55)
at io.cattle.platform.process.containerevent.ContainerEventCreate$1.doWithLock(ContainerEventCreate.java:181)
at io.cattle.platform.process.containerevent.ContainerEventCreate$1.doWithLock(ContainerEventCreate.java:144)
at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40)
at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33)
at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13)
at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37)
at io.cattle.platform.process.containerevent.ContainerEventCreate.handleInternal(ContainerEventCreate.java:144)
at io.cattle.platform.process.containerevent.ContainerEventCreate.handle(ContainerEventCreate.java:118)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:470)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:421)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:415)
at io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandlers(DefaultProcessInstanceImpl.java:415)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runLogic(DefaultProcessInstanceImpl.java:517)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:345)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:245)
at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7)
at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3)
at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40)
at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33)
at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13)
at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:242)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:184)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:157)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:116)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:113)
at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40)
at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33)
at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13)
at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37)
at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:113)
at io.cattle.platform.engine.server.impl.ProcessInstanceDispatcherImpl.processExecuteWithLock(ProcessInstanceDispatcherImpl.java:136)
at io.cattle.platform.engine.server.impl.ProcessInstanceDispatcherImpl$1.doWithLock(ProcessInstanceDispatcherImpl.java:123)
at io.cattle.platform.engine.server.impl.ProcessInstanceDispatcherImpl$1.doWithLock(ProcessInstanceDispatcherImpl.java:120)
at io.cattle.platform.lock.impl.AbstractLockManagerImpl$4.doWithLock(AbstractLockManagerImpl.java:50)
at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33)
at io.cattle.platform.lock.impl.AbstractLockManagerImpl.tryLock(AbstractLockManagerImpl.java:25)
at io.cattle.platform.lock.impl.AbstractLockManagerImpl.tryLock(AbstractLockManagerImpl.java:47)
at io.cattle.platform.engine.server.impl.ProcessInstanceDispatcherImpl.execute(ProcessInstanceDispatcherImpl.java:120)
at io.cattle.platform.engine.server.ProcessInstanceReference.doRun(ProcessInstanceReference.java:59)
at org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:15)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: (conn:31742) Lock wait timeout exceeded; try restarting transaction
Query is: select `instance`.`id`, `instance`.`name`, `instance`.`account_id`, `instance`.`kind`, `instance`.`uuid`, `instance`.`description`, `instance`.`state`, `instance`.`created`, `instance`.`removed`, `instance`.`remove_time`, `instance`.`data`, `instance`.`allocation_state`, `instance`.`compute`, `instance`.`memory_mb`, `instance`.`image_id`, `instance`.`hostname`, `instance`.`zone_id`, `instance`.`instance_triggered_stop`, `instance`.`agent_id`, `instance`.`domain`, `instance`.`first_running`, `instance`.`token`, `instance`.`userdata`, `instance`.`registry_credential_id`, `instance`.`external_id`, `instance`.`native_container`, `instance`.`network_container_id`, `instance`.`health_state`, `instance`.`start_count`, `instance`.`create_index`, `instance`.`deployment_unit_uuid`, `instance`.`version`, `instance`.`health_updated`, `instance`.`service_index_id`, `instance`.`dns_internal`, `instance`.`dns_search_internal`, `instance`.`memory_reservation`, `instance`.`milli_cpu_reservation`, `instance`.`system`, `inst...
at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:150)
at org.mariadb.jdbc.internal.util.ExceptionMapper.getException(ExceptionMapper.java:101)
at org.mariadb.jdbc.internal.util.ExceptionMapper.throwAndLogException(ExceptionMapper.java:77)
at org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:224)
at org.mariadb.jdbc.MariaDbServerPreparedStatement.executeInternal(MariaDbServerPreparedStatement.java:411)
at org.mariadb.jdbc.MariaDbServerPreparedStatement.execute(MariaDbServerPreparedStatement.java:361)
at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
at org.jooq.tools.jdbc.DefaultPreparedStatement.execute(DefaultPreparedStatement.java:194)
at org.jooq.impl.AbstractResultQuery.execute(AbstractResultQuery.java:261)
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:322)
... 91 common frames omitted
Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Lock wait timeout exceeded; try restarting transaction
Query is: select `instance`.`id`, `instance`.`name`, `instance`.`account_id`, `instance`.`kind`, `instance`.`uuid`, `instance`.`description`, `instance`.`state`, `instance`.`created`, `instance`.`removed`, `instance`.`remove_time`, `instance`.`data`, `instance`.`allocation_state`, `instance`.`compute`, `instance`.`memory_mb`, `instance`.`image_id`, `instance`.`hostname`, `instance`.`zone_id`, `instance`.`instance_triggered_stop`, `instance`.`agent_id`, `instance`.`domain`, `instance`.`first_running`, `instance`.`token`, `instance`.`userdata`, `instance`.`registry_credential_id`, `instance`.`external_id`, `instance`.`native_container`, `instance`.`network_container_id`, `instance`.`health_state`, `instance`.`start_count`, `instance`.`create_index`, `instance`.`deployment_unit_uuid`, `instance`.`version`, `instance`.`health_updated`, `instance`.`service_index_id`, `instance`.`dns_internal`, `instance`.`dns_search_internal`, `instance`.`memory_reservation`, `instance`.`milli_cpu_reservation`, `instance`.`system`, `inst...
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:1115)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1047)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1002)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executePreparedQuery(AbstractQueryProtocol.java:588)
at org.mariadb.jdbc.MariaDbServerPreparedStatement.executeInternal(MariaDbServerPreparedStatement.java:399)
... 97 common frames omitted