Rancher Server Failing to Start

Please help, using rancher in Production v0.46.0 and I tried adding a new environment and the server just crashed. Tried restarting the container but keep getting redirected to /fail.

The logs from the container show this below

         2015-12-17 02:15:24,779 ERROR [:] [] [] [] [235763758-24074] [i.g.i.g.r.handler.ExceptionHandler  ] Exception in API for request [io.github.ibuildthecloud.gdapi.request.ApiRequest@511c4a9f] org.jooq.exception.DataAccessException: SQL [select distinct `account`.`id`, `account`.`name`, `account`.`kind`, `account`.`uuid`, `account`.`description`, `account`.`state`, `account`.`created`, `account`.`removed`, `account`.`remove_time`, `account`.`data`, `account`.`external_id`, `account`.`external_id_type`, `project_member`.`id`, `project_member`.`name`, `project_member`.`account_id`, `project_member`.`kind`, `project_member`.`uuid`, `project_member`.`description`, `project_member`.`state`, `project_member`.`created`, `project_member`.`removed`, `project_member`.`remove_time`, `project_member`.`data`, `project_member`.`external_id`, `project_member`.`project_id`, `project_member`.`external_id_type`, `project_member`.`role` from `account` join `project_member` on `project_member`.`project_id` = `account`.`id` where (1 = 0 or (`project_member`.`external_id` = ? and `project_member`.`external_id_type` = ? and `project_member`.`removed` is null and `project_member`.`state` = ?) or (`project_member`.`external_id` = ? and `project_member`.`external_id_type` = ? and `project_member`.`removed` is null and `project_member`.`state` = ?) or (`project_member`.`external_id` = ? and `project_member`.`external_id_type` = ? and `project_member`.`removed` is null and `project_member`.`state` = ?) or (`project_member`.`external_id` = ? and `project_member`.`external_id_type` = ? and `project_member`.`removed` is null and `project_member`.`state` = ?) or (`project_member`.`external_id` = ? and `project_member`.`external_id_type` = ? and `project_member`.`removed` is null and `project_member`.`state` = ?))]; Got error 28 from storage engine
at org.jooq.impl.Utils.translate(Utils.java:1287) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:495) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:326) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.AbstractResultQuery.fetch(AbstractResultQuery.java:330) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.AbstractResultQuery.fetchInto(AbstractResultQuery.java:645) ~[jooq-3.3.0.jar:na]
at io.cattle.platform.iaas.api.auth.dao.impl.AuthDaoImpl.getAccessibleProjects(AuthDaoImpl.java:306) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.dao.impl.AuthDaoImpl.hasAccessToAnyProject(AuthDaoImpl.java:490) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.TokenUtils.getOrCreateAccount(TokenUtils.java:258) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.integration.github.GithubTokenCreator.getGithubToken(GithubTokenCreator.java:72) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.integration.github.GithubTokenCreator.getToken(GithubTokenCreator.java:102) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.identity.TokenResourceManager.createToken(TokenResourceManager.java:52) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.identity.TokenResourceManager.createInternal(TokenResourceManager.java:41) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.resource.impl.AbstractBaseResourceManager.create(AbstractBaseResourceManager.java:115) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.ResourceManagerRequestHandler.generate(ResourceManagerRequestHandler.java:39) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.AbstractResponseGenerator.handle(AbstractResponseGenerator.java:14) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.write.DefaultReadWriteApiDelegate.handle(DefaultReadWriteApiDelegate.java:27) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.write.DefaultReadWriteApiDelegate.write(DefaultReadWriteApiDelegate.java:22) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at sun.reflect.GeneratedMethodAccessor652.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_79]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_79]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) ~[    spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
              at    org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96) ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260) ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94) ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at com.sun.proxy.$Proxy43.write(Unknown Source) ~[na:na]
at io.github.ibuildthecloud.gdapi.request.handler.write.ReadWriteApiHandler.handle(ReadWriteApiHandler.java:19) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.servlet.ApiRequestFilterDelegate.doFilter(ApiRequestFilterDelegate.java:86) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.api.servlet.ApiRequestFilter$1.runInContext(ApiRequestFilter.java:79) [cattle-framework-api-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.api.servlet.ApiRequestFilter.doFilter(ApiRequestFilter.java:72) [cattle-framework-api-0.5.0-SNAPSHOT.jar:na]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300) [jetty-servlets-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) [jetty-security-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [jetty-io-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]

Caused by: java.sql.SQLException: Got error 28 from storage engine
at org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:149) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.MySQLStatement.executeQueryEpilog(MySQLStatement.java:264) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:288) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.MySQLPreparedStatement.execute(MySQLPreparedStatement.java:141) ~[mariadb-java-client-1.1.7.jar:na]
at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172) ~[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172) ~[commons-dbcp-1.4.jar:1.4]
at org.jooq.tools.jdbc.DefaultPreparedStatement.execute(DefaultPreparedStatement.java:194) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.AbstractResultQuery.execute(AbstractResultQuery.java:261) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:322) ~[jooq-3.3.0.jar:na]
… 58 common frames omitted
Caused by: org.mariadb.jdbc.internal.common.QueryException: Got error 28 from storage engine
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.getResult(MySQLProtocol.java:942) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:991) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:281) ~[mariadb-java-client-1.1.7.jar:na]
… 64 common frames omitted

time=“2015-12-17T02:15:38Z” level=error msg=“Unable to start EventRouter” Err="Post
time=“2015-12-17T02:15:38Z” level=fatal msg=“Unable to start event router” error="Post "
time=“2015-12-17T02:15:38Z” level=info msg="Exiting go-machine-service…"
151217 2:15:40 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
151217 2:15:42 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
151217 2:15:45 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
s6-svscanctl: fatal: unable to control /service: supervisor not listening
151217 2:15:48 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
151217 2:15:52 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
151217 2:15:57 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
151217 2:16:06 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
151217 2:16:21 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 22 …
151217 2:16:49 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
s6-svscanctl: fatal: unable to control /service: supervisor not listening
151217 2:17:42 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
151217 2:19:27 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 24 …
151217 2:22:54 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
s6-svscanctl: fatal: unable to control /service: supervisor not listening
^[[A^Cadmin@ip-172-31-12-117:~$ sudo docker logs -f --tail=200 3f6871c22982
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) ~[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) ~[commons-dbcp-1.4.jar:1.4]
at org.jooq.tools.jdbc.DefaultPreparedStatement.executeUpdate(DefaultPreparedStatement.java:88) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.AbstractStoreQuery.execute(AbstractStoreQuery.java:346) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:322) ~[jooq-3.3.0.jar:na]
… 67 common frames omitted
Caused by: org.mariadb.jdbc.internal.common.QueryException: Lock wait timeout exceeded; try restarting transaction
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.getResult(MySQLProtocol.java:942) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:991) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:281) ~[mariadb-java-client-1.1.7.jar:na]
… 74 common frames omitted

2015-12-17 02:15:24,034 ERROR [:] [] [] [] [235763758-24065] [i.g.i.g.r.handler.ExceptionHandler ] Exception in API for request [io.github.ibuildthecloud.gdapi.request.ApiRequest@435f4c12] org.jooq.exception.DataAccessException: SQL [insert into service_event (account_id, kind, uuid, state, created, data, host_id, healthcheck_uuid, instance_id, healthcheck_instance_id, reported_health, external_timestamp) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; Lock wait timeout exceeded; try restarting transaction
at org.jooq.impl.Utils.translate(Utils.java:1287) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:495) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:326) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.UpdatableRecordImpl.storeInsert0(UpdatableRecordImpl.java:230) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.UpdatableRecordImpl.access$100(UpdatableRecordImpl.java:90) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.UpdatableRecordImpl$2.operate(UpdatableRecordImpl.java:201) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.RecordDelegate.operate(RecordDelegate.java:123) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.UpdatableRecordImpl.storeInsert(UpdatableRecordImpl.java:196) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.UpdatableRecordImpl.insert(UpdatableRecordImpl.java:144) ~[jooq-3.3.0.jar:na]
at io.cattle.platform.object.impl.JooqObjectManager$1.executeNoResult(JooqObjectManager.java:80) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.engine.idempotent.IdempotentExecutionNoReturn.execute(IdempotentExecutionNoReturn.java:7) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.engine.idempotent.IdempotentExecutionNoReturn.execute(IdempotentExecutionNoReturn.java:3) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.engine.idempotent.Idempotent.change(Idempotent.java:88) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.object.impl.JooqObjectManager.insert(JooqObjectManager.java:77) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.object.impl.AbstractObjectManager.create(AbstractObjectManager.java:62) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.api.resource.AbstractObjectResourceManager.createAndScheduleObject(AbstractObjectResourceManager.java:103) ~[cattle-framework-api-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.api.resource.AbstractObjectResourceManager.doCreate(AbstractObjectResourceManager.java:98) ~[cattle-framework-api-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.api.resource.AbstractObjectResourceManager.createInternal(AbstractObjectResourceManager.java:89) ~[cattle-framework-api-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.resource.impl.AbstractBaseResourceManager.create(AbstractBaseResourceManager.java:115) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.resource.AbstractResourceManagerFilter.create(AbstractResourceManagerFilter.java:43) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.filter.serviceevent.ServiceEventFilter.create(ServiceEventFilter.java:81) ~[cattle-iaas-api-logic-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.resource.impl.FilteredResourceManager.create(FilteredResourceManager.java:56) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.ResourceManagerRequestHandler.generate(ResourceManagerRequestHandler.java:39) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.AbstractResponseGenerator.handle(AbstractResponseGenerator.java:14) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.write.DefaultReadWriteApiDelegate.handle(DefaultReadWriteApiDelegate.java:27) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.write.DefaultReadWriteApiDelegate.write(DefaultReadWriteApiDelegate.java:22) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at sun.reflect.GeneratedMethodAccessor652.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_79]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_79]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96) ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260) ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94) ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at com.sun.proxy.$Proxy43.write(Unknown Source) ~[na:na]
at io.github.ibuildthecloud.gdapi.request.handler.write.ReadWriteApiHandler.handle(ReadWriteApiHandler.java:19) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.servlet.ApiRequestFilterDelegate.doFilter(ApiRequestFilterDelegate.java:86) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.api.servlet.ApiRequestFilter$1.runInContext(ApiRequestFilter.java:79) [cattle-framework-api-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.api.servlet.ApiRequestFilter.doFilter(ApiRequestFilter.java:72) [cattle-framework-api-0.5.0-SNAPSHOT.jar:na]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300) [jetty-servlets-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) [jetty-security-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [jetty-io-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
at org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:149) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.MySQLStatement.executeQueryEpilog(MySQLStatement.java:264) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:288) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.MySQLStatement.executeUpdate(MySQLStatement.java:317) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.MySQLPreparedStatement.executeUpdate(MySQLPreparedStatement.java:156) ~[mariadb-java-client-1.1.7.jar:na]
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) ~[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) ~[commons-dbcp-1.4.jar:1.4]
at org.jooq.tools.jdbc.DefaultPreparedStatement.executeUpdate(DefaultPreparedStatement.java:88) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.AbstractStoreQuery.execute(AbstractStoreQuery.java:346) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:322) ~[jooq-3.3.0.jar:na]
… 67 common frames omitted
Caused by: org.mariadb.jdbc.internal.common.QueryException: Lock wait timeout exceeded; try restarting transaction
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.getResult(MySQLProtocol.java:942) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:991) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:281) ~[mariadb-java-client-1.1.7.jar:na]
… 74 common frames omitted

2015-12-17 02:15:24,779 ERROR [:] [] [] [] [235763758-24074] [i.g.i.g.r.handler.ExceptionHandler ] Exception in API for request [io.github.ibuildthecloud.gdapi.request.ApiRequest@511c4a9f] org.jooq.exception.DataAccessException: SQL [select distinct account.id, account.name, account.kind, account.uuid, account.description, account.state, account.created, account.removed, account.remove_time, account.data, account.external_id, account.external_id_type, project_member.id, project_member.name, project_member.account_id, project_member.kind, project_member.uuid, project_member.description, project_member.state, project_member.created, project_member.removed, project_member.remove_time, project_member.data, project_member.external_id, project_member.project_id, project_member.external_id_type, project_member.role from account join project_member on project_member.project_id = account.id where (1 = 0 or (project_member.external_id = ? and project_member.external_id_type = ? and project_member.removed is null and project_member.state = ?) or (project_member.external_id = ? and project_member.external_id_type = ? and project_member.removed is null and project_member.state = ?) or (project_member.external_id = ? and project_member.external_id_type = ? and project_member.removed is null and project_member.state = ?) or (project_member.external_id = ? and project_member.external_id_type = ? and project_member.removed is null and project_member.state = ?) or (project_member.external_id = ? and project_member.external_id_type = ? and project_member.removed is null and project_member.state = ?))]; Got error 28 from storage engine
at org.jooq.impl.Utils.translate(Utils.java:1287) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:495) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:326) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.AbstractResultQuery.fetch(AbstractResultQuery.java:330) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.AbstractResultQuery.fetchInto(AbstractResultQuery.java:645) ~[jooq-3.3.0.jar:na]
at io.cattle.platform.iaas.api.auth.dao.impl.AuthDaoImpl.getAccessibleProjects(AuthDaoImpl.java:306) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.dao.impl.AuthDaoImpl.hasAccessToAnyProject(AuthDaoImpl.java:490) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.TokenUtils.getOrCreateAccount(TokenUtils.java:258) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.integration.github.GithubTokenCreator.getGithubToken(GithubTokenCreator.java:72) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.integration.github.GithubTokenCreator.getToken(GithubTokenCreator.java:102) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.identity.TokenResourceManager.createToken(TokenResourceManager.java:52) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.identity.TokenResourceManager.createInternal(TokenResourceManager.java:41) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.resource.impl.AbstractBaseResourceManager.create(AbstractBaseResourceManager.java:115) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.ResourceManagerRequestHandler.generate(ResourceManagerRequestHandler.java:39) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.AbstractResponseGenerator.handle(AbstractResponseGenerator.java:14) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.write.DefaultReadWriteApiDelegate.handle(DefaultReadWriteApiDelegate.java:27) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.write.DefaultReadWriteApiDelegate.write(DefaultReadWriteApiDelegate.java:22) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at sun.reflect.GeneratedMethodAccessor652.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_79]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_79]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96) ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260) ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94) ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at com.sun.proxy.$Proxy43.write(Unknown Source) ~[na:na]
at io.github.ibuildthecloud.gdapi.request.handler.write.ReadWriteApiHandler.handle(ReadWriteApiHandler.java:19) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.servlet.ApiRequestFilterDelegate.doFilter(ApiRequestFilterDelegate.java:86) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.api.servlet.ApiRequestFilter$1.runInContext(ApiRequestFilter.java:79) [cattle-framework-api-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.api.servlet.ApiRequestFilter.doFilter(ApiRequestFilter.java:72) [cattle-framework-api-0.5.0-SNAPSHOT.jar:na]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300) [jetty-servlets-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) [jetty-security-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [jetty-io-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
Caused by: java.sql.SQLException: Got error 28 from storage engine
at org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:149) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.MySQLStatement.executeQueryEpilog(MySQLStatement.java:264) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:288) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.MySQLPreparedStatement.execute(MySQLPreparedStatement.java:141) ~[mariadb-java-client-1.1.7.jar:na]
at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172) ~[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172) ~[commons-dbcp-1.4.jar:1.4]
at org.jooq.tools.jdbc.DefaultPreparedStatement.execute(DefaultPreparedStatement.java:194) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.AbstractResultQuery.execute(AbstractResultQuery.java:261) ~[jooq-3.3.0.jar:na]
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:322) ~[jooq-3.3.0.jar:na]
… 58 common frames omitted
Caused by: org.mariadb.jdbc.internal.common.QueryException: Got error 28 from storage engine
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.getResult(MySQLProtocol.java:942) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:991) ~[mariadb-java-client-1.1.7.jar:na]
at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:281) ~[mariadb-java-client-1.1.7.jar:na]
… 64 common frames omitted

time=“2015-12-17T02:15:38Z” level=error msg=“Unable to start EventRouter” Err="Post
time=“2015-12-17T02:15:38Z” level=fatal msg=“Unable to start event router” error="Post
time=“2015-12-17T02:15:38Z” level=info msg="Exiting go-machine-service…"
151217 2:15:40 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
151217 2:15:42 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
151217 2:15:45 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
s6-svscanctl: fatal: unable to control /service: supervisor not listening
151217 2:15:48 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
151217 2:15:52 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
151217 2:15:57 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
151217 2:16:06 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
151217 2:16:21 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 22 …
151217 2:16:49 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
s6-svscanctl: fatal: unable to control /service: supervisor not listening
151217 2:17:42 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
151217 2:19:27 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 24 …
151217 2:22:54 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1-log) starting as process 23 …
s6-svscanctl: fatal: unable to control /service: supervisor not listening

@oyemike This looks like an underlying problem with mysql. After a little googling, this message Caused by: java.sql.SQLException: Got error 28 from storage engine leads me to believe that you are out of disk space. Can you check that?

@cjellick thank for replying, I figured that was the issue and had to move the sql database. Also found https://github.com/rancher/rancher/issues/2136 , Thanks for your help. When I moved the db I also updated rancher does that mean mysql bin logs will be purged now as well?