Rancher Server errors: Multiple dynamic schemas found of the same role and account id combination

Hi there,

I’m trying to startup a stack containing 29 services and 53 containers. When doing so, occasionally lose connection to my hosts (I have two) and the containers take an awfully long time to provision and invariably fail to do so, going into a deleted state. During the provision, I also see the following error in the Rancher Server logs very frequently:

ERROR [:] [] [] [] [tp330552435-446] [i.c.p.c.d.impl.DynamicSchemaDaoImpl ] Failed to get a schema record. org.jooq.exception.InvalidResultException: Multiple dynamic schemas found of the same role and account id combination: Name:digitaloceanConfigRole: owner AccountId: null
4/11/2016 4:37:20 PM	at io.cattle.platform.core.dao.impl.DynamicSchemaDaoImpl.pickRecordOnPriority(DynamicSchemaDaoImpl.java:154) ~[cattle-iaas-model-0.5.0-SNAPSHOT.jar:na]
4/11/2016 4:37:20 PM	at io.cattle.platform.core.dao.impl.DynamicSchemaDaoImpl.getSchemas(DynamicSchemaDaoImpl.java:55) ~[cattle-iaas-model-0.5.0-SNAPSHOT.jar:na]
4/11/2016 4:37:20 PM	at io.cattle.platform.iaas.api.auth.dynamic.DynamicSchemaFactory.listSchemas(DynamicSchemaFactory.java:47) [cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
4/11/2016 4:37:20 PM	at io.github.ibuildthecloud.gdapi.request.handler.SchemasHandler.generate(SchemasHandler.java:21) [cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
4/11/2016 4:37:20 PM	at io.github.ibuildthecloud.gdapi.request.handler.AbstractResponseGenerator.handle(AbstractResponseGenerator.java:14) [cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
4/11/2016 4:37:20 PM	at io.github.ibuildthecloud.gdapi.servlet.ApiRequestFilterDelegate.doFilter(ApiRequestFilterDelegate.java:86) [cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
4/11/2016 4:37:20 PM	at io.cattle.platform.api.servlet.ApiRequestFilter$1.runInContext(ApiRequestFilter.java:90) [cattle-framework-api-0.5.0-SNAPSHOT.jar:na]
4/11/2016 4:37:20 PM	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
4/11/2016 4:37:20 PM	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
4/11/2016 4:37:20 PM	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
4/11/2016 4:37:20 PM	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
4/11/2016 4:37:20 PM	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
4/11/2016 4:37:20 PM	at io.cattle.platform.api.servlet.ApiRequestFilter.doFilter(ApiRequestFilter.java:83) [cattle-framework-api-0.5.0-SNAPSHOT.jar:na]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364) [jetty-servlets-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) [jetty-security-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [jetty-io-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
4/11/2016 4:37:20 PM	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_95]

I am running two Ubuntu 14.04 Azure D2v2 VMs for the hosts if that helps.

I assume it’s likely to be a localized issue, most probably resource starvation or something but I’m unsure how to diagnose it. Any pointers would be very helpful indeed, especially with regards to the error.

Many thanks in advance.

Sam

Should be fixed in next release. See https://github.com/rancher/rancher/issues/4190