Problem Bringing up Rancher Server with MySQL

Archive Rancher 1.x
1

Running as root: (endpoints and passwords redacted)
Running: v1.5.2 rancher/server:latest

 docker run -d --restart=unless-stopped -p 8080:8080 rancher/server \
--db-host mysql.cmqqpolfjue5.us-east-1.rds.amazonaws.com --db-port 3306 --db-user cattle --db-pass <redacted> --db-name cattle
  • MySQL db is accessible from 0.0.0.0/0
  • I can successfully connect from a mysql client
  • Rancher did create the tables successfully
  • Web service is never coming up on 8080
  • Works fine without the mysql arguments

Logs

root@onprem:/home/rancher# docker logs 26
CATTLE_AGENT_PACKAGE_HOST_API_URL=/usr/share/cattle/artifacts/host-api.tar.gz
CATTLE_AGENT_PACKAGE_PYTHON_AGENT_URL=/usr/share/cattle/artifacts/go-agent.tar.gz
CATTLE_API_UI_URL=//releases.rancher.com/api-ui/1.0.8
CATTLE_CATTLE_VERSION=v0.177.9
CATTLE_DB_CATTLE_DATABASE=mysql
CATTLE_DB_CATTLE_MYSQL_HOST=mysql.cmqqpolfjue5.us-east-1.rds.amazonaws.com
CATTLE_DB_CATTLE_MYSQL_NAME=cattle
CATTLE_DB_CATTLE_MYSQL_PORT=3306
CATTLE_DB_CATTLE_USERNAME=cattle
CATTLE_GRAPHITE_HOST=
CATTLE_GRAPHITE_PORT=
CATTLE_HOME=/var/lib/cattle
CATTLE_HOST_API_PROXY_MODE=embedded
CATTLE_LOGBACK_OUTPUT_GELF_HOST=
CATTLE_LOGBACK_OUTPUT_GELF_PORT=
CATTLE_RANCHER_CLI_VERSION=v0.5.1
CATTLE_RANCHER_COMPOSE_VERSION=v0.12.4
CATTLE_RANCHER_SERVER_IMAGE=rancher/server
CATTLE_RANCHER_SERVER_VERSION=v1.5.2
CATTLE_USE_LOCAL_ARTIFACTS=true
DEFAULT_CATTLE_API_UI_CSS_URL=/api-ui/ui.min.css
DEFAULT_CATTLE_API_UI_INDEX=//releases.rancher.com/ui/1.5.8
DEFAULT_CATTLE_API_UI_JS_URL=/api-ui/ui.min.js
DEFAULT_CATTLE_AUTH_SERVICE_EXECUTE=true
DEFAULT_CATTLE_CATALOG_EXECUTE=true
DEFAULT_CATTLE_CATALOG_URL={"catalogs":{"community":{"url":"https://git.rancher.io/community-catalog.git","branch":"master"},"library":{"url":"https://git.rancher.io/rancher-catalog.git","branch":"master"}}}
DEFAULT_CATTLE_COMPOSE_EXECUTOR_EXECUTE=true
DEFAULT_CATTLE_MACHINE_EXECUTE=true
DEFAULT_CATTLE_RANCHER_CLI_DARWIN_URL=https://releases.rancher.com/cli/v0.5.1/rancher-darwin-amd64-v0.5.1.tar.gz
DEFAULT_CATTLE_RANCHER_CLI_LINUX_URL=https://releases.rancher.com/cli/v0.5.1/rancher-linux-amd64-v0.5.1.tar.gz
DEFAULT_CATTLE_RANCHER_CLI_WINDOWS_URL=https://releases.rancher.com/cli/v0.5.1/rancher-windows-386-v0.5.1.zip
DEFAULT_CATTLE_RANCHER_COMPOSE_DARWIN_URL=https://releases.rancher.com/compose/v0.12.4/rancher-compose-darwin-amd64-v0.12.4.tar.gz
DEFAULT_CATTLE_RANCHER_COMPOSE_LINUX_URL=https://releases.rancher.com/compose/v0.12.4/rancher-compose-linux-amd64-v0.12.4.tar.gz
DEFAULT_CATTLE_RANCHER_COMPOSE_WINDOWS_URL=https://releases.rancher.com/compose/v0.12.4/rancher-compose-windows-386-v0.12.4.zip
DEFAULT_CATTLE_SECRETS_API_EXECUTE=true
DEFAULT_CATTLE_WEBHOOK_SERVICE_EXECUTE=true
05:36:53.488 [main] INFO  ConsoleStatus - Loading configuration
2017-03-25 05:36:59,009 INFO    [main] [ConsoleStatus] Starting DB migration

Some more logs:

2017-03-25 05:52:43,762 WARN  [:] [] [] [] [main           ] [nnotationConfigWebApplicationContext] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibaseCore' defined in io.cattle.platform.app.SystemConfig: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by 7a16604877bb (fe80:0:0:0:42:acff:fe11:1%eth0) since 3/25/17 4:28 AM 
2017-03-25 05:52:43,845 ERROR [:] [] [] [] [main           ] [o.s.web.context.ContextLoader       ] Context initialization failed org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibaseCore' defined in io.cattle.platform.app.SystemConfig: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by 7a16604877bb (fe80:0:0:0:42:acff:fe11:1%eth0) since 3/25/17 4:28 AM
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1578) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:776) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:861) ~[spring-context-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:541) ~[spring-context-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:444) ~[spring-web-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:326) ~[spring-web-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107) [spring-web-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349) [jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342) [jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505) [jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.Server.start(Server.java:387) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.Server.doStart(Server.java:354) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at io.cattle.platform.launcher.jetty.Main.main(Main.java:146) [9949e600a7bcdd427ae7bc9d33bf477c/:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_72]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_72]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_72]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_72]
	at io.cattle.platform.launcher.Main.run(Main.java:188) [9949e600a7bcdd427ae7bc9d33bf477c/:na]
	at io.cattle.platform.launcher.Main.main(Main.java:251) [9949e600a7bcdd427ae7bc9d33bf477c/:na]
Caused by: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by 7a16604877bb (fe80:0:0:0:42:acff:fe11:1%eth0) since 3/25/17 4:28 AM
	at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:168) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.Liquibase.update(Liquibase.java:189) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.Liquibase.update(Liquibase.java:181) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:342) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:299) ~[liquibase-core-3.1.0.jar:na]
	at io.cattle.platform.liquibase.Loader.afterPropertiesSet(Loader.java:66) ~[cattle-framework-db-loader-0.5.0-SNAPSHOT.jar:na]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1637) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	... 34 common frames omitted

2017-03-25 05:52:43,846 WARN  [:] [] [] [] [main           ] [o.eclipse.jetty.webapp.WebAppContext] Failed startup of context o.e.j.w.WebAppContext@91161c7{/,file:/usr/share/cattle/9949e600a7bcdd427ae7bc9d33bf477c/,STARTING}{file:/usr/share/cattle/9949e600a7bcdd427ae7bc9d33bf477c/} org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibaseCore' defined in io.cattle.platform.app.SystemConfig: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by 7a16604877bb (fe80:0:0:0:42:acff:fe11:1%eth0) since 3/25/17 4:28 AM
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1578) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:776) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:861) ~[spring-context-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:541) ~[spring-context-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:444) ~[spring-web-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:326) ~[spring-web-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107) ~[spring-web-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444) ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294) ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.Server.start(Server.java:387) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.Server.doStart(Server.java:354) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at io.cattle.platform.launcher.jetty.Main.main(Main.java:146) [9949e600a7bcdd427ae7bc9d33bf477c/:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_72]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_72]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_72]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_72]
	at io.cattle.platform.launcher.Main.run(Main.java:188) [9949e600a7bcdd427ae7bc9d33bf477c/:na]
	at io.cattle.platform.launcher.Main.main(Main.java:251) [9949e600a7bcdd427ae7bc9d33bf477c/:na]
Caused by: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by 7a16604877bb (fe80:0:0:0:42:acff:fe11:1%eth0) since 3/25/17 4:28 AM
	at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:168) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.Liquibase.update(Liquibase.java:189) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.Liquibase.update(Liquibase.java:181) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:342) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:299) ~[liquibase-core-3.1.0.jar:na]
	at io.cattle.platform.liquibase.Loader.afterPropertiesSet(Loader.java:66) ~[cattle-framework-db-loader-0.5.0-SNAPSHOT.jar:na]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1637) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	... 34 common frames omitted

2017-03-25 05:52:43,847 WARN  [:] [] [] [] [main           ] [o.e.j.u.component.AbstractLifeCycle ] FAILED o.e.j.w.WebAppContext@91161c7{/,file:/usr/share/cattle/9949e600a7bcdd427ae7bc9d33bf477c/,UNAVAILABLE}{file:/usr/share/cattle/9949e600a7bcdd427ae7bc9d33bf477c/}: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibaseCore' defined in io.cattle.platform.app.SystemConfig: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by 7a16604877bb (fe80:0:0:0:42:acff:fe11:1%eth0) since 3/25/17 4:28 AM org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibaseCore' defined in io.cattle.platform.app.SystemConfig: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by 7a16604877bb (fe80:0:0:0:42:acff:fe11:1%eth0) since 3/25/17 4:28 AM
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1578) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:776) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:861) ~[spring-context-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:541) ~[spring-context-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:444) ~[spring-web-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:326) ~[spring-web-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107) ~[spring-web-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444) ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294) ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.Server.start(Server.java:387) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.Server.doStart(Server.java:354) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at io.cattle.platform.launcher.jetty.Main.main(Main.java:146) [9949e600a7bcdd427ae7bc9d33bf477c/:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_72]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_72]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_72]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_72]
	at io.cattle.platform.launcher.Main.run(Main.java:188) [9949e600a7bcdd427ae7bc9d33bf477c/:na]
	at io.cattle.platform.launcher.Main.main(Main.java:251) [9949e600a7bcdd427ae7bc9d33bf477c/:na]
Caused by: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by 7a16604877bb (fe80:0:0:0:42:acff:fe11:1%eth0) since 3/25/17 4:28 AM
	at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:168) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.Liquibase.update(Liquibase.java:189) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.Liquibase.update(Liquibase.java:181) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:342) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:299) ~[liquibase-core-3.1.0.jar:na]
	at io.cattle.platform.liquibase.Loader.afterPropertiesSet(Loader.java:66) ~[cattle-framework-db-loader-0.5.0-SNAPSHOT.jar:na]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1637) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	... 34 common frames omitted

2017-03-25 05:52:43,867 WARN  [:] [] [] [] [main           ] [o.e.j.u.component.AbstractLifeCycle ] FAILED org.eclipse.jetty.server.Server@3cbbc1e0: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibaseCore' defined in io.cattle.platform.app.SystemConfig: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by 7a16604877bb (fe80:0:0:0:42:acff:fe11:1%eth0) since 3/25/17 4:28 AM org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibaseCore' defined in io.cattle.platform.app.SystemConfig: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by 7a16604877bb (fe80:0:0:0:42:acff:fe11:1%eth0) since 3/25/17 4:28 AM
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1578) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:776) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:861) ~[spring-context-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:541) ~[spring-context-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:444) ~[spring-web-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:326) ~[spring-web-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107) ~[spring-web-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444) ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294) ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.Server.start(Server.java:387) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibaseCore' defined in io.cattle.platform.app.SystemConfig: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by 7a16604877bb (fe80:0:0:0:42:acff:fe11:1%eth0) since 3/25/17 4:28 AM
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1578)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.Server.doStart(Server.java:354) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at io.cattle.platform.launcher.jetty.Main.main(Main.java:146) [9949e600a7bcdd427ae7bc9d33bf477c/:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_72]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_72]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_72]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_72]
	at io.cattle.platform.launcher.Main.run(Main.java:188) [9949e600a7bcdd427ae7bc9d33bf477c/:na]
	at io.cattle.platform.launcher.Main.main(Main.java:251) [9949e600a7bcdd427ae7bc9d33bf477c/:na]
Caused by: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by 7a16604877bb (fe80:0:0:0:42:acff:fe11:1%eth0) since 3/25/17 4:28 AM
	at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:168) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.Liquibase.update(Liquibase.java:189) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.Liquibase.update(Liquibase.java:181) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:342) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:299) ~[liquibase-core-3.1.0.jar:na]
	at io.cattle.platform.liquibase.Loader.afterPropertiesSet(Loader.java:66) ~[cattle-framework-db-loader-0.5.0-SNAPSHOT.jar:na]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1637) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	... 34 common frames omitted

	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:776)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:861)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:541)
	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:444)
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:326)
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800)
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444)
	at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.server.Server.start(Server.java:387)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.Server.doStart(Server.java:354)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at io.cattle.platform.launcher.jetty.Main.main(Main.java:146)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at io.cattle.platform.launcher.Main.run(Main.java:188)
	at io.cattle.platform.launcher.Main.main(Main.java:251)
Caused by: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by 7a16604877bb (fe80:0:0:0:42:acff:fe11:1%eth0) since 3/25/17 4:28 AM
	at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:168)
	at liquibase.Liquibase.update(Liquibase.java:189)
	at liquibase.Liquibase.update(Liquibase.java:181)
	at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:342)
	at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:299)
	at io.cattle.platform.liquibase.Loader.afterPropertiesSet(Loader.java:66)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1637)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574)
	... 34 more
STARTUP FAILED [316586] ms
2017-03-25 05:52:43,874 ERROR   [main] [ConsoleStatus] Startup Failed [316582ms] org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibaseCore' defined in io.cattle.platform.app.SystemConfig: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by 7a16604877bb (fe80:0:0:0:42:acff:fe11:1%eth0) since 3/25/17 4:28 AM
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1578) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:776) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:861) ~[spring-context-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:541) ~[spring-context-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:444) ~[spring-web-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:326) ~[spring-web-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107) ~[spring-web-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444) ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294) ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.Server.start(Server.java:387) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.server.Server.doStart(Server.java:354) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
	at io.cattle.platform.launcher.jetty.Main.main(Main.java:146) ~[9949e600a7bcdd427ae7bc9d33bf477c/:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_72]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_72]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_72]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_72]
	at io.cattle.platform.launcher.Main.run(Main.java:188) [9949e600a7bcdd427ae7bc9d33bf477c/:na]
	at io.cattle.platform.launcher.Main.main(Main.java:251) [9949e600a7bcdd427ae7bc9d33bf477c/:na]
Caused by: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by 7a16604877bb (fe80:0:0:0:42:acff:fe11:1%eth0) since 3/25/17 4:28 AM
	at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:168) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.Liquibase.update(Liquibase.java:189) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.Liquibase.update(Liquibase.java:181) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:342) ~[liquibase-core-3.1.0.jar:na]
	at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:299) ~[liquibase-core-3.1.0.jar:na]
	at io.cattle.platform.liquibase.Loader.afterPropertiesSet(Loader.java:66) ~[cattle-framework-db-loader-0.5.0-SNAPSHOT.jar:na]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1637) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
	... 34 common frames omitted
2

http://docs.rancher.com/rancher/v1.5/en/faqs/server/#why-is-rancher-server-frozen-or-why-could-my-upgrade-have-failed

3

Initially, I had a row lock on DATABASECHANGELOGLOCK. Ultimately, I resolved this by:

  • Deleting the cattle database on mysql
  • Allowing rancher to recreate the database from scratch
  • Letting it sit overnight

I woke up to a working installation. I will say that an Amazon t2.micro instance is not sufficient for this task. I’ll report back as I get to a reasonably performant connection. I did choose the MultiAZ deployment to ensure that it would have redundancy.

1 Like
4

Yes T2’s and particularly the small ones are really a poor choice for a lot of workloads. There’s a very fine line between working fine and cascading death spiral, I call it T2-itis.

If you try to use any significant sustained CPU you eventually run out of credits and get zero run time for seconds at a time. If you’re lucky it’s while you’re still looking and not after having walked away since it was fine for a while. In our case this just causes more work because the agent connections drop and have to be reestablished, which leads to more CPU burn, etc.

5

Based upon your experience, what is a reasonable entry point for:

  • 10 node cluster
  • 30 node cluster
6

your answer helped me figure out what was wrong. I had wrong DB collation so mysql initialization wasn’t successfull. I changed the collation and end up on this forum because of the same error. Turned out rancher locked one of the tables/rows so I just had to delete those. thanks