I am having issues making the rancher server work with remote mysql. I have verified that all db parameters are set correctly, getting following exception while creating/starting the rancher server(1.6.5). Anyways, it doesn’t affect the working of the rancher server as all services appears to be running fine inside the rancher. Even deployment of custom catalog is fine. When I tried using multi container catalog with higher number of containers , its failing with same proxy error. Everything working fine with Rancher Packaged image with internal mysql server.
Rancher Server Version: 1.6.5
Mysql Version:- 5.7
2017-12-27 10:07:38,724 INFO [main] [ConsoleStatus] Starting [89/90]: WebhookServiceLauncher
2017-12-27 10:07:38,724 INFO [main] [ConsoleStatus] Starting [90/90]: project.template.reload
2017-12-27 10:07:38,793 INFO [main] [ConsoleStatus] [DONE ] [138400ms] Startup Succeeded, Listening on port 8081
2017-12-27 10:07:48,112 ERROR [:] [] [] [] [ecutorService-2] [o.a.c.m.context.NoExceptionRunnable ] Uncaught exception io.cattle.platform.lock.ex
ception.FailedToAcquireLockException: Failed to acquire lock [SERVICE.ACCOUNT.CREATE.machineServiceAccount]
at io.cattle.platform.lock.provider.impl.StandardLock.lock(StandardLock.java:56) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.lock.impl.AbstractLockManagerImpl$1.withLock(AbstractLockManagerImpl.java:17) ~[cattle-framework-lock-0.5.0-SNAPSHO
T.jar:na]
at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:30) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:
na]
at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:
na]
at io.cattle.platform.service.launcher.GenericServiceLauncher.getCredential(GenericServiceLauncher.java:116) ~[cattle-iaas-logic-common-0
.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.docker.machine.launch.ComposeExecutorLauncher.setEnvironment(ComposeExecutorLauncher.java:40) ~[cattle-docker-machi
ne-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.service.launcher.GenericServiceLauncher.doRun(GenericServiceLauncher.java:200) ~[cattle-iaas-logic-common-0.5.0-SNA
PSHOT.jar:na]
at org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:15) ~[cattle-framework-managed-context
-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0
.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-conte
xt-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-man
aged-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-manag
ed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5
.0-SNAPSHOT.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_72]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
time=“2017-12-27T10:08:13Z” level=fatal msg="Failed to configure cattle client: Get http:// localhost:8080/v2-beta: net/http: request canceled (Cl
**ient.Timeout exceeded while awaiting headers)" **
**2017/12/27 10:08:13 http: proxy error: net/http: request canceled **
**time=“2017-12-27T10:08:14Z” level=fatal msg=“Exiting go-machine-service: Timed out waiting for transtion.” service=gms **
**2017/12/27 10:08:14 http: proxy error: net/http: request canceled **
time=“2017-12-27T10:08:15Z” level=info msg=“Starting go-machine-service…” gitcommit=v0.39.0 service=gms
time=“2017-12-27T10:08:15Z” level=info msg=“Waiting for handler registration (1/2)” service=gms
time=“2017-12-27T10:08:15Z” level=info msg="Starting Rancher Auth service"
time=“2017-12-27T10:08:15Z” level=info msg="Listening on :8090"
time=“2017-12-27T10:08:15Z” level=info msg=“Initializing event router” workerCount=5
time=“2017-12-27T10:08:16Z” level=info msg="Connection established"
time=“2017-12-27T10:08:16Z” level=info msg=“Waiting for handler registration (2/2)” service=gms
time=“2017-12-27T10:08:16Z” level=info msg="Starting websocket pings"
time=“2017-12-27T10:08:16Z” level=info msg=“Initializing event router” workerCount=250
time=“2017-12-27T10:08:16Z” level=info msg="Connection established"
time=“2017-12-27T10:08:16Z” level=info msg="Starting websocket pings"
time=“2017-12-27T10:08:17Z” level=info msg=“Installing builtin drivers” service=gms
time=“2017-12-27T10:08:17Z” level=info msg=“Downloading all drivers” service=gms
time=“2017-12-27T10:08:17Z” level=info msg=“Done downloading all drivers” service=gms
time=“2017-12-27T10:08:24Z” level=info msg=“Starting Catalog Service (port 8088, refresh interval 300 seconds)”
Sometimes I am getting following error too, in addition to the above one
time=“2017-12-27T11:27:26Z” level=fatal msg="Failed to configure cattle client: Get http:// localhost:8080/v2-beta/schemas: net/http: request canc
eled (Client.Timeout exceeded while awaiting headers)"
time=“2017-12-27T12:09:08Z” level=error msg="Error reading the setting api.auth.provider.configured , error: Get http:// localhost:8080/v2-beta/se
ttings/api.auth.provider.configured: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
time=“2017-12-27T12:09:08Z” level=error msg="UpgradeSettings: Error reading existing DB settings Get http:// localhost:8080/v2-beta/settings/api.a
uth.provider.configured: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
time=“2017-12-27T12:09:08Z” level=fatal msg=“Failed to upgrade the existing auth settings in db to new: Get http:// localhost:8080/v2-beta/setting
s/api.auth.provider.configured: net/http: request canceled (Client.Timeout exceeded while awaiting headers)”