Since upgrading from Rancher 1.4.2 to 1.5.3, I get the following error in my rancher server logs once an hour:
2017-03-30 19:34:32,606 ERROR [:] [] [] [] [ecutorService-4] [i.c.p.task.impl.TaskManagerImpl ] Task [run.scheduled] failed io.cattle.platform.lock.exception.FailedToAcquireLockException: Failed to acquire lock [SCHEDULED.UPGRADE]
at io.cattle.platform.lock.provider.impl.StandardLock.lock(StandardLock.java:56) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.lock.impl.AbstractLockManagerImpl$1.withLock(AbstractLockManagerImpl.java:17) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:30) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.systemstack.service.UpgradeManager.run(UpgradeManager.java:112) ~[cattle-system-stack-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.systemstack.task.RunScheduledTask.run(RunScheduledTask.java:23) ~[cattle-system-stack-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.task.impl.TaskManagerImpl$1.doRun(TaskManagerImpl.java:100) ~[cattle-framework-system-task-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:15) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_72]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
I don’t know what’s holding the lock. Or even what its trying to upgrade (all the stacks show up to date).
MariaDB [cattle]> select * from scheduled_upgrade;
Empty set (0.03 sec)
MariaDB [cattle]> select * from DATABASECHANGELOGLOCK\G
*************************** 1. row ***************************
ID: 1
LOCKED:
LOCKGRANTED: NULL
LOCKEDBY: NULL
1 row in set (0.03 sec)
Any ideas would be appreciated.