[.p.c.v.i.ConfigItemStatusManagerImpl] Failed null

I have a rancher server running on a VM created in Vsphere and I’m getting the following error. I think this is also related to why I can’t seem to get some of the Rancher features to work properly. Mostly the features that require Rancher internal routing and discovery.

Here’s the error:
2016-01-04 20:50:45,642 ERROR [:] [] [] [] [ecutorService-7] [.p.c.v.i.ConfigItemStatusManagerImpl] Failed null, exit code [1] output [INFO: Updating host-iptables

Any clues will be helpful…

I don’t have a vsphere set up to test on, but is this a fresh setup or an upgraded setup? I haven’t seen this issue before, so a little harder for me to debug.

I’ve upgraded Rancher on this server several times, if that’s what you are asking.

Here is more of the log entry. Let me know if this give you any clues…
I just did a fresh stop, rm, pull and run on the server.

2016-01-13
20:21:15,354 ERROR [d868009c-68eb-4136-8d8b-c24ec03356c1:175591]
[instance:3947] [instance.start->(InstanceStart)] [] [cutorService-22]
[i.c.p.process.in

stance.InstanceStart]
Failed to Scheduling for instance [3947]

2016-01-13
20:21:15,490 ERROR [548b2ff9-1b0a-4266-a27c-ddc0b5e89573:175593]
[instance:3949]
[instance.start->(InstanceStart)->instance.allocate->(InstanceAllocate)]

 [] [cutorService-46]
[c.p.a.e.i.AllocatorEventListenerImpl] No allocator handled [EventVO
[id=dc39d4a3-9375-421b-a3e0-19e8553096cb, name=instance.allocate, previou

sNames=null,
replyTo=reply.3933803215941404492, resourceId=3949, resourceType=instance,
publisher=null, transitioning=null, transitioningMessage=null, transitioning

InternalMessage=null,
previousIds=null, data={}, time=Wed Jan 13 20:21:15 UTC 2016, listenerKey=null,
transitioningProgress=null]]

2016-01-13
20:21:15,494 ERROR [548b2ff9-1b0a-4266-a27c-ddc0b5e89573:175593]
[instance:3949] [instance.start->(InstanceStart)->instance.allocate] []
[cutorService-19

]
[c.p.e.p.i.DefaultProcessInstanceImpl] Unknown exception
io.cattle.platform.eventing.exception.EventExecutionException: Failed to find a
placement

        at
io.cattle.platform.eventing.exception.EventExecutionException.fromEvent(EventExecutionException.java:53)
~[cattle-framework-eventing-0.5.0-SNAPSHOT.jar:n

a]

        at
io.cattle.platform.eventing.impl.AbstractEventService.callSync(AbstractEventService.java:263)
~[cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.process.common.handler.EventBasedProcessHandler.handle(EventBasedProcessHandler.java:109)
~[cattle-iaas-logic-common-0.5.0-SNAPSHOT.ja

r:na]

        at
io.cattle.platform.process.instance.InstanceAllocate.handle(InstanceAllocate.java:47)
~[cattle-iaas-logic-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:456)
[cattle-framework-engine-0.5.0-SNAPSHOT

.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:396)
[cattle-framework-engine-0.5.0-SNAPSHOT.

jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:390)
[cattle-framework-engine-0.5.0-SNAPSHOT.

jar:na]

        at
io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandlers(DefaultProcessInstanceImpl.java:390)
[cattle-framework-engine-0.5.0-SNAPSHO

T.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runLogic(DefaultProcessInstanceImpl.java:520)
[cattle-framework-engine-0.5.0-SNAPSHOT.j

ar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:323)
[cattle-framework-engine-0.5.0-

SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:259)
[cattle-framework-engine-0.5.

0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:256)
[cattle-framework-engine-0.5.0-S

NAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:188)
[cattle-framework-engine-0.5.0-SNA

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:188)
[cattle-framework-engine-0.5.0-SNA

PSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:161)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:110)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:107)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:107)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.process.common.handler.AbstractObjectProcessLogic.execute(AbstractObjectProcessLogic.java:131)
[cattle-iaas-logic-common-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.process.instance.InstanceStart.allocate(InstanceStart.java:205)
[cattle-iaas-logic-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.process.instance.InstanceStart.handle(InstanceStart.java:70)
[cattle-iaas-logic-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:456)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:396)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:390)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandlers(DefaultProcessInstanceImpl.java:390)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runLogic(DefaultProcessInstanceImpl.java:520)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:323)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:259)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:256)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:188)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:161)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:110)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:107)
[cattle-framework-engine-0.5.0-SNAPSH

OT.jar:na]

        at
io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:107)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.engine.eventing.impl.ProcessEventListenerImpl.processExecute(ProcessEventListenerImpl.java:66)
[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

        at
sun.reflect.GeneratedMethodAccessor496.invoke(Unknown Source) ~[na:na]

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.7.0_91]

        at
java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_91]

        at
io.cattle.platform.eventing.annotation.MethodInvokingListener$1.doWithLockNoResult(MethodInvokingListener.java:76)
[cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37)
[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.eventing.annotation.MethodInvokingListener.onEvent(MethodInvokingListener.java:72)
[cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]

        at
io.cattle.platform.eventing.impl.AbstractThreadPoolingEventService$2.doRun(AbstractThreadPoolingEventService.java:135)
[cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]

        at
org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:13)
[cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]

        at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
[cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
[cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108)
[cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
[cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]

        at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
[cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]

        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_91]

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_91]

        at
java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]

 

2016-01-13
20:21:15,494 ERROR [548b2ff9-1b0a-4266-a27c-ddc0b5e89573:175593]
[instance:3949] [instance.start->(InstanceStart)] [] [cutorService-19]
[i.c.p.process.instance.InstanceStart] Failed to Scheduling for instance [3949]

The second batch of logs indicate that you have a service that cannot find a place to launch it’s instance. Usually this is a port conflict.

Failed to Scheduling for instance [3947]

Do you have any services stuck in Activating state? Could you scale that service down?

Ok, I’ll take a look.

Ok, here’s something the may not be related, but still a mystery…to me.

The logs mention having problems executing this script.
/var/lib/cattle/download/host-iptables/host-iptables-314-d049e47d422e7c2556577ff35624f118e182ec70793d23712724afa2f3165683/apply.sh

however, on my system, this file doesn’t exit. In fact, /var/lib/cattle doesn’t exist. Does this make sense?