What is convoy Longhorn?

rafael.sisweb · April 19, 2016, 7:47pm

Hi, I wanna know more about convoy-longhorn. What is this?

ibuildthecloud · April 20, 2016, 3:34am

Convoy Longhorn is the code name for a storage system we’ve been working on for a bit. Think of it as a cloud agnostic EBS. It’s very raw, very early, but we just recently published the source. https://github.com/rancher/longhorn There will be more details to follow, but right now just think of it as a R&D project.

rafael.sisweb · April 20, 2016, 11:27am

Great, thanks for that.

pwFoo · June 24, 2016, 4:35pm

Hi @ibuildthecloud

any updates about longhorn? I’m searching a multi host shared / synced volume solution

ibuildthecloud · June 24, 2016, 5:15pm

@pwFoo It’s progressing well It’s all open source so we aren’t hiding anything and if you’re extremely brave you can try https://github.com/rancher/longhorn but honestly that project is half the solution and trying it now might give you the wrong impression. We started with a technology that only works on bleeding edge kernels (4.6+) so we are looking at different approaches that allow us to bring this to all Linux distros. We expect in a couple months we will announce the public alpha/beta of this.

pwFoo · June 24, 2016, 6:00pm

So I have to wait until RancherOS 0.5 is released I think, because RancherOS 0.4.5 is running with 4.2.8-ckt4-rancher kernel?

A friend tested torus in the last days. So maybe he have some time to test longhorn too g

ibuildthecloud · June 24, 2016, 8:12pm

@pwFoo I works on RancherOS 0.4.5. We backported it to that kernel.

pwFoo · June 24, 2016, 8:14pm

Nice
So we should try it soon

pwFoo · July 19, 2016, 8:36am

Tried to test longhorn from longhorn-catalog, but volumes stucks in requested state.
I’ll upgrade rancher server and rancher os (0.4.5 -> 0.5) and test it again…

pwFoo · July 25, 2016, 9:39am

Is there any more information about longhorn? Could it be downloaded as binary or running inside an docker container?

I tried the longhorn catalog, but it doesn’t work

volumes stuck in requested state
driver / storagepool stucks initializing state

Setup:

RancherOS 0.5
Hosts connected by OpenVPN without firewall restrictions

Host1 (with rancher server running too!) doesn’t start any longhorn related container!

Host2 starts a longhorn storagepool container
81f24e735d13 rancher/longhorn:dev-c464662 "launch-driver storag" 9 minutes ago Up 9 minutes r-longhorn_storagepool_1

cjellick · July 25, 2016, 3:59pm

As a complete solution, longhorn is designed primarily to work with Rancher, but it can be ran “manaully” as a series of binaries. This is largely a development tool. The readme here offers some guidance: https://github.com/rancher/longhorn, but the documentation is pretty sparse since 1) we are still working on longhorn and 2) the standalone use case isn’t our focus beyond allowing us to develop and test.

If you’re docker-longhorn-driver stack is stuck in initializing, then no volumes are going to deploy successfully, as they depend on that stack working properly. If you post a snipped of your storage pool and driver logs, there might be something that I can give guidance on.

pwFoo · July 25, 2016, 6:36pm

Hi @cjellick,

I tried to install it from the longhorn catalog. RancherOS Hosts and deployed by a Rancher Server
So “working” (sucessful install to do first tests) deployment by Rancher Server would be fine.

cjellick · July 25, 2016, 7:06pm

Deploying from the catalog has worked pretty consistently for our in-house QA. If you have any logs from the containers in the stack or from the ranchers-server, that would be useful.

It sounds like the docker-longhorn-driver container cannot be started for some reason. I’m guessing either the rancher-server or rancher-agent logs will have some error in them saying that a container could not be created/started.

pwFoo · July 26, 2016, 1:54pm

Host1 (with rancher server running)

time="2016-07-27T05:51:55Z" level=fatal msg="open /host/var/run/docker/execdriver/native: no such file or directory" 
+ '[' volumedriver == volumedriver ']'
+ /usr/bin/share-mnt /var/lib/rancher/longhorn -- docker-longhorn-driver volumedriver
time="2016-07-27T05:52:12Z" level=fatal msg="open /host/var/run/docker/execdriver/native: no such file or directory" 
+ '[' volumedriver == volumedriver ']'
+ /usr/bin/share-mnt /var/lib/rancher/longhorn -- docker-longhorn-driver volumedriver
time="2016-07-27T05:52:26Z" level=fatal msg="open /host/var/run/docker/execdriver/native: no such file or directory" 
+ '[' volumedriver == volumedriver ']'
+ /usr/bin/share-mnt /var/lib/rancher/longhorn -- docker-longhorn-driver volumedriver
time="2016-07-27T05:52:42Z" level=fatal msg="open /host/var/run/docker/execdriver/native: no such file or directory" 
+ '[' volumedriver == volumedriver ']'
+ /usr/bin/share-mnt /var/lib/rancher/longhorn -- docker-longhorn-driver volumedriver
+ '[' volumedriver == volumedriver ']'
+ /usr/bin/share-mnt /var/lib/rancher/longhorn -- docker-longhorn-driver volumedriver
time="2016-07-27T05:52:56Z" level=fatal msg="open /host/var/run/docker/execdriver/native: no such file or directory" 
+ '[' volumedriver == volumedriver ']'
+ /usr/bin/share-mnt /var/lib/rancher/longhorn -- docker-longhorn-driver volumedriver
time="2016-07-27T05:53:12Z" level=fatal msg="open /host/var/run/docker/execdriver/native: no such file or directory" 
time="2016-07-27T05:53:26Z" level=fatal msg="open /host/var/run/docker/execdriver/native: no such file or directory" 
+ '[' volumedriver == volumedriver ']'
+ /usr/bin/share-mnt /var/lib/rancher/longhorn -- docker-longhorn-driver volumedriver
time="2016-07-27T05:53:42Z" level=fatal msg="open /host/var/run/docker/execdriver/native: no such file or directory" 
+ '[' volumedriver == volumedriver ']'
+ /usr/bin/share-mnt /var/lib/rancher/longhorn -- docker-longhorn-driver volumedriver
+ '[' volumedriver == volumedriver ']'
+ /usr/bin/share-mnt /var/lib/rancher/longhorn -- docker-longhorn-driver volumedriver
time="2016-07-27T05:53:56Z" level=fatal msg="open /host/var/run/docker/execdriver/native: no such file or directory" 
+ '[' volumedriver == volumedriver ']'
+ /usr/bin/share-mnt /var/lib/rancher/longhorn -- docker-longhorn-driver volumedriver
time="2016-07-27T05:54:11Z" level=fatal msg="open /host/var/run/docker/execdriver/native: no such file or directory" 
time="2016-07-27T05:54:26Z" level=fatal msg="open /host/var/run/docker/execdriver/native: no such file or directory" 
+ '[' volumedriver == volumedriver ']'
+ /usr/bin/share-mnt /var/lib/rancher/longhorn -- docker-longhorn-driver volumedriver
+ '[' volumedriver == volumedriver ']'
+ /usr/bin/share-mnt /var/lib/rancher/longhorn -- docker-longhorn-driver volumedriver
time="2016-07-27T05:54:42Z" level=fatal msg="open /host/var/run/docker/execdriver/native: no such file or directory" 
time="2016-07-27T05:54:56Z" level=fatal msg="open /host/var/run/docker/execdriver/native: no such file or directory" 
+ '[' volumedriver == volumedriver ']'
+ /usr/bin/share-mnt /var/lib/rancher/longhorn -- docker-longhorn-driver volumedriver
time="2016-07-27T05:55:11Z" level=fatal msg="open /host/var/run/docker/execdriver/native: no such file or directory" 
+ '[' volumedriver == volumedriver ']'
+ /usr/bin/share-mnt /var/lib/rancher/longhorn -- docker-longhorn-driver volumedriver
time="2016-07-27T05:55:27Z" level=fatal msg="open /host/var/run/docker/execdriver/native: no such file or directory" 
+ '[' volumedriver == volumedriver ']'
+ /usr/bin/share-mnt /var/lib/rancher/longhorn -- docker-longhorn-driver volumedriver

Host2 Storagepool container logs

+ '[' storagepool == volumedriver ']'
+ docker-longhorn-driver storagepool
time="2016-07-26T13:46:40Z" level=info msg="Launching plugin" 
time="2016-07-26T13:46:40Z" level=info msg="Listening for health checks on 0.0.0.0:10241/healthcheck" 
time="2016-07-26T13:46:40Z" level=info msg="Listening for cattle events" 
time="2016-07-26T13:46:40Z" level=info msg="Initializing event router" workerCount=10 
time="2016-07-26T13:46:40Z" level=info msg="Connection established"

Rancher Server

time="2016-07-27T05:51:48Z" level=info msg="Stack Create Event Received" eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:51:49Z" level=info msg="[longhorn:]: Creating project " eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:51:49Z" level=info msg="[longhorn:storagepool]: Creating " eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:51:49Z" level=info msg="[longhorn:driver]: Creating " eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:51:49Z" level=info msg="Creating service driver" 
time="2016-07-27T05:51:49Z" level=info msg="Creating service storagepool" 
time="2016-07-27T05:51:49Z" level=info msg="[longhorn:driver]: Created " eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:51:49Z" level=info msg="[longhorn:storagepool]: Created " eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:51:49Z" level=info msg="[longhorn:]: Project created " eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:51:49Z" level=info msg="[longhorn:]: Creating project " eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:51:49Z" level=info msg="[longhorn:driver]: Creating " eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:51:49Z" level=info msg="[longhorn:storagepool]: Creating " eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:51:49Z" level=info msg="[longhorn:storagepool]: Created " eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:51:49Z" level=info msg="[longhorn:driver]: Created " eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:51:49Z" level=info msg="[longhorn:]: Project created " eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:51:49Z" level=info msg="[longhorn:]: Starting project " eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:51:49Z" level=info msg="[longhorn:storagepool]: Starting " eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:51:49Z" level=info msg="[longhorn:driver]: Starting " eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:51:57Z" level=info msg="[longhorn:storagepool]: Started " eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:52:00Z" level=info msg="[longhorn:driver]: Started " eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:52:00Z" level=info msg="[longhorn:]: Project started " eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 
time="2016-07-27T05:52:00Z" level=info msg="Stack Create Event Done" eventId=a5c882f3-d116-4775-8f10-7f8c870165de resourceId=1e55 




2016-07-27 05:52:28,523 ERROR [:] [] [] [] [ServiceReplay-4] [c.p.e.p.i.DefaultProcessInstanceImpl] final ExitReason is null, should not be 
2016-07-27 05:52:28,524 ERROR [:] [] [] [] [ServiceReplay-4] [i.c.p.e.e.i.ProcessEventListenerImpl] Unknown exception running process [service.remove:143324] on [44] java.lang.IllegalStateException: Attempt to cancel when process is still transitioning
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:190) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:157) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:107) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:104) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:104) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.eventing.impl.ProcessEventListenerImpl.processExecute(ProcessEventListenerImpl.java:74) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.server.impl.ProcessInstanceParallelDispatcher$1.runInContext(ProcessInstanceParallelDispatcher.java:27) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_95]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_95]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_95]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_95]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_95]
Caused by: io.cattle.platform.engine.process.impl.ProcessCancelException: State [restarting] is not valid
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.preRunStateCheck(DefaultProcessInstanceImpl.java:267) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:182) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:157) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:107) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:104) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:104) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.manager.impl.DefaultProcessManager.scheduleProcessInstance(DefaultProcessManager.java:60) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.object.process.impl.DefaultObjectProcessManager.scheduleProcessInstance(DefaultObjectProcessManager.java:47) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.object.process.impl.DefaultObjectProcessManager.scheduleProcessInstance(DefaultObjectProcessManager.java:52) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.object.process.impl.DefaultObjectProcessManager$1.run(DefaultObjectProcessManager.java:115) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.deferred.util.DeferredUtils$3.call(DeferredUtils.java:43) ~[cattle-framework-deferred-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.deferred.util.DeferredUtils.nest(DeferredUtils.java:53) ~[cattle-framework-deferred-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.deferred.util.DeferredUtils.nest(DeferredUtils.java:40) ~[cattle-framework-deferred-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.object.process.impl.DefaultObjectProcessManager.scheduleProcessInstanceAsync(DefaultObjectProcessManager.java:112) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.servicediscovery.upgrade.impl.UpgradeManagerImpl.cleanupUpgradedInstances(UpgradeManagerImpl.java:459) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.servicediscovery.upgrade.impl.UpgradeManagerImpl.finishUpgrade(UpgradeManagerImpl.java:421) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.servicediscovery.process.ServiceRemove.handle(ServiceRemove.java:41) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:424) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:375) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:369) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandlers(DefaultProcessInstanceImpl.java:369) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runLogic(DefaultProcessInstanceImpl.java:471) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:305) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:245) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:242) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:184) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        ... 20 common frames omitted

2016-07-27 05:52:28,539 ERROR [:] [] [] [] [erviceReplay-24] [c.p.e.p.i.DefaultProcessInstanceImpl] final ExitReason is null, should not be 
2016-07-27 05:52:28,541 ERROR [:] [] [] [] [erviceReplay-24] [i.c.p.e.e.i.ProcessEventListenerImpl] Unknown exception running process [service.remove:236204] on [81] java.lang.IllegalStateException: Attempt to cancel when process is still transitioning
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:190) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:157) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:107) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:104) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:104) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.eventing.impl.ProcessEventListenerImpl.processExecute(ProcessEventListenerImpl.java:74) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.server.impl.ProcessInstanceParallelDispatcher$1.runInContext(ProcessInstanceParallelDispatcher.java:27) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_95]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_95]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_95]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_95]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_95]
Caused by: io.cattle.platform.engine.process.impl.ProcessCancelException: State [restarting] is not valid
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.preRunStateCheck(DefaultProcessInstanceImpl.java:267) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:182) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:157) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:107) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:104) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:104) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.manager.impl.DefaultProcessManager.scheduleProcessInstance(DefaultProcessManager.java:60) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.object.process.impl.DefaultObjectProcessManager.scheduleProcessInstance(DefaultObjectProcessManager.java:47) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.object.process.impl.DefaultObjectProcessManager.scheduleProcessInstance(DefaultObjectProcessManager.java:52) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.object.process.impl.DefaultObjectProcessManager$1.run(DefaultObjectProcessManager.java:115) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.deferred.util.DeferredUtils$3.call(DeferredUtils.java:43) ~[cattle-framework-deferred-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.deferred.util.DeferredUtils.nest(DeferredUtils.java:53) ~[cattle-framework-deferred-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.deferred.util.DeferredUtils.nest(DeferredUtils.java:40) ~[cattle-framework-deferred-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.object.process.impl.DefaultObjectProcessManager.scheduleProcessInstanceAsync(DefaultObjectProcessManager.java:112) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.servicediscovery.deployment.impl.unit.DefaultDeploymentUnitInstance.removeInstance(DefaultDeploymentUnitInstance.java:93) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.servicediscovery.deployment.impl.unit.DefaultDeploymentUnitInstance.removeUnitInstance(DefaultDeploymentUnitInstance.java:83) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.servicediscovery.deployment.DeploymentUnitInstance.remove(DeploymentUnitInstance.java:39) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.servicediscovery.deployment.impl.unit.DeploymentUnit.remove(DeploymentUnit.java:167) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.servicediscovery.deployment.impl.DeploymentManagerImpl$3.deleteServiceInstances(DeploymentManagerImpl.java:389) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.servicediscovery.deployment.impl.DeploymentManagerImpl$3.doWithLockNoResult(DeploymentManagerImpl.java:376) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.servicediscovery.deployment.impl.DeploymentManagerImpl.remove(DeploymentManagerImpl.java:372) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.servicediscovery.process.ServiceRemove.handle(ServiceRemove.java:42) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:424) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:375) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:369) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandlers(DefaultProcessInstanceImpl.java:369) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runLogic(DefaultProcessInstanceImpl.java:471) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:305) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:245) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:242) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:184) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        ... 20 common frames omitted


2016-07-27 05:53:28,583 ERROR [:] [] [] [] [erviceReplay-14] [c.p.e.p.i.DefaultProcessInstanceImpl] final ExitReason is null, should not be 
2016-07-27 05:53:28,585 ERROR [:] [] [] [] [erviceReplay-14] [i.c.p.e.e.i.ProcessEventListenerImpl] Unknown exception running process [service.remove:143356] on [52] java.lang.IllegalStateException: Attempt to cancel when process is still transitioning

Both hosts are RancherOS 0.5, Host1 also running Rancher Server and fails to start because of:
time="2016-07-27T05:51:55Z" level=fatal msg="open /host/var/run/docker/execdriver/native: no such file or directory"

Both hosts have a /var/lib/rancher/longhorn/ directory, but there is no /host/var/run/docker/execdriver/native (looked at host and inside running container @ host2).

Is it a bug or a problem with my environment?

cjellick · July 26, 2016, 3:39pm

Ah, that’s been fixed on master of longhorn here: https://github.com/rancher/longhorn/pull/200 but actually hasn’t made it into master of longhorn-catalog.

I’m not sure how soon we’ll be updating the master catalog as we have a lot of development currently going on in longhorn. You’re on the bleeding edge and feeling the pain

It would be nice to get that particular fix into the primary catalog though. I’ll have some conversations today and see if that is feasible.

pwFoo · July 26, 2016, 4:01pm

Thanks, great
It’s ok because it’s just a longhorn dev release, but with a new catalog release more and simple testing will be possible.

Topic		Replies	Views
Convoy is gone from docs, what replaces it? Rancher 1.x	3	1304	April 19, 2017
Convoy with RancherOS? RancherOS	3	1791	October 25, 2016
About the Longhorn category Longhorn	0	2485	April 15, 2017
Convoy EBS (rancher catalog) Convoy	6	3337	June 23, 2016
Using longhorn with Rancher Longhorn	3	2342	July 7, 2017

What is convoy Longhorn?

Related topics