Can't delete storage pool volumes and reuse the volume names

Archive Rancher 1.x
1

I’ve set up glusterfs and convoy-gluster. Creating volumes by specifying them in a docker-compose.yml works. :thumbsup:

I’m having issues when trying to delete the volumes after tearing down a stack.

1.) Deleting Volume but not Purging doesn’t delete the data.
To reproduce:

  1. Create a stack with a volume using convoy-gluster storage driver
  2. Put some data in the volume
  3. Delete the stack
  4. From the Infrastructure -> Storage Pools page. Delete but do not Purge the volume.
  • Volume should now show “Removed”
  1. Create a new stack using the same volume name.
  2. The data is not deleted and the volume returns to Active.

2.) Can’t reuse volume name after Deleting Volume and Purging

  1. Create a stack with a volume using convoy-gluster storage driver
  2. Put some data in the volume
  3. Delete the stack
  4. From the Infrastructure -> Storage Pools page. Delete and then Purge the volume.
  • Volume should no longer be listed
  1. Recreate the stack. Use the same volume name.
  2. The container creation will fail and you might see a 500 “Couldn’t find Volume” or a 404 “no such file or directory” .

Sample 500 error from the rancher server logs:

2016-02-21 00:31:13,114 ERROR [:] [] [] [] [rviceReplay-103] [i.c.p.e.e.i.ProcessEventListenerImpl] Unknown exception running process [instance.start:253938] on [10577] io.cattle.platform.eventing.exception.EventExecutionException: 500 Server Error: Internal Server Error ("Couldn't find volume.")

Sample 404 error from the rancher server logs:

2016-02-20 23:40:59,975 ERROR [c33013a0-c24d-4cb9-9a25-4851e5f68c2d:253519] [instance:10569] [instance.start] [] [rviceReplay-110] [c.p.e.p.i.DefaultProcessInstanceImpl] Unknown exception io.cattle.platform.eventing.exception.EventExecutionException: 404 Client Error: Not Found ("open /var/lib/rancher/convoy/convoy-gluster-f84579f3-1548-43b5-b934-f445bd6011e5/glusterfs/mounts/data01/test: no such file or directory")

There are a lot of containers involved in gluster and convoy, so I’m not sure which logs might help. Please let me know if you would like me to post other logs.

Rancher Versions:
Rancher v0.59.0
Cattle v0.148.0
User Interface v0.90.0
Rancher Compose v0.7.2

Test docker-compose.yml

test:
  labels:
    io.rancher.container.pull_image: always
  tty: true
  image: busybox
  volumes:
    - test01:/test
  stdin_open: true
  volume_driver: convoy-gluster
  command: 
    - /bin/sh
    - -c
    - while true; do echo hello world; sleep 1; done

Test rancher-compose.yml

test:
  scale: 1
1 Like
2

Same for me. Thanks for the great explanation!

3

Thanks for the excellent write ups.

RE issue #1 Deleting Volume but not Purging doesn’t delete the data:
That is by design, a volume isn’t truly gone until you purge.

RE Issue #2, I went ahead and created a github issue for it (bc thats where we track bugs):
https://github.com/rancher/rancher/issues/3671
If you have additional info for this issue or any follow ups, can you post as a comment in the github issue as opposed to this forum topic?

I’ll try to take a look sometime soon.