GlusterFS Catalog continually restarting (0.47.0 using catalog template 3.7.5-rancher1)

Not entirely sure how to go about debugging – but I just upgraded from 0.46.0 to 0.47.0 on GCE and tried to launch a GlusterFS Stack using the Catalog 3.7.5-rancher1.

The Service launches the first container and started it once. The second service (server) seems to be continually restarting… eventually showing the error:

State: Stopped ( : a7e401a5-8427-4042-a050-be9843ae3690 : 404 Client Error: Not Found (“Cannot start container 678ae9cd2012034a8f463d8b637c56cfb3c9e066997532289d8faef11bc02e91: error gathering device information while adding custom device “/dev/fuse”: lstat /dev/fuse: no such file or directory”))

From the Rancher UI. Any ideas as to what might help debug this issue? Thoughts on resolutions?

Thanks!

The same problem here but also the first container does not start?
I tried to launch the exported service and it’s working??

I have encountered this, and have resolved it down to these pitfalls

  1. What OS you running?
    • What user is the docker daemon running under?
      • if running as a non-root user, the service does not have the permission to created the device. if you manually create the device see mknod/makedev the service then start.

My two test cases:
1 Host running RancherOS - failed to deploy, see above resolution
another host running ubuntu:14.0.4.3, scheduled fine with no issues.

There are further issues when it comes then to deploying the convoy-gluster service, and it not created and seeing volumes, but have submitted issues through to Rancher Labs.

Yep, I’m also running RancherOS v0.4.0 on hosts in GCE. Haven’t tried on other platforms yet but I’d completely believe thats an issue. @Moridinmhael did you open a ticket for the RancherOS issue anywhere? Thoughts on how to gather debug info on this to create a bug if not?

I think i have the same problem. I run rancher on Ubuntu and the agent on the same machine. It seams that convoy-gluster did not start and have no containers. How can i get out any error logs from the startup of the stack ?

Here’s the output of /var/log/docker.log in my stack.

  • RancherOS v0.4.1 on a KVM VM
  • Rancher 0.47.0
time="2015-11-27T08:17:05.956478206Z" level=error msg="HTTP Error" err="Cannot start container 46692e7091874b4dcac7f0bc1bc52b06fe4a6de224a9775faae9ad6bd0a3733e: error gathering device information while adding custom device \"/dev/fuse\": lstat /dev/fuse: no such file or directory" statusCode=500

What’s curios is, is that if I start the stack through rancher-compose it’s working ??

@lkoenigsberger : are you using sudo when running the rancher-compose? since I have that the commands should work via sudo

@Moridinmhael No i did not run the command wiith sudo it worked so. but only tried the glusterfs service not the convoy service.

Hi,
a little late here but modprobe fuse wil sort this out.

Also, check rancher storage example in particular the section regarding host label…

This worked fine for me.

Br hw

1 Like

@hwaastad Thanks! That was the resolution to this specific issue (I guess I should have actually taken the hint from the error :smile:). I did see the host label stuff as well – thanks for the tip.

My container get this error after some digging level=error msg=“Error while reading host info [json: cannot unmarshal object into Go value of type string]”.

As i describe above i started the service directly from the catalog but the convoy-gluster will not start but convoy-gluster-storagepool started.

@Mikael_Svensson What OS are you running? Your issue seems different from the others.

For others, I’ve created an issue in Github.

FYI, we’ve made a fix for GlusterFS catalog item so it works on RancherOS and is compatible with systems that don’t have fuse loaded. The updated catalog entry is already available.

@Mikael_Svensson: If you are still having issues, please open another topic as your issue is different.

1 Like