GlusterFS Catalog continually restarting (0.47.0 using catalog template 3.7.5-rancher1)

Rancher 1.x
1

Not entirely sure how to go about debugging – but I just upgraded from 0.46.0 to 0.47.0 on GCE and tried to launch a GlusterFS Stack using the Catalog 3.7.5-rancher1.

The Service launches the first container and started it once. The second service (server) seems to be continually restarting… eventually showing the error:

State: Stopped ( : a7e401a5-8427-4042-a050-be9843ae3690 : 404 Client Error: Not Found (“Cannot start container 678ae9cd2012034a8f463d8b637c56cfb3c9e066997532289d8faef11bc02e91: error gathering device information while adding custom device “/dev/fuse”: lstat /dev/fuse: no such file or directory”))

From the Rancher UI. Any ideas as to what might help debug this issue? Thoughts on resolutions?

Thanks!

2

The same problem here but also the first container does not start?
I tried to launch the exported service and it’s working??

3

I have encountered this, and have resolved it down to these pitfalls

  1. What OS you running?
    • What user is the docker daemon running under?
      • if running as a non-root user, the service does not have the permission to created the device. if you manually create the device see mknod/makedev the service then start.

My two test cases:
1 Host running RancherOS - failed to deploy, see above resolution
another host running ubuntu:14.0.4.3, scheduled fine with no issues.

There are further issues when it comes then to deploying the convoy-gluster service, and it not created and seeing volumes, but have submitted issues through to Rancher Labs.

4

Yep, I’m also running RancherOS v0.4.0 on hosts in GCE. Haven’t tried on other platforms yet but I’d completely believe thats an issue. @Moridinmhael did you open a ticket for the RancherOS issue anywhere? Thoughts on how to gather debug info on this to create a bug if not?

5

I think i have the same problem. I run rancher on Ubuntu and the agent on the same machine. It seams that convoy-gluster did not start and have no containers. How can i get out any error logs from the startup of the stack ?

6

Here’s the output of /var/log/docker.log in my stack.

  • RancherOS v0.4.1 on a KVM VM
  • Rancher 0.47.0
time="2015-11-27T08:17:05.956478206Z" level=error msg="HTTP Error" err="Cannot start container 46692e7091874b4dcac7f0bc1bc52b06fe4a6de224a9775faae9ad6bd0a3733e: error gathering device information while adding custom device \"/dev/fuse\": lstat /dev/fuse: no such file or directory" statusCode=500

What’s curios is, is that if I start the stack through rancher-compose it’s working ??

7

@lkoenigsberger : are you using sudo when running the rancher-compose? since I have that the commands should work via sudo

8

@Moridinmhael No i did not run the command wiith sudo it worked so. but only tried the glusterfs service not the convoy service.

9

Hi,
a little late here but modprobe fuse wil sort this out.

Also, check rancher storage example in particular the section regarding host label…

This worked fine for me.

Br hw

1 Like
10

@hwaastad Thanks! That was the resolution to this specific issue (I guess I should have actually taken the hint from the error :smile:). I did see the host label stuff as well – thanks for the tip.

11

My container get this error after some digging level=error msg=“Error while reading host info [json: cannot unmarshal object into Go value of type string]”.

As i describe above i started the service directly from the catalog but the convoy-gluster will not start but convoy-gluster-storagepool started.

12

@Mikael_Svensson What OS are you running? Your issue seems different from the others.

For others, I’ve created an issue in Github.

13

FYI, we’ve made a fix for GlusterFS catalog item so it works on RancherOS and is compatible with systems that don’t have fuse loaded. The updated catalog entry is already available.

@Mikael_Svensson: If you are still having issues, please open another topic as your issue is different.

1 Like