I am going through the intro documentation, and it seems that I am getting hung up when trying to start the database container using the mysql image.
Name: database
Image: mysql
Environment Vars: MYSQL_ROOT_PASSWORD = pass1
It has been sitting “Starting” for almost 30 mins now. Is there anyway to see what step or process it is on? or how to view the logs?
Thanks!
@ben_humangeo to answer the question in the subject, container startup should be pretty fast (matter of seconds). 30 mins and container being stuck indicates there is some error in container start process. We are working on better error reporting right now, so today the best way to check for the failures would be to look at the log:
/var/lib/cattle/logs/cattle-debug.log
Or you can share your docker-compose template (omitting sensitive info), so I can help to debug it.
Alena,
This is actually coming from your quick start guide. I cannot open up a terminal for the container because it is not running yet.
@ben_humangeo this log file is not located in the service container that is stuck in Starting state. This is the log on the Rancher server. There you can track “service.activate” followed by “instance.start” processes for your service/instance. Let me know if you see any exceptions, and I’ll help to debug further.
@ben_humangeo If you are following the Quick Start Guide, I’m assuming you are adding a container?
What version of Rancher are you using? You can click on the cow in the upper left hand corner to get that information.
How is your Rancher server set up? Do you have Rancher server running on the same host as Rancher agent? If so, is the IP of the host the same as the IP of the server? If the IP is not the same, you’ll need to follow these instructions to re-register the agent. http://docs.rancher.com/rancher/rancher-ui/infrastructure/hosts/custom/#samehost
And finally, can you take a snapshot of the host and the stuck container?
Alena,
I don’t see anything that points me to an exception or error. I can see something that resembles what I am requesting Rancher to do, but nothing stands out to me.
I am trying to implement the “Create a Multi-Container Application” section, specifically where I am creating a new service called database and using the mysql image.
Rancher v0.32.0
Cattle v0.75.0
User Interface v0.40.0
Rancher Compose beta/latest
Once again, following the Quick Start Guide, I ran docker run -d --restart=always -p 8080:8080 rancher/server to get the Rancher server running.
Running docker ps shows me the following:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
41d3508f814f rancher/agent-instance:v0.3.1 "/etc/init.d/agent-in" 20 minutes ago Up 20 minutes 0.0.0.0:500->500/udp, 0.0.0.0:4500->4500/udp 1c0930f5-d269-4ee4-b016-492cde355de2
7e7ada2aa082 rancher/agent:v0.7.11 "/run.sh run" 25 minutes ago Up 25 minutes rancher-agent
4f116531fc16 rancher/server "/usr/bin/s6-svscan /" 28 minutes ago Up 28 minutes 3306/tcp, 0.0.0.0:8090->8080/tcp admiring_franklin
This was the command that I ran to set the CATTLE_AGENT_IP:
docker run -e CATTLE_AGENT_IP=192.168.99.100 -d --privileged -v /var/run/docker.sock:/var/run/docker.sock rancher/agent:v0.7.11 http://192.168.99.100:8090/v1/scripts/18BF003BB5EE86A2C048:1440093600000:kKXe2vQBjtfUJbOEvwLC1kmwUw
Can you share me a screenshot of the hosts screen? Click on the Infrastructure tab to view the hosts.
Typically, when you start a service on a new host, Rancher will start a network agent container, but it’s not in your docker ps so that might be part of the issue.
Can you do a docker ps -a for me? It’s strange that the Network Agent container is in your screen but not in your list of containers. It typically starts a rancher/agent-instance container.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
41d3508f814f rancher/agent-instance:v0.3.1 "/etc/init.d/agent-in" 50 minutes ago Up 50 minutes 0.0.0.0:500->500/udp, 0.0.0.0:4500->4500/udp 1c0930f5-d269-4ee4-b016-492cde355de2
7e7ada2aa082 rancher/agent:v0.7.11 "/run.sh run" 55 minutes ago Up 55 minutes rancher-agent
53789f2cf8c0 rancher/agent:v0.7.11 "/run.sh state" 55 minutes ago Exited (0) 55 minutes ago rancher-agent-state
225a5ac30526 rancher/agent:v0.7.11 "/run.sh http://192.1" 55 minutes ago Exited (0) 55 minutes ago agitated_hoover
4f116531fc16 rancher/server "/usr/bin/s6-svscan /" 58 minutes ago Up 58 minutes 3306/tcp, 0.0.0.0:8090->8080/tcp admiring_franklin
There is a rancher/agent-instance running. Did you mean something else?
Can you do a docker images to see if the mysql image was downloaded? I wonder if it somehow got stuck at that step.
You are correct. No mysql image exists in the local images.
That’s why it was stuck in storage. Can you try starting another service with the same image? You can click on “clone” from the dropdown of the service.
Alternatively, you could directly pull the image on to the host and start the service. You shouldn’t need to do this, but to get you started with Rancher, that might be a current workaround if you continue to have issues pulling images.
I haven’t seen any one else have any direct issues with having Rancher pull images for them.
Something was definitely hung in the background due to Rancher. I was unable to do docker pull mysql from Docker CLI. I tried a different image mysql:5.7.8 and that downloaded fine through the CLI, so I stopped it before it was finished downloading, and then tried to use that image for Rancher, and it hung again. This time I was unable to pull that image from Docker CLI.
I am using the new Docker Toolbox for Mac OS X if that matters.
After restart the docker-machine instance, I was able to add the service if I successfully pulled the image first from Docker CLI.
Just to chime in, I’ve seen similar issues, where newly added containers (or cloned, but with updated image tag, i.e. the image does not exist locally yet and has to be pulled) would fail to start. On a second attempt it would work.
It feels like the start procedure fails when the image is missing and has to be pulled, but the pull operation is triggered. So on the subsequent attempt, the image is in place and the start up goes through.
I’ve not posted anything about this earlier as I wasn’t sure were the issue was, or how repeatable it was (using a private registry, so thought it might be related to that). But reading of a very similar issue here, I thought it might be worth mentioning… 
EDIT: just to be clear, I don’t know why/when this happens, as it doesn’t always cause troubles (no issue when doing a rancher-compose up for instance). But, one of the things I would’ve looked into before reporting was if I could spot there actually being two start attempts for the container when it was launched by a service. The issues I have described above was when I was manually adding a container directly in infrastructure page using the Add Container button, so I guess there’s not as much retries and fixups going on as when using a Service.
