We had some trouble deploying a catalog across three hosts. Firstly deployment went fine. Then we removed that stack and redeployed it again. As the images are already present in the host, containers got created, but the stack remains in “unhealthy” state with error as below.
http: proxy error: net/http: request canceled
level=error msg="Failed Creating service1 : Get http://localhost:8080/v2-beta/projects/1a83132/services?name=service1&removed_null=<nil%3E&stackId=1st376: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
level=error msg="Failed Creating service2 : Post http://localhost:8080/v2-beta/projects/1a83132/services/1s921/?action=setservicelinks:net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
On checking, ports appears to be not listening at all for the online containers after the deployment. All internal containers including ipsec, healthcheck are up on those hosts and we had to remove those hosts from rancher, remove all containers, remove /var/lib/rancher and re add the host to deploy it again.
Any one have any idea about what happened? And is there any better solution in scenarios other than re adding the hosts? Please help