Launch container very slow (40min)

Archive Rancher 1.x
1

Hi,

I have a problem with rancher (or with my own docker images).
In my case, I have a stack with one big service (docker Image uncompressed size is 1.22GB) and, when rancher start the container for the first time, it takes 45 minutes to go from start to run status. (In details I see Waiting: container starting [container:1i2734]).

Also, time to time, it seems like starting this big container is taking too much ressources and rancher-agent is going wry, passing others containers in the same host in “unhealthy” state

But, with another large docker image (725MB) the time is 10-20 more faster. And I hav’nt this problem with rancher-agent

Is it normal ?

I run rancher on an Azure VM scale set and the VM size are E2_v3 : 2x2.29Ghz CPU and 15.7GB RAM.

Rancher version : 1.6.5
Docker version : 1.12.6
Linux Kernel : 4.4.0

Thanks

2

Issue appears to be with secrets stack creating thousands of volume , which went in detached mode. I had tried removing it with rancher cli. Though it appears as getting deleted while cli runs, its not even removing a single volume, getting the following error in rancher logs continuously with each volume delete from rancher cli

2018-09-24 16:53:10,982 ERROR [:] [] [] [] [cutorService-31] [.e.s.i.ProcessInstanceDispatcherImpl] Agent error for [agent.request]: Agent [1287] is removed
2018-09-24 16:53:11,244 ERROR [c4f9ca42-4657-4feb-b30f-dd1715859cec:40686172] [volume:703371] [volume.remove] [] [cutorService-31]

[c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [agent.request]: Agent [375916] is removed
2018-09-24 16:53:11,250 ERROR [:] [] [] [] [cutorService-31] [.e.s.i.ProcessInstanceDispatcherImpl] Agent error for [agent.request]: Agent [375916] is removed
2018-09-24 16:53:11,507 ERROR [f27ee7e0-8589-40ba-97ba-cd175f6d3b69:40686174] [volume:703373] [volume.remove] [] [cutorService-31]

[c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [agent.request]: Agent [1287] is removed
2018-09-24 16:53:11,515 ERROR [:] [] [] [] [cutorService-31] [.e.s.i.ProcessInstanceDispatcherImpl] Agent error for [agent.request]: Agent [1287] is removed