Make sure new image gets pulled using "branch" image tags

Rancher 1.x
1

Hello,

i’m using Rancher 1.6 and I’m having problems having it pull the new image from the gitlab’s docker repository where the CI runner uploads the images.

I’m using images labelled like “product_name:git_branch_or_tag_name” and i’m using rancher-compose to setup or upgrade a stack. On the images i’ve set the “io.rancher.container.pull_image: always” label for all the services and I’m calling rancher-compose with “rancher-compose up -u -p” so trying to force the pull on the command line also.

What happens is that at the end of the CI build/test steps, i run the deployment but many times the service don’t update to the latest image available in the repository. If I manually stop and delete the stack, remove the images directly from docker on the host and then repeat the deploy, then rancher pulls the most recent images.

In the past I’ve used different labels like “product_name:git_commit_hash” and this problem never arised, probably because the tags change every time and so rancher is forced to pull. I would prefer using branch/tag labels though, as it makes the cleanup of both the registry and the hosts simpler.

Anyone knows I’m missing something or if I’m doing something wrong or wants to share his experience on the subject?
Maybe I should add the --force-recreate flag to ranche-compose, or maybe try the new rancher cli?

2

We too have this problem (Rancher 1.5.7). Same thing as you, we’ve set “io.rancher.container.pull_image: always” and using “–pull” option to rancher-compose. Sometimes the new image gets pulled sometimes it doesn’t.

I have noticed though that if I do an “Upgrade” to the service through the Rancher UI, don’t change any values, just hit “OK”, then the image always gets pulled properly. This would seem to indicate a problem with rancher compose perhaps?

3

I should note, we’re using Nexus as our Docker registry.

4

We fixed a related bug in this issue for rancher 1.6. @azazel75 Can you share the logs from rancher-agent next time you reproduce this?

5

Great! I’ll upgrade to 1.6 and see how it looks.

6

@StrongMonkey here it is:

5/25/2017 2:49:40 AMtime=“2017-05-25T00:49:40Z” level=info msg="Received event: Name: compute.instance.pull, Event Id: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, Resource Id: "
5/25/2017 2:49:42 AMtime=“2017-05-25T00:49:42Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Pulling from arstecnica/ytefas/ytefas_db”
5/25/2017 2:49:42 AMtime=“2017-05-25T00:49:42Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Already exists”
5/25/2017 2:49:42 AMtime=“2017-05-25T00:49:42Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Pulling fs layer”
5/25/2017 2:49:42 AMtime=“2017-05-25T00:49:42Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Waiting”
5/25/2017 2:49:43 AMtime=“2017-05-25T00:49:43Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Download complete”
5/25/2017 2:49:43 AMtime=“2017-05-25T00:49:43Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Verifying Checksum”
5/25/2017 2:49:43 AMtime=“2017-05-25T00:49:43Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Download complete”
5/25/2017 2:49:43 AMtime=“2017-05-25T00:49:43Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Downloading”
5/25/2017 2:49:43 AMtime=“2017-05-25T00:49:43Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Verifying Checksum”
5/25/2017 2:49:43 AMtime=“2017-05-25T00:49:43Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Download complete”
5/25/2017 2:49:43 AMtime=“2017-05-25T00:49:43Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Extracting”
5/25/2017 2:49:43 AMtime=“2017-05-25T00:49:43Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Pull complete”
5/25/2017 2:49:43 AMtime=“2017-05-25T00:49:43Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Extracting”
5/25/2017 2:49:44 AMtime=“2017-05-25T00:49:44Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Pull complete”
5/25/2017 2:49:44 AMtime=“2017-05-25T00:49:44Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Extracting”
5/25/2017 2:49:44 AMtime=“2017-05-25T00:49:44Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Pull complete”
5/25/2017 2:49:44 AMtime=“2017-05-25T00:49:44Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Downloading”
5/25/2017 2:49:44 AMtime=“2017-05-25T00:49:44Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Verifying Checksum”
5/25/2017 2:49:44 AMtime=“2017-05-25T00:49:44Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Download complete”
5/25/2017 2:49:44 AMtime=“2017-05-25T00:49:44Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Downloading”
5/25/2017 2:49:45 AMtime=“2017-05-25T00:49:45Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Download complete”
5/25/2017 2:49:45 AMtime=“2017-05-25T00:49:45Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Downloading”
5/25/2017 2:49:46 AMtime=“2017-05-25T00:49:46Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Verifying Checksum”
5/25/2017 2:49:46 AMtime=“2017-05-25T00:49:46Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Download complete”
5/25/2017 2:49:46 AMtime=“2017-05-25T00:49:46Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Downloading”
5/25/2017 2:49:46 AMtime=“2017-05-25T00:49:46Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Verifying Checksum”
5/25/2017 2:49:46 AMtime=“2017-05-25T00:49:46Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Download complete”
5/25/2017 2:49:46 AMtime=“2017-05-25T00:49:46Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Downloading”
5/25/2017 2:49:47 AMtime=“2017-05-25T00:49:47Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Verifying Checksum”
5/25/2017 2:49:47 AMtime=“2017-05-25T00:49:47Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Download complete”
5/25/2017 2:49:47 AMtime=“2017-05-25T00:49:47Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Extracting”
5/25/2017 2:49:50 AMtime=“2017-05-25T00:49:50Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Pull complete”
5/25/2017 2:49:50 AMtime=“2017-05-25T00:49:50Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Extracting”
5/25/2017 2:49:50 AMtime=“2017-05-25T00:49:50Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Pull complete”
5/25/2017 2:49:50 AMtime=“2017-05-25T00:49:50Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Extracting”
5/25/2017 2:49:50 AMtime=“2017-05-25T00:49:50Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Pull complete”
5/25/2017 2:49:50 AMtime=“2017-05-25T00:49:50Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Extracting”
5/25/2017 2:49:50 AMtime=“2017-05-25T00:49:50Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Pull complete”
5/25/2017 2:49:50 AMtime=“2017-05-25T00:49:50Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Extracting”
5/25/2017 2:49:51 AMtime=“2017-05-25T00:49:51Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Pull complete”
5/25/2017 2:49:51 AMtime=“2017-05-25T00:49:51Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Digest: sha256:d33076b0bb66b58931f932a8d172ac9fa9fc58f39ec2ff588e5417696573d219”
5/25/2017 2:49:51 AMtime=“2017-05-25T00:49:51Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull, transitioning: yes: Status: Downloaded newer image for registry.gitlab.com/arstecnica/ytefas/ytefas_db:porto626”
5/25/2017 2:49:51 AMtime=“2017-05-25T00:49:51Z” level=info msg=“rancher id : Image with docker id [sha256:33bb3a5720158d2dce48f2cba48dcefc5d42485dcca56d5ea193bab06bf07f61] has been pulled”
5/25/2017 2:49:51 AMtime=“2017-05-25T00:49:51Z” level=info msg=“Reply: c18ec9c5-0be6-4f03-83ca-6b6caf6518d3, compute.instance.pull, :resourcePull”
5/25/2017 2:49:51 AMtime=“2017-05-25T00:49:51Z” level=info msg="Received event: Name: compute.instance.pull, Event Id: d23a511a-9ded-4571-8591-51774297fec2, Resource Id: "
5/25/2017 2:49:51 AMtime=“2017-05-25T00:49:51Z” level=info msg=“rancher id : Image with docker id has been pulled”
5/25/2017 2:49:51 AMtime=“2017-05-25T00:49:51Z” level=info msg=“Reply: d23a511a-9ded-4571-8591-51774297fec2, compute.instance.pull, :resourcePull”
5/25/2017 2:49:52 AMtime=“2017-05-25T00:49:52Z” level=info msg=“Received event: Name: storage.image.activate, Event Id: 3e550b18-e3fd-40ea-93a1-fd3cc3e3a09a, Resource Id: 1ispm275”
5/25/2017 2:49:52 AMtime=“2017-05-25T00:49:52Z” level=info msg=“Reply: 3e550b18-e3fd-40ea-93a1-fd3cc3e3a09a, storage.image.activate, 1ispm275:imageStoragePoolMap”
5/25/2017 2:49:52 AMtime=“2017-05-25T00:49:52Z” level=info msg=“Received event: Name: storage.volume.activate, Event Id: ec15c570-d42d-44f5-bb53-b9a5b4a368f5, Resource Id: 1vspm391”
5/25/2017 2:49:52 AMtime=“2017-05-25T00:49:52Z” level=info msg=“Reply: ec15c570-d42d-44f5-bb53-b9a5b4a368f5, storage.volume.activate, 1vspm391:volumeStoragePoolMap”
5/25/2017 2:49:52 AMtime=“2017-05-25T00:49:52Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: f650fef0-4300-4174-af0d-f5a55fe33b67, Resource Id: 1ihm276”
5/25/2017 2:49:55 AMtime=“2017-05-25T00:49:55Z” level=info msg=“Reply: f650fef0-4300-4174-af0d-f5a55fe33b67, compute.instance.activate, 1ihm276:instanceHostMap, transitioning: yes: Pulling from arstecnica/ytefas/ytefas_db”
5/25/2017 2:49:55 AMtime=“2017-05-25T00:49:55Z” level=info msg=“Reply: f650fef0-4300-4174-af0d-f5a55fe33b67, compute.instance.activate, 1ihm276:instanceHostMap, transitioning: yes: Digest: sha256:d33076b0bb66b58931f932a8d172ac9fa9fc58f39ec2ff588e5417696573d219”
5/25/2017 2:49:55 AMtime=“2017-05-25T00:49:55Z” level=info msg=“Reply: f650fef0-4300-4174-af0d-f5a55fe33b67, compute.instance.activate, 1ihm276:instanceHostMap, transitioning: yes: Status: Image is up to date for registry.gitlab.com/arstecnica/ytefas/ytefas_db:porto626”
5/25/2017 2:49:56 AMtime=“2017-05-25T00:49:56Z” level=info msg=“rancher id [277]: Container with docker id [e7ecc093ac42bb40c8f5aa395ed24369bbbbd4df887f1b636fdd3b8b0c7ec43c] has been started”
5/25/2017 2:49:56 AMtime=“2017-05-25T00:49:56Z” level=info msg=“Reply: f650fef0-4300-4174-af0d-f5a55fe33b67, compute.instance.activate, 1ihm276:instanceHostMap”

7

This time i added labels with the commit sha inside and the problem is “proved”, adding a --force-recreate in a second deploy helped: the correct images have been pulled. Tell me if you need more info

8

Have tried this with 1.6.5, and the only way I can get the image to be re-pulled if the service definition hasn’t changed is to use the --force-upgrade flag. Although this “works”, I think it is far from ideal as it recreates the container even if the image has not changed.

I think a solution that allows the image/container to be updated only if it has changed (even if the service definition has not changed) would be very helpful. This issue seems to be relevant: https://github.com/rancher/rancher/issues/5516