I recently upgraded to Rancher v2.2.2 so that I can use the added valuesYaml parameter in the payload for deployment requests of Catalog Apps via Rancher’s Rest API.
I have made changes to helm charts stored in our Custom Catalog (Helm Repo) and then run a refreshCatalog for Rancher via the UI. The helm charts fail to update and still show the older versions.
Our helm repo is in Amazon S3. I can see from S3 that the charts have been successfully updated so i suspect that the issue lies within Rancher. Specifically, I believe it is the caching mechanism that has been enabled.
Is there anyway we can do either of the following:
disable the caching mechanism in place
forcefully clear the catalog cache
I have tried removing the catalog and re-adding it with no success. Even more, I create a new Catalog with a new name but pointing to the same URL and the charts still show up as the older versions.
Any help is appreciated. At the moment, we have to wait at least a day to see changes made to the helm charts in Rancher.
Thanks so much,
Zach
UPDATE: it seems the issue only lies with some of my charts. Is there any reason why Rancher would choose to update some charts but not the others?
I would expect that helm repos are using the same cache mechanic in Rancher so it might be your trouble maker too.
My workaround until I’m able to upgrade to 2.2.4 is to redeploy the rancher deployment in my HA setup. This is purging the cache since the old pods with the old cache are deleted.
Did you ever get this working with v2.2.4? I have upgraded to v2.2.6 and am still struggling to see the helm charts update when previewing after updating them in my Custom Catalog Repo (Amazon S3) and performing a catalog refresh. I have actually opened a new issue that is specific to clearing the catalog-cache.
If I can’t disable it, I am probably going to have to clear the catalog-cache directory inside the rancher containers prior to each deployment (and refresh catalog) to ensure I deploy the most up-to-date helm charts.
It turns out the way we handle our helm charts for our dev environments is to just use a single chart versio for all our helm charts. Our helm charts for our test / ops environments will be using chart version that correspond to our builds.
Since the charts get staged with the same chart version for our dev environments, I believe rancher is not actually picking up those changes because is just assuming that a chart version will never have its contents changed.
We have just added a catalog-cache clearing mechanism for our dev deployments as a workaround.
I can try and explain what we’ve done. Basically, inside of the rancher containers, there is a directory: /var/lib/rancher/management-state/catalog-cache which acts as a cache for the helm chart repos. I have observed that the helm charts will not get automatically updated when performing a catalogRefresh IF the helm chart version has not changed. However, if we wipe that directory, the rancher servers are forced to re-pull/sync all the helm charts from the provided repo.
With this in mind, I created a workload that actually gets kubectl installed on it and configured to talk to the local cluster. When the workload is ran (Deployment of replica 1), it starts by immediately executing into each rancher container (assuming HA config) and removing that directory. It then sleeps the rest of the time. This means that I can just kill that pod whenever I want to refresh the cache because a new pod will spawn and perform that same startup sequence.
The other idea was to use service-tokens to have the workload perform that sequence without having to install kubectl. I am sure thats probably a better design, however, given this only occurs for us in DEV, its not the worst workaround.
At least for me, this problem would appear to still exist.
Changes to both git and http based repositories were not picked up regardless of refresh attempts at the menu points of the catalogs or repositories. rancher catalog refresh also did nothing.
I remember reading somewhere that this might have something to do with using a HA rancher installation?
We are running three rancher nodes (v2.5.1) on Ubuntu.