Hey, folks.
I’m working on a PR for the Cattle Jenkins Catalog entry and I’ve apparently goofed something as I’m not seeing changes in my Catalog entry in my custom Catalog (for anyone interested : https://github.com/nrvale0/community-catalog).
Q1: Can someone point me to a log files, either in the server console or perhaps via docker logs <container> where I might spot problems on custom Catalog refresh?
Q2: Related…are there tools for syntax checking, linting, etc the various components of a Catalog during development? In the Puppet module world that would look something like:
- bundle exec rake syntax
- bundle exec rake lint
- bundle exec rake test (syntax -> lint -> lots of other stuff)
I’ve done similar things in Python projects with the “invoke” tool from GH:pyinvoke/invoke:
- invoke syntax
- invoke lint
- invoke test (syntax -> lint -> lots of other stuff)
If that tooling does not currently exist would it be better to go down the path of developing it with Python? Go?
Thx
Answering my own questions again… 
A1: The rancher/server interface for adding repos for custom Catalogs does not allow you to specify the desired branch in the repo. I was developing on a feature branch which was not named ‘master’. I was able to get around this by tweaking the ‘default branch’ in GitHub to be my feature branch, changing the name of the repo in the server interface to a purposefully broken URL, and then changing back to the correct URL thus triggering another download of the repo.
Also, FWIW, I was able to see some logs for Catalog-y stuff in /var/log on RancherOS but not enough to trouble shoot this particular issue.
Q2 is still open though.
A1: Your solution to this problem is interesting, I’ve not seen anyone else adjusting the default branch.
Typically, I run rancher/server locally, bind mount my catalog repo into that container, add the catalog as a FS path in Rancher Settings, and lower the polling frequency to something reasonable for local development, maybe 10 seconds (inspect the rancher-catalog-service running inside rancher/server, there is a -refreshInterval flag initially set to 300 seconds). That allows me to work on a specific branch for development. You still have to commit your changes in order for Rancher to pick it up, though, and it is not always apparent if your catalog got updated. A file watcher implementation would be MUCH better. We accept pull requests 
A2: Short answer: not really
There are a few python tests that we run to verify simple things, it might expose syntax problems but not in a very helpful way. We run these as Drone builds to enforce a few semantics in our catalog, but it is not a full featured syntax checker.
You could use docker-compose to look for compilation errors in the docker-compose.yml file (python). This doesn’t help us with rancher-compose.yml, though…
Better yet, we could implement some schema validation in our rancher-compose tool (golang). I would recommend using this project as a starting point.
2 Likes
LLParse,
Only because I’m more comfortable with Python than Go at the moment, I will likely prototype something w/ Python and post to the forums for feedback. If I get +1’s it’ll make for a good excuse to learn Go and re-implement.
Thanks for your bind mount trick.
BTW, I suppose the UI repo would be the one for a PR to add a field for Git branch name into the console?
@nrvale0,
Sounds like a plan! I would love to have this sort of tooling for catalog work. If you get +1’s we might get some folks at Rancher to lend a helping hand.
The git branch functionality I would also love to use for working with remote Rancher clusters. It will require a change to the UI and the rancher-catalog-service.
It occurs to me that you might get away with using the existing UI, and instead inventing some special syntax for describing a git branch which could be parsed in the catalog service.
The FS trick should go into documentation! This is something many people might not come about.
In more recent versions, we’ve added a Refresh button to the catalog UI screen so you can click that now.
We also support overriding the ‘Rancher Certified’ and ‘Community Contributed’ repositories by defining custom catalogs with the respective names ‘library’ and ‘community’.
Hang tight, catalog branch support is also coming to an official release in the near future. 
1 Like
Rather than branch, maybe working directory support would be nice (for local developement) so you can easily test different versions by simply checking them out… Instead of ammending to the same commit over and over again.
I wholeheartedly agree. At the very least, we should document the limitations along with FS trick.
I still intend to work on the tooling I mentioned above re linting etc but as to fast iteration during Catalog template development I’ve take to using rancher-compose --env-file . That’s at least helpful for Cattle. As for other orchestrators…not so much.
Heres another gotcha (which at least fit’s under the title of this thread):
When you do confd and need the metadata service,
you can join the docker to the rancher network with the docker run -l io.rancher.networking=true, this would make the metadataservice available and you can test your bind mounted confd templates.
However on docker-for windows, this did not work, so I ended up creating a regular stack paying attention to the special bind mounting syntax of docker-for-windows to get everything in sync. Awkward, but works.