Hello guys,
we have some servers still on Rancher 1.6.x and on one of this was been renewed SSL cert and all nodes can’t communicate with server. Is there any way to temporary disable ssl and make release with new ca-certs?
THX
1.x has no built in TLS support, so you have nginx or similar in front of it doing TLS termination. One of LetsEncrypt’s root certificates expired today.
Hi sirbesir,
Did you find any workaround on your side?
That would help us tremendously, br,
Jeff
Yes! You have 100% true. We have Rancher behind Nginx with SSL termination. But the core problem is outdated package ca-certificates in container. I rebuild the rancher-agent container and found why is replaced with rancher-agent:v1.2.21 and found the same problem with network-services stack, ipsec etc…
I can simple copy docker and rancher compose files, create new containers witch concrete bases, but I don’t know if I will must solve the same issue with recreating containers to concrete version for Rancher Server.
Hello, Yes, I temporary disable the SSL termination on proxy in front of Rancher. But its not great fix 
Please, consult my reply for @vincent . If I will not have the trouble with running others container then I can try build my own.
Thanks for your Time. I really appreciate it.
Besir
We are trying in fact also to change the proxy rules to accept this, and to rebuild the agent with the certificate. In case you would be willing to share your rebuild image, that would be great. If we find something on our side on how-to I will also share.
Good luck
rancher-agent is there registry.gitlab.mondayfactory.cz/besir/rancher-agent BUT its hacked and ignore required version of Rancher. Tested on Rancher 1.6.30 and 1.6.28. Tonight I will test other images and try to create other containers. This will work with TLS, but another e.g. network-servicer will be failing. This is first step.
Thanks a lot for that, but get an error on your
We are also checking to use http but behind vpns then and with iptables limitations
@jfburr
What mistake?
Sorry works, that already great!
The “core problem” is that you’re still running versions of software (probably extending far beyond the Rancher components) that have not been patched or maintained or supported for years. Things will break, the container ecosystem was young in 1.x days and things move and get abandoned quickly (Docker supported versions for a total of 3 months then!). You’re eventually going to get turned into a botnet when one of the many unpatched vulnerabilities up and down the stack get exploited.
It would be a lot easier and safer to switch the cert being served up to a different ssl provider (like ZeroSSL, which does ACME just like LetsEncrypt), vs rebuilding every image and getting them to run everywhere.
For anybody else, I have no reason to trust or distrust @sirbesir, but frankly running a random image someone from the internet gave you as effectively root on every node to fix a problem is insane.
Fully agree, this is clear, and then comes the reality where we try to do our best in our choices and with our ressources. Clearly your security case is to be highly considered and the point is more that if someone else manages a POC to rebuild we should be able too…
About the cert beeing served, we did update the caddy service to latest in front of the rancher v1 server, but that did not make it, you mean that a proxy in front of the server with Zerossl would avoid the re-building of the agent?
Yes! Absolutely yes! You have true. This situation is my big ass kick. We have in long term plan migrate to k8s, but… You know 
… Is it woks? Don’t repair it.
For anybody else, I have no reason to trust or distrust @sirbesir, but frankly running a random image someone from the internet gave you as effectively root on every node to fix a problem is insane.
I have actualy preparing complet public repo with public build for this repo where will be to see what happening with container.
AND if is anybody scaried to use our image, I mean… you can buy any SSL cert and solve this problem with any else authority. This solution will be temporary fix.
Yes, LetsEncrypt’s cert chain is X3 → X1 → R3 → your-issued-cert. X1 is both a self-signed “root” and cross-signed by an older X3, so that it could work during the probably several years they worked to get X1 directly bundled into the ca-cert stores of all the major OSes and browsers. But the Rancher 1.x images do not contain the X1 cert to know to trust things issued by it. So verification continues up the chain to X3, which it does know about, but is no longer valid because it expired yesterday.
Using any cert from another provider will come from a different chain that is presumably still valid (they are typically for decades) is the simplest way to fix this. ZeroSSL supports the ACME protocol and does all the same free 90-day auto-updating cert stuff as LetsEncrypt, but with their own separate CA (which is good into 2030). So it is relatively easy to drop-in in place of LetsEncrypt.
Nothing against you personally, but nobody should trust a random image, especially for such a privileged context. I’m sure it does exactly what you say now but for all anybody knows it could get re-pulled next month and be a bitcoin miner or a remote shell.
Thanks for the efforts, would be great to see the images.
On our side we try also to switch to zerossl served by caddy, works better (initial handshake with agent works), but then the host does not show up, investigating…
Hi there,
does anybody have a solution to this ?
I tried a differen cert for rancher server - didn’t work, host don’t connect
I tried commenting out
# !mozilla/DST_Root_CA_X3.crt
in agent image didn’t work too.
Wer are on a desperate search for getting a quickfix for this problem, any help appreciated.
Hi,
we moved from lets encrypt to zerossl on caddy V2 reverse proxy.
Howdy all. my team and I are also getting hammered by the Let’s Encrypt issue. We’ve applied some software patches, but really - the best answer (as suggested several times in this forum thread) is to swap from Let’s Encrypt to another ACME SSL provider, like ZeroSSL. On that…
We’re running Rancher 2.6.8, managing four clusters. Only one cluster is adversely affeccted by the LE root SSL issue (and specifically only one namespace in one project in that cluster)…
That teammate who set this up is long-gone, and I’m an SSL novice (if that). I’ve inherited this rather complex setup and am doing my best to figure out what to do.
I read an intro article on Encrypting HTTP Communication then Rancher Docs: Updating a Private CA Certificate (and got a bit confused and terrified).
Then I discovered the cert-manager Dashboard and tried following the instructions in Alternative ACME via cert-manager | by Mark McWhirter | Medium and used the dashboard to add the YAML, but that didn’t work out.
My point? If one of y’all can give me some (more) pointers for how to switch from Let’s Encrypt to ZeroSSL, I’d sincerely apprecaite it. I can follow instructions and I’m not afraid to do read and do the work. If you need additional information, I’m happy to share it.
That article seems to cover everything… I converted our git mirror to use ZeroSSL without issue (so existing older installs would be able to pull our catalogs).
make sure you create the secret in the right format and in the right namespace, and look at the cert-manager’s pod logs and the status of the cert and cluster user to see what’s going on.
1 Like
Hi there, the best I can do is share part of my caddy configuration with zerossl rather than letsencrypt (by default). But notice that for us rancher is deployed within a container.
- Caddy conf (Caddyfile):
{
acme_ca https://acme.zerossl.com/v2/DV90
email your@email
}
your.rancher.com {
reverse_proxy rancher:8080 {
transport http {
tls_insecure_skip_verify
}
header_up Host {host}
header_up X-Real-IP {remote}
}
tls your@email
}
- And we did deploy this with docker-compose:
version: '2'
services:
caddy:
image: caddy:2.4.5
depends_on:
- rancher
ports:
- 80:80
- 443:443
volumes:
- /opt/caddy/Caddyfile:/etc/caddy/Caddyfile
- caddy2_data:/data
- caddy2_config:/config
environment:
- ACME_AGREE=true
rancher:
image: rancher/server:v1.6.30
restart: unless-stopped
volumes:
- /opt/rancher:/var/lib/mysql
volumes:
caddy2_data: {}
caddy2_config: {}
We still have an issue with the stats not beeing displayed within the containers in the rancher GUI (V1), due to a websocket handshake issue. But in general all required functions are working.