Rancher HA 2.4 Cert-Manager / Lets Encrypt 404

Rancher 2.x
1

Hi there,

I’m trying to get up and running with a rancher 2.4 installation following the instructions here: https://rancher.com/docs/rancher/v2.x/en/installation/k8s-install/helm-rancher/

I’m using K3’s with 2x ubuntu 18.04 nodes and have setup an nginx loadbalancer on a seperate ubuntu 18.04 node.

Setting things up with the default self signed rancher cert with cert manager seems to work fine, but when I try to get SSL going with Lets Encrypt I’m getting some issues, namely 404’s in the challenge response

kubectl describe challenge tls-rancher-ingress-977329062-64000749-2488832917 -n cattle-system

Name:         tls-rancher-ingress-977329062-64000749-2488832917
Namespace:    cattle-system
Labels:       <none>
Annotations:  <none>
API Version:  acme.cert-manager.io/v1alpha3
Kind:         Challenge
Metadata:
  Creation Timestamp:  2020-05-03T05:51:36Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
  Owner References:
    API Version:           acme.cert-manager.io/v1alpha2
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  tls-rancher-ingress-977329062-64000749
    UID:                   1e9f4a0f-8c34-45b4-8e38-8a35db829d45
  Resource Version:        141461
  Self Link:               /apis/acme.cert-manager.io/v1alpha3/namespaces/cattle-system/challenges/tls-rancher-ingress-977329062-64000749-2488832917
  UID:                     e14b5d0f-9db7-4be1-b409-a8066b7930de
Spec:
  Authz URL:  https://acme-v02.api.letsencrypt.org/acme/authz-v3/4324735091
  Dns Name:   fake.domain.com
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   Issuer
    Name:   rancher
  Key:      mqTWtd9oZXXDxuDCOKJyr7OBf5KlkrNp3kuZr8Edmqs.UBjGB40VIY3MOiTrKFUg8Z6V4-_nhIInLBvkcKo6BoQ
  Solver:
    http01:
      Ingress:
        Class:  nginx
  Token:        mqTWtd9oZXXDxuDCOKJyr7OBf5KlkrNp3kuZr8Edmqs
  Type:         http-01
  URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/4324735091/-J9sjw
  Wildcard:     false
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for http-01 challenge propagation: wrong status code '404', expected '200'
  State:       pending
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Started    12m   cert-manager  Challenge scheduled for processing
  Normal  Presented  12m   cert-manager  Presented challenge using http-01 challenge mechanism

Appreciate any insights or ideas, thanks!

1 Like
2

A couple of other odd things I’ve noticed:

It looks like there are other acme solver objects being referenced in my cert-manager logs that aren’t running, is this normal?

When I run kubectl -n cattle-system describe ingress
I get a solver that isn’t running either

3

This is probably caused by https://github.com/rancher/rancher/issues/26323

4

That was it - I upgraded to 2.4.3rc7 from 2.4.x and it appears to be fixed. Thanks for your help!