Letsencrypt Certificates not Updating

Hello together, I just realized that my certificates from letsencrypt are not updating anymore. Can someone explain me how I can force letsencrypt to update all certifcates in my cluster and how I could debug things that fail?
Thanks a lot for your help.

Best regards,
Christoph

I am currently running cert-manager 0.11.0.

kc get certificate --all-namespaces -o custom-columns=“NAMESPACE:.metdata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind,OLD FORMAT:.spec.acme”

NAMESPACE NAME OWNER OLD FORMAT
my-nutri-diary.de-crt Ingress
www.my-nutri-diary.de-crt Ingress
dev-my-nutri-diary-crt Ingress
dev.my-nutri-diary-crt Ingress
dev-my-nutri-diary-crt Ingress
dev-my-nutri-diary-crt Ingress
gitlab-tls-crt Ingress
minio-tls-crt Ingress
registry-tls-crt Ingress
prod.my-nutri-diary-crt Ingress
prod-my-nutri-diary-crt Ingress
prod-my-nutri-diary-crt Ingress
staging.my-nutri-diary-crt Ingress
staging-my-nutri-diary-crt Ingress
staging-my-nutri-diary-crt Ingress
I had a look into the logs of cert-manager. I got lots of error of the following kind. Can someone tell what these errors mean and what might go wrong here?

E0526 21:48:59.272036 1 sync.go:184] cert-manager/controller/challenges “msg”=“propagation check failed” “error”=“wrong status code ‘503’, expected ‘200’” “dnsName”=“gitlab.my-nutri-diary.de” “resource_kind”=“Challenge” “resource_name”=“gitlab-tls-crt-3702062889-1085796334-810123510” “resource_namespace”=“gitlab” “type”=“http-01”

26.5.2020 23:48:59 I0526 21:48:59.272104 1 controller.go:135] cert-manager/controller/challenges “level”=0 “msg”=“finished processing work item” “key”=“gitlab/gitlab-tls-crt-3702062889-1085796334-810123510”

26.5.2020 23:48:59 I0526 21:48:59.324791 1 controller.go:129] cert-manager/controller/challenges “level”=0 “msg”=“syncing item” “key”=“staging-nginx/staging-my-nutri-diary-crt-603438673-3589057284-2184142743”

26.5.2020 23:48:59 I0526 21:48:59.325186 1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod “level”=0 “msg”=“found one existing HTTP01 solver pod” “dnsName”=“staging.my-nutri-diary.de” “related_resource_kind”=“Pod” “related_resource_name”=“cm-acme-http-solver-vn64d” “related_resource_namespace”=“staging-nginx” “resource_kind”=“Challenge” “resource_name”=“staging-my-nutri-diary-crt-603438673-3589057284-2184142743” “resource_namespace”=“staging-nginx” “type”=“http-01”

26.5.2020 23:48:59 I0526 21:48:59.325282 1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService “level”=0 “msg”=“found one existing HTTP01 solver Service for challenge resource” “dnsName”=“staging.my-nutri-diary.de” “related_resource_kind”=“Service” “related_resource_name”=“cm-acme-http-solver-nfmdx” “related_resource_namespace”=“staging-nginx” “resource_kind”=“Challenge” “resource_name”=“staging-my-nutri-diary-crt-603438673-3589057284-2184142743” “resource_namespace”=“staging-nginx” “type”=“http-01”

Can you check https://gitlab.com/gitlab-org/gitlab/-/issues/194355 / https://github.com/jetstack/cert-manager/issues/2442 ?

Hello @superseb,
thanks for the link, I am not sure if this is my problem as all of my certificates are not updated anymore. It is not the gitlab one alone. It might be that I have broken something when I upgraded my cert-manager from version 7 to version 11 a while ago. I am now going to try to completely remove cert-manager and reinstall it again. As I have never done this before and as I am not very familiar with kubernetes yet, I have a question where you might be able to help me with. I would like to install the most recent cert-manager version but would like to keep all my certificate resources which are currently installed in the cluster. Can I simply backup the resources like described here:

Backup and Restore Resources - cert-manager Documentation

than remove cert-manager, install the latest version, and restore the backup of the resources like described in the link, or will this not work and I will break things as the resources were created against the old cert-manager version? Unfortunately, I have not found any descriptions yet that explain how to uninstall and reinstall cert-manager when an installation broke. The docs only described how to do upgrades of a correctly working installation. Are you aware of a description regarding what I am going to try which might help me to do things right ? Thanks for your help!

Best regards,
Christoph

@suberseb,
thanks once again for your support. I now upgraded cert-manager to version 0.15.0. This solved my problems. cert-manager started updating my certs again :slight_smile: