Rancker Kuberntes Custom Cluster deployment Nginx Ingress Controller on CrashLoopBackOff Release:nginx-0.25.1-rancher1

I deployed a custom Kubernetes Cluster using a Rancher Single Node installation

Kubernetes Version 1.15.2
Rancher Version 2.3.2
Nginx release: nginx-0.25.1-rancher1

I’m using docker behind a proxy

My issue is with the nginx-ingress controller containers, they are permanently on a CrashLoopBackOff status.

kubectl get pods -n ingress-nginx
NAME                                    READY   STATUS             RESTARTS   AGE
default-http-backend-5bcc9fd598-jcp6z   1/1     Running            1          24h
nginx-ingress-controller-g8hll          0/1     CrashLoopBackOff   439        24h
nginx-ingress-controller-pghqs          0/1     CrashLoopBackOff   437        24h

This is my Cluster config file:
#
# Cluster Config
#
answers: {}
docker_root_dir: /var/lib/docker
enable_cluster_alerting: false
enable_cluster_monitoring: true
enable_network_policy: false
local_cluster_auth_endpoint:
enabled: true
name: cujae
#
# Rancher Config
#
rancher_kubernetes_engine_config:
addon_job_timeout: 30
authentication:
strategy: x509|webhook
authorization: {}
bastion_host:
ssh_agent_auth: false
cloud_provider: {}
ignore_docker_version: true
#
# # Currently only nginx ingress provider is supported.
# # To disable ingress controller, set provider: none
# # To enable ingress on specific nodes, use the node_selector, eg:
# provider: nginx
# node_selector:
# app: ingress
#
ingress:
provider: nginx
kubernetes_version: v1.15.5-rancher1-2
monitoring:
provider: metrics-server
#
# If you are using calico on AWS
#
# network:
# plugin: calico
# calico_network_provider:
# cloud_provider: aws
#
# # To specify flannel interface
#
# network:
# plugin: flannel
# flannel_network_provider:
# iface: eth1
#
# # To specify flannel interface for canal plugin
#
# network:
# plugin: canal
# canal_network_provider:
# iface: eth1
#
network:
options:
flannel_backend_type: vxlan
plugin: calico
restore:
restore: false
#
# services:
# kube-api:
# service_cluster_ip_range: 10.43.0.0/16
# kube-controller:
# cluster_cidr: 10.42.0.0/16
# service_cluster_ip_range: 10.43.0.0/16
# kubelet:
# cluster_domain: cluster.local
# cluster_dns_server: 10.43.0.10
#
services:
etcd:
backup_config:
enabled: true
interval_hours: 12
retention: 6
safe_timestamp: false
creation: 12h
extra_args:
election-timeout: ‘5000’
heartbeat-interval: ‘500’
gid: 0
retention: 72h
snapshot: false
uid: 0
kube-api:
always_pull_images: false
pod_security_policy: false
service_node_port_range: 30000-32767
kube-controller: {}
kubelet:
fail_swap_on: false
kubeproxy: {}
scheduler: {}
ssh_agent_auth: false

This is one of the containers logs:

kubectl logs -n ingress-nginx nginx-ingress-controller-g8hll 
E1113 15:55:47.987155       6 main.go:165] unexpected error obtaining NGINX version: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       nginx-0.25.1-rancher1
  Build:         
  Repository:    https://github.com/rancher/ingress-nginx.git
  N/A
-------------------------------------------------------------------------------

W1113 15:55:47.987908       6 flags.go:221] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W1113 15:55:48.200592       6 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1113 15:55:48.200880       6 main.go:183] Creating API client for https://10.43.0.1:443
I1113 15:55:48.212334       6 main.go:227] Running in Kubernetes cluster version v1.15 (v1.15.5) - git (clean) commit 20c265fef0741dd71a66480e35bd69f18351daea - platform linux/amd64
I1113 15:55:48.215949       6 main.go:91] Validated ingress-nginx/default-http-backend as the default backend.
I1113 15:55:48.894786       6 main.go:102] Created fake certificate with PemFileName: /etc/ingress-controller/ssl/default-fake-certificate.pem
I1113 15:55:48.931734       6 nginx.go:274] Starting NGINX Ingress controller
I1113 15:55:48.960753       6 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"1b33c0ef-0e42-4d5b-aec2-6127d4ad72f7", APIVersion:"v1", ResourceVersion:"542", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/nginx-configuration
I1113 15:55:48.964448       6 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"01406293-14b4-430d-b42a-7cf5b23a73b6", APIVersion:"v1", ResourceVersion:"545", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services
I1113 15:55:48.964508       6 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"3b6e5029-1f98-4dff-a499-5d4acdbbe70a", APIVersion:"v1", ResourceVersion:"546", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services
I1113 15:55:50.132155       6 nginx.go:318] Starting NGINX process
I1113 15:55:50.132183       6 leaderelection.go:235] attempting to acquire leader lease  ingress-nginx/ingress-controller-leader-nginx...
I1113 15:55:50.132952       6 controller.go:133] Configuration changes detected, backend reload required.
I1113 15:55:50.138605       6 leaderelection.go:245] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I1113 15:55:50.138713       6 status.go:86] new leader elected: nginx-ingress-controller-g8hll
E1113 15:55:50.363819       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:55:50.363864       6 queue.go:130] requeuing initial-sync, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
I1113 15:55:53.466414       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:55:53.689303       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:55:53.689329       6 queue.go:130] requeuing kube-system/metrics-server, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
I1113 15:55:56.799713       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:55:57.022714       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:55:57.022751       6 queue.go:130] requeuing default/kubernetes, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E1113 15:55:57.071861       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1113 15:56:00.133039       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:00.356490       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:56:00.356521       6 queue.go:130] requeuing ingress-nginx/default-http-backend, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
I1113 15:56:03.466347       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:03.690128       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:56:03.690157       6 queue.go:130] requeuing kube-system/kube-controller-manager, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E1113 15:56:05.285981       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1113 15:56:06.799663       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:07.023039       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:56:07.023069       6 queue.go:130] requeuing kube-system/kube-dns, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E1113 15:56:07.072056       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1113 15:56:10.133111       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:10.355413       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:56:10.355452       6 queue.go:130] requeuing kube-system/kube-scheduler, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
I1113 15:56:13.466330       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:13.690602       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:56:13.690634       6 queue.go:130] requeuing ingress-nginx/nginx-ingress-controller-pghqs, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E1113 15:56:15.286079       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1113 15:56:16.799667       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:17.026462       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:56:17.026551       6 queue.go:130] requeuing ingress-nginx/nginx-ingress-controller-g8hll, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E1113 15:56:17.071928       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1113 15:56:20.133066       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:20.356144       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:56:20.356181       6 queue.go:130] requeuing configmap-change, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
I1113 15:56:23.466368       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:23.690407       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:56:23.690438       6 queue.go:130] requeuing configmap-change, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E1113 15:56:25.286204       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1113 15:56:25.324233       6 main.go:154] Received SIGTERM, shutting down
I1113 15:56:25.324265       6 nginx.go:401] Shutting down controller queues
I1113 15:56:25.324296       6 status.go:117] updating status of Ingress rules (remove)
I1113 15:56:25.442537       6 status.go:127] leaving status update for next leader (2)
I1113 15:56:25.442569       6 nginx.go:417] Stopping NGINX process
I1113 15:56:25.660041       6 main.go:158] Error during shutdown: signal: illegal instruction (core dumped)
I1113 15:56:25.660085       6 main.go:162] Handled quit, awaiting Pod deletion
E1113 15:56:27.071711       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory

I seems to be be seeing the same issue, on rancher 2.3.3 single instance install, k8s v1.16.3-rancher-1.

I also has Istio installed via racnher, traffic seems to work properly.

Anyone has idea how to debug that?

In my case, it was caused because the latest versions of Nginx use microprocessor modules that are not available on old servers, refer to this post https://github.com/rancher/rancher/issues/23307

I couldn’t get new servers, my workaround was to install an older Rancher version (2.2.0) and it worked perfectly because it uses an older Nginx version

hum… then I might be in the same case… my bare metal is really old 1U (Dell CS-24SC)

Fortunately, I switched to metalLB and can just disabled the nginx ingress… however, I am still having issue with metalLB, but that’s another problem.