I deployed a custom Kubernetes Cluster using a Rancher Single Node installation
Kubernetes Version 1.15.2
Rancher Version 2.3.2
Nginx release: nginx-0.25.1-rancher1
I’m using docker behind a proxy
My issue is with the nginx-ingress controller containers, they are permanently on a CrashLoopBackOff status.
kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
default-http-backend-5bcc9fd598-jcp6z 1/1 Running 1 24h
nginx-ingress-controller-g8hll 0/1 CrashLoopBackOff 439 24h
nginx-ingress-controller-pghqs 0/1 CrashLoopBackOff 437 24h
This is my Cluster config file:
#
# Cluster Config
#
answers: {}
docker_root_dir: /var/lib/docker
enable_cluster_alerting: false
enable_cluster_monitoring: true
enable_network_policy: false
local_cluster_auth_endpoint:
enabled: true
name: cujae
#
# Rancher Config
#
rancher_kubernetes_engine_config:
addon_job_timeout: 30
authentication:
strategy: x509|webhook
authorization: {}
bastion_host:
ssh_agent_auth: false
cloud_provider: {}
ignore_docker_version: true
#
# # Currently only nginx ingress provider is supported.
# # To disable ingress controller, set provider: none
# # To enable ingress on specific nodes, use the node_selector, eg:
# provider: nginx
# node_selector:
# app: ingress
#
ingress:
provider: nginx
kubernetes_version: v1.15.5-rancher1-2
monitoring:
provider: metrics-server
#
# If you are using calico on AWS
#
# network:
# plugin: calico
# calico_network_provider:
# cloud_provider: aws
#
# # To specify flannel interface
#
# network:
# plugin: flannel
# flannel_network_provider:
# iface: eth1
#
# # To specify flannel interface for canal plugin
#
# network:
# plugin: canal
# canal_network_provider:
# iface: eth1
#
network:
options:
flannel_backend_type: vxlan
plugin: calico
restore:
restore: false
#
# services:
# kube-api:
# service_cluster_ip_range: 10.43.0.0/16
# kube-controller:
# cluster_cidr: 10.42.0.0/16
# service_cluster_ip_range: 10.43.0.0/16
# kubelet:
# cluster_domain: cluster.local
# cluster_dns_server: 10.43.0.10
#
services:
etcd:
backup_config:
enabled: true
interval_hours: 12
retention: 6
safe_timestamp: false
creation: 12h
extra_args:
election-timeout: ‘5000’
heartbeat-interval: ‘500’
gid: 0
retention: 72h
snapshot: false
uid: 0
kube-api:
always_pull_images: false
pod_security_policy: false
service_node_port_range: 30000-32767
kube-controller: {}
kubelet:
fail_swap_on: false
kubeproxy: {}
scheduler: {}
ssh_agent_auth: false
This is one of the containers logs:
kubectl logs -n ingress-nginx nginx-ingress-controller-g8hll
E1113 15:55:47.987155 6 main.go:165] unexpected error obtaining NGINX version: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: nginx-0.25.1-rancher1
Build:
Repository: https://github.com/rancher/ingress-nginx.git
N/A
-------------------------------------------------------------------------------
W1113 15:55:47.987908 6 flags.go:221] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W1113 15:55:48.200592 6 client_config.go:541] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1113 15:55:48.200880 6 main.go:183] Creating API client for https://10.43.0.1:443
I1113 15:55:48.212334 6 main.go:227] Running in Kubernetes cluster version v1.15 (v1.15.5) - git (clean) commit 20c265fef0741dd71a66480e35bd69f18351daea - platform linux/amd64
I1113 15:55:48.215949 6 main.go:91] Validated ingress-nginx/default-http-backend as the default backend.
I1113 15:55:48.894786 6 main.go:102] Created fake certificate with PemFileName: /etc/ingress-controller/ssl/default-fake-certificate.pem
I1113 15:55:48.931734 6 nginx.go:274] Starting NGINX Ingress controller
I1113 15:55:48.960753 6 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"1b33c0ef-0e42-4d5b-aec2-6127d4ad72f7", APIVersion:"v1", ResourceVersion:"542", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/nginx-configuration
I1113 15:55:48.964448 6 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"01406293-14b4-430d-b42a-7cf5b23a73b6", APIVersion:"v1", ResourceVersion:"545", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services
I1113 15:55:48.964508 6 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"3b6e5029-1f98-4dff-a499-5d4acdbbe70a", APIVersion:"v1", ResourceVersion:"546", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services
I1113 15:55:50.132155 6 nginx.go:318] Starting NGINX process
I1113 15:55:50.132183 6 leaderelection.go:235] attempting to acquire leader lease ingress-nginx/ingress-controller-leader-nginx...
I1113 15:55:50.132952 6 controller.go:133] Configuration changes detected, backend reload required.
I1113 15:55:50.138605 6 leaderelection.go:245] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I1113 15:55:50.138713 6 status.go:86] new leader elected: nginx-ingress-controller-g8hll
E1113 15:55:50.363819 6 controller.go:145] Unexpected failure reloading the backend:
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
W1113 15:55:50.363864 6 queue.go:130] requeuing initial-sync, err
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
I1113 15:55:53.466414 6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:55:53.689303 6 controller.go:145] Unexpected failure reloading the backend:
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
W1113 15:55:53.689329 6 queue.go:130] requeuing kube-system/metrics-server, err
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
I1113 15:55:56.799713 6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:55:57.022714 6 controller.go:145] Unexpected failure reloading the backend:
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
W1113 15:55:57.022751 6 queue.go:130] requeuing default/kubernetes, err
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
E1113 15:55:57.071861 6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1113 15:56:00.133039 6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:00.356490 6 controller.go:145] Unexpected failure reloading the backend:
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
W1113 15:56:00.356521 6 queue.go:130] requeuing ingress-nginx/default-http-backend, err
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
I1113 15:56:03.466347 6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:03.690128 6 controller.go:145] Unexpected failure reloading the backend:
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
W1113 15:56:03.690157 6 queue.go:130] requeuing kube-system/kube-controller-manager, err
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
E1113 15:56:05.285981 6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1113 15:56:06.799663 6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:07.023039 6 controller.go:145] Unexpected failure reloading the backend:
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
W1113 15:56:07.023069 6 queue.go:130] requeuing kube-system/kube-dns, err
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
E1113 15:56:07.072056 6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1113 15:56:10.133111 6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:10.355413 6 controller.go:145] Unexpected failure reloading the backend:
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
W1113 15:56:10.355452 6 queue.go:130] requeuing kube-system/kube-scheduler, err
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
I1113 15:56:13.466330 6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:13.690602 6 controller.go:145] Unexpected failure reloading the backend:
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
W1113 15:56:13.690634 6 queue.go:130] requeuing ingress-nginx/nginx-ingress-controller-pghqs, err
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
E1113 15:56:15.286079 6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1113 15:56:16.799667 6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:17.026462 6 controller.go:145] Unexpected failure reloading the backend:
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
W1113 15:56:17.026551 6 queue.go:130] requeuing ingress-nginx/nginx-ingress-controller-g8hll, err
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
E1113 15:56:17.071928 6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1113 15:56:20.133066 6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:20.356144 6 controller.go:145] Unexpected failure reloading the backend:
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
W1113 15:56:20.356181 6 queue.go:130] requeuing configmap-change, err
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
I1113 15:56:23.466368 6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:23.690407 6 controller.go:145] Unexpected failure reloading the backend:
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
W1113 15:56:23.690438 6 queue.go:130] requeuing configmap-change, err
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
E1113 15:56:25.286204 6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1113 15:56:25.324233 6 main.go:154] Received SIGTERM, shutting down
I1113 15:56:25.324265 6 nginx.go:401] Shutting down controller queues
I1113 15:56:25.324296 6 status.go:117] updating status of Ingress rules (remove)
I1113 15:56:25.442537 6 status.go:127] leaving status update for next leader (2)
I1113 15:56:25.442569 6 nginx.go:417] Stopping NGINX process
I1113 15:56:25.660041 6 main.go:158] Error during shutdown: signal: illegal instruction (core dumped)
I1113 15:56:25.660085 6 main.go:162] Handled quit, awaiting Pod deletion
E1113 15:56:27.071711 6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory