Random Timeouts in Ingress nginx affecting Cluster Performance

1

Hello:

Has any one had an issue similar to below? If so, how did you resolve this issue?

Issue

• Very slow performance on a fresh installation running minio
    ◦ Expected speed: 1.5GB/sec
    ◦ Actual speed: 350MB/sec
• Discovered that when HTTP request was routed to certain nodes, a timeout occurs

Steps to re-create:

  1. Create any HTTP server deployment
  2. Expose the HTTP service through ingress
  3. Access the HTTP server using the ingress URL from browser or command line

Configuration

Cluster

• Kubernetes cluster launched via Rancher
• 10+ nodes

Software

• Red Hat Enterprise Linux 3.10.0-1127.19.1.el7.x86_64
• Docker v1.13.1
• Kublet v1.17.2
• Kube Proxy v1.17.2
• Rancher v2.3.5

Tests Performed

• Test: curl -v -k [url entry point]
    ◦ Result
        ▪ connected to [entry point] port 443 (#0)
        ▪ ALPN, offering h2
        ▪ ALPN, offering http/1/1
        ▪ successfully set certificate very locations 
        ▪ ....
        ▪ Capath: none
        ▪ TLSv1.2 (OUT), TLS handshake, Client hello (1):
        ▪ Operation timed out after 300002 milliseconds with 0 of 0 bytes received
        ▪ Closing connection 0
• Test: Point ingress nginx to a single node
    ◦ Result
        ▪ So far, works fine
• Test: Restart nginx controller on a node identified as bad
    ◦ Result
        ▪ Node works fine for an undetermined length of time