Rancher 2.2.3, ingress-nginx config and client_max_body_size

Rancher 2.x
1

I am testing hosting a docker repo on a cluster (nexus3) where routing to the docker service is using Ingress.

Everything seems to work except:
604cbde1a4c8: Pushing [==================================================>] 101.7MB/101.7MB
error parsing HTTP 413 response body: invalid character ‘<’ looking for beginning of value: “\r\n413 Request Entity Too Large\r\n\r\n

413 Request Entity Too Large

\r\n
nginx/1.15.6\r\n\r\n\r\n”

Looks like I need to modify the “client_max_body_size” parameter. According to stock Kubernetes docs, there are 2 ways to do that:

  1. Using an annotation on the Ingress object for a per-ingress setting
  2. Editing the configMap for nginx-config (or “nginx-configuration”, as Rancher seems to call it by default)

configMap currently:
apiVersion: v1
data:
client-max-body-size: 1024m
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{“apiVersion”:“v1”,“data”:null,“kind”:“ConfigMap”,“metadata”:{“annotations”:{},“labels”:{“app”:“ingress-nginx”},“name”:“nginx-configuration”,“namespace”:“ingress-nginx”}}
creationTimestamp: “2019-05-17T16:11:17Z”
labels:
app: ingress-nginx
name: nginx-configuration
namespace: ingress-nginx
resourceVersion: “3406200”
selfLink: /api/v1/namespaces/ingress-nginx/configmaps/nginx-configuration
uid: 66a355c2-78be-11e9-ad8b-0050569fab93

Neither option has worked for me. Whether I add a data item to nginx-configuration (and kill the pods to force them to re-load), or if I add the nginx.org/client-max-body-size annotation to my Ingress, I exec a /bin/sh shell on one of the nginx pods and always see under my ingress object’s config clause:

…
# enforce ssl on server side
if ($redirect_to_https) {

                            return 308 https://$best_http_host$request_uri;

                    }

                    client_max_body_size                    1m;

                    proxy_set_header Host                   $best_http_host;

                    # Pass the extracted client certificate to the backend

                    # Allow websocket connections
                    proxy_set_header                        Upgrade           $http_upgrade;

…

So how do you tune those parameters correctly with Rancher-deployed kubernetes?

2

OK, figured it out…

The ConfigMap attribute is “proxy-body-size” in this case and it works:

–
$ grep client_max_body_size *
grep: geoip: Is a directory
grep: lua: Is a directory
grep: modsecurity: Is a directory
grep: modules: Is a directory
nginx.conf: client_max_body_size 1024m;
nginx.conf: client_max_body_size 1024m;
nginx.conf: client_max_body_size 1024m;
nginx.conf: client_max_body_size 10m;
grep: owasp-modsecurity-crs: Is a directory
grep: template: Is a directory

3

I assume then that this is the “real” list of annotations and configmap entries:

https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/

as we’re using Kubernetes’ ingress-nginx, not NGINX Inc’s version of the kubernetes ingress controller.

4

Hi,

In my setup I have added the annotation nginx.ingress.kubernetes.io/proxy-body-size: 50m to the ingrtess definition, as documented in the link to the nginx ingress annotations you mentioned above. I have also had some success in the past using the following command:

$ kubectl patch configmap nginx-configuration -n ingress-nginx -p '{"data":{"proxy-body-size":"50m"}}'

but iirc this must be executed before creating the ingress. Therefore I have recently used the annotation only.

1 Like