Custom kubernetes set up

1

I’m trying to create a customer kubernetes orchestration service as we want to change the hard coded k8s service CIDR (10.43.0.0/16) and DNS address (10.43.0.10) to something else. I’ve forked the k8s infra-template (and called it k8s-custom but left the versioning the same) and updated the values in the template. However when I try to create an environment template I run into the issue that my template isn’t supported by the other default infra-services (healthcheck, network-services, etc).

Am I going to need to fork all of these and the base environment template as well, or is there a way I can get my custom k8s changes injected? Can I make it a new version and will it get merged with the library template in my rancher server when I add my custom repo?

Hope this makes sense

1 Like
2

Hey there, while I don’t know the answer… It’s very interesting what you are trying to do. Can you point me to these templates? I’d mess around too with kube system params, just out of curiosity.

4

I just pulled the template files from the catalog with the preview option but I think they are stored here:

The specific file we modified was docker-compose.yml.tpl - this contains all the options for running the containers that k8s requires. Unfortunately our changes are stored in a private server so I can’t post them here.

5

So to answer my own question I played around with it. If I call my custom template Kubernetes then it all works OK :slight_smile:

It shows up separately because it’s in a different catalog.

6

I’m just going to use this thread to document my experience trying to modify this. So I have the template up and running but the add on starter that rancher uses is still trying to set the DNS IP to 10.43.0.10. It’s set by the build script using this file:

Should be able to modify the run command of the addon-starter to do a quick and dirty replace in the dns service yaml though and go from there.

7

Awesome! Thanks for the pointer. I will try this too.

Happy it works for you!

8

It fascinates me how Kubernetes is just a Cattle template, as anything else.

9

Yeah it’s good. For what it’s worth here is the updated command for the addon starter:

{{- if eq .Values.ENABLE_ADDONS "true" }}
addon-starter:
    image: rancher/k8s:v1.6.6-rancher1-4
    labels:
        {{- if eq .Values.CONSTRAINT_TYPE "required" }}
        io.rancher.scheduler.affinity:host_label: orchestration=true
        {{- end }}
        io.rancher.container.create_agent: 'true'
        io.rancher.container.agent.role: environmentAdmin
    environment:
        KUBERNETES_URL: https://kubernetes.kubernetes.rancher.internal:6443
        REGISTRY: ${REGISTRY}
        INFLUXDB_HOST_PATH: ${INFLUXDB_HOST_PATH}
    command:
        - bash
        - -c
        - 'sed -i "s|10\.43|172\.19|g" /etc/kubernetes/addons/dns/kubedns-svc.yaml && sed -i "s|10\.43|172\.19|g" /etc/kubernetes/addons/heapster/heapster-deployment.yaml && addons-update.sh'
    links:
        - kubernetes
{{- end }}

Edited to show update to heapster as well

1 Like