Hi,
Long time listener, first time caller. I’ve been tasked with the infrastructure component of an elastic docker environment. I am heavily focused on Rancher right now as a smooth management component with easily implemented hooks to enroll host devices. I’m also currently using RancherOS as the minimal network boot framwork for my hosts. I’ve got a host of things on my wish list but for now I’m focusing on a single debilitating limitation with the cloud-config. On my hosts I have a 1G interface with DHCP that is used for ICMP and iPXE booting RancherOS. During boot the host reaches out to grab a cloud config file that is (for convenience) also hosted from my iPXE boot server. The cloud-config holds network config items that enable, bond, and configure LACP across two 10G interfaces on the hosts. Lastly, it runs the command provided in the Rancher management GUI to auto-enroll the host within the infrastructure stack for the given environment.
The problem:
The hosts are enrolling with the Rancher management tool with connection info from the 1G ‘boot’ network rather than the 10G ‘service layer’ network. This means that all container activity also crosses 1G instead of 10G. This will be a big issue once we reach production with this technology.
Mitigating Steps:
I’ve attempted cloud-config items to disable (ifdown) the 1G interface, wipe its config, set it to a static non-functional subnet, etc. The result was continued registration with Rancher on the 1G interface that ignored the wipe config.
I’ve made the Rancher management tool unavailable to the 1G network. This resulted in no host registration and unavailability.
I’ve attempted to reload the networking services at the end of the cloud config. I have no idea if this command actually ran but I didn’t note any change in behavior.
Core Issue:
The cloud-config documentation and standards of use are REALLY lacking. Usually I would fall back on community posts for guidance but it seems like elastic/cloud config deployments aren’t quite popular or widely talked about with RancherOS or even CoreOS that I can see. Does anyone have any better resources for cloud-config and/or network layer manipulation on RancherOS?