In May 2019, I saw a demo of Rancher 2.3 Beta at rancher.com. There a Linux/Windows Mixed Cluster was installed and working. Now I’m trying to rebuild the Final 2.3.2 for an important project. Unfortunately the Linux-Worker-Nodes are locked with “cattle.io/os=linux:NoSchedule” for the Container-Deploymend. Is it possible to set up a mixed cluster with Rancher 2.3.x? So it was shown in the Rancher.com advertisement! Or is there a trick for the configuration?
THX
Tino Brandt
The early betas did allow mixing worker nodes, but there were various issues with that so what shipped in 2.3.0 does not support mixing. Linux nodes in a Windows cluster are only meant for running system services.
Thanks for the quick reply 
Is there a plan that you can use the mixed mode at any time?
It’s not exactly my area but not that I’m aware of. The most basic problem is just trying to do simple kubectl apply deployments no longer works because everything has to have a scheduling rule or toleration to select only the right OS for the thing you want to run.
Thank you, I understand
My problem is that I’m working on a project where I need Linux and Windows containers. In these containers are services that need to communicate between Linux- and Windows-Services. If the Linux containers are in one cluster and the Windows containers are in the other cluster, then I have a no idea about communication. I think windows is very limited with the network communication!
Do you know a description (Link) of how to solve the problem?
THX
Hi Vincent,
is it possible to have the prefered node selector to Linux instead of Windows so to remove the taint of cattle.io/os=linux and add the taint to the Windows nodes cattle.io/os=windows ?
Because most of the Apps like Longhorn are only available for linux nodes. And the only Windows stuff that needs to run on our Kubernetes cluster is our own software so we can set a taint tolerance on that one
With kind regards,
JW
I also need that for having Jenkins agent on linux and windows. The application will run on windows node which should not be the case. It cannot create the application due to some port mapping. It should be executed on linux nodes and best on master node.
Has there been any progress / decision with this option? The recommendation from Kubernetes is to taint Windows nodes rather than both - https://kubernetes.io/docs/setup/production-environment/windows/user-guide-windows-containers/#ensuring-os-specific-workloads-land-on-the-appropriate-container-host. The overhead to tolerate both for Linux and Windows seems a little over engineered.
The Linux machines in a Windows cluster are required to run etcd and various other system services (for which no Windows build exists). Otherwise from the user perspective we treat it as a “Windows-only” cluster.
You can override it if you want but we do not encourage mixing in Linux workloads in a Windows cluster, so the configuration reflects that.
Thanks for the extra info. Our goal is to use mixed node clusters. Is there any specifics you can share as to why Rancher does not recommend this? From Kubernetes’ perspective it does not sound like this approach is not recommended.
The simplest approach still sounds like to taint Windows nodes only so all Linux system services stay on Linux and that way you only need to tolerate explicitly for Windows.
Thanks for the responses on this.
More and more people start to try out Windows Containers with Kubernetes. As the process of adding Windows nodes following the Kubernetes docs is still very manual and tedious, Rancher could make a great difference here in my opinion.
There are a lot of benefits with running mixed workloads and as Windows Containers are still a niche, users almost always need some Linux workloads in their cluster because most of the existing applications like databases etc. are only available for Linux.
I think that making it hard for Rancher users due to the need to add tolerations to all those Linux workload, people will try to find alternatives although Rancher could be a superior solution.
There is a tremendous potential for heterogeneous workloads on the same cluster. Though Rancher is a very attractive choice for cluster management but is hard to choose when it comes to mixed workloads. In my particular use case, having a separate windows cluster is not an option as it requires modifying all Linux workloads and keep adding annoying toleration conditions.
Actually, it’s already possible to add windows nodes on a regular linux cluster and configure taints as you like.
Just create a Linux cluster using flannel network, or use the GUI to create a Windows cluster but editing the yaml before applying it:
Flannel network options
network:
mtu: 0
options:
flannel_backend_port: '4789'
flannel_backend_type: vxlan
flannel_backend_vni: '4096'
plugin: flannel
Set “windows prefered” as false
windows_prefered_cluster: false
Be aware that you won’t get any GUI support to add Windows nodes on this cluster.
You just have to replace the url, token and checksum on the standard join command line and run it on CMD
PowerShell -NoLogo -NonInteractive -Command "& {docker run -v c:\:c:\host rancher/rancher-agent:v2.4.8 bootstrap --server https://<<URL>> --token <<TOKEN>> --ca-checksum <<CHECKSUM>> --worker | iex}"
Don’t forget to add taints to all Windows nodes.
Hi, not sure if this is the most recent information regarding this topic?
It seems adding both Linux and Windows worker nodes in one cluster leads to taints being defined on the OS. While taints are a perfect way to pass hints to the scheduler, in fact a label would be sufficient. Each Helm chart should use a selector to make sure it gets scheduled on Linux. I consider it bad practice to enforce all helm chart builders to include tolerations for a taint on OS as well.
Is anyone working on this topic?