The only thing that gives you a GUI is the main Rancher itself, which runs as a container either in Docker or in an existing Kubernetes cluster. So you’re talking about using at least 2 things.
Rancher: UI and API for creating and managing multiple clusters. Runs as a container somewhere.
RKE: Command line binary that creates a Kubernetes clusters out of a set of remote machines that are running Docker.
K3s: Command line binary that creates a Kubernetes cluster out of the local machine it’s run on with no other dependencies (does not need or use Docker). Can be pointed at an existing k3s node to join it and make multi-node clusters.
RancherOS and k3os are operating systems. You download an ISO and can boot that in proxmox and install to virtual disk. RancherOS gives you a machine with docker installed. K3os gives you a machine with a running kubernetes cluster via k3s. Since everything we do involves Kubernetes, docker/rancheros are more the past and k3s/k3os the future.
Docker-machine (and our fork rancher-machine) is a command line binary which talks to different providers (or addon plug-ins for more choices) to create a virtual machine, install an operating system (boot2docker by default) and install docker on it. You can use this to tell proxmox to make a VM running RancherOS. But not k3os, because it does not use or install Docker.