Clusters won't create - role cannot be found

Rancher 2.x
1

Getting the following in the logs:

2018/11/01 18:27:08 [INFO] [mgmt-auth-prtb-controller] Creating role admin in namespace p-86dbn
2018/11/01 18:27:08 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject u-appq5auq3l with role project-owner in namespace
2018/11/01 18:27:08 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace p-86dbn
2018/11/01 18:27:08 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace p-xqxq8
2018/11/01 18:27:08 [INFO] [mgmt-auth-prtb-controller] Creating role admin in namespace p-xqxq8
2018/11/01 18:27:08 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject u-appq5auq3l with role admin in namespace
2018/11/01 18:27:08 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject u-appq5auq3l with role project-owner in namespace
2018/11/01 18:27:08 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject u-appq5auq3l with role project-owner in namespace
2018/11/01 18:27:08 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject u-appq5auq3l with role admin in namespace
2018/11/01 18:27:08 [INFO] [mgmt-cluster-rbac-delete] Updating cluster c-6msx7
2018/11/01 18:27:08 [INFO] [mgmt-auth-prtb-controller] Updating clusterRoleBinding clusterrolebinding-58tc6 for cluster membership in cluster c-6msx7 for subject u-appq5auq3l
2018/11/01 18:27:08 [INFO] Provisioning cluster [c-6msx7]
2018/11/01 18:27:08 [INFO] Creating cluster [c-6msx7]
2018/11/01 18:27:08 [INFO] Starting create
2018/11/01 18:27:08 [INFO] VPC info provided, skipping create
2018/11/01 18:27:08 [INFO] Retrieving existing service role
2018/11/01 18:27:08 [ERROR] ClusterController c-6msx7 [cluster-provisioner-controller] failed with : error getting role: NoSuchEntity: The role with name AROAJLVEFOMODFEMLP7NI cannot be found.
	status code: 404, request id: bda81ef0-de03-11e8-986e-5fbe089ad211
2018/11/01 18:27:38 [INFO] Provisioning cluster [c-6msx7]
2018/11/01 18:27:38 [INFO] Creating cluster [c-6msx7]
2018/11/01 18:27:38 [ERROR] Cluster c-6msx7 previously failed to create

Not sure which role it’s looking for here, but it seems to be internal to Rancher & not EKS-related. Attempting to create this cluster as the admin/account owner.

** Edit - I should add that I was able to create/delete EKS clusters previously via Rancher without issue. I think this is something that may have started happening after the 2.1 upgrade, but not positive.

2

I found that the string above for the role was being interpreted by the system as the “service role,” as I found that string in: /var/lib/rancher/etcd/member/snap/db :

"serviceRole":"AROAJLVEFOMODFEMLP7NI"

The problem is that the service role that I chose (& had previously worked) has no such string associated with it. Additionally, I just reconfirmed that the configured role (EKS-Resource-Mgr) is correctly configured, with both the AmazonEKSClusterPolicy & AmazonEKSServicePolicy attached.

To test, I created a cluster without choosing my pre-configured role, and the cluster was successfully provisioned. Rancher automatically created a role with identical permissions to my previously configured role, and used it to configure the cluster. HOWEVER, as another test, I deleted that cluster, and went back to create another cluster using that same, newly-created role… and it failed again. Same error (role cannot be found), but this time it was searching for a different string.

3

Turns out this is a known issue.