Upgraded from rancher 2.1.0 to 2.2.7 and cluster still hungs in provisioning state

After the upgrade we are unable to access the clusters again:

Error shown is :

Exit status 1, unable to recognize “management-statefile_path_redacted”: Unauthorized unable to recognize “management-statefile_path_redacted”: Unauthorized unable to recognize “management-statefile_path_redacted”: Unauthorized unable to recognize “management-statefile_path_redacted”: Unauthorized unable to recognize “management-statefile_path_redacted”: Unauthorized unable to recognize “management-statefile_path_redacted”: Unauthorized unable to recognize “management-statefile_path_redacted”: Unauthorized unable to recognize “management-statefile_path_redacted”: Unauthorized unable to recognize “management-statefile_path_redacted”: Unauthorized ; waiting for nodes; exit status 1, unable to recognize “management-statefile_path_redacted”: Unauthorized unable to recognize “management-statefile_path_redacted”: Unauthorized unable to recognize “management-statefile_path_redacted”: Unauthorized unable to recognize “management-statefile_path_redacted”: Unauthorized unable to recognize “management-statefile_path_redacted”: Unauthorized unable to recognize “management-statefile_path_redacted”: Unauthorized unable to recognize “management-statefile_path_redacted”: Unauthorized unable to recognize “management-statefile_path_redacted”: Unauthorized unable to recognize “management-statefile_path_redacted”: Unauthorized