Can custom cluster etcd snapshots be backed up to S3?

Greetings,
I have a Rancher custom cluster and I’d like to set the ‘etcd Snapshot Backup Target’ to backup to S3. I believe I’ve setup proper permissions in IAM and configured Rancher accordingly but I’m not seeing any backups showing up in the bucket. I’ve enabled logging for the bucket and I don’t see Rancher even attempting to put anything there, so I’m assuming I’m hitting a problem in Rancher. Per the documentation here:

https://rancher.com/docs/rancher/v2.x/en/cluster-admin/backing-up-etcd/

When referencing requirements for S3, it sounds like the assumption is that the cluster is running on EC2 instances. Is it possible to backup etcd snapshots from a custom on-prem cluster to S3?

Are there any logs Rancher side I can look at to gain some insight on why this isn’t working?

Can you share the sanitized config for s3 backups? The error is currently not surfaced in the UI (which is fixed in https://github.com/rancher/rancher/issues/20685), but there should be an error in the Rancher log regarding EtcdBackupController.

Thanks @superseb. Once I saw this in the logs:

level=fatal msg="failed to set s3 server: bucket ******-etcd-backups is not found"

It was quickly apparent I’d made a typo mistake: backup != backups.

Also good to know there’s a fix in master for the error to get up to the UI.