After add catalog rancher cannot start

Rancher 2.x
1

Hi,
After i added a catalog to rancher, it’s broken, and now i receive the following error:

2021/07/05 21:27:35 [ERROR] failed to start cluster controllers c-8vm88: context canceled
2021/07/05 21:27:45 [ERROR] error syncing ‘rancher-partner-charts’: handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 reset --hard FETCH_HEAD error: exit status 128, detail: fatal: Unable to create ‘/var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974/.git/index.lock’: File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by ‘git commit’. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
, handler helm-clusterrepo-download: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 reset --hard HEAD error: exit status 128, detail: fatal: Unable to create ‘/var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974/.git/index.lock’: File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by ‘git commit’. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
, requeuing
2021/07/05 21:27:47 [ERROR] error syncing ‘rancher-charts’: handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721 reset --hard FETCH_HEAD error: exit status 128, detail: fatal: Unable to create ‘/var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721/.git/index.lock’: File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by ‘git commit’. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
, handler helm-clusterrepo-download: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721 reset --hard HEAD error: exit status 128, detail: fatal: Unable to create ‘/var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721/.git/index.lock’: File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by ‘git commit’. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
, requeuing
W0705 21:28:52.064373 26 watcher.go:207] watch chan error: etcdserver: mvcc: required revision has been compacted
2021-07-05 21:28:54.625828 I | mvcc: store.index: compact 29007475
2021-07-05 21:28:55.041027 I | mvcc: finished scheduled compaction at 29007475 (took 404.107797ms)
2021/07/05 21:29:32 [INFO] Stopping cluster agent for c-8vm88
2021/07/05 21:29:32 [ERROR] failed to start cluster controllers c-8vm88: context canceled
2021/07/05 21:29:42 [ERROR] error syncing ‘c-8vm88’: handler cluster-deploy: Get “https://10.100.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent”: waiting for cluster [c-8vm88] agent to connect, requeuing

Rancher 2.5.8

I cannot remove the catalog or enter at rancher interface, i create a backup and try to update to rancher 2.5.9, but the errors persist.

Thanks

2

Hello,
Did you solve your issue? I get the same by adding a catalog in cluster scope, Rancher 2.5.8 too.
Thanks

3

I’m struggling with the same issue! Rancher keeps restarting, it makes it totally useless :frowning:

Also v2.5.8

4

To fix this, just run below commands on your rancher node where teh docker is running
docker exec -it <container_ID> bash
Then inside the running rancher docker, remove the specific .lock file which is shown in error and it should work fine

cd ‘/var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721/.git"
then rm index.lock

after this, no error are cropping up in the racnher running docker.

2 Likes
5

Thank you 3 year old post for saving my morning. Simple problems can be a big deal when you are over tired.