I have configured Rancher cluster with 3 nodes.
I am using the internal docker registry.
After the Rancher Pipeline execution is complete, only certain node can pull the docker image at deployment time, and the remaining nodes will have ErrImagePull errors.
502 Gateway error occurs and docker image cannot be pulled.
What kind of problem is this?
https://rancher.com/docs/rancher/v2.x/en/k8s-in-rancher/pipelines/
The internal Docker Registry is only accessible from cluster nodes and cannot be directly accessed by users.
The above documents suggest that all nodes in the cluster are accessible.
Configuring Persistent Data for Pipeline Components
I have finished setting up Persistent Data by referring to the document.
10.8.181.12 server works well.
But nodes at different addresses have problems.
Below is the event logs of the node that failed image pull.
| Type | Reason | Message | Last Seen |
|---|---|---|---|
| Normal | BackOff | Back-off pulling image “127.0.0.1:34293/rancher-node-test2:15” | a few seconds ago |
| Warning | Failed | Error: ImagePullBackOff | a few seconds ago |
| Warning | Failed | Failed to pull image “127.0.0.1:34293/rancher-node-test2:15”: rpc error: code = Unknown desc = failed to pull and unpack image “127.0.0.1:34293/rancher-node-test2:15”: failed to resolve reference “127.0.0.1:34293/rancher-node-test2:15”: unexpected status code [manifests 15]: 502 Bad Gateway | a few seconds ago |
| Warning | Failed | Error: ErrImagePull | a few seconds ago |
| Normal | Pulling | Pulling image “127.0.0.1:34293/rancher-node-test2:15” | a minute ago |
| Normal | Scheduled | Successfully assigned p-8xm4j-pipeline/rancher-node-test2-app-7c79999dd5-7jcv2 to gcloud-seoul-2ae659bde981b0ad9c0f24835a8d1232 | Invalid date |
Below is the event logs of the node that succeeded in image pull.
| Type | Reason | Message | Last Seen |
|---|---|---|---|
| Normal | Pulled | Container image “127.0.0.1:34293/rancher-node-test2:15” already present on machine | a minute ago |
| Normal | Created | Created container rancher-node-test2 | a minute ago |
| Normal | Started | Started container rancher-node-test2 | a minute ago |
| Normal | Scheduled | Successfully assigned p-8xm4j-pipeline/rancher-node-test2-app-7c79999dd5-5m72w to gcloud-seoul-78853ecbfdcbfed58ed8483bf205990b | Invalid date |
I want to know how to solve this problem.
| Component | Version |
|---|---|
| Rancher | v2.4.4 |
| User Interface | v2.4.22 |
| Helm | v2.16.5-rancher1 |
| Machine | v0.15.0-rancher36 |
| k3s | v1.17.5+k3s1 |