[Solved] K3s problem with pulling an image from private repo

I’m having a hard time with pulling image from private repository. Here’s the drill down:

The pod:

apiVersion: v1
kind: Pod
metadata:
[...]
spec:
  containers:
  - image: gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main
    imagePullPolicy: Always
[...]
  imagePullSecrets:
    - name: gitlab-dc-cdndemo-2

And the pull secret:

$ base64 -d <(kubectl -n test-cdndemo get secret gitlab-dc-cdndemo-2 -o json | jq -r '.data.".dockerconfigjson"') | jq
{
  "auths": {
    "https://gitlab.private:31443": {
      "username": "gitlab+deploy-token-22",
      "password": "EGDLqGKJwBtfYYf9cDFg",
      "email": "example@example.com",
      "auth": "Z2l0bGFiK2RlcGxveS10b2tlbi0yMjpFR0RMcUdLSndCdGZZWWY5Y0RGZw=="
    }
  }
}

It’s a playbook example of how it should be done. But when I deploy this I get:

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  14m                   default-scheduler  Successfully assigned test-cdndemo/appcdnmanagerui-68c8f8c6dd-qcxr5 to node-waw107
  Normal   Pulling    13m (x4 over 14m)     kubelet            Pulling image "gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main"
  Warning  Failed     13m (x4 over 14m)     kubelet            Failed to pull image "gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main": rpc error: code = Unknown desc = failed to pull and unpack image "gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest": failed to resolve referen
ce "gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
  Warning  Failed     13m (x4 over 14m)     kubelet            Error: ErrImagePull
  Warning  Failed     12m (x6 over 14m)     kubelet            Error: ImagePullBackOff
  Normal   BackOff    4m41s (x43 over 14m)  kubelet            Back-off pulling image "gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main"
                                                                                       gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main

Notice the error, it’s 403 Forbidden, not 401 Unauthorized so the credentials do work. Despite this, the image cannot be pulled from my private repo. But when I do this manually on a worker node everything goes smoothly:

$ crictl --debug pull --creds gitlab+deploy-token-22:EGDLqGKJwBtfYYf9cDFg gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest
DEBU[0000] get image connection
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest,Annotations:map[string]string{},},Auth:&AuthConfig{Username:gitlab+deploy-token-22,Password:EGDLqGKJwBtfYYf9cDFg,Auth:,ServerAddress:,IdentityToken:,RegistryToken:,},SandboxConfig:nil,}
DEBU[0006] PullImageResponse: &PullImageResponse{ImageRef:sha256:0c3b5d355c164d02aaa7b6cbe91bbfa12bd35826566472317efac63cb467d260,}
Image is up to date for sha256:0c3b5d355c164d02aaa7b6cbe91bbfa12bd35826566472317efac63cb467d260

$ crictl image
IMAGE                                                   TAG                  IMAGE ID            SIZE
[...]
gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main   latest               0c3b5d355c164       105MB```

It alwo works when I use auth data instead (as expected):

crictl --debug pull --auth Z2l0bGFiK2RlcGxveS10b2tlbi0yMjpFR0RMcUdLSndCdGZZWWY5Y0RGZw== gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest

Now I’m stuck. The only thing that comes to my mind is some kind of weird bug that’s in the k3s release I’m using:

# k3s -v
k3s version v1.25.4+k3s1 (0dc63334)
go version go1.19.3
# crictl -v
crictl version v1.25.0-k3s1

The bug hypothesis is based on previous experience with older k8s deployment, where this approach is used and it works. But on a fresh cluster all I get is 403 error from k3s despite crictl pulling the image with no problems.

Anyone had this kind of problem before and solved it?

OK, I’ve found the problem. Instead of:

{
  "auths": {
    "https://gitlab.private:31443": {
      "username": "gitlab+deploy-token-22",
      "password": "EGDLqGKJwBtfYYf9cDFg",
      "email": "example@example.com",
      "auth": "Z2l0bGFiK2RlcGxveS10b2tlbi0yMjpFR0RMcUdLSndCdGZZWWY5Y0RGZw=="
    }
  }
}

It should have been:

    {
      "auths": {
        "gitlab.private:31443": {
          "username": "gitlab+deploy-token-22",
          "password": "EGDLqGKJwBtfYYf9cDFg",
          "email": "example@example.com",
          "auth": "Z2l0bGFiK2RlcGxveS10b2tlbi0yMjpFR0RMcUdLSndCdGZZWWY5Y0RGZw=="
        }
      }
    }

Apparently the documentation at Pull an Image from a Private Registry | Kubernetes is a bit misleading since the example there does contain https:// in the URL.

{
    "auths": {
        "https://index.docker.io/v1/": {
            "auth": "c3R...zE2"
        }
    }
}

In a nutshell, it’s a bug but not where I was expecting it. And yes, my private repo works over HTTPS.

1 Like