[Solved] K3s problem with pulling an image from private repo

k3s, k3OS, and k3d
1

I’m having a hard time with pulling image from private repository. Here’s the drill down:

The pod:

apiVersion: v1
kind: Pod
metadata:
[...]
spec:
  containers:
  - image: gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main
    imagePullPolicy: Always
[...]
  imagePullSecrets:
    - name: gitlab-dc-cdndemo-2

And the pull secret:

$ base64 -d <(kubectl -n test-cdndemo get secret gitlab-dc-cdndemo-2 -o json | jq -r '.data.".dockerconfigjson"') | jq
{
  "auths": {
    "https://gitlab.private:31443": {
      "username": "gitlab+deploy-token-22",
      "password": "EGDLqGKJwBtfYYf9cDFg",
      "email": "example@example.com",
      "auth": "Z2l0bGFiK2RlcGxveS10b2tlbi0yMjpFR0RMcUdLSndCdGZZWWY5Y0RGZw=="
    }
  }
}

It’s a playbook example of how it should be done. But when I deploy this I get:

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  14m                   default-scheduler  Successfully assigned test-cdndemo/appcdnmanagerui-68c8f8c6dd-qcxr5 to node-waw107
  Normal   Pulling    13m (x4 over 14m)     kubelet            Pulling image "gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main"
  Warning  Failed     13m (x4 over 14m)     kubelet            Failed to pull image "gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main": rpc error: code = Unknown desc = failed to pull and unpack image "gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest": failed to resolve referen
ce "gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
  Warning  Failed     13m (x4 over 14m)     kubelet            Error: ErrImagePull
  Warning  Failed     12m (x6 over 14m)     kubelet            Error: ImagePullBackOff
  Normal   BackOff    4m41s (x43 over 14m)  kubelet            Back-off pulling image "gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main"
                                                                                       gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main

Notice the error, it’s 403 Forbidden, not 401 Unauthorized so the credentials do work. Despite this, the image cannot be pulled from my private repo. But when I do this manually on a worker node everything goes smoothly:

$ crictl --debug pull --creds gitlab+deploy-token-22:EGDLqGKJwBtfYYf9cDFg gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest
DEBU[0000] get image connection
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest,Annotations:map[string]string{},},Auth:&AuthConfig{Username:gitlab+deploy-token-22,Password:EGDLqGKJwBtfYYf9cDFg,Auth:,ServerAddress:,IdentityToken:,RegistryToken:,},SandboxConfig:nil,}
DEBU[0006] PullImageResponse: &PullImageResponse{ImageRef:sha256:0c3b5d355c164d02aaa7b6cbe91bbfa12bd35826566472317efac63cb467d260,}
Image is up to date for sha256:0c3b5d355c164d02aaa7b6cbe91bbfa12bd35826566472317efac63cb467d260

$ crictl image
IMAGE                                                   TAG                  IMAGE ID            SIZE
[...]
gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main   latest               0c3b5d355c164       105MB```

It alwo works when I use auth data instead (as expected):

crictl --debug pull --auth Z2l0bGFiK2RlcGxveS10b2tlbi0yMjpFR0RMcUdLSndCdGZZWWY5Y0RGZw== gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest

Now I’m stuck. The only thing that comes to my mind is some kind of weird bug that’s in the k3s release I’m using:

# k3s -v
k3s version v1.25.4+k3s1 (0dc63334)
go version go1.19.3
# crictl -v
crictl version v1.25.0-k3s1

The bug hypothesis is based on previous experience with older k8s deployment, where this approach is used and it works. But on a fresh cluster all I get is 403 error from k3s despite crictl pulling the image with no problems.

Anyone had this kind of problem before and solved it?

2

OK, I’ve found the problem. Instead of:

{
  "auths": {
    "https://gitlab.private:31443": {
      "username": "gitlab+deploy-token-22",
      "password": "EGDLqGKJwBtfYYf9cDFg",
      "email": "example@example.com",
      "auth": "Z2l0bGFiK2RlcGxveS10b2tlbi0yMjpFR0RMcUdLSndCdGZZWWY5Y0RGZw=="
    }
  }
}

It should have been:

    {
      "auths": {
        "gitlab.private:31443": {
          "username": "gitlab+deploy-token-22",
          "password": "EGDLqGKJwBtfYYf9cDFg",
          "email": "example@example.com",
          "auth": "Z2l0bGFiK2RlcGxveS10b2tlbi0yMjpFR0RMcUdLSndCdGZZWWY5Y0RGZw=="
        }
      }
    }

Apparently the documentation at Pull an Image from a Private Registry | Kubernetes is a bit misleading since the example there does contain https:// in the URL.

{
    "auths": {
        "https://index.docker.io/v1/": {
            "auth": "c3R...zE2"
        }
    }
}

In a nutshell, it’s a bug but not where I was expecting it. And yes, my private repo works over HTTPS.

2 Likes