Traefik dashboard 404

I’m no expert in kubernetes; why is exposing the dashboard to the built in traefik so poorly documented and have so many different answers everywhere? I’ve included as much info as I can think of below. somebody please help why is this so difficult

I heard that we’re supposed to use traefik-config.yaml so that the setting persists:
/var/lib/rancher/k3s/server/manifests.traefik-config.yaml contents:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    dashboard:
      enabled: true
      domain: "traefik.domain.com"

I’ve tried accessing the dashboard both locally, with traefik.domain.com pointing to 127.0.0.1 in /etc/hosts and over a local network (DNS serves the local IP correctly) keep in mind rancher.domain.com is working perfectly fine

I’ve tried port forwarding 9000:9000 as well as using the recommended ingress route:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: dashboard
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`traefik.localhost`) && (PathPrefix(`/dashboard`) || PathPrefix(`/api`))
      kind: Rule
      services:
        - name: api@internal
          kind: TraefikService

Everything returns as 404 Not found. But weirdly my tls-traefik-ingress cert works fine so I get a valid https cert but a 404 response on traefik.domain.com/dashboard

helm -n kube-system get values traefik shows the values are there:

USER-SUPPLIED VALUES:
dashboard:
  domain: traefik.domain.com
  enabled: true
global:
  systemDefaultRegistry: ""
image:
  name: rancher/library-traefik
podAnnotations:
  prometheus.io/port: "8082"
  prometheus.io/scrape: "true"
ports:
  websecure:
    tls:
      enabled: true
priorityClassName: system-cluster-critical
providers:
  kubernetesIngress:
    publishedService:
      enabled: true
rbac:
  enabled: true
tolerations:
- key: CriticalAddonsOnly
  operator: Exists
- effect: NoSchedule
  key: node-role.kubernetes.io/control-plane
  operator: Exists
- effect: NoSchedule
  key: node-role.kubernetes.io/master
  operator: Exists

An ingress route already exists which I didn’t create, the one I created was dashboard kubectl -n kube-system describe ingressRoute traefik-dashboard:

Name:         traefik-dashboard
Namespace:    kube-system
Labels:       app.kubernetes.io/instance=traefik
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=traefik
              helm.sh/chart=traefik-9.18.2
Annotations:  helm.sh/hook: post-install,post-upgrade
API Version:  traefik.containo.us/v1alpha1
Kind:         IngressRoute
Metadata:
  Creation Timestamp:  2021-11-15T15:53:19Z
  Generation:          1
  Managed Fields:
    API Version:  traefik.containo.us/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:helm.sh/hook:
        f:labels:
          .:
          f:app.kubernetes.io/instance:
          f:app.kubernetes.io/managed-by:
          f:app.kubernetes.io/name:
          f:helm.sh/chart:
      f:spec:
        .:
        f:entryPoints:
        f:routes:
    Manager:         helm
    Operation:       Update
    Time:            2021-11-15T15:53:19Z
  Resource Version:  3415133
  UID:               04ae0639-0aa4-41af-80a7-f7353155ff98
Spec:
  Entry Points:
    traefik
  Routes:
    Kind:   Rule
    Match:  PathPrefix(`/dashboard`) || PathPrefix(`/api`)
    Services:
      Kind:  TraefikService
      Name:  api@internal
Events:      <none>

Deployment info kubectl -n kube-system describe deployment traefik:

Name:                   traefik
Namespace:              kube-system
CreationTimestamp:      Mon, 01 Nov 2021 00:27:49 +0800
Labels:                 app.kubernetes.io/instance=traefik
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=traefik
                        helm.sh/chart=traefik-9.18.2
Annotations:            deployment.kubernetes.io/revision: 3
                        field.cattle.io/publicEndpoints:
                          [{"addresses":["192.168.2.5"],"port":80,"protocol":"TCP","serviceName":"kube-system:traefik","allNodes":false},{"addresses":["192.168.2.5"...
                        meta.helm.sh/release-name: traefik
                        meta.helm.sh/release-namespace: kube-system
Selector:               app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:           app.kubernetes.io/instance=traefik
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=traefik
                    helm.sh/chart=traefik-9.18.2
  Service Account:  traefik
  Containers:
   traefik:
    Image:       rancher/library-traefik:2.4.8
    Ports:       9000/TCP, 8000/TCP, 8443/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP
    Args:
      --global.checknewversion
      --global.sendanonymoususage
      --entryPoints.traefik.address=:9000/tcp
      --entryPoints.web.address=:8000/tcp
      --entryPoints.websecure.address=:8443/tcp
      --api.dashboard=true
      --ping=true
      --providers.kubernetescrd
      --providers.kubernetesingress
      --providers.kubernetesingress.ingressendpoint.publishedservice=kube-system/traefik
      --entrypoints.websecure.http.tls=true
    Liveness:     http-get http://:9000/ping delay=10s timeout=2s period=10s #success=1 #failure=3
    Readiness:    http-get http://:9000/ping delay=10s timeout=2s period=10s #success=1 #failure=1
    Environment:  <none>
    Mounts:
      /data from data (rw)
      /tmp from tmp (rw)
  Volumes:
   data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
   tmp:
    Type:               EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:          <unset>
  Priority Class Name:  system-cluster-critical
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   traefik-97b44b794 (1/1 replicas created)
Events:          <none>

Not sure if the App itself is useful, quite a long yaml but shows ports:traefik:expose=false, shortened it here kubectl -n kube-system describe app traefik:

      Ports:
        Traefik:
          Expose:        false
          Exposed Port:  9000
          Port:          9000
          Protocol:      TCP
        Web:
          Expose:        true
          Exposed Port:  80
          Port:          8000
          Protocol:      TCP
        Websecure:
          Expose:        true
          Exposed Port:  443
          Port:          8443
          Protocol:      TCP
      Providers:
        Kubernetes CRD:
          Enabled:     true
          Namespaces:  <nil>
        Kubernetes Ingress:
          Enabled:     true
          Namespaces:  <nil>
          Published Service:
            Enabled:  false
  Values:
    Dashboard:
      Domain:   traefik.domain.com
      Enabled:  true

/var/lib/rancher/k3s/server/manifests/traefik.yaml contents:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik-crd
  namespace: kube-system
spec:
  chart: https://%{KUBERNETES_API}%/static/charts/traefik-crd-9.18.2.tgz
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik
  namespace: kube-system
spec:
  chart: https://%{KUBERNETES_API}%/static/charts/traefik-9.18.2.tgz
  set:
    global.systemDefaultRegistry: ""
  valuesContent: |-
    rbac:
      enabled: true
    ports:
      websecure:
        tls:
          enabled: true
    podAnnotations:
      prometheus.io/port: "8082"
      prometheus.io/scrape: "true"
    providers:
      kubernetesIngress:
        publishedService:
          enabled: true
    priorityClassName: "system-cluster-critical"
    image:
      name: "rancher/library-traefik"
    tolerations:
    - key: "CriticalAddonsOnly"
      operator: "Exists"
    - key: "node-role.kubernetes.io/control-plane"
      operator: "Exists"
      effect: "NoSchedule"
    - key: "node-role.kubernetes.io/master"
      operator: "Exists"
      effect: "NoSchedule"
k3s --version
k3s version v1.21.5+k3s2 (724ef700)
go version go1.16.8

Had the same problem. Found the solution here: k3s - Traefik & k3d: Dashboard is not reachable - Stack Overflow

This worked for me:
kubectl patch ingressroute -n kube-system traefik-dashboard --type=merge -p '{"spec":{"entryPoints":["web"]}}'