We’re using the Rancher Load Balancer to redirect incoming requests to named microservices based on their path. I thought that each Balancer Rule was applied in order of priority, and therefore I’ve arranged them so that more specific rules will be applied first (i.e. the path /users-usage-api has a higher priority than /user).
However, we’ve been seeing strange 404 errors for some services when we add new rules. On digging deeper, we’ve realised that the HAProxy.cfg which is being created in the container for our LB service looks to have acl rules which are in the “wrong” order (at least as far as my understanding of HAProxy goes) and as a result, a request to /users-usage-api is being caught by the rule which matches paths beginning with “/user” and sent to the wrong Target.
I’ve pasted a copy of the Rancher rules and part of the HAProxy.cfg which was generated as a result.
We have 3 environments (Test/UAT/Live) and although they have identical Rancher Load Balancer Rules, the HAProxy.cfg files are different, so the example above fails with a 404 on Test, but works OK on UAT and Live. It seems like the order of the HAProxy.cfg rules is “random”. Our only option at the moment seems to be to make sure that none of our pathnames begin with the same characters so that the rule ordering doesn’t matter.
We’re running Rancher v1.5.9 and rancher/lb-service-haproxy:v0.6.4, but I’ve tried creating a test setup with the latest Rancher version and I’m seeing the same issue. I don’t know if I’m misunderstanding how the priority rules are supposed to work (either in Rancher or HAProxy) but any help would be greatly appreciated.
Rancher Balancer Rules
HAProxy.cfg (acl rules section)
frontend 80
bind *:80
mode http
acl 80_fake_path path_beg -i /fake
use_backend 80_fake if 80_fake_path
frontend 443
bind *:443 ssl crt /etc/haproxy/certs/current
mode http
acl 443_form_actors_readmodel_path path_beg -i /form-actors-readmodel
use_backend 443_form_actors_readmodel if 443_form_actors_readmodel_path
acl 443_queues_processor_path path_beg -i /queues-processor
use_backend 443_queues_processor if 443_queues_processor_path
acl 443_organisations_readmodel_path path_beg -i /organisations-readmodel
use_backend 443_organisations_readmodel if 443_organisations_readmodel_path
acl 443_worklist_readmodel_path path_beg -i /worklist-readmodel
use_backend 443_worklist_readmodel if 443_worklist_readmodel_path
acl 443_user_path path_beg -i /user
use_backend 443_user if 443_user_path
acl 443_users_usage_readmodel_path path_beg -i /users-usage-readmodel
use_backend 443_users_usage_readmodel if 443_users_usage_readmodel_path
acl 443_users_path path_beg -i /users
use_backend 443_users if 443_users_path
acl 443_worklist_path path_beg -i /worklist
use_backend 443_worklist if 443_worklist_path
acl 443_user_permissions_readmodel_path path_beg -i /user-permissions-readmodel
use_backend 443_user_permissions_readmodel if 443_user_permissions_readmodel_path
acl 443_users_readmodel_path path_beg -i /users-readmodel
use_backend 443_users_readmodel if 443_users_readmodel_path
acl 443_paperforms_path path_beg -i /paperforms
use_backend 443_paperforms if 443_paperforms_path
acl 443_usersusage_path path_beg -i /usersusage
use_backend 443_usersusage if 443_usersusage_path
acl 443_badgeradmin_path path_beg -i /badgeradmin
use_backend 443_badgeradmin if 443_badgeradmin_path
