Azure LB and vms in multiple availability sets

I’m trying to setup K8S on azure with Racnher (with my own masters - not AKS).
I have one node pool for masters (availability set “masters”) and another one for compute nodes (availability set “nodes”).
After some trial and error I’ve found a way to successfully install LB with this setup (primary_availability_set_name = “nodes”), but still have problem with contents of Network Security Group.

Service LoadBalancer/Azure LB adds its own address to NSG for each VM, but with multiple availability sets they get added only for masters VMs and not to nodes VMs.

I assume I forgot something to configure on cluster deploy and can’t find what…

Help appreciated!

PS. I can’t find good documentation for Azure specific settings (short descriptions on form are not enough).