that’s an easy one: In /etc/ldap.conf on the client (the server you want to limit access to) you can set “pam_groupdn” to the DN of the group that contains the list of permitted users (as fully qualified DNs), i.e.
The actual member of that node to contain the list of users is defined in the same file, i.e.
pam_member_attribute memberIf OES stores the values at other locations, you need to adopt these statements.
PS: I advise to keep an open root ssh session to the server while doing these changes. If you limit server ssh access to non-root users and incorrectly set those ldap.conf parameters, there’s no way to get into the system rather than via console… which may or may not be available at that time