The registration token (long URL in Add Host -> Custom) is used by the agent to connect to the server for the first time and generate an agent account and API key pair. That key pair is then used for all subsequent communication using the same authentication and authorization logic as there is for other kinds of accounts, like environment API keys.
The design is that the agent is untrusted because it is running on outside and potentially hostile (to the server) hardware. So the agent accounts have access to only the resources they need in the API, replies to events are checked that the event was actually sent to that agent, etc. There is not as much in the opposite direction for the agent to verify the host. You’d want to setup TLS (which should work now in 0.30 without those buffer settings ) and the cert will be verified. We plan on making this easier to setup as more of a managed solution option (vs configuring nginx on the side on your own).
The IPSec key is per-environment (the UI term, the drop down in the upper right) generated on the server, stored in the database, and sent to the host as part of the agent registration with the API key pair. The connections are point to point between hosts and AES encrypted, which is accelerated by most modern CPUs and can do at least a couple gbps