3. Encrypt HA server communications
There are a couple of approaches you can take to encrypt the communication between your HA nodes. The easiest one is to use a reverse proxy like NGINX, Traefik, Caddy, etc. The second easiest is to use some kind of peer-to-peer VPN, like Nebula, WireGuard or ZeroTier.
Manual HTTPs approach is always at the bottom of my list, because it's error-prone, and you'll need to manage your certificates by hand (including tracking the expiration dates).
API/HA service binds itself to 0.0.0.0, so please take care of the pf.conf record accordingly and only allow subnets or peers that absolutely need to communicate with the REST API or HA.
This is a rule that I use quite often:
# This line is a part of incoming Hoster node rules (usually at the very bottom of the file)
pass in quick log (all) on { bge0 } proto tcp from { 172.16.0.1 172.16.0.2 172.16.0.3 } to port 3000 keep state # Allow REST API/HA Mode
Reverse proxy
Using reverse proxies is the easiest way to encrypt the communication between the REST API/HA Hoster nodes, but not the best one considering the fact that most of the time HA mode is used within the same LAN network.
I'll publish some example/ready-to-go templates here, for the most popular reverse proxies (that should be fully compatible with our REST API server). Stay tuned for that, and remember to perform the IP filtering for the incoming connections in order to boost the HA service security even further.
VPN/Overlay networking with over-the-wire encryption
ZeroTier will be the easiest to manage, while the WireGuard will be the most stable (which is my preference, when it comes to HA).
Also keep in mind that all candidate nodes will have to be on the same VPN network with each other (and all workers) to function correctly and communicate securely.
I'll publish a short WG configuration here soon (that will include 3 candidate nodes, and 2 workers). Stay tuned.
Manual HTTP/SSL certificate management
SSL (HTTPs) support has not been implemented yet. Stay tuned.
Prepare the certificates
Create a new CA
Generate a CA Key and Certificate
Create a directory to store your CA files:
Generate a private key for your CA. You'll be prompted for a passphrase:
Create a self-signed CA certificate using the private key:
Fill in the requested information (Common Name, Organization, etc.) as appropriate for your CA.
Generate Server Keys and Certificate Signing Requests (CSRs)
For each server, generate a CSR as follows:
When prompted for the Common Name (CN), use the IP address of the server.
Sign Server Certificates with the Common CA
Use your common CA's private key and certificate to sign each server's CSR:
openssl x509 -req -in server-csr.pem -CA ca-cert.pem -CAkey ca-key.pem -out server-cert.pem -CAcreateserial
Repeat this step for each server, generating a separate server certificate for each.
Distribute Certificates to Servers
Copy each server's certificate (server-cert.pem) and its private key (server-key.pem) to the respective server.
Trust the Common CA on All Servers
Import the CA certificate (ca-cert.pem) of your common CA into the trust store of each server. This ensures that all servers trust the certificates signed by the common CA.
cp ca-cert.pem /usr/local/share/certs/
/usr/sbin/mtree -u -f /etc/mtree/BSD.local.dist -p /usr/local/share/certs
/usr/local/sbin/ca_root_nss_util install
With this approach, you have a common CA that signs certificates for all your servers, and all servers trust certificates signed by this CA, even when using IP addresses. This provides a secure way for your servers to communicate with each other within your firewall-protected network.
Add a new certificate file to your HA config
Start the REST API in HA mode
Start the REST API in HA mode over an encrypted SSL connection: