Cluster and failover

1. On the running node, go to Cluster and press Add a node. You get a command with a token in it. It works once and expires in two hours.
2. On the new machine, run it:

sudo git clone https://git.failoverlb.net/failoverlb.git /data/docker/failoverlb \
  && sudo /data/docker/failoverlb/failoverlb_setup.sh \
       --join 203.0.113.10:7444 --token <token> \
       --allowlist 198.51.100.25/32

Then build nginx on it too. The new node joins as standby and pulls the current configuration, so you do not set your sites up twice.

It contains an .env file with that node's identity and its secret key. The second machine comes up believing it is the first one.

The installer now refuses to join when it spots this, and a machine set up before that check will sit there failing quietly. It also means both machines share one secret key, which they should not, so treat that key as needing to be changed.

Look at the Cluster page on the first node.

• Missing entirely: the join did not finish and the installer output says why. The usual cause is port 7444 being blocked between the machines.
• There but saying its credentials were refused: nearly always a name problem rather than a certificate problem. Check what the second node thinks it is called.

grep NFM_NODE_NAME /data/docker/failoverlb/.env

If that says the name of your first node, the settings file was copied. Delete it and install again so it makes its own.

Exactly the same as adding the second. Get a token from any node already in the cluster and run the installer on the new machine with the join details. Quorum works better with an odd number, so three is a genuinely better shape than two if you have the machines.

Cluster page, use the failover button. Doing it deliberately is calmer than pulling the plug and letting the software work it out, and it lets you confirm the other node is healthy first. Use it before any planned work.

Turn on forced standby for that node. It stays in the fleet and stays in sync, and it will not take over.

Remember to turn it off afterwards. A fleet where one node can never take over is a fleet with no failover, and nothing will remind you.

Order matters, and this order is the whole point.

1. Check both nodes are online, and that nothing is saved but unapplied.
2. Turn off its DNS answer and wait at least twice the TTL.
3. Hand over if it is the active one.
4. Turn on forced standby.
5. Do the work.
6. Coming back: check it is online, turn off forced standby, turn the DNS answer back on, then watch it for a few minutes.

Health first, then being allowed to be in charge, then traffic.

The standby notices the heartbeats have stopped, promotes itself, and carries on. Nobody has to be woken up and nothing has to be typed. Traffic already going to the survivor is unaffected throughout.

What this does not fix is traffic that was going to the dead node. That decision was made before the connection was attempted, so only DNS can move it.

Cluster page, Remove next to the node. Do it while the fleet is healthy rather than after the machine has already gone, so the roster is tidy and DNS answers tied to that node get cleaned up.

Cluster page, Sync now. Normally you do not need it, because every apply goes to every node. It is useful after a node has been offline and you want to confirm it has caught up rather than waiting.

Port 7444 between them is the usual answer. Neither will promote itself over the other in a two node cluster, because neither can see a majority, so both keep serving with the config they have. Fix the link and they catch up.