Set up DNS failover end to end
This is the longest guide here because it involves your domain host as well as this software. It is also the one that makes a whole server dying survivable rather than merely noticed.
The example
| Thing | Value |
|---|---|
| First load balancer | lb-01, public address 203.0.113.10 |
| Second load balancer | lb-02, public address 203.0.113.11 |
| Part of the domain to delegate | gslb.example.com |
| Nameserver names to invent | ns1.example.com and ns2.example.com |
| The name people actually use | shop.example.com |
Delegate a piece, never the whole domain
Delegating example.com itself moves every other record you have, including your mail records, onto these nodes. Delegating gslb.example.com moves nothing except what you put there.
Step 1: the zone
- DNS Failover, Add a zone.
- Zone name:
gslb.example.com. - Nameserver names:
ns1.example.com,ns2.example.com, one per node, in the same order as your nodes. - Save.
You are inventing those nameserver names now and creating them at your domain host in step 3.
Step 2: the name
- Press Add a name on the zone.
- Name:
www. Type: A. TTL: 30. - When a node is down: hand out all the healthy ones.
- If every node is down: hand them out anyway.
- Add an answer and pick lb-01 from the list. Its address fills in.
- Add a second answer and pick lb-02.
- Save.
The page now shows the name with two answers, each with a green dot and the words node is up, handed out.
Why hand them out anyway when everything is down
If the health check itself is what broke, an empty answer takes the site down when it did not have to be. Handing out addresses that might be fine is better than handing out nothing that definitely is not.
Step 3: the delegation
This is the step people miss, and missing it looks exactly like the feature being broken: everything on the page reads correctly and the name still does not work, because nothing has told the internet to ask your servers.
The page prints the exact records to create, with your addresses already filled in. In Route 53,
open the hosted zone for example.com, which is the parent of the part you are
delegating, and create them there.
| Name | Type | Value | TTL |
|---|---|---|---|
| gslb | NS | ns1.example.com. and ns2.example.com. | 300 |
| ns1 | A | 203.0.113.10 | 300 |
| ns2 | A | 203.0.113.11 | 300 |
| shop | CNAME | www.gslb.example.com. | 60 |
The last row is what points the address people actually use at the new zone. Until you add it, everything works and nobody is using it.
Do not create a hosted zone for the delegated part
Your nodes are the nameservers for gslb.example.com now. A hosted zone at your provider as well gives two different sets of servers claiming the same name, and which one answers depends on who is asking.
The A records for the nameservers are called glue. They are needed because a nameserver named inside the zone it answers for cannot be looked up without them, which is a loop. They are also the records people forget.
Step 4: check the nodes answer
Ask each node directly, with no port on the end.
dig @203.0.113.10 www.gslb.example.com +short
dig @203.0.113.11 www.gslb.example.com +shortBoth should answer, and both should list every healthy node. If one answers and the other does not, look at the DNS listener panel on the quiet one. That panel reports what the node is actually doing rather than what was typed in, and the gap between those two is where nearly all confusion here lives.
Never test against port 5353
The part that answers DNS runs without root on purpose, so it listens on 5353 instead, on UDP and TCP both, and a firewall rule the software manages sends UDP and TCP port 53 to it. Testing against 5353 gives you an answer whether or not the public path works, which tells you nothing.
Step 5: check the delegation took
Now ask the internet rather than the nodes, so you are testing the whole path. Leave it a few minutes after making the records.
dig shop.example.com +short
dig +trace www.gslb.example.comIf this returns nothing, the delegation is the problem and not the software. Check the NS records and the two address records for the nameservers. Those address records are the ones people forget.
Step 6: test the failover
while true; do date +%T; dig @8.8.8.8 shop.example.com +short; sleep 5; done- Leave that running.
- Stop the manager on lb-01, or shut the machine down entirely.
- Watch the answers. The dead node drops out in about thirty seconds.
- Start lb-01 again. It is back in the answer within about five seconds.
Removal is deliberately slower than restoring, so a brief hiccup does not move everybody, and a recovered node is put back as soon as it is well.
What to check afterwards
- The DNS Failover page on both nodes says it is answering, not just one.
- Both answers show a green dot.
- The address people use still resolves through to both load balancers.
Be honest about the limits
DNS failover is not instant and cannot be made instant. How long a computer remembers an answer is a request from you, not a rule. Some providers refuse very short values, browsers keep their own copy, and some programs remember an address for as long as they are running. Expect most visitors to move within a minute or two and a small number to take much longer.