Worth reading before you install
The design assumes the web part will be attacked
A management screen that can move all of your traffic is a valuable target. Everything below is arranged so that reaching the web interface is not the same as owning the servers.
The privilege boundary
There are two programs on each node and only one of them has root.
| Manager container | Host agent | |
|---|---|---|
| Runs as | uid 10001 | root |
| Capabilities | All dropped | Only what the job needs, via systemd |
| Filesystem | Read only | Normal, but every path is checked |
| Faces | Your staff, over the network | A unix socket, and nothing else |
| Can write /etc/nginx | No | Yes |
| Can restart services | No | Yes |
| Size | A web application | Small enough to read start to finish |
What the agent will and will not do
The agent accepts a fixed list of verbs. Stage a config. Test a config. Commit. Roll back. Write certificate files. Run the build. Install a data file to a fixed path.
There is no "run this command" verb, no verb that takes a URL, and no verb that takes an arbitrary path. Every path is resolved, symlinks and all, and checked against an allow list before anything is touched.
This is the part to audit
Everything else in the system is a program that asks the agent to do something. The agent is the program that decides whether it is allowed. It is deliberately small for exactly that reason.
Node to node
Three separate things have to line up on every peer call:
- A client certificate signed by this cluster's own certificate authority and nothing else.
- A source address that is on the node roster.
- An HMAC signature made with a key only that node holds.
The timestamp and a nonce are inside the signed part, so a captured request cannot be replayed. A browser pointed at that port gets nowhere at all.
Joining a cluster
The join token works once and expires in two hours. Tokens are stored hashed, so a copy of the database does not hand anybody a working one. There is no long lived cluster password to leak.
A joining node makes its own key pair and the private half never leaves that box. It sends a signing request, gets back a certificate signed by the cluster authority, and then asks for the whole roster so it learns about every member rather than only the one it joined through.
A node that already has its own cluster will refuse to join another one rather than silently throw your configuration away.
Getting to the login page at all
NFM_ADMIN_ALLOWLIST is checked before the login page renders. That
ordering is the whole point: it is enforced earlier than anything in the database can be read,
so it works even if the database is unavailable and it works against somebody with a valid
password.
It lives in the settings file on each node rather than in the screen, for the same reason. A setting that can be changed from the screen it protects is not really protecting the screen.
Include IPv6
An allowlist that names only IPv4 addresses is the most common way people lock themselves out of their own system, because their browser quietly preferred IPv6.
Accounts
- Two factor is required for local accounts. A password on its own is not enough to reach a screen that can move all of your traffic somewhere else.
- Three roles, and viewer is the right one for most people.
- Recovery codes are shown once and each works once.
- Single sign on with OIDC and PKCE, where groups map to roles and removing somebody at the provider removes them from every node at once.
- Turning an account off ends its sessions immediately, so a browser left open somewhere stops working.
There is a console recovery command for the case where an administrator has lost their phone and used all their codes. It needs root on the machine, which anybody who has it could use to read the database anyway, so it hands out nothing new, and every use is written to the audit log as coming from the console.
Secrets at rest
Authenticator seeds, peer keys, single sign on client secrets, DNS provider credentials, paid authority credentials and backend trust tokens are encrypted with a key in each node's own settings file.
There is a check in the test suite for the audit log keeping secrets out of itself. Tokens, keys and passwords are recorded as having changed, never as values, because a log that contains the secrets is a second copy of the secrets.
Why a bad config cannot take the fleet down
Every change is staged on every node and tested with that node's own nginx binary before any node commits. If one node refuses, both staged copies are thrown away and nothing goes live.
That applies to hand written config too. The worst case is a configuration that refuses to save. It is not a fleet that stops serving.
What it gives you at the edge
| Feature | What it stops |
|---|---|
| Web application firewall | Injection, traversal and the long tail of application attacks, using the OWASP rules. |
| Rate and connection limits | Rapid fire requests, password guessing, and one visitor holding hundreds of slow connections. |
| Login watch | Password spraying, which slips past both a lockout and a rate limit. |
| Country blocking | Scanner noise, and serving somewhere an agreement says you must not. |
| Access lists | Anybody outside your networks reaching an admin area. |
| Backend trust tokens | Anything that can reach your backend port pretending to be the fleet. |
| Post quantum key exchange | Traffic captured today being decrypted later. |
What this does not claim
There has been no third party security audit
The design above is written down so you can judge it, and the source is short enough to read start to finish. If you are putting this in front of something where an independent audit is a requirement, that requirement is not met, and no amount of description changes it.
Some more honest limits:
- A backend trust token is a bearer secret. Anybody holding it can pretend to be the fleet. It belongs behind a firewall rather than instead of one.
- Country blocking is not a security control. A VPN gets around it in one click. It is for cutting noise and meeting a rule about where you serve.
- A backup file is equivalent to your private keys. The zip encryption derives its key with fewer rounds than a modern password vault, so it is only as strong as the password against somebody with time to spend.
- A compromised node should be treated as a compromised fleet. That node holds its own keys and certificates. Rebuild both.
- The GUI has not been through a penetration test by anybody independent.
Reporting something
If you find a security problem, please report it privately rather than opening a public issue, and give it a reasonable amount of time before publishing. The support page has the details of how to get in touch.
There is no bounty program. There is somebody who will read what you send and fix it.