fleetd design

fleetd manages a small fleet of Linux servers and routers: metrics, alerts, remote commands and a web terminal. Its defining constraint is that a compromised server must not be able to run commands on any machine.

Components

browser ──HTTPS──▶ fleetd-server ──▶ PostgreSQL
                      ▲
                      │ WebSocket + Noise (agents dial out; no inbound ports on agents)
      ┌───────────────┼────────────────┐
 fleetd-agent    fleetd-agent     fleetd-agent      (systemd hosts, OpenWrt / iStoreOS routers)

Threat model

Principles:

1. Agents do not trust the server. They run only commands signed by a key in their own trusted_signers.
2. Each agent's permission level lives in its own config file; the server cannot raise it.
3. Agents authenticate the server (pinned key) and the server authenticates agents (registered key).
4. New agents only report metrics until an admin approves them.

Transport

• Agents connect out over WebSocket (ws:// on a private network, wss:// through a reverse proxy), so no machine needs an inbound port and routers behind NAT work.
• Inside the WebSocket runs Noise_IK_25519_ChaChaPoly_BLAKE2s (the WireGuard primitives, via github.com/flynn/noise). The agent pins the server's static key; the handshake fails against any other server.
• The server pings every 30 s; either side drops a connection silent for 75 s. Agents reconnect with exponential backoff (1–60 s, jittered).

Registration

1. On first start the agent generates a Curve25519 key (/etc/fleetd/agent.key, mode 600).
2. Any unknown key that completes the handshake is added as pending. Nothing it sends is stored and it receives no commands. At most 50 machines can be pending at once.
3. The admin compares the fingerprint printed by the installer with the one in the UI and approves, or rejects. Rejected keys go on a block list and are refused at the handshake.
4. Deleting an approved machine removes its record; a still-running agent re-registers as pending.

Commands and the web terminal

• The admin's browser generates a non-extractable Ed25519 key with WebCrypto and keeps it in IndexedDB. Its public key is registered with the server, which only uses it to pre-fill trusted_signers in install.sh.

• A command is a JSON payload signed in the browser. The server relays the exact bytes; the agent verifies the signature over those bytes before parsing them. The payload carries:

  Field	Purpose
  id	128-bit random, single use
  targets	agent fingerprints; a command for one machine is refused by every other
  kind	exec or shell
  argv / action	program and arguments (no shell), or an allowlisted action name
  issued_at, expires_at	at most 10 minutes validity, 2 minutes clock skew allowed
  timeout	exec timeout, default 60 s, at most 1 hour

• Used ids are persisted in state_dir/seen.json until they expire, so replay protection survives restarts.

• Permission levels, set per agent:

  Level	Allows
  monitor (default)	nothing — metrics only
  allowlist	named actions defined in the agent's config, with fixed arguments
  full	any command, and the web terminal

• Output is streamed back, capped at 1 MB; at most 4 commands and 4 terminals run at once per agent. Every command is logged by the server and in the agent's own journal.

• The terminal starts bash -l (or sh) in a PTY on the agent; the browser ⇄ server ⇄ agent relay carries keystrokes and output.

Agent updates

• A release bundle is manifest (version, arch, size, sha256) + Ed25519 signature + binary (proto/release.go). The signing key stays on the maintainer's machine; the server only stores bundles.
• The server decides when an agent updates, never what it installs: the agent refuses bundles that are not signed by its release_key, target another architecture, are not newer than itself, or whose binary does not report the signed version when run.
• The binary is swapped with an atomic rename and the previous one is kept as fleetd-agent.old; the agent then exits and systemd / procd starts the new version. Updates are triggered by hand in the UI.

One-line install

• GET /install.sh renders a POSIX shell script with the server's address, server key, release key and the registered signing keys. It contains no secrets.
• GET /agent/download/<arch> serves the newest valid signed build. The first install trusts the server and HTTPS, like any curl | sh; later updates are verified against release_key.
• All script arguments are checked against a character allowlist before anything is written. Re-running the script on an installed machine only replaces the binary.

Monitoring and alerts

• Every 10 s: CPU, load, memory, swap, uptime, processes, TCP connections, per-interface traffic and rates. Every 60 s: disks and inodes, failed systemd units, Docker containers, WireGuard peers.
• Raw samples are kept for 7 days and downsampled when charted.
• Alerts: offline (40 s), CPU / memory above a threshold for a sustained period, disk and inode usage, failed units, unhealthy / restarting / crashed containers, stale WireGuard handshakes, connection IP changes. Changes from one evaluation are sent as a single e-mail; without SMTP they are only shown in the UI and logged.

Admin login

• Username and password (argon2id). TOTP two-factor is optional per admin (admin-add -totp); each code works once.
• Session tokens are 256-bit random values stored as hashes; cookies are HttpOnly, SameSite=Strict, and Secure when served over HTTPS. State-changing requests require a CSRF header.
• The admin UI is in English and Chinese (per browser). Alerts, e-mails and API errors use the server's language setting (en or zh).
• Five failed logins lock a username or IP for 15 minutes.

Known limitations