Architecture

The clients do the clustering. Nodes store data, discovery servers track membership, and every routing decision happens in the SDK — so the data path has no extra hop.

The three roles

An SDK bootstraps by connecting to any seed. The server's handshake reply says whether it is a node or a discovery server; against a discovery server the client fetches the node list (which carries the replication factor R) and then talks to nodes directly.

Routing: rendezvous hashing

For each key, every node gets a score — fmix64(fnv1a(node-name) XOR fnv1a(key)) — and the nodes ranked by descending score are the key's owner list. The top-R nodes hold the key.

Rendezvous (highest-random-weight) hashing has a property that matters here: adding or removing a node never reorders the surviving nodes in any key's ranking. A new node takes exactly its share of keys and nothing else moves; a dead node's keys fall to the next owner that already holds a replica.

All six SDKs and the server implement this pipeline bit-for-bit and assert the same published test vectors, so a polyglot deployment routes identically from every language.

Replication

Writes fan out from the client to all R owners; the primary's result decides the operation, and a dead replica never fails a write. Reads go to the primary and fail over to the next owner only when the holder is unreachable.

Two self-healing behaviors close the failure gaps:

• A node that is asked for a key it does not own (stale client routing) answers W; the client refreshes its node list and retries once.
• A write whose primary just died fails its connection, which triggers the same refresh-and-retry — so writes recover as soon as discovery drops the dead node (bounded by the liveness timeout, seconds).

Node identity is a random per-process name, decoupled from the network address — routing hashes the name, connections use the address — so a node restarting on the same address is correctly treated as a new member.

Joining a node: staged handoff

Adding a node must not drop the keys it takes ownership of, so joins are orchestrated by discovery in stages:

1. The joining node registers; discovery tells the existing nodes to migrate the keys the newcomer now owns.
2. While migration runs, existing nodes keep serving those keys and forward concurrent writes to the joiner, so no update is lost.
3. When every node reports completion, discovery promotes the joiner into the served node list; clients pick it up on their next refresh.

Migrated-away entries are swept only after the handoff completes, and the forwarding window stays open past completion to cover clients still routing with the older node list.

Discovery without a single point of failure

The registry is deliberately soft state: nodes announce themselves and heartbeat continuously, so a discovery server's knowledge is always rebuildable from the nodes alone. That allows N independent replicas with zero coordination — no consensus, no leader election, no shared storage:

• Every node and client is configured with the same replica list. Nodes heartbeat to all replicas; clients try seeds in order.
• Losing any replica — including the first-listed primary — costs neither cache traffic nor client bootstrap. Only joins pause until the primary is back, because exactly one orchestrator keeps handoffs simple.
• A restarted replica answers B (busy) during a startup grace while live members re-announce, so clients never bootstrap from a half-recovered list; SDKs skip a busy seed like a dead one.

Pair this with a supervisor that restarts a dead replica (systemd, Kubernetes) and two replicas make discovery effectively always available.

Design decisions

Every one of these choices is recorded as an ADR in the repository, with the alternatives that were rejected and why: