Architecture

The clients do the clustering. Nodes store data, discovery servers track membership, and every routing decision happens in the SDK — so the data path has no extra hop.

The three roles

RoleHoldsOn the data path?
nanocached-nodeCache entries (memory-bounded, LRU)Yes — clients connect directly
nanocached-discoverySoft-state membership registryNo — consulted only at bootstrap and on refresh
SDK clientThe node list, R, and open connectionsIs the data path

An SDK bootstraps by connecting to any configured address. The server's handshake reply says whether it is a node or a discovery server; against a discovery server the client fetches the node list (which carries the replication factor R) and then talks to nodes directly.

Routing: rendezvous hashing

For each key, every node gets a score — fmix64(fnv1a(node-name) XOR fnv1a(key)) — and the nodes ranked by descending score are the key's owner list. The top-R nodes hold the key. A namespaced key hashes be32(namespace-length) || namespace || key in place of the bare key, so one namespace's entries still spread over every node and two namespaces' identical key names land on different ones; the default namespace keeps the bare form so existing placements survive an upgrade.

Rendezvous (highest-random-weight) hashing has a property that matters here: adding or removing a node never reorders the surviving nodes in any key's ranking. A new node takes exactly its share of keys and nothing else moves; a dead node's keys fall to the next owner that already holds a replica.

All six SDKs and the server implement this pipeline bit-for-bit and assert the same published test vectors, so a polyglot deployment routes identically from every language.

Replication

Writes fan out from the client to all R owners; the primary's result decides the operation, and a dead replica never fails a write. Reads go to the primary and fail over to the next owner only when the holder is unreachable.

Two self-healing behaviors close the failure gaps:

These healers share one consistency model: each acts on a snapshot of membership taken when it starts, and no two of them coordinate. A planned leave (the SIGTERM decommission — see Deployment) computes its handoffs the same way, from one roster fetch as the drain begins — the roster clients route by, so a node whose join is still pending is not in it: the leaver's V abandons that join, and keys are handed to the members that will own them, not to the joiner. Two membership changes landing inside one drain window — two nodes scaled in at once, or an eviction while a third drains — can therefore leave keys transiently below R: each leaver picks entrants from a roster that still lists the other, so a key whose owners included both comes out of the double-leave one handoff short, and that double-leave case never arises when scale-in retires one node at a time. A put-if-absent handoff cannot overwrite a newer value that the receiver already holds, so reads keep landing on the surviving copies, and a key that does lose its last copy is a cache miss. But the handoff carries no version either: if two survivors hold different copies of a key and both re-replicate it to a fresh node, the first to arrive is kept and the other is refused, newer or not, so a key whose surviving copies disagree can be served with its older value. There is no anti-entropy sweep: the gap closes per key when a client rewrites it or its TTL turns it over, and a key that is only read stays as it is.

Two more ways copies come to disagree or go missing were measured in chaos runs that join, decommission and kill nodes back to back. The first is bounded by the ownership reclaim pass described below; the second has no confirmed cause and no code fix, and is accepted as it stands:

Both are repaired by the next write to the key; a key that is only read is not. Spreading the membership changes out does not remove the leftover copies of the first case, and how often a refused replica write happens when the changes are spread out has not been measured.

Liveness is binary, though: a node that is slow but still answering is never evicted, and every request that touches it waits out its round trip — with R copies on N nodes, about R/N of them. The SDKs' hedged reads (opt-in) send a read to the next owner once the primary has been silent for a configured interval and take the first answer, so reads route around a slow owner; writes cannot, since every copy must be written. An owner that keeps leaving hedged reads unanswered (32 legs still waiting when the interval elapsed) is tried last, so reads for its keys go to the replica first instead of paying the interval each time; its answer is still waited for to settle a miss. Merely having many reads in flight to a healthy owner does not count.

Node identity is a random per-process name, decoupled from the network address — routing hashes the name, connections use the address — so a node restarting on the same address is correctly treated as a new member. When it registers while the old name still holds that address as a member, discovery drops the old name at once if it has sent no heartbeat for about two heartbeat intervals (9 s), instead of waiting for the rest of its liveness timeout: a handoff asked of that address would reach the new node itself. If the old name was heard from more recently, the collision is undecided: the new node’s join is held, and the nodes queued behind it wait too. The old name is dropped as soon as it has been silent for 9 s, which is what a restart looks like, and the join then starts without it. If the old name is heard from again, the address is shared by two live processes: the old name stays, and the new one joins as it would have without the replacement. The hold ends at most 9 s after the old name’s last heartbeat, and a pause of discovery itself does not count towards that silence.

Joining a node: staged handoff

Adding a node must not drop the keys it takes ownership of, so joins are orchestrated by discovery in stages:

  1. The joining node registers; discovery tells the existing nodes to migrate the keys the newcomer now owns.
  2. While migration runs, existing nodes keep serving those keys and forward concurrent writes to the joiner, so no update is dropped. A forwarded delete also leaves a tombstone on the joiner for two minutes: the transfer reads each key as it goes and sends it put-if-absent on its own connection (and with replication, from several old owners at once), so a copy read before the delete can land after it, and the tombstone refuses it; an unconditional forward and a client write are newer than the delete and go in. A forwarded touch that expires the key (a TTL of zero) is applied as such a delete, removal and tombstone alike, rather than stored as an entry that has just expired: an expired entry counts as absent to a put-if-absent copy — the transfer's, or one an old owner relays from its forward queue after a re-replication delivered it — and such a copy landing after the expiry brought the key back on the joiner. What is queued for one joiner is bounded at 64 MiB; once a joiner that has stopped answering has that much waiting, a client write for it is held before its ack until the queue drains, so the backlog waits in the clients, not in the node's memory. The forward is unconditional and may lag, so during the forwarding window a key rewritten or invalidated within a few seconds of a previous write via an old owner can keep the previous value until the next write or TTL expiry; frequently updated keys (counters via INCR, last-seen timestamps) are the realistic exposure. This is accepted rather than fixed: removing it would need versioned writes across the server and every SDK, or an extra joiner round trip on every forwarded write, for a stale that is bounded by TTL and needs a join, a double write with changed content and a lagging forward to coincide.
  3. When every node reports completion, discovery promotes the joiner into the served node list; clients pick it up on their next refresh. Until it is promoted, the joiner answers W to any client request that reaches it — no node list names it yet, but a connection a client dialed to the previous occupant of the same address can land on it — so the client refreshes and retries on an owner that has the key, instead of reading a miss from a node still being filled.
  4. A source keeps the copies it handed off, marked dead, until discovery confirms the join (the joiner shows up in its heartbeat acknowledgment, or in the roster of the next join's migrate request); an abandoned join restores them instead, so no key is lost to a half-finished handoff. A joiner that is promoted and gone again between two of a source's heartbeats never appears in a roster that source sees, so the heartbeat names such a joiner and discovery answers whether it has left; if it has, the source restores the copies and closes the window the same way.

Migrated-away entries are swept only after the join is confirmed, and the forwarding window stays open past completion to cover clients still routing with the older node list. A put-if-absent handoff that reaches the source after its snapshot was taken (a survivor’s re-replication promoting it under a ring the join has already superseded) is stored and relayed to the joiner like any concurrent write, and — while the join is still unconfirmed, so its ring is the newest this node knows — marked like the keys the snapshot sent when that ring gives the key to a joiner and not to this node, so the sweep removes it instead of a later join handing it on. The re-replication pass the join itself triggers on the source never sends a dead copy to the joiner that took it, confirmed or not: the joiner holds the key from the handoff, and a put-if-absent landing after a client expired or deleted it there would bring it back. A decommissioning node relays a handoff it receives to its entrant only when it changed something, so its own drain echoing back through the entrant cannot circle; a TTL given, removed or set to zero counts as a change, a TTL merely shorter than the one held (what an echo carries) does not. Joins are serialized — one handoff at a time, cluster-wide — so a fleet started at once simply queues; a source that has finished one handoff accepts the next join's request while it is still forwarding for the previous one, and keeps forwarding to the earlier joiner until its own window ends — also when that later handoff is abandoned, its joiner evicted, or its transfer cut short. A joiner is tracked by its connection to discovery, which enables TCP keepalive on it, so a joiner whose host vanishes without closing the connection (power loss, a partition) is dropped once keepalive gives up, about 25 seconds on Linux, rather than holding up the joiners queued behind it.

Discovery without a single point of failure

The registry is deliberately soft state: nodes announce themselves and heartbeat continuously, so a discovery server's knowledge is always rebuildable from the nodes alone. That allows N independent replicas with zero coordination — no consensus, no leader election, no shared storage:

Pair this with a supervisor that restarts a dead replica (systemd, Kubernetes) and two replicas make discovery effectively always available.

Membership carries a second, per-node secret alongside the shared one: each node registers with a random membership token, and the frames that skip a node's wrong-node check — a join's M, a decommission's or re-replication's U/u — must present the receiver's token, so holding the cluster-wide shared secret alone doesn't let a client forge them. The flip side is deliberate: any registered node can fetch every member's token from discovery (the T frame), because a draining node must authorize handoffs to every peer, and a survivor re-replicating after an eviction must authorize them to owners it never met at registration time. Compromising one node therefore yields enough to impersonate any node in the cluster — the tokens narrow who can speak the internal frames from "anything holding the shared secret" down to "cluster members", not further. Size the blast radius accordingly: keep the data and operations ports internal (see Deployment's security groups), and treat a node-level compromise as a cluster-level event.

The proxy tier (optional)

Proxyless client-side sharding holds one connection per node per client process — a node-side connection count that grows with fleet size, capped by each node's connection limit. For large fleets, nanocached-proxy terminates many client connections and routes on their behalf: it fetches the roster from discovery, performs the same rendezvous-hash routing the SDKs do (namespaced keys included), fans writes out to all R owners, retries a stale-view W after a roster refresh, and fans c/F to every member. To a client it looks like a single node that owns every key, so any SDK pointed at one address — and any bare protocol client — works against it unchanged, with no ring view or discovery client of its own. Proxies are stateless and scale horizontally: each registers itself with discovery, and a client in proxy mode asks discovery for the proxy list and picks one — the only address anything ever needs is discovery's (a LB/VIP in front of the proxies works too). Clients that prefer cluster-direct connections keep working as before. Every client connection shares one tagged, pipelined backend connection per node, so the node-side connection count is the proxy count — not the fleet size — with bounded per-client and per-backend in-flight windows providing fairness and backpressure.

One documented limit of “looks like a single node”: the proxy enqueues every request on its backend connection in the order the client sent it, but a request the proxy has to retry (a stale-view W after a roster refresh, a replica fallback) is re-enqueued afterwards, outside that order. A client that pipelines a G for a key and, without awaiting its reply, an S for the same key on the same connection can — if the G lands on a stale view during a ring change — receive the value its own S just wrote. That is the same outcome an SDK’s own W retry produces in cluster-direct mode and is not a consistency violation (the write was already on the wire when the read was answered), so it is left as is rather than serialising same-key requests per connection. A client that needs the read to strictly precede the write awaits the G reply before sending the S.

The result of an i, k or x is inside that order. The proxy runs these on the key’s primary and writes the result to the replicas afterwards, as an ordinary set or delete, which it can only do once the primary has answered. A later request on the same connection that touches the same namespace and key — a single-key request, a multi-key one (m, o, y) carrying the key, or a c/F covering it — is held back until that result has been enqueued on the replicas (enqueued, not yet acknowledged). A D pipelined behind an i is therefore not undone by the i’s result, and the results of two pipelined increments reach each replica in the order of the requests. Only the request on the same key waits: other keys on the connection, and every other connection, are not held, and a connection that never sends a same-key request behind an i, k or x sees no added latency. For one that does, a request that is not itself an i, k or x goes out after a round trip to the primary. An i, k or x behind another one is not held: its frame is sent to the primary at once, so a burst of increments of one key on one connection stays pipelined at the primary, and only the write of its result to the replicas waits for the earlier result’s. The wait ends when the primary has answered or the proxy has given up on it, so a dead primary does not hold the connection longer than the proxy’s backend timeouts. What stays outside the order is a retry: a replica write that fails and is sent again can arrive after a later result for the key, and an i, k or x the proxy re-runs after a stale-view W is not waited for. Two connections incrementing one counter are not ordered against each other, through a proxy as in cluster-direct mode (see the accepted behaviour below).

An expiring touch (t or y with a TTL of 0) is inside the order too. The proxy sends the key’s replicas their touch only after the primary has answered, since a replica that expired its copy ahead of a failing primary would lose the value. A later request on the same key would otherwise reach a replica first and have its write expired by the late touch, so it waits until those replica frames are enqueued, as for an i (enqueued, not acknowledged). Touches with any other TTL send every owner its frame at once. Like a g or s, such a touch is in the order only when it has to wait itself, and it releases the key once its frames are enqueued, not once the owners answer.

One case is wider than the key. A multi-key request (m, o, y) that has to wait, or an expiring y, and carries more than 64 keys is not registered key by key, which would leave an entry per key on the connection until the next sweep, outside every in-flight budget (tens of megabytes for a 1 MiB batch of short keys). It is registered for its whole namespace instead, and a later request on that connection in the same namespace waits for it whatever its key. An i, k or x that would have been sent to its primary at once is held until the batch has been enqueued, which lasts as long as the batch’s own wait (bounded by the proxy’s backend timeouts). Other namespaces on the connection and other connections are not held, and a batch of 64 keys or fewer, or one that waits for nothing, registers nothing of this kind.

Known limits

A ring change stalls a node for a few tens of milliseconds per million keys. Re-replication, a staged handoff, a decommission, and the ownership reclaim pass each begin by taking a snapshot of every live key the node holds, and that walk runs on the node’s single cache task, so requests that arrive during it wait. Measured in release builds at about 25 ms per million live keys (5 ms at 200k, 52 ms at 2M). A staged handoff also counts, from that snapshot, how many entries this node will send to the joiner, about 60 ms per million keys; that count runs in 2,000-key slices that yield to other tasks between them, so it spreads out over the request traffic instead of adding to the pause. At the default --max-memory of 256 MiB a node holds on the order of 1.3–1.5M small entries, so one membership change costs a few pauses of some tens of milliseconds each (one per walker that runs, and the reclaim pass runs about twice per change: once after it, once when the grace window moves past the old membership) — the same order as the expiry sweep’s own periodic refill, and only on membership changes, never on the request path or on a timer. The snapshot itself is a transient allocation of 64 bytes per key that --max-memory does not account for (about 180 MiB at 3M keys), so leave that much headroom above the limit. Bounding it would need an ordered per-namespace key index that every write maintains; that standing cost on the hot path was judged not worth removing a rare, bounded pause. Size the node count so a single node’s keyspace stays in the low millions if even that pause matters to you.

Accepted behaviour

Each item below is a behaviour that was designed in or measured, judged, and left as it is. The sections above describe most of them in context; this list is the short form, one line per item with why it stays, so the next such decision has a place to go. Configuration facts (request size, connection caps, idle timeout) are in the per-page limits sections, not here.

Consistency and membership changes

Data model and operations

SDKs and adapters

Design decisions

The main choices, in summary:

Decision
Client-side hashing with a lightweight discovery server
Shared-secret authentication via environment variable
TLS via rustls, required once configured
Server type in the auth response → one unified connect
Staged node join with orchestrated data handoff
Node identity decoupled from network address
Discovery HA via soft-state replicas and announces
Client-side replication via rendezvous hashing