Architecture
The clients do the clustering. Nodes store data, discovery servers track membership, and every routing decision happens in the SDK — so the data path has no extra hop.
The three roles
| Role | Holds | On the data path? |
|---|---|---|
nanocached-node | Cache entries (memory-bounded, LRU) | Yes — clients connect directly |
nanocached-discovery | Soft-state membership registry | No — consulted only at bootstrap and on refresh |
| SDK client | The node list, R, and open connections | Is the data path |
An SDK bootstraps by connecting to any configured address. The server's handshake reply says whether it is a node or a discovery server; against a discovery server the client fetches the node list (which carries the replication factor R) and then talks to nodes directly.
Routing: rendezvous hashing
For each key, every node gets a score —
fmix64(fnv1a(node-name) XOR fnv1a(key)) — and the nodes
ranked by descending score are the key's owner list. The top-R nodes
hold the key. A namespaced key
hashes be32(namespace-length) || namespace || key in place
of the bare key, so one namespace's entries still spread over every node
and two namespaces' identical key names land on different ones; the
default namespace keeps the bare form so existing placements survive an
upgrade.
Rendezvous (highest-random-weight) hashing has a property that matters here: adding or removing a node never reorders the surviving nodes in any key's ranking. A new node takes exactly its share of keys and nothing else moves; a dead node's keys fall to the next owner that already holds a replica.
All six SDKs and the server implement this pipeline bit-for-bit and assert the same published test vectors, so a polyglot deployment routes identically from every language.
Replication
Writes fan out from the client to all R owners; the primary's result decides the operation, and a dead replica never fails a write. Reads go to the primary and fail over to the next owner only when the holder is unreachable.
Two self-healing behaviors close the failure gaps:
- A node that is asked for a key it does not own (stale client
routing) answers
W; the client refreshes its node list and retries once. - A write whose primary just died fails its connection, which triggers the same refresh-and-retry — so writes recover as soon as discovery drops the dead node (bounded by the liveness timeout, seconds).
- Every heartbeat acknowledgment carries the current member list, so
the surviving nodes' own ring view follows that eviction within one
heartbeat interval too — a key re-homed from the dead node is served
by its new owner rather than answered
W. Inside that interval, a node asked for a key its own view does not grant asks discovery for the current list before answeringW(at most once a second, each ask bounded at a second), so a client that already routes by the new list is served, not refused. That holds for increments and compare-and-set too: a replica that the removal promoted to a key's primary runs them as soon as the published list makes it the primary, without waiting for its own ring to catch up. - That same ring-view update triggers re-replication:
rendezvous hashing's removal property means an eviction (or any leave)
can only ever promote a node into a key's top-R, never leave a
slot vacant, but nothing had sent that promoted node a copy — so at
R ≥ 2 the cluster silently ran one replica short after the
first failure, and a second failure on the sole remaining
copy lost the key. Once a survivor notices the drop, the highest-ranked
surviving old owner of each affected key streams it to the newly
promoted owner(s) as a put-if-absent handoff (
U's trailingAtoken — see Protocol), so a re-replication racing a newer client write on the receiver can never regress it. While the pass runs, and for 60 s after it, each sender also mirrors the client writes it accepts to that promoted owner, as it would to a decommission’s entrant: a client keeps its node list for up to 30 s after the eviction, and until it refreshes its writes reach only the surviving old owners (the leg to the evicted node fails and is swallowed), so without the mirror the promoted owner kept the copy the pass handed it, missing the write, delete or increment acknowledged meanwhile. This runs independently of, and is ordered against, a join in progress: a join's sweep of its own dead-copy marks waits for any in-flight re-replication to finish first (so it can't reclaim a survivor's only copy of a key re-replication hasn't delivered yet), and a re-replication that starts while a join is running waits for the join to finish first, since the join changes the ring again. - A node that comes back after being evicted starts over
instead of resuming. A process that was stopped, or cut off from
discovery, for longer than the liveness timeout is evicted while it is
silent; the survivors re-replicate its keys and the clients write to
them. If it were put back in the ring with its old cache it would serve
the older values (and the deleted keys), and the survivors, no longer
owners, would drop the newer copies at their next reclaim pass. So
discovery, which remembers for an hour the names it evicted, answers
such a node’s re-announce with
Ginstead of registering it. The node answers every key requestW, clears its cache, and joins again like a new node: the owners hand it the keys it owns now. What it held before the eviction is not kept, at any replication factor. A discovery that has just been restarted remembers nothing, and the nodes that kept heartbeating through the restart re-register at once, as before. Nodes are not evicted by a sweep that came after discovery itself had not been running for longer than half the liveness timeout (a stopped or suspended discovery process, which finds every heartbeat old when it resumes): that sweep gives every node and proxy a fresh liveness window instead, since they were not silent and nobody moved on without them. A join in progress gets the same: the pause is not counted against its migration timeout. Nor are nodes sent back when every one of them was evicted together, as when the path between discovery and the whole fleet was cut: discovery answersGonly while some node has been a member in unbroken contact since before that node was evicted, since only such a node was served a roster without it and can have taken its keys over, and otherwise registers the node as it announces. A member whose contact was broken across the eviction does not count: discovery keeps the gaps of nine seconds or more between a member’s five-second heartbeats (at least one did not arrive), and a gap that began before the eviction and ended after it means the member saw no roster without the evicted node. So when the path to the whole fleet heals after the first evictions and before the last, the nodes still listed were cut off just the same, and the evicted ones are registered with their caches rather than sent back. A member that has not been heard from since the eviction and has been silent for nine seconds or more already counts as cut off when the evicted node announces, before its own heartbeat arrives, so the outcome does not depend on which of them reaches discovery first. A gap that began after the eviction, or that spans some other node’s departure, changes nothing, since the member had already been served the roster without the evicted node: one dropped heartbeat connection later does not make a node that was evicted earlier come back with its old cache. A member whose single heartbeat was a few seconds late still counts as in contact throughout, so a node evicted while another was out of contact for less than nine seconds across its eviction, or announcing while another that has not been heard from since the eviction has been silent for less than nine seconds, is still sent back. A node evicted longer ago than that hour, or by a standby discovery replica (which only lists nodes), is registered as before.
These healers share one consistency model: each acts on a snapshot of
membership taken when it starts, and no two of them coordinate. A
planned leave (the SIGTERM decommission — see
Deployment) computes its handoffs the same
way, from one roster fetch as the drain begins — the roster clients route
by, so a node whose join is still pending is not in it: the leaver's
V abandons that join, and keys are handed to the members that
will own them, not to the joiner. Two membership changes
landing inside one drain window — two nodes scaled in at once, or an
eviction while a third drains — can therefore leave keys transiently
below R: each leaver picks entrants from a roster that still lists the
other, so a key whose owners included both comes out of the double-leave
one handoff short, and that double-leave case never arises when scale-in
retires one node at a time. A put-if-absent handoff cannot overwrite a
newer value that the receiver already holds, so reads keep landing on
the surviving copies, and a key that does lose its last copy is a cache
miss. But the handoff carries no version either: if two survivors hold
different copies of a key and both re-replicate it to a fresh node, the
first to arrive is kept and the other is refused, newer or not, so a key
whose surviving copies disagree can be served with its older value.
There is no anti-entropy sweep: the gap closes per key when a client
rewrites it or its TTL turns it over, and a key that is only read stays
as it is.
Two more ways copies come to disagree or go missing were measured in chaos runs that join, decommission and kill nodes back to back. The first is bounded by the ownership reclaim pass described below; the second has no confirmed cause and no code fix, and is accepted as it stands:
- A client write that reaches a node a join has displaced releases that copy's dead-copy mark, so the mark sweep never reclaims it (the release is deliberate: a mark that outlives its value once deleted fresh writes). Such a copy is removed instead by the ownership reclaim pass: after a membership change (and once more when the 120 s grace moves past the previous membership) a node drops the keys it has owned under no membership discovery published to it during the last 120 s, provided no handoff write for a key it did not yet own under its adopted membership reached it in that time (a handoff write for a key it already owns counts, to the pass, as a client write) and it is not itself leaving, mid-handoff or still forwarding for a join. The pass is checked every 30 s but skipped while nothing it judges against has moved. Until then the node keeps more entries than its share of the keys, and if an eviction makes it an owner again inside that window the copy is live again — by design, since a membership that flips back within the grace is exactly what the grace waits for — so an old value can still come back and be served or re-replicated. Under back-to-back membership changes the pass keeps postponing and the leftovers look as they did before; they go once the cluster has been quiet for the grace. A dead copy that was not rewritten (still marked) is judged differently when the node owns its key again: the mark sweep keeps it only when no other current owner was an owner under the previously published ring — the one case in which nobody else can re-replicate a fresher state (a replication factor of 1 with the joiner gone, or every other holder gone). Otherwise the dead copy goes, because every write since the join went to the owners and a delete or an expiring touch leaves nothing a re-replication could carry back over it; the re-replication after the change, which overwrites a copy that is still marked, brings the key back only if an owner still has it. The sweep runs as soon as a membership is published, so a dead copy is served live for at most the moment between the two. Nor is a dead copy handed on: a decommission drain, a re-replication after a membership change and a join transfer leave out a copy whose mark has been released (while the join that made the mark is still undecided, a drain or a re-replication sends it as before, since an abandoned join would make it live again).
- A replica write refused by a live node (
WrongNodeError, while the client's node list and the nodes' views of the ring disagree) is swallowed by the SDK, so the key sits on one copy until its next write. If the node holding that copy is decommissioned or killed first, the newest value is lost and the surviving owners serve the older one. The cause was a node whose own ring lagged the list discovery had published — by up to a heartbeat after a leave or an eviction, and for as long as a join was pending on it — refusing a key it was about to own; earlier runs (Python SDK, Docker) saw about 15–17 % of writes with a replica leg refused, a later run about 1.5 %. A node now asks discovery for the current list before refusing such a key, so the refusal is left only for the ask failing or taking over a second (discovery unreachable), and for a client whose list is older than the node's.
Both are repaired by the next write to the key; a key that is only read is not. Spreading the membership changes out does not remove the leftover copies of the first case, and how often a refused replica write happens when the changes are spread out has not been measured.
Liveness is binary, though: a node that is slow but still answering is never evicted, and every request that touches it waits out its round trip — with R copies on N nodes, about R/N of them. The SDKs' hedged reads (opt-in) send a read to the next owner once the primary has been silent for a configured interval and take the first answer, so reads route around a slow owner; writes cannot, since every copy must be written. An owner that keeps leaving hedged reads unanswered (32 legs still waiting when the interval elapsed) is tried last, so reads for its keys go to the replica first instead of paying the interval each time; its answer is still waited for to settle a miss. Merely having many reads in flight to a healthy owner does not count.
Node identity is a random per-process name, decoupled from the network address — routing hashes the name, connections use the address — so a node restarting on the same address is correctly treated as a new member. When it registers while the old name still holds that address as a member, discovery drops the old name at once if it has sent no heartbeat for about two heartbeat intervals (9 s), instead of waiting for the rest of its liveness timeout: a handoff asked of that address would reach the new node itself. If the old name was heard from more recently, the collision is undecided: the new node’s join is held, and the nodes queued behind it wait too. The old name is dropped as soon as it has been silent for 9 s, which is what a restart looks like, and the join then starts without it. If the old name is heard from again, the address is shared by two live processes: the old name stays, and the new one joins as it would have without the replacement. The hold ends at most 9 s after the old name’s last heartbeat, and a pause of discovery itself does not count towards that silence.
Joining a node: staged handoff
Adding a node must not drop the keys it takes ownership of, so joins are orchestrated by discovery in stages:
- The joining node registers; discovery tells the existing nodes to migrate the keys the newcomer now owns.
- While migration runs, existing nodes keep serving those keys and
forward concurrent writes to the joiner, so no update is dropped. A
forwarded delete also leaves a tombstone on the joiner for two
minutes: the transfer reads each key as it goes and sends it
put-if-absent on its own connection (and with replication, from
several old owners at once), so a copy read before the delete can
land after it, and the tombstone refuses it; an unconditional forward
and a client write are newer than the delete and go in. A forwarded
touch that expires the key (a TTL of zero) is applied as such a
delete, removal and tombstone alike, rather than stored as an entry
that has just expired: an expired entry counts as absent to a
put-if-absent copy — the transfer's, or one an old owner relays from
its forward queue after a re-replication delivered it — and such a
copy landing after the expiry brought the key back on the joiner. What
is queued for one joiner is bounded at 64 MiB; once a joiner
that has stopped answering has that much waiting, a client write for
it is held before its ack until the queue drains, so the backlog
waits in the clients, not in the node's memory. The
forward is unconditional and may lag, so during the forwarding window
a key rewritten or invalidated within a few seconds of a previous
write via an old owner can keep the previous value until the next
write or TTL expiry; frequently updated keys (counters via
INCR, last-seen timestamps) are the realistic exposure. This is accepted rather than fixed: removing it would need versioned writes across the server and every SDK, or an extra joiner round trip on every forwarded write, for a stale that is bounded by TTL and needs a join, a double write with changed content and a lagging forward to coincide. - When every node reports completion, discovery promotes the joiner
into the served node list; clients pick it up on their next
refresh. Until it is promoted, the joiner answers
Wto any client request that reaches it — no node list names it yet, but a connection a client dialed to the previous occupant of the same address can land on it — so the client refreshes and retries on an owner that has the key, instead of reading a miss from a node still being filled. - A source keeps the copies it handed off, marked dead, until discovery confirms the join (the joiner shows up in its heartbeat acknowledgment, or in the roster of the next join's migrate request); an abandoned join restores them instead, so no key is lost to a half-finished handoff. A joiner that is promoted and gone again between two of a source's heartbeats never appears in a roster that source sees, so the heartbeat names such a joiner and discovery answers whether it has left; if it has, the source restores the copies and closes the window the same way.
Migrated-away entries are swept only after the join is confirmed, and the forwarding window stays open past completion to cover clients still routing with the older node list. A put-if-absent handoff that reaches the source after its snapshot was taken (a survivor’s re-replication promoting it under a ring the join has already superseded) is stored and relayed to the joiner like any concurrent write, and — while the join is still unconfirmed, so its ring is the newest this node knows — marked like the keys the snapshot sent when that ring gives the key to a joiner and not to this node, so the sweep removes it instead of a later join handing it on. The re-replication pass the join itself triggers on the source never sends a dead copy to the joiner that took it, confirmed or not: the joiner holds the key from the handoff, and a put-if-absent landing after a client expired or deleted it there would bring it back. A decommissioning node relays a handoff it receives to its entrant only when it changed something, so its own drain echoing back through the entrant cannot circle; a TTL given, removed or set to zero counts as a change, a TTL merely shorter than the one held (what an echo carries) does not. Joins are serialized — one handoff at a time, cluster-wide — so a fleet started at once simply queues; a source that has finished one handoff accepts the next join's request while it is still forwarding for the previous one, and keeps forwarding to the earlier joiner until its own window ends — also when that later handoff is abandoned, its joiner evicted, or its transfer cut short. A joiner is tracked by its connection to discovery, which enables TCP keepalive on it, so a joiner whose host vanishes without closing the connection (power loss, a partition) is dropped once keepalive gives up, about 25 seconds on Linux, rather than holding up the joiners queued behind it.
Discovery without a single point of failure
The registry is deliberately soft state: nodes announce themselves and heartbeat continuously, so a discovery server's knowledge is always rebuildable from the nodes alone. That allows N independent replicas with zero coordination — no consensus, no leader election, no shared storage:
- Every node and client is configured with the same replica list. Nodes heartbeat to all replicas; clients try their configured addresses in order.
- Losing any replica — including the first-listed primary — costs neither cache traffic nor client bootstrap. Only joins pause until the primary is back, because exactly one orchestrator keeps handoffs simple.
- A restarted replica answers
B(busy) during a startup grace while live members re-announce, so clients never bootstrap from a half-recovered list; SDKs skip a busy replica like a dead one. A join requested during that grace is held until it ends, for the same reason: its handoff must come from every member, not from the few that have re-announced so far.
Pair this with a supervisor that restarts a dead replica (systemd, Kubernetes) and two replicas make discovery effectively always available.
Membership carries a second, per-node secret alongside the shared
one: each node registers with a random membership token, and
the frames that skip a node's wrong-node check — a join's
M, a decommission's or re-replication's
U/u — must present the receiver's
token, so holding the cluster-wide shared secret alone doesn't let a
client forge them. The flip side is deliberate: any registered node can
fetch every member's token from discovery (the T frame),
because a draining node must authorize handoffs to every peer, and a
survivor re-replicating after an eviction must authorize them to owners
it never met at registration time. Compromising one node therefore
yields enough to impersonate any node in the cluster — the tokens
narrow who can speak the internal frames from "anything holding the
shared secret" down to "cluster members", not further. Size the blast
radius accordingly: keep the data and operations ports internal (see
Deployment's security groups), and treat
a node-level compromise as a cluster-level event.
The proxy tier (optional)
Proxyless client-side sharding holds one connection per node per
client process — a node-side connection count that grows with fleet
size, capped by each node's connection limit. For large fleets,
nanocached-proxy terminates many client connections and
routes on their behalf: it fetches the roster from discovery, performs
the same rendezvous-hash routing the SDKs do (namespaced keys included),
fans writes out to all R owners, retries a stale-view W
after a roster refresh, and fans c/F to every
member. To a client it looks like a single node that owns every key, so
any SDK pointed at one address — and any bare protocol client — works
against it unchanged, with no ring view or discovery client of its own.
Proxies are stateless and scale horizontally: each registers itself with
discovery, and a client in proxy mode asks discovery for the proxy list
and picks one — the only address anything ever needs is discovery's (a
LB/VIP in front of the proxies works too). Clients that prefer
cluster-direct connections keep working as before. Every client
connection shares one tagged, pipelined backend connection per node, so
the node-side connection count is the proxy count — not the fleet size —
with bounded per-client and per-backend in-flight windows providing
fairness and backpressure.
One documented limit of “looks like a single node”: the
proxy enqueues every request on its backend connection in the order the
client sent it, but a request the proxy has to retry (a
stale-view W after a roster refresh, a replica fallback) is
re-enqueued afterwards, outside that order. A client that pipelines a
G for a key and, without awaiting its reply, an
S for the same key on the same connection can — if
the G lands on a stale view during a ring change —
receive the value its own S just wrote. That is the same
outcome an SDK’s own W retry produces in
cluster-direct mode and is not a consistency violation (the write was
already on the wire when the read was answered), so it is left as is
rather than serialising same-key requests per connection. A client
that needs the read to strictly precede the write awaits the
G reply before sending the S.
The result of an i, k or x is
inside that order. The proxy runs these on the key’s primary and
writes the result to the replicas afterwards, as an ordinary set or delete,
which it can only do once the primary has answered. A later request on the
same connection that touches the same namespace and key — a
single-key request, a multi-key one (m, o,
y) carrying the key, or a c/F
covering it — is held back until that result has been enqueued on
the replicas (enqueued, not yet acknowledged). A D
pipelined behind an i is therefore not undone by the
i’s result, and the results of two pipelined increments
reach each replica in the order of the requests. Only the request on the
same key waits: other keys on the connection, and every other connection,
are not held, and a connection that never sends a same-key request behind
an i, k or x sees no added latency.
For one that does, a request that is not itself an i,
k or x goes out after a round trip to the primary.
An i, k or x behind another one is
not held: its frame is sent to the primary at once, so a burst of
increments of one key on one connection stays pipelined at the primary, and
only the write of its result to the replicas waits for the earlier
result’s.
The wait ends when the primary has answered or the proxy has given up on
it, so a dead primary does not hold the connection longer than the
proxy’s backend timeouts. What stays outside the order is a retry: a
replica write that fails and is sent again can arrive after a later result
for the key, and an i, k or x the
proxy re-runs after a stale-view W is not waited for. Two
connections incrementing one counter are not ordered against each other,
through a proxy as in cluster-direct mode (see the accepted behaviour
below).
An expiring touch (t or y with a TTL of 0) is
inside the order too. The proxy sends the key’s replicas their touch only
after the primary has answered, since a replica that expired its copy ahead of
a failing primary would lose the value. A later request on the same key would
otherwise reach a replica first and have its write expired by the late touch,
so it waits until those replica frames are enqueued, as for an i
(enqueued, not acknowledged). Touches with any other TTL send every owner its
frame at once. Like a g or s, such a touch is in the
order only when it has to wait itself, and it releases the key once its frames
are enqueued, not once the owners answer.
One case is wider than the key. A multi-key request (m,
o, y) that has to wait, or an expiring
y, and carries more than 64
keys is not registered key by key, which would leave an entry per key on the
connection until the next sweep, outside every in-flight budget (tens of
megabytes for a 1 MiB batch of short keys). It is registered for its
whole namespace instead, and a later request on that connection in the same
namespace waits for it whatever its key. An i, k or
x that would have been sent to its primary at once is held until
the batch has been enqueued, which lasts as long as the batch’s own wait
(bounded by the proxy’s backend timeouts). Other namespaces on the
connection and other connections are not held, and a batch of 64 keys or
fewer, or one that waits for nothing, registers nothing of this kind.
Known limits
A ring change stalls a node for a few tens of milliseconds
per million keys. Re-replication, a staged handoff, a
decommission, and the ownership reclaim pass each begin by taking a
snapshot of every live key the node holds, and that walk runs on the
node’s single cache task, so requests that arrive during it wait.
Measured in release builds at about 25 ms per million live keys
(5 ms at 200k, 52 ms at 2M). A staged handoff also counts, from
that snapshot, how many entries this node will send to the joiner, about
60 ms per million keys; that count runs in 2,000-key slices that
yield to other tasks between them, so it spreads out over the request
traffic instead of adding to the pause. At the default
--max-memory of 256 MiB a node holds on the order of
1.3–1.5M small entries, so one membership change costs a few pauses of
some tens of milliseconds each (one per walker that runs, and the
reclaim pass runs about twice per change: once after it, once when the
grace window moves past the old membership) — the same order as
the expiry sweep’s own periodic refill, and only on membership
changes, never on the request path or on a timer. The snapshot itself
is a transient allocation of 64 bytes per key that
--max-memory does not account for (about 180 MiB at
3M keys), so leave that much headroom above the limit. Bounding it would need an ordered
per-namespace key index that every write maintains; that standing cost
on the hot path was judged not worth removing a rare, bounded pause.
Size the node count so a single node’s keyspace stays in the low
millions if even that pause matters to you.
Accepted behaviour
Each item below is a behaviour that was designed in or measured, judged, and left as it is. The sections above describe most of them in context; this list is the short form, one line per item with why it stays, so the next such decision has a place to go. Configuration facts (request size, connection caps, idle timeout) are in the per-page limits sections, not here.
Consistency and membership changes
- During a join’s or a decommission’s forwarding window,
a key rewritten or invalidated within seconds of a write via an old
owner may keep the previous value until the next write or TTL expiry
(a lagging unconditional forward; relay coalescing). Measured on
2026-11-01 over 21 chaos cells (join, decommission and kill rounds under
load): 2 cells showed it, each on keys acknowledged within seconds of a
decommissioning node completing its handoff — one deleted key was
served again, three keys set again after a delete read back as missing,
and two keys rewritten with a TTL kept the earlier no-TTL copy, which
then never expired. A clear is
forwarded through every window (a join’s, a decommission’s,
and an eviction’s re-replication) behind the writes queued before
it, so a cleared key does not come back from such a forward; the mirror case
remains: a client whose node list dropped the old owner between its
write and its clear sends the clear only to the new owners, and the
old owner’s late forward can bring that one write back. The
realistic exposure is a counter incremented concurrently by several
callers during the window: the result the primary replicates to an
old owner is forwarded back to the primary as an unconditional write,
and when it lands after a later increment it rewinds the counter, so
usually one to three increments are lost (five were seen among twelve
parallel increments) on about one counter per join in a
loaded cluster. One caller is exposed too when its increments follow
each other within a few milliseconds, as a forward lags that long:
with four back-to-back increments about 1 ms apart on each counter,
357 to 395 of 2,000 counters lost an acknowledged increment in
six-round join, decommission and kill runs. With increments spaced well
above the lag they are practically exact (about one in 12,000 in a
join window, measured). A compare-and-set loop is exposed the same way: the
result of an acknowledged
replacethat a lagging forward rewinds is gone, and the nextreplacesucceeds on the older value (one to seven of 2,000 chains lost an acknowledged update, out of about 11,000 acknowledged replaces, in each of three six-round join, decommission and kill runs with two clients appending to the same key). The old owner cannot tell a replicated result from a client’s write it must forward. Accepted 2026-10-17, and for compare-and-set 2026-10-28: the stale is bounded by TTL and needs a membership change, a double write with changed content and a lagging forward to coincide, and removing it would need versioned writes in the server and every SDK. Versioned writes are not part of 1.0.0. The plan is a capability a client opts into on itsAframe (tokenV), which a 1.x release can add without touching any frame the protocol freezes: the primary of a key stamps each write with a hybrid logical clock, replica legs and forwards carry the stamp and apply a write only if it is newer, a tombstone keeps the stamp of its delete, and a clear becomes “remove what is older than this time”, so a copy of it applied late cannot drop a newer write. The price is one extra round trip on replicated writes that do not run fire-and-forget, about 8 bytes per entry and a stamp on each write. The six oracles intests/e2ealready print the anomaly of each item on this list; an item counts as closed when its printed counter is 0 over the nightly chaos matrix with the capability on. - Two clients incrementing the same counter at the same time can leave a replica behind the primary, with no membership change involved. The primary applies the increments in one order, but each client writes the result it got to the replica as an ordinary set once its own answer arrives, and two clients read their answers from different connections, so the replica can receive the larger result first and the smaller one after it. The primary holds the right value; the replica holds one or two increments less until the next increment of the key writes it again, and a primary lost before then leaves the replica’s value as the counter. Measured with the Python SDK on three nodes, after the join window had passed, with counters each incremented 12 times in turns of four concurrent increments: 4 of 600, 7 of 2,000 and 21 of 2,000 counters had a replica behind in three runs (in 11 of the 21 the replica writes were issued in the opposite order of the primary’s values), against none of 2,000 with one client (one connection per node) and none with serial increments. Accepted 2026-10-29: ordering the replica writes would need a version on every replica write in the server and every SDK. A single client, or one client per counter, does not see it, through a proxy too: the proxy enqueues the replica write of an increment’s result before it sends the next request of that connection for the key (see the proxy tier above). Two connections behind one proxy are exposed like two clients.
- During a join’s or a decommission’s forwarding window, a
touch (
t, or a key of aybatch) is forwarded as the value the old owner holds with the new TTL, written like a set, not as a touch: the joiner may not hold the key yet, and a touch it could not apply would be lost with the copy left on the handoff’s old TTL. A touch that runs on the old owner just before a client’s write to the same key, through another connection, can have its forward queued after that write’s, and then the joiner holds the earlier value. Measured with 20 keys written with a 1,000 s TTL, handed to a joiner, and shortened to 10 s on the old owner alone while its window was open: the joiner expired all 20 together with the old owner (with the forward disabled it still held all 20 once the TTL had passed). Accepted 2026-10-24: the same lagging unconditional forward as above, with the same bounds (a TTL, a membership change, and a concurrent write to a key that is also being touched), and a touch that could not create the key would make the shortened TTL depend on whether the handoff had delivered it yet. - During a join’s, a decommission’s or an eviction’s
forwarding window, writes a client makes right after
clear()returned can be missing on one or more owners until they are written again or expire. A node passes a clear on to the nodes of the window through the same queue as the writes it mirrors, and a forwarded clear is passed on again from there; the client’s own clear is acknowledged once the nodes it knows have applied it, not once those copies have, so a copy can land on a node after the client’s next writes there and drop them. Measured with a client that sets 20 keys into a namespace, clears it and sets 20 others at once, fifteen times in a run, around a decommission, an eviction and two mixed join/decommission sequences: a clear that broke the invariants in 12 of 16 runs, mostly one whose new keys were missing on every owner (a cache miss, not a stale read); the same client with no membership change did not (6 runs). The size of it, measured with 600 such cycles of 20 keys each in a namespace of its own, spread over six join, decommission and kill rounds with the clears landing in the forwarding windows: 7,300 to 8,300 of the 12,000 keys set after a clear were missing on every owner (three runs), in 400 to 450 of the 600 cycles. In two traced runs a clear was not applied once on each node but 30 to 50 times, because a forwarded clear goes on to every joiner still inside its grace, and the copies arrive a median 5 to 7 ms and up to 1.8 s after the client’s clear was acknowledged; nearly all of the new keys that were dropped were dropped by such a copy. The same cycles in a steady three-node cluster lost none. Accepted 2026-10-23: the missing write is a miss on a cache, and acknowledging a clear only after its copies were applied would make every clear wait for every window’s queue and needs a delivery signal on the forward path. Not passing a forwarded clear on again was tried and rejected: it removed nearly all of the whole-namespace misses but let a cleared key come back with its old value on a join followed by a decommission, which is the worse outcome. - During an eviction’s re-replication pass or a decommission’s
drain, a
clear()can leave one key per sender connection back with its old value. Both read each key just before they send it, on a connection of their own, so a copy read before the clear can land after it. A delete or an expiry is not exposed: the window passes it on ahead of the copy and the receiver keeps a tombstone that refuses the copy, and a clear has no counterpart of that. Measured on three nodes with the pass held at its first key while the client changed every key on both owners: a delete, an expiry, a rewrite and a TTL removal left nothing behind; a clear left that one key with its old value (the pass; the drain reads and sends the same way and was not measured). A join’s transfer is not exposed, because a clear that arrives during it is replayed on the transfer’s own stream, in order. Accepted 2026-10-28: the exposure is the one key in flight per clear during a membership change, and closing it would need each copy to carry when it was read, or the receiver to refuse every put-if-absent of the namespace for a while, which would also refuse the copies of keys written after the clear. The same runs left 0 to 44 keys with their old value (up to 10 in one cycle, 0 to 8 cycles of 600), more than one per sender connection; the path of the extra copies was not traced. - A key deleted and set again within two minutes of the delete can be one copy short after a membership change, until the next re-replication or the next write. A delete relayed through a window leaves a tombstone for two minutes on the node that received it, and the set that follows reaches only the owners the sender knew at that moment. If a membership change then makes that node an owner, the sender’s put-if-absent copy is refused by the tombstone, because the receiver cannot tell it from a copy read before the delete. The SDKs read the other owner and see the value. Measured with 2,000 keys deleted and set again 1–6 seconds later across join, decommission and kill/restart rounds: 3 to 8 keys per run were missing on one owner (the refusal came 40 to 45 seconds after the delete, on a node with no entry written after the tombstone). A tombstone that sat on top of a copy written after the delete is dropped when that copy is swept or reclaimed, which removed the larger share of the same loss (37 to 42 keys per run before). Accepted 2026-10-28: the copy stays short only for keys recreated that quickly during a membership change, and telling the two copies apart would need each copy to carry when it was last written, a wire change; a shorter tombstone would let a copy read before a delete come back.
- A write a client sends to an old owner is relayed to that owner’s joiner, but not on to the joiner’s own joiner: a node relays a write that reached it as a handoff only to a joiner discovery has not confirmed yet. With two joins inside one forwarding window and a client whose node list predates both, the write reaches the first joiner and is missing from the second (the key’s owner now) until the next write or TTL expiry. Accepted 2026-10-21: relaying handoffs to confirmed joiners is what sent a decommissioning node’s drained value back to it through its entrant and overwrote the newer value it held, and the two-join case needs a list older than two membership changes within one window. A clear still chains through every joiner.
- During a re-replication’s forwarding window (the pass and 60 s after it), a second eviction replaces the window with its own snapshot, so a client still routing by the first eviction’s node list can miss the first promoted owner for the rest of its refresh interval; and with R ≥ 3 two senders’ unconditional forwards of one key can interleave at the promoted owner, which is the lagging-forward item above. Accepted: both are the same exposure the join and decommission windows carry. A join adopted inside the window is not one of them: a write is mirrored to a promoted owner only while that owner still owns the key under the newest membership the node has adopted, so an owner the join has just displaced gets nothing that would revive the dead copy it holds.
- Concurrent writes to one key have no order across its owners: each
owner keeps the write that reached it last, so two callers (threads,
tasks, or clients) racing a
setor asetagainst adeleteon the same key can leave the owners holding different values, or one holding the key and the other not, with every leg acknowledged and nothing instats()— until the next write or TTL expiry. Measured at up to a third of the raced keys with 8 concurrent writers per key from one client. Accepted 2026-10-20: ordering it would need a version on every write in the server and every SDK, and a per-key lock inside one SDK would not cover two clients.nanocached-proxydoes not remove it: it sends each write to the owners concurrently and orders nothing between requests of different connections, so two connections writing one key can reach one owner in one order and the other owner in the other. Through one proxy in front of a 2-node, 2-replica cluster, 8 connections writing one key for 300 rounds left no owners disagreeing, 16 for 3,000 rounds left 2 rounds, and 24 for 6,000 rounds left 8 and 57 in two runs. Compare-and-set runs on the primary alone and showed no disagreement between owners in the measurements below (about 34,000 acknowledged replaces in three runs). A counter is exposed in one case, the next item. - Re-replication handoff carries no version: when two survivors hold different copies, the first to reach the fresh node is kept, newer or not. A version would have to ride on every entry and every SDK write path; not adopted.
- No anti-entropy sweep: diverged copies converge on the next write or TTL turnover, and a read-only key stays as it is. A sweep would be a standing cross-node scan for entries that TTL turns over anyway.
- Two membership changes inside one drain window (two scale-ins at once, an eviction while a third drains) leave keys transiently below R. Each healer acts on one membership snapshot and none coordinate; retire nodes one at a time and the case does not arise.
- A drain that runs out of its budget leaves the keys it could not hand
off at R=1 (the leaver logs how many:
could not be handed off within the drain budget) until the survivors’ re-replication pass for the leave has run, which takes seconds to tens of seconds for tens of thousands of entries. A crash of the remaining owner inside that window loses those keys for good: at R=2 that is two copy losses. The budget is a soft limit by design (the leaver must exit), so the operator side is to size--drain-timeoutso the WARN does not appear, and to let the cluster settle after a leave before the next change. - A displaced copy loses its dead-copy mark on rewrite and is removed by the ownership reclaim pass after the 120 s grace; an eviction that flips membership back inside the grace makes it live again, so a key deleted on the owners after that rewrite can come back with the rewritten value. The mark is released on rewrite because a mark that outlived its value once deleted fresh writes, and the grace exists so that a membership that flips back is not treated as a loss. A displaced copy that was never rewritten does not come back this way: the mark sweep lets it go whenever another current owner saw the writes it missed, and that owner's re-replication decides whether the key still exists.
- A replica leg refused with
Wby a live node is swallowed by the SDK; the key sits on one copy until its next write, and if that node dies first the newest value is lost. Seen on 1.5–17 % of writes in chaos runs that join, decommission and kill nodes back to back. The cause found on 2026-10-20 — a node's own ring lagging the published list by up to a heartbeat, longer while a join is pending on it — is closed by the node asking discovery before it refuses; what remains accepted is the refusal when that ask fails or takes over a second, and when the client's list is the older one. Multi-set takes the same path: the answer of a key's primary decides its outcome and a replica leg's failure is only counted. In chaos runs that write 2,000 keys per class in batches of ten, 5–25 keys per class ended with an owner one generation behind, the same as when the batches were sent as single sets (almost all on the replica). - The proxy re-enqueues a retried request (stale-view
W, replica fallback) outside the client’s send order; a pipelinedGthenSon one connection can read its ownS. The same outcome an SDK’s ownWretry produces in cluster-direct mode; serialising same-key requests per connection was judged not worth it. A client that needs the order awaits theGreply first. The replica write of ani,korxresult is in the order (a later request on the same key waits for it to be enqueued), as are the replica legs of an expiring touch (torywith a TTL of 0) and of a delete; a re-sent replica write is not. - A multi-key request (
m,o,y) of more than 64 keys that the proxy has to defer, because it waits for an earlieri,korxresult or other ordered request on a key it carries, is registered for its whole namespace rather than key by key. A later request on the same client connection and in the same namespace then waits until the batch has been enqueued, even for a key the batch does not carry: anithat would have been sent to its primary at once is deferred, for as long as the batch waits (up to the proxy’s backend timeouts). Registering each key would leave an entry per key on the connection (about 25 MB for a 1 MiB batch of short keys) outside the in-flight byte budget, so the bounded memory was judged worth the wait. Other namespaces, other connections and batches of 64 keys or fewer are not affected. - A delete whose answer was lost after the proxy wrote it is sent again,
and when the first attempt had already removed the key the client is told
N(not found) although the key existed. The proxy cannot tell that from a key that never existed, so no fixed answer is right for both; it reports what the attempt that answered found, as an SDK’s own retry of a delete does. The key is gone either way. - A ring change pauses a node about 25 ms per million live keys (a snapshot walk on the single cache task), with a transient allocation of 64 bytes per key; a staged handoff’s count of the entries to send (about 60 ms per million) runs in slices that yield between them and does not add to the pause. Bounding the snapshot would need an ordered per-namespace key index maintained on every write; see Known limits above.
- Hedged and fallback reads: a miss means the primary answered miss; a non-hedged read falls to the next owner only on connection failure. The primary’s answer is the answer; probing the other owners on a miss is what read repair, off by default, opts into.
Data model and operations
- A touch is not ordered against the fire-and-forget replica leg of
an earlier write to the same key. A client with fire-and-forget replica
writes sends the two legs independently, so under concurrency the touch
can reach a replica before the write does, find nothing to touch, and
leave the write to land with its own TTL (none, for a plain
set); that copy then outlives the primary’s. Rare (it needs a touch right after a write of the same key and concurrent load) and the same class as a replica that answersNand is never repaired; ordering the legs per key would put a lock on every background write. Without fire-and-forget the write waits for its replica legs and the race does not exist. - A touch that changes only the TTL of a key (removes it, shortens it, or expires the key) and reaches one owner but not the other is not repaired. The leg can fail while a node is leaving or restarting, or the copy can be one the sender no longer counts as an owner until a ring change makes it one again. Re-replication writes only where the key is absent, so the copy that missed the change keeps its old TTL: it expires later than it should (a key expired by a touch lingers) or earlier (a key whose TTL was removed is missing on that owner), and if the copy that took the change is lost before the other expires, the key is gone from both. A repair would have to know which TTL is newer, which takes a version on every write; taking the longer TTL instead would bring back keys a touch had expired. In six-round join, decommission and kill runs, 1 to 19 of 2,000 keys whose TTL was removed ended missing on an owner in the CI runs that showed it (2,000 keys, two-core runners), and in one run 46 of 2,000 expired keys lingered on a copy; on a laptop it showed in a handful of runs.
- A touch forwarded as a
Uto a leaving node that only moves the TTL of the value it holds (same value, a longer or shorter TTL) is not relayed to the node’s entrant, which keeps the TTL it had. Such a frame cannot be told apart from the delayed echo of the node’s own drain, which carries the same value with the TTL it left with, and relaying it as a change makes the echo circle between the node and its entrant for as long as the delays last, so how long it circles would depend on timing. It needs an old-list client that touches only the lingering joiner’s old owner while that joiner decommissions, and is the same class as a touch that reaches one owner but not the other. A new value is relayed. - A batched touch with
Gleaves a value that does not fit the 16 MiB reply untouched on the primary, but the plain form its other owners receive has no such bound, so their copies of that key get the new TTL and the primary’s keeps the old one. The client reads the key as a miss either way; keep the batch under the reply bound, or touch the key again. INCRand compare-and-set are as volatile as a plain set: eviction and TTL reclaim counters, and CAS is not a distributed lock since eviction breaks mutual exclusion. Neither is a lock across a join either: for a few seconds after discovery promotes a joiner, a client that has not refreshed its ring yet still sends to the old owner while one that has sends to the joiner, each takes the key as absent because the old owner’s forward has not arrived, and twoputIfAbsenton the same key both answer true (one to seven of 2,000 racing keys per six-round join, decommission and kill run, two clients). Replicas receive the result as a set, because replaying the operation could drift a replica from the primary.- The memory bound is approximate: key and value bytes plus a per-entry and a per-namespace overhead measured on the container image (allocator share included), so resident memory tracks the bound for tiny entries and namespaces; transient allocations (snapshots, in-flight requests, forward queues) are outside it. Leave headroom for those.
- Replica-leg writes, read-repair writes and node-list refreshes fail
silently, so that a dead replica never fails a write; the per-client
counters (
stats()) are the signal. - A standalone node has nobody to hand entries to and nothing is persisted: a restart is a cold cache.
- Asymmetric port mapping (
-p 9999:8356) is unsupported: it is NAT, which the cluster design excludes, since nodes are reached at the address they registered. - Any registered node can fetch every member’s token
(
Tframe), because a draining node must authorize handoffs to peers it met after registration; registering a node is the trust step. - The operations endpoint (
/healthz,/readyz,/metrics) is unauthenticated; bind it internally. - Cluster-internal frames are not frozen; a mixed-version cluster is a transient rolling-upgrade state only, discovery first. Policy, see Compatibility.
SDKs and adapters
- Go and Java spell “remove the TTL” of a touch as the
integer
-1(RemoveTTL/REMOVE_TTL), because the argument has no default and0is a real value (expire at once). A computed TTL of exactly-1is therefore accepted and removes the TTL, where every other negative value is rejected. The other four SDKs useNone/null/an option and cannot collide; changing Go and Java would be an API break for a mistake that needs arithmetic on the TTL to make. - The
connect()forgotten-close warning is kept per address within a process, not per client instance, so several clients to one single address trip it even when each is closed in turn. Tracking per instance would never fire (a new instance has no previous connection), so the choice is this warning or none; kept. Share one client, or use a multi-address config. - Keyv adapter:
has, thegetManyfamily anditeratorare deliberately omitted; JCache: honest subset (iterator, management and several configurations throwUnsupportedOperationException). The protocol has no key enumeration and no existence check short of a read; each adapter’s README lists what is omitted and why. - SDK production constants (timeouts, in-flight budgets) are mutable, documented-but-hidden test knobs so each SDK’s own test suite can shrink them; not an API, and not a connect option.
- A node or discovery server answers a connection turned away by a
connection cap (the global one or the per-IP one) with
Band keeps it open for up to 200 ms to read what the peer has already sent, so that closing does not reset the reply away. At most 128 rejected connections are in that state at once, process-wide. A source that keeps 128 of them lingering (about 640 rejected connections a second, each held for the 200 ms) makes any other client rejected at that moment get a reset instead ofB: past the 128 a rejection gets a non-blocking reply and is closed at once, and a client that has already sent its first frame sees the close as a reset. Only connections that are being refused anyway are affected; accepted connections, and a listener with TLS (which sends no reply to a plaintext peer), are not. Per-source slots were not added: the bound exists so that rejected connections cannot pile up without limit, and who is refused is already decided by the caps. - A proxy’s first registration (
Y) under a new name is paced per source like a node’s (P), with an allowance of its own: 64 at once, then one every 2 seconds, and an IPv6 source counts per /64. More than 64 proxies behind one NAT or SNAT address, or in one /64, that all register at the same time (after a discovery restart, say) are admitted one per 2 seconds beyond the 64: discovery refuses the others and closes the connection, the proxy logs a warning and sends the sameYagain at its next refresh (every 5 seconds, or sooner after a forced refresh), and the proxy list (Q) is partial until the last one is in, about 2 seconds per proxy beyond 64. A refresh of a name already registered is never refused. The pacing stops one connection from filling the proxy registry with made-up names in milliseconds; a fleet larger than the burst behind one address re-registers slowly here exactly as nodes do.
Design decisions
The main choices, in summary:
| Decision |
|---|
| Client-side hashing with a lightweight discovery server |
| Shared-secret authentication via environment variable |
| TLS via rustls, required once configured |
| Server type in the auth response → one unified connect |
| Staged node join with orchestrated data handoff |
| Node identity decoupled from network address |
| Discovery HA via soft-state replicas and announces |
| Client-side replication via rendezvous hashing |