What every deployment needs

The three things an orchestrator — or a person with a shell — needs from nanocached before anything else on the deployment pages: probes to route around instances that aren't ready, metrics to scale on, and a SIGTERM-driven drain so scale-in never loses data.

The operations endpoint

Each binary takes --metrics-port <port> and serves three paths on it (plain HTTP, on --host):

PathWhat it returns
GET /metricsPrometheus text format.
GET /healthz200 while the process is serving — a liveness probe.
GET /readyz200 only when the instance can do useful work — a readiness probe. A node is ready once it has adopted cluster membership; a proxy once it has a node roster, and it stops being ready as soon as it starts draining; a discovery server once its startup grace period is over.

The endpoint is unauthenticated by design (probes and scrapers don't do protocol handshakes) — bind it to an internal interface and don't expose it outside the task network. Omitting the flag disables the endpoint entirely. Collecting what it serves — which agent, with which configuration, and what the numbers look like once stored — is the subject of metrics collection.

Metrics to scale on

Nodes scale on the memory watermark. Sum nanocached_node_memory_used_bytes over the fleet, divide by the summed nanocached_node_memory_max_bytes, and target a utilization (say 70%) with your autoscaler. Entry counts, hit/miss/set/ delete/eviction/expiration counters, connection counts (nanocached_node_connections against …_connections_max, the --max-connections bound — raise --max-connections-per-ip too when clients share a source IP, as behind NAT or on Kubernetes), and per-namespace breakdowns (nanocached_node_namespace_used_bytes, …_namespace_entries) are exported alongside for dashboards and alerting.

Proxies scale on connections: nanocached_proxy_client_connections against …_client_connections_max (plus CPU). …_requests_total and …_upstream_failures_total give request and error rates. …_replica_refusals_total counts replica legs of a write a node answered W or E: the client saw success, but that replica's copy is missing or stale until the next write — a rise after a membership change is the proxy-side trace of a stale owner. The log is not that record: a multi-key write logs one WARN line for its failed replica legs and one for its refused ones, and the proxy writes at most five replica-leg warnings a second, reporting how many it dropped, so a node that is down does not fill the log; the counter counts every leg.

Discovery exports roster sizes (nanocached_discovery_members, …_proxies, …_waiting_nodes, …_joining_nodes) and join counters — not scaling signals, but the first thing to graph when a scale event misbehaves.

Sizing a node process

Size a node process at --max-memory (the cache only) plus --max-in-flight-bytes plus headroom. The in-flight budget is what client connections hold outside the cache, node-wide, 256 MiB by default (at least 1 MiB):

A frame or a reply that costs more than the budget has room for waits for that room — what is left after every connection's receive buffer and the 64 bytes per key its reply keeps, bytes whose holders may be waiting themselves — and then runs alone with it. When receive buffers pin the whole budget, nothing is left to wait for: a connection that holds a buffer itself then takes a single node-wide overdraft slot and answers without a charge, one at a time, because its buffer is only released after the replies it has buffered and two such connections would otherwise wait for each other. A receive buffer whose next doubling the room cannot cover does the same: it grows under that slot without a charge until its frame has arrived and been answered, since connections each part-way through a large frame would otherwise wait for each other's buffers. That overshoots the budget by one frame or reply, never more, and a connection with no buffer charged keeps waiting for the budget. A connection that cannot get its share in time is closed, like a stalled one.

Discovery: registration pacing

A discovery server paces how fast one source can add new node names. A source may register up to 64 new names at once and one more every 2 seconds after that; a node refreshing its own entry is never counted. The source is the peer address, with an IPv6 source counting per /64 prefix, so a fleet behind one NAT address or in one IPv6 subnet shares the allowance.

After a discovery restart the nodes re-register within the startup grace (--liveness-timeout, 15 seconds by default), which admits 64 plus one per 2 seconds of grace from one source: 71 nodes at the default. A larger fleet behind one address is still re-registering when the grace ends, so the first join after it is handed off against a partial roster; the rest follow at one node per 2 seconds. Set --liveness-timeout to 2 seconds per node beyond 64 (72 seconds for 100 nodes, since 36 nodes are beyond 64; a dead node is then evicted that much later too; the minimum is 15), or spread the nodes over more source addresses.

A proxy registering with discovery (Y) is paced the same way, with an allowance of its own: 64 new names at once and one more every 2 seconds per source, an IPv6 source counting per /64, and a refresh of a name already registered never counted. With more than 64 proxies behind one NAT or SNAT address, or in one /64, those registering together (after a discovery restart) get in at one per 2 seconds beyond the 64. Discovery refuses the others and closes the connection; the proxy logs WARN proxy announce … failed and announces again at its next refresh (every 5 seconds), so no restart is needed, and the proxy list an SDK in proxy mode fetches is partial meanwhile (about 2 seconds per proxy beyond 64, so 72 seconds for 100 proxies). The startup grace only delays the list (B); it does not widen this allowance. Spread the proxies over more source addresses if that matters.

Graceful scale-in

Scale-in terminates instances that are not dead, and orchestrators signal that with SIGTERM followed by SIGKILL after a grace period. Both server binaries treat SIGTERM as "drain, then exit" — so ordinary deployments, autoscaler scale-in, and spot/Spot reclaim all get the graceful path with no extra hooks.

Node: decommission

On SIGTERM a clustered node runs a planned leave within --drain-timeout seconds (default 25):

  1. It hands every key it owns to the node that becomes the new owner once it's gone — so even at R=1 the only copy moves rather than dies.
  2. It leaves the cluster membership immediately once the handoff is done (no liveness timeout to wait out), so clients refresh onto the surviving nodes. Until then it stays a member and keeps heartbeating, so a handoff longer than the discovery's liveness timeout does not get the node evicted halfway through.
  3. It keeps serving and forwards writes it no longer owns for the rest of the budget while clients catch up, then exits 0.

The budget is checked between keys and between the attempts on one key, so it is a soft limit. An attempt that is under way when the budget runs out is allowed to finish, no further attempt starts, and every key after it is left behind. An attempt is bounded by the sum of its legs, each limited to 10 s: the TCP connect, the TLS handshake (with TLS), the auth exchange (with an auth secret) and the send. That is 20 s on a plain connection and 40 s with both TLS and a secret. A receiver that refuses the connection fails at once, but one that stops answering without closing it (a suspended process, a saturated host) can stretch the leave past the budget by that one attempt. A receiver that cannot be reached on all three attempts at a key (a node that is down but still listed by discovery) is written off for 30 s: its remaining keys are left behind without further attempts, so the keys bound for the live nodes still get the budget. So is a receiver that answers with a refusal and closes the connection (its membership token does not match, because the address now belongs to a different process), after the first key and without a retry. Entries deleted or expired by their turn are not counted as left behind. The node logs the entries it could not hand off and exits 0. The discovery servers are told of the leave at the same time, and the notification gets what is left of the budget. If the handoff used the budget up, it still gets 2 s, which comes on top of the attempt that overran. A discovery server that does not answer in that time is logged and keeps listing the node until its liveness timeout.

Treat that log line (WARN decommission: N entr(ies) could not be handed off within the drain budget) as a sizing signal, not as a loss: the surviving replicas hold those keys, and the survivors’ re-replication pass for the leave brings them back to R within seconds to tens of seconds. Until it has, they have one copy, and a node failure in that window loses them. If the line appears, raise --drain-timeout (and the stop grace with it) so the handoff completes with room to spare, and let the cluster settle after a leave before the next membership change.

--drain-timeout 0 skips the handoff (the old crash-equivalent shutdown) — acceptable for fast restarts at R ≥ 2, where replicas cover the gap.

Scale in one node at a time. A draining node plans its entire handoff from one roster snapshot taken as the drain begins. Two nodes draining at once each plan against a roster that still lists the other, so a key whose owners included both can come out of the double-leave one replica short until a later membership change, a rewrite, or its TTL turns it over — and at R=1, a handoff that lands on the other leaver can be dropped outright (see Architecture on the self-healing consistency model). Scale-in policies that step the desired count down one instance per action avoid the window entirely; a larger single step (or a spot reclaim of several instances at once) trades that safety for speed and should be reserved for R ≥ 2.

A node that was evicted

A node that stops heartbeating for longer than the discovery’s --liveness-timeout (15 s by default) is evicted, whether the process was stopped (a paused VM, SIGSTOP, a long garbage collection of the host) or the network between it and discovery was cut. If the process is still running when it can reach discovery again, it drops its cache and joins again as a new node (its log says evicted this node earlier); it does not resume with what it held, because the cluster has moved on without it. At R=1 the keys it held and nobody rewrote in the meantime are therefore gone with it, as they were when it was evicted. When nobody could have moved on — every node was cut off from discovery at once, or discovery itself was not running — it is registered with its cache instead; the accepted behaviour list says exactly when. Keep the heartbeat path short enough, and the liveness timeout long enough, that a pause you expect does not evict it. A node heartbeats every 5 s, so discovery refuses a --liveness-timeout below 15 s (three heartbeats): with less, one lost heartbeat, or a short pause of discovery itself, would evict healthy nodes and cost them their caches.

Proxy: drain

On SIGTERM a proxy deregisters from discovery's proxy roster immediately (new clients stop being handed its address), finishes in-flight requests within --drain-timeout (default 25), and exits 0. Clients connected to it reconnect to a surviving proxy transparently — at most one reconnect per connection. Where the proxy tier runs and how many to run is its own page.

Sizing the stop grace

The orchestrator's grace period must outlast the drain: set the orchestrator’s stop grace (a container service’s stopTimeout, Kubernetes’ terminationGracePeriodSeconds) to at least --drain-timeout plus a margin (twice it is a comfortable rule while the nodes answer or refuse promptly; to cover a receiver that stops answering, which can stretch the leave past the budget by one attempt (up to 40 s) and the 2 s for the leave notification, as described above, use the budget plus 45 s — for a 20 s budget, 65 s). A node's handoff time scales with the data it holds — a 20 s budget drains hundreds of MB comfortably on an in-network transfer, but measure with your value sizes; keys that miss the budget are logged and left to the surviving replicas. The default 25 s fits inside a spot interruption's 2-minute warning with room to spare.

Upgrading a cluster

Roll a new server version out discovery first, then the nodes, then the proxies, one component at a time. The frames nodes, discovery servers and proxies exchange with each other are versioned with the server and may change in any release (see the compatibility policy); a newer discovery accepts what an older node sends, but the reverse does not hold. In 0.4.5 a node's heartbeat gained an optional field that an older discovery rejects, so a node upgraded ahead of its discovery servers could lose its heartbeats and be evicted. A mixed-version cluster is only ever the transient state of such a roll-out; each release's changelog notes any exception to this order.

Re-pointing :latest by hand

Each release moves :latest of the node, discovery and proxy images to the new version in one step, and never moves it backwards. If the three ever disagree — for example because a build of an unreleased main was pushed as :latest earlier — a maintainer can bring them back together without cutting a release. Run the Release server workflow by hand (Actions tab, Run workflow, or from a terminal):

gh workflow run release-server.yaml -f version=0.6.2
gh run watch

The version input is optional; left empty, it is the newest stable server/v* tag. A manual run builds and pushes nothing: it checks that the node, discovery and proxy images exist at that version, and only then points all three :latest tags at it. A version that is not X.Y.Z (a pre-release, a typo) or whose images do not exist fails the run before anything moves. A version older than the one the node image's :latest already carries changes nothing and the run ends green with a notice. Check the result with docker buildx imagetools inspect ghcr.io/nanocached/nanocached-node:latest (and the same for the other two): the digests match the version tag's.

Verifying a deployment

The repository ships runnable scale-in scenarios under tests/e2e: scalein.sh (a node leaves under load at R=2 — zero loss, zero corruption, membership follows) and proxydrain.sh (a proxy drains under via-proxy load — immediate roster removal, exit 0, clients reconnect). They document the exact guarantees the drain makes and are the fastest way to see the choreography before wiring up an autoscaler. Read them as examples: nothing under tests/ or tools/ carries a compatibility promise (see Compatibility), so copy what you need rather than calling them from your own automation.