What every deployment needs
The three things an orchestrator — or a person with a shell — needs from nanocached before anything else on the deployment pages: probes to route around instances that aren't ready, metrics to scale on, and a SIGTERM-driven drain so scale-in never loses data.
The operations endpoint
Each binary takes --metrics-port <port> and serves
three paths on it (plain HTTP, on --host):
| Path | What it returns |
|---|---|
GET /metrics | Prometheus text format. |
GET /healthz | 200 while the process is serving — a liveness probe. |
GET /readyz | 200 only when the instance can do useful work — a readiness probe. A node is ready once it has adopted cluster membership; a proxy once it has a node roster, and it stops being ready as soon as it starts draining; a discovery server once its startup grace period is over. |
The endpoint is unauthenticated by design (probes and scrapers don't do protocol handshakes) — bind it to an internal interface and don't expose it outside the task network. Omitting the flag disables the endpoint entirely. Collecting what it serves — which agent, with which configuration, and what the numbers look like once stored — is the subject of metrics collection.
Metrics to scale on
Nodes scale on the memory watermark. Sum
nanocached_node_memory_used_bytes over the fleet, divide by
the summed nanocached_node_memory_max_bytes, and target a
utilization (say 70%) with your autoscaler. Entry counts, hit/miss/set/
delete/eviction/expiration counters, connection counts
(nanocached_node_connections against
…_connections_max, the --max-connections bound
— raise --max-connections-per-ip too when clients share a
source IP, as behind NAT or on Kubernetes), and per-namespace
breakdowns (nanocached_node_namespace_used_bytes,
…_namespace_entries) are exported alongside for dashboards
and alerting.
Proxies scale on connections:
nanocached_proxy_client_connections against
…_client_connections_max (plus CPU).
…_requests_total and …_upstream_failures_total
give request and error rates. …_replica_refusals_total
counts replica legs of a write a node answered W or
E: the client saw success, but that replica's copy is
missing or stale until the next write — a rise after a membership
change is the proxy-side trace of a stale owner. The log is not
that record: a multi-key write logs one WARN line for its
failed replica legs and one for its refused ones, and the proxy writes at
most five replica-leg warnings a second, reporting how many it dropped, so
a node that is down does not fill the log; the counter counts every leg.
Discovery exports roster sizes
(nanocached_discovery_members, …_proxies,
…_waiting_nodes, …_joining_nodes) and join
counters — not scaling signals, but the first thing to graph when a
scale event misbehaves.
Sizing a node process
Size a node process at --max-memory (the cache only) plus
--max-in-flight-bytes plus headroom. The in-flight budget is
what client connections hold outside the cache, node-wide, 256 MiB by
default (at least 1 MiB):
- A reply's values are charged before it is built and released once it is written.
- A receive buffer's growth past 64 KiB is charged before it is allocated.
- A multi-key frame (
m,oory) is charged 256 bytes per key (288 foro) for the per-key lists the node builds while it runs, taken as soon as the frame is fully received and before those lists exist — a frame of one-byte keys can carry about 350,000 keys, so about 85 MiB. Once the reply is built, 64 bytes per key of that stay charged until the reply is written (the reply's entries and its encoded copy, which a client that does not read keeps alive), and the rest is released.
A frame or a reply that costs more than the budget has room for waits for that room — what is left after every connection's receive buffer and the 64 bytes per key its reply keeps, bytes whose holders may be waiting themselves — and then runs alone with it. When receive buffers pin the whole budget, nothing is left to wait for: a connection that holds a buffer itself then takes a single node-wide overdraft slot and answers without a charge, one at a time, because its buffer is only released after the replies it has buffered and two such connections would otherwise wait for each other. A receive buffer whose next doubling the room cannot cover does the same: it grows under that slot without a charge until its frame has arrived and been answered, since connections each part-way through a large frame would otherwise wait for each other's buffers. That overshoots the budget by one frame or reply, never more, and a connection with no buffer charged keeps waiting for the budget. A connection that cannot get its share in time is closed, like a stalled one.
Discovery: registration pacing
A discovery server paces how fast one source can add new node names. A source may register up to 64 new names at once and one more every 2 seconds after that; a node refreshing its own entry is never counted. The source is the peer address, with an IPv6 source counting per /64 prefix, so a fleet behind one NAT address or in one IPv6 subnet shares the allowance.
After a discovery restart the nodes re-register within the startup
grace (--liveness-timeout, 15 seconds by default), which
admits 64 plus one per 2 seconds of grace from one source: 71 nodes
at the default. A larger fleet behind one address is still re-registering
when the grace ends, so the first join after it is handed off against a
partial roster; the rest follow at one node per 2 seconds. Set
--liveness-timeout to 2 seconds per node beyond 64
(72 seconds for 100 nodes, since 36 nodes are beyond 64; a dead node
is then evicted that much later too; the minimum is 15), or spread the
nodes over more source addresses.
A proxy registering with discovery (Y) is paced the same
way, with an allowance of its own: 64 new names at once and one more every
2 seconds per source, an IPv6 source counting per /64, and a refresh of
a name already registered never counted. With more than 64 proxies behind one
NAT or SNAT address, or in one /64, those registering together (after a
discovery restart) get in at one per 2 seconds beyond the 64. Discovery
refuses the others and closes the connection; the proxy logs
WARN proxy announce … failed and announces again at its
next refresh (every 5 seconds), so no restart is needed, and the proxy
list an SDK in proxy mode fetches is partial meanwhile (about 2 seconds
per proxy beyond 64, so 72 seconds for 100 proxies). The startup grace
only delays the list (B); it does not widen this allowance.
Spread the proxies over more source addresses if that matters.
Graceful scale-in
Scale-in terminates instances that are not dead, and orchestrators signal that with SIGTERM followed by SIGKILL after a grace period. Both server binaries treat SIGTERM as "drain, then exit" — so ordinary deployments, autoscaler scale-in, and spot/Spot reclaim all get the graceful path with no extra hooks.
Node: decommission
On SIGTERM a clustered node runs a planned leave within
--drain-timeout seconds (default 25):
- It hands every key it owns to the node that becomes the new owner once it's gone — so even at R=1 the only copy moves rather than dies.
- It leaves the cluster membership immediately once the handoff is done (no liveness timeout to wait out), so clients refresh onto the surviving nodes. Until then it stays a member and keeps heartbeating, so a handoff longer than the discovery's liveness timeout does not get the node evicted halfway through.
- It keeps serving and forwards writes it no longer owns for the rest of the budget while clients catch up, then exits 0.
The budget is checked between keys and between the attempts on one key, so it is a soft limit. An attempt that is under way when the budget runs out is allowed to finish, no further attempt starts, and every key after it is left behind. An attempt is bounded by the sum of its legs, each limited to 10 s: the TCP connect, the TLS handshake (with TLS), the auth exchange (with an auth secret) and the send. That is 20 s on a plain connection and 40 s with both TLS and a secret. A receiver that refuses the connection fails at once, but one that stops answering without closing it (a suspended process, a saturated host) can stretch the leave past the budget by that one attempt. A receiver that cannot be reached on all three attempts at a key (a node that is down but still listed by discovery) is written off for 30 s: its remaining keys are left behind without further attempts, so the keys bound for the live nodes still get the budget. So is a receiver that answers with a refusal and closes the connection (its membership token does not match, because the address now belongs to a different process), after the first key and without a retry. Entries deleted or expired by their turn are not counted as left behind. The node logs the entries it could not hand off and exits 0. The discovery servers are told of the leave at the same time, and the notification gets what is left of the budget. If the handoff used the budget up, it still gets 2 s, which comes on top of the attempt that overran. A discovery server that does not answer in that time is logged and keeps listing the node until its liveness timeout.
Treat that log line (WARN decommission: N entr(ies) could not be
handed off within the drain budget) as a sizing signal, not as a
loss: the surviving replicas hold those keys, and the survivors’
re-replication pass for the leave brings them back to R within seconds
to tens of seconds. Until it has, they have one copy, and a node failure
in that window loses them. If the line appears, raise
--drain-timeout (and the stop grace with it) so the handoff
completes with room to spare, and let the cluster settle after a leave
before the next membership change.
--drain-timeout 0 skips the handoff (the old
crash-equivalent shutdown) — acceptable for fast restarts at
R ≥ 2, where replicas cover the gap.
Scale in one node at a time. A draining node plans its entire handoff from one roster snapshot taken as the drain begins. Two nodes draining at once each plan against a roster that still lists the other, so a key whose owners included both can come out of the double-leave one replica short until a later membership change, a rewrite, or its TTL turns it over — and at R=1, a handoff that lands on the other leaver can be dropped outright (see Architecture on the self-healing consistency model). Scale-in policies that step the desired count down one instance per action avoid the window entirely; a larger single step (or a spot reclaim of several instances at once) trades that safety for speed and should be reserved for R ≥ 2.
A node that was evicted
A node that stops heartbeating for longer than the discovery’s
--liveness-timeout (15 s by default) is evicted, whether
the process was stopped (a paused VM, SIGSTOP, a long garbage
collection of the host) or the network between it and discovery was cut.
If the process is still running when it can reach discovery again, it drops
its cache and joins again as a new node (its log says
evicted this node earlier); it does not resume with what it
held, because the cluster has moved on without it. At R=1 the keys it held
and nobody rewrote in the meantime are therefore gone with it, as they were
when it was evicted. When nobody could have moved on — every node was
cut off from discovery at once, or discovery itself was not running —
it is registered with its cache instead; the accepted behaviour list
says exactly when. Keep the heartbeat path short enough, and the liveness
timeout long enough, that a pause you expect does not evict it. A node
heartbeats every 5 s, so discovery refuses a
--liveness-timeout below 15 s (three heartbeats): with
less, one lost heartbeat, or a short pause of discovery itself, would
evict healthy nodes and cost them their caches.
Proxy: drain
On SIGTERM a proxy deregisters from discovery's proxy roster
immediately (new clients stop being handed its address), finishes
in-flight requests within --drain-timeout (default 25),
and exits 0. Clients connected to it reconnect to a surviving proxy
transparently — at most one reconnect per connection. Where the
proxy tier runs and how many to run is its own page.
Sizing the stop grace
The orchestrator's grace period must outlast the drain:
set the orchestrator’s stop grace (a container
service’s stopTimeout, Kubernetes’
terminationGracePeriodSeconds) to at least
--drain-timeout plus a margin (twice it is a
comfortable rule while the nodes answer or refuse promptly; to cover a
receiver that stops answering, which can stretch the leave past the budget
by one attempt (up to 40 s) and the 2 s for the leave
notification, as described above, use the budget plus 45 s — for a
20 s budget, 65 s). A node's handoff time scales with the data it holds
— a 20 s budget drains hundreds of MB comfortably on an
in-network transfer, but measure with your value sizes; keys that miss
the budget are logged and left to the surviving replicas. The default
25 s fits inside a spot interruption's 2-minute warning with room
to spare.
Upgrading a cluster
Roll a new server version out discovery first, then the nodes, then the proxies, one component at a time. The frames nodes, discovery servers and proxies exchange with each other are versioned with the server and may change in any release (see the compatibility policy); a newer discovery accepts what an older node sends, but the reverse does not hold. In 0.4.5 a node's heartbeat gained an optional field that an older discovery rejects, so a node upgraded ahead of its discovery servers could lose its heartbeats and be evicted. A mixed-version cluster is only ever the transient state of such a roll-out; each release's changelog notes any exception to this order.
Re-pointing :latest by hand
Each release moves :latest of the node, discovery and
proxy images to the new version in one step, and never moves it
backwards. If the three ever disagree — for example because a build of
an unreleased main was pushed as :latest
earlier — a maintainer can bring them back together without cutting a
release. Run the Release server workflow by hand
(Actions tab, Run workflow, or from a terminal):
gh workflow run release-server.yaml -f version=0.6.2
gh run watch
The version input is optional; left empty, it is the
newest stable server/v* tag. A manual run builds and pushes
nothing: it checks that the node, discovery and proxy images exist at
that version, and only then points all three :latest tags
at it. A version that is not X.Y.Z (a pre-release, a typo) or
whose images do not exist fails the run before anything moves. A version
older than the one the node image's :latest already carries
changes nothing and the run ends green with a notice. Check the result with
docker buildx imagetools inspect ghcr.io/nanocached/nanocached-node:latest
(and the same for the other two): the digests match the version tag's.
Verifying a deployment
The repository ships runnable scale-in scenarios under
tests/e2e:
scalein.sh (a node leaves under load at R=2 — zero loss,
zero corruption, membership follows) and proxydrain.sh (a
proxy drains under via-proxy load — immediate roster removal, exit 0,
clients reconnect). They document the exact guarantees the drain makes
and are the fastest way to see the choreography before wiring up an
autoscaler. Read them as examples: nothing under tests/ or
tools/ carries a compatibility promise (see
Compatibility), so copy what you
need rather than calling them from your own automation.