Wire protocol

Each request starts with an ASCII header terminated by \n; key and value bodies follow immediately, read by their declared byte lengths. Bodies have no terminator and may contain arbitrary bytes. One-byte commands keep overhead minimal while frames stay readable during development.

From 1.0.0 the client-facing protocol on this page is frozen; the compatibility policy says what that covers.

The transcripts on this page show the bytes on the wire: a line starting with > is sent, a line starting with < is what the server answers, \n stands for a newline and \0 for a zero byte, a line starting with # is a comment, and = closed means the server then closes the connection. A line ending carries no newline of its own.

Multiple requests per connection and pipelining are both supported. A <key-length> of 0 is rejected for every command; there is no dedicated key or value length limit beyond the overall request-size limit: 1 MiB (1,048,576 bytes) per request on a node, the header line and the bodies together, counted per frame (what is pipelined behind a frame does not count towards it). A request over it, like one the server cannot parse, is answered En\n and the connection closed. The header line has no limit of its own: a header that runs past the limit without its \n is refused the same way. Through nanocached-proxy the limit is 11 bytes lower, 1,048,565: the proxy puts a tag on the frame it passes to a node, and the node’s limit still has to hold for the longer frame; the proxy answers a bare E\n. An i through the proxy is further limited to 1,048,512 bytes of namespace and key together, because the proxy sends its result to the replicas as a set, which carries the new value and the TTL that the i does not; a longer one closes the connection like any other request the proxy cannot accept. An o or y is also limited by its namespace: the namespace length times the number of keys in the frame (all of them, whichever node they route to) may not exceed 64 MiB (67,108,864). A node forwards each key of such a frame as a frame of its own while a join, a decommission or an eviction changes the cluster, and every one repeats the namespace, so without the limit a frame of 1 MiB with a 500 KB namespace and about 90,000 one-byte keys would make the node send tens of gigabytes. A frame over it is refused from its header, before its body is read, like any request the server cannot parse: a node answers En\n and closes the connection, the proxy answers a bare E\n and closes the client’s connection without sending anything to a node. The limit does not depend on whether the cluster is changing. The SDKs send at most 400 keys in a frame and, for o and y, no more than 64 MiB divided by the namespace length (rounded down), so their frames stay under the limit with any namespace; a namespace of 180 bytes or less cannot reach it at any frame size. The frames only another node sends (U, u, f) may exceed it by 4 KiB: the U that moves a value to another node carries the receiver’s token on top of what the client’s S carried, and a value stored at the limit must still be movable.

Numbers in headers

Every number in a header — a length, an entry count, a TTL, a tag, a namespace length — is an unsigned decimal: one or more ASCII digits 0–9 and nothing else. A sign (+ or -), hex, an exponent, a non-ASCII digit, an empty field and a number too large for its field are each a malformed request, answered En\n with the connection closed. Fields are separated by exactly one space, so a doubled or trailing space leaves an empty field and is malformed too. A TTL (on S, s, o, k, t and y) goes up to 264−1 and a tag up to 232−1; a length or count has no limit of its own beyond the request-size limit above.

Leading zeros are accepted and mean the same number: G 01 is G 1, a TTL of 007 is 7, and the tag 007 is echoed back as 7. No SDK sends them and no server does, but they are part of the frozen protocol: a server keeps accepting them.

i’s <delta> is the one exception. It is signed and is held to the grammar of a stored counter (see i): an optional -, then digits with no leading zero other than a lone 0, within the range of an i64. -0 is accepted (it is 0); +3, 007 and -007 are malformed. The proxy and the discovery server read numbers the same way (the proxy answers a bare E\n where a node answers En\n).

> S 4 5\nnameAlice
< S\n
# leading zeros are the same number
> G 04\nname
< V 5\nAlice
# a delta takes none
> i 0 4 007\nname
< En\n
= closed

A — authenticate

A <secret-length>\n<secret>

Responds On\n on success or En\n followed by the server closing the connection (Od\n/Ed\n from a discovery server — the second byte always names the server type, see the note below). If the server has no auth secret configured, any well-formed A succeeds (an empty secret, A 0, is malformed and answered En\n like any other). A client that has no secret therefore sends a one-byte placeholder: every SDK sends a single NUL byte, A 1 T R\n followed by \0. If the server does have a secret, every other command is rejected with En\n (and the connection closed) until a matching A has been sent. Until then the node accepts a frame of at most 4096 bytes, which bounds the A frame itself (header and secret); a server refuses to start with a secret too long to fit.

A failed A that asked for tags (a T field) is answered EnT\n by a node and by the proxy and EdT\n by a discovery server, echoing the capability as a success does. Either way the connection is then closed.

Send A once, as the first frame. A node and a discovery server accept a later A too and run it afresh: a matching one is answered On/OnT (Od/OdT) again, and the connection continues in the mode that latest A asked for, tags or none; a failing one is answered En and the connection is closed, even if an earlier A had succeeded. nanocached-proxy accepts no second A at all: it answers a bare E\n, even for a correct secret, and closes the connection.

The second byte (n/d — node/discovery) is how a client discovers what kind of server it reached and adapts, with no configuration; it is sent whether or not the secret was accepted, and whatever form of A was used. SDKs additionally append a T field (A <secret-length> T\n) to request response tags; a server that supports them echoes the capability as OnT/OdT.

The capability fields after the length are T and R, in that order. Every other token is reserved: a node answers En, a discovery server Ed and nanocached-proxy a bare E, and the connection is closed. That refusal is also how an SDK learns that a server does not know a capability it asked for, and falls back to asking for less. A later release may define a new token, only as an opt-in capability a client asks for, never as a change to what a request without it does. V is set aside for versioned writes (a version carried on write requests and replies, so that the owners of a key can order the writes to it); no release defines it yet, and nothing should send it.

A server with no secret, asked for tags with the placeholder secret:

> A 1 T R\n\0
< OnT\n

A server started with the secret hunter2:

> G 1\nk
< En\n
= closed
> A 7\nhunter2
< On\n
> G 1\nk
< N\n
> A 5\nwrong
< En\n
= closed

G — get

G <key-length>\n<key>

Responds with the value:

V <value-length>\n<value>

or N\n when the key is missing or expired.

S — set

S <key-length> <value-length>\n<key><value>
S <key-length> <value-length> <ttl-seconds>\n<key><value>

Responds S\n. <ttl-seconds> is an unsigned 64-bit decimal (up to 18446744073709551615; a larger number is a malformed request). Without a TTL the entry lives until evicted (LRU under the memory bound) or deleted; with one it also expires <ttl-seconds> after the write — a literal 0 is a TTL that has already elapsed, so the entry expires immediately (the next access sees a miss).

A value may be empty (<value-length> of 0; a <key-length> of 0 is not allowed). It is stored like any other, by S or by a batched o: G answers V 0\n — a hit with no bytes after the header, not a miss — and m a result of 0.

A TTL of 0 has already elapsed, and an empty value is a hit:

> S 4 5 0\nnameAlice
< S\n
> G 4\nname
< N\n
> S 5 0\nempty
< S\n
> G 5\nempty
< V 0\n

D — delete

D <key-length>\n<key>

Responds D\n if the key existed, N\n otherwise.

Set, read, delete, and read again:

> S 4 5\nnameAlice
< S\n
> G 4\nname
< V 5\nAlice
> D 4\nname
< D\n
> D 4\nname
< N\n
> G 4\nname
< N\n

g / s / d — namespaced get, set, delete

A namespace is a flat, opaque byte string that scopes a key: the same key name in two namespaces is two independent entries. The lowercase command letters are the namespaced counterparts of G/S/D: the header gains one leading <namespace-length> field and the namespace bytes lead the body.

g <namespace-length> <key-length>\n<namespace><key>
s <namespace-length> <key-length> <value-length>\n<namespace><key><value>
s <namespace-length> <key-length> <value-length> <ttl-seconds>\n<namespace><key><value>
d <namespace-length> <key-length>\n<namespace><key>

Responses are exactly those of the uppercase commands. A <namespace-length> of 0 addresses the default namespace — the same one every G/S/D addresses, so g 0 4\nname and G 4\nname are the same request. <key-length> 0 is rejected as always. The namespace is never interpreted: there is no delimiter, no escaping, no hierarchy — it is sliced by its declared length like every other body field, and may contain any bytes.

Namespaces also enter routing (clusters only). Each node's score is fmix64(fnv1a(node-name) XOR key-hash), where the key hash is

Every SDK and the server pin the same test vectors for both forms. Namespaced frames need a server that understands them; a pre-namespace node (0.3.0 or earlier) closes the connection without a reply, so upgrade every node before clients start using namespaces.

The same key name in the default namespace and in users:

> s 5 4 5\nusersnameAlice
< S\n
> g 5 4\nusersname
< V 5\nAlice
> G 4\nname
< N\n
> d 5 4\nusersname
< D\n

c / F — clear a namespace, flush everything

c <namespace-length>\n<namespace>
F\n

Both respond C\n. c drops every entry in one namespace — a <namespace-length> of 0 clears the default namespace — and F drops every namespace, the default one included. On the node this is a single sub-map removal (no key scan); a map of 4096 entries or more is then freed on a thread of its own (a smaller one inline, which takes a fraction of a millisecond), so neither command stalls other requests the way a scan-and-delete would. The memory is returned over the moments that follow, in proportion to the entries dropped (about 0.3 µs per small entry), and until it is the node can sit above its memory limit by the amount it just dropped. Neither is key-addressed, so W never applies: in a cluster a client sends the command to every member node (a namespace's keys are spread over all of them by rendezvous hashing) and each node clears its own share. A node that is mid-handoff to a joining node replays the clear there too, in order with the keys it is still sending, so a clear racing a join can't leave stale entries on the joiner.

> s 5 4 5\nusersnameAlice
< S\n
> c 5\nusers
< C\n
> g 5 4\nusersname
< N\n
> F\n
< C\n

i — increment/decrement a counter

i <namespace-length> <key-length> <delta>\n<namespace><key>

Adds <delta> — a signed decimal i64; a negative <delta> decrements, so there is no separate decrement command — to the key's stored value, in place, and returns the new value:

I <value-length> [<ttl-seconds>]\n<value>

or N\n if the key is missing or expired (INCR never creates a key), or T\n if the key exists but its stored value isn't a decimal-ASCII i64 — or applying <delta> would overflow one. Values, and the <delta>, must match the same grammar INCR itself writes back: an optional leading -, then digits with no leading zero (other than a lone 0) — no +, no internal sign, no whitespace. A leading zero is refused here although it is accepted in every other number (see numbers in headers). <namespace-length> is always present, unlike G/S/D: INCR has no pre-namespace legacy form, so a <namespace-length> of 0 addresses the default namespace like everywhere else. The optional <ttl-seconds> field on a successful reply is the entry's remaining TTL, rounded up to whole seconds, present only when the entry has one — INCR never resets or clears an existing TTL, unlike S. The TTL isn't cosmetic: a client or nanocached-proxy replicating the result to a key's other owners (see below) needs it to avoid silently making a TTL'd counter immortal on every replica.

This is exactly as volatile as a plain SET. LRU eviction and TTL expiry reclaim an incremented value the same as any other entry — INCR makes an operation atomic against concurrent requests on the node that owns the key, not durable or resistant to memory pressure. It is a good fit for rate limiting and approximate counters; it is not a fit for billing or inventory counts, which need an update to survive.

In a cluster, replication is client-side (see Architecture) and INCR is not fanned out like a plain write: only the key's primary owner runs the increment, and a node that is one of the key's owners but not its primary answers W (unlike a plain write, which every owner accepts), so a client whose ring is stale — after a join the old primary is still an owner — is sent to refresh rather than served a second, independent increment. Once it succeeds, the SDK (or nanocached-proxy in proxy mode) writes the result — the new value and its TTL — to the remaining owners as an ordinary S/s, never by replaying i itself. Re-running the increment on a replica could drift it from the primary (an eviction, or a replica leg that missed an earlier write, leaves it starting from a different base), so every replica always ends up with the exact bytes the primary computed. One exposure remains: for at least a minute after a node joins (60 s from the end of its handoff, plus 5 ms for each entry it sent), an old owner forwards every write it receives to the joiner, the replicated result of an increment included, and that unconditional forward can land after a later increment and rewind the counter — a counter incremented concurrently by several callers during that window can lose a few increments; so can one caller whose increments follow each other within a few milliseconds, while increments spaced well above that are practically exact. Separately, two clients incrementing the same key at the same time can leave a replica one or two increments behind the primary, because each writes its result to the replica on its own and the writes are not ordered. See accepted behaviour.

A counter is an ordinary entry holding decimal text; i never creates one:

> S 7 1\ncounter5
< S\n
> i 0 7 3\ncounter
< I 1\n8
> i 0 7 -10\ncounter
< I 2\n-2
> i 0 5 1\nother
< N\n

k / x — compare-and-set

k <namespace-length> <key-length> <value-length> <cond> [<ttl-seconds>]\n<namespace><key><value>

Stores <value> only if <cond> holds against the key's current stored bytes; otherwise nothing changes. <cond> is one of three bare tokens (not length-prefixed — its own shape identifies it):

On success: S\n, the same acknowledgement a plain S gives. On a condition mismatch: N\n, reusing the same "nothing here to act on" status G/D already use for a miss — k introduces no new response marker. <ttl-seconds> means exactly what it means for S — omitted = no expiry, a literal 0 is a TTL that has already elapsed and the entry expires immediately; unlike INCR, the new value is supplied whole by the caller, so there is no old TTL to preserve.

x <namespace-length> <key-length> <cond> [<tag>]\n<namespace><key>

Both k and x are evaluated by the key's primary owner alone, like INCR: an owner that is not the primary answers W, so two clients with different rings cannot both win the same condition on two owners.

Removes the key only if <cond> holds — this is the two-argument remove(key, old). <cond> here is always a digest: an absent- or present-only conditioned delete is already the plain, unconditional D. On success: D\n, the same acknowledgement a plain D gives for a key that existed. On a mismatch or a missing key: N\n, the same status D already gives when there was nothing to delete.

Both are always namespaced, same as INCR: neither op has a pre-namespace legacy form, so <namespace-length> is unconditionally present (0 = the default namespace).

The digest. SHA-256 of the key's exact stored bytes — the same bytes a V/I response body would carry (for a compression-enabled client, that includes its marker byte, since the server never decompresses) — truncated to the first 16 bytes (128 bits), lowercase hex-encoded (32 characters). Computed identically by the server and every SDK; a fixed cross-language test vector pins the agreement (SHA-256 of the UTF-8 bytes nanocached-cas-vector truncates to 36287141940ca57acbd7695ccdde9d43).

A client obtains an expected digest from an ordinary GET by hashing the response body itself — there is no separate wire round-trip to fetch one, and GET's response shape is unchanged. This is content-based CAS, so it is exactly as sensitive to encoding as memcached's own value-based CAS: an expected digest reconstructed by re-serializing/re-compressing a value a caller already holds (rather than one taken from a real prior read) is only correct if that reconstruction produces byte-identical output to what the server actually stores — true within one client sharing one serializer/compressor, not guaranteed across languages with client-side compression enabled.

This is not a distributed lock. LRU eviction reclaims a key exactly as it would after a plain SET, CAS or not: if a key used as a lock (add to acquire, a TTL to eventually release) is evicted under memory pressure, a second caller's A-conditioned k succeeds while the first caller still believes it holds the lock — a silent double-acquisition CAS cannot detect. k/x are atomic against concurrent requests on the node that currently owns the key, the same guarantee INCR makes and no stronger.

Replication follows INCR's rule exactly: only the key's primary owner evaluates <cond>. Once it succeeds, the SDK (or nanocached-proxy in proxy mode) writes the result to the remaining owners as an ordinary S/s (for k) or D/d (for x) — never by replaying k/x itself. A replica evaluating the same condition against its own possibly-different copy could reach a different outcome than the primary just did.

Using the digest of the test vector above, and the digest of v4:

> S 1 21\nknanocached-cas-vector
< S\n
# the digest matches: replaced
> k 0 1 2 36287141940ca57acbd7695ccdde9d43\nkv2
< S\n
# the key now holds v2, so the same digest no longer matches
> k 0 1 2 36287141940ca57acbd7695ccdde9d43\nkv3
< N\n
# A: only if absent; P: only if present
> k 0 1 2 A\nkv4
< N\n
> k 0 1 2 P\nkv4
< S\n
> x 0 1 8e38a1ea5c681c8e9a08f1af465f1f07\nk
< D\n

t — refresh or remove an entry's TTL

t <namespace-length> <key-length> [<ttl-seconds>] [G]\n<namespace><key>

Replaces the TTL of an existing entry in place, without rewriting its value. <ttl-seconds> means what it means for S: counted from the moment the node receives the request, and 0 expires the entry at once (the next read misses). Omitting it removes the TTL, so the entry never expires. Without G the reply is S\n, or N\n if the key is missing or expired: t never creates a key, and since it never changes the value either, it cannot overwrite a write that lands at the same time. <namespace-length> is always present, like i: a <namespace-length> of 0 addresses the default namespace. The node never interprets the value, so a touch works on a compressed value as well.

With G (get and touch) the request is a read as well, and the reply is that of a read:

V <value-length> [<ttl-seconds>]\n<value>

or N\n. The optional <ttl-seconds> is the TTL now in effect, rounded up to whole seconds, present only when the entry has one. A get and touch counts as a hit or a miss and makes the entry the most recently used, as G does; a plain touch changes only the expiry, so it counts as neither and does not protect the entry from eviction.

In a cluster, replication is client-side, and a touch is not primary-only like i, k and x: it is idempotent (the new expiry does not depend on what the entry holds), so every owner of the key accepts it, as it accepts an S, and a client sends it to each owner rather than writing a result to the others — with one ordering rule for a TTL of 0: a client (and nanocached-proxy) sends an expiring touch to the primary alone and the plain t to the other owners only once the primary has answered. Sent together, a replica would expire its copy while the primary leg was still failing (unreachable, or W from a stale ring), and the retry over the refreshed ring would then ask that replica, now the primary, for a key it no longer holds. Those deferred legs run outside the request order a pipelined connection otherwise keeps. A node that owns no copy of the key answers W. An owner that answers N has no live copy; the touch does not create one and nothing repairs it, while the owners that answered S keep the new expiry. That includes a replica whose copy of the key is still on its way: with fire-and-forget replica writes a client sends the replica legs of a set and of a following touch of the same key independently, so the touch can arrive first, find nothing, and the set then land with its own TTL — none, for a plain set — so that copy never expires. During a join’s or a decommission’s forwarding window an old owner passes a touch on as the value it holds with the new TTL, written like an S, not as t itself: the joiner may not hold the key yet. As for every forwarded write, that unconditional copy can land after a later write to the key and bring back the earlier value — see accepted behaviour.

t needs a node that knows it: an older node closes the connection (from 0.4.8 on it answers En\n first; before that it closed without a reply), so upgrade every node before clients start sending it.

> S 4 5\nnameAlice
< S\n
> t 0 4 60\nname
< S\n
# with G: the value, and the TTL now in effect
> t 0 4 60 G\nname
< V 5 60\nAlice
# no TTL removes it; 0 expires the entry at once
> t 0 4\nname
< S\n
> t 0 4 0\nname
< S\n
> G 4\nname
< N\n
> t 0 4 60\nname
< N\n

y — batched touch

y <namespace-length> <n> <key-length-1> ... <key-length-n> [<ttl-seconds>] [G]\n<namespace><key-1>...<key-n>

t for n keys of one namespace under one frame and, on the node that owns them, one round trip through the cache. <ttl-seconds> means what it means for t — counted from receipt, 0 expires every key at once, omitted removes the TTL — and is one TTL for the whole batch, as it is for o. A key that is missing or expired is a miss and is never created. <namespace-length> is always present, like m.

Without G the reply carries no values:

Y <n> <result-1> ... <result-n>\n

Each <result-i> is S (touched), N (no live entry) or W (this node doesn't own this particular key), in the order of the request's keys.

With G the request is a read as well and the reply is m's, for the keys it touched:

M <n> <result-1> ... <result-n>\n<values of the hits, concatenated in request order>

Each <result-i> is a decimal byte length, - or W. Unlike a single t with G, there is no per-key TTL in the reply: every key has the TTL the frame carried. The values in one reply total at most 16 MiB, as for m, and a key whose value does not fit is answered - and left untouched, so a reply that leaves a value out never refreshed it on that node. Only the node that was asked for values has this bound: the plain form a client sends to a key’s other owners has none, so those owners do refresh the key and its TTL can then differ between the primary and a replica. Each touched key counts as a hit and becomes the most recently used, as for G; a miss counts as a miss. Without G a key counts as neither and is not protected from eviction.

y is idempotent and not primary-only, like t: every owner of a key accepts it, and a client sends it to each owner of the keys in the batch. A client that wants the values sends the G form to a key's primary and the plain form to its other owners. During a join's or a decommission's forwarding window the old owner passes each key it touched on as the value it holds with the new TTL, written like an S, not as y itself. As for m/o, a batch never fails as a whole: a W for some keys leaves the answers for the others standing.

Through nanocached-proxy, a batch spanning several owners is split into one sub-frame per node and the answers are put back in the client's order, as for m/o. Every owner of a key is sent it; with G, a key's primary is sent G for the keys it is primary of (its answer is the client's) and every other owner the plain form. A key whose primary answers W, or whose sub-frame fails, is sent again once over a refreshed ring, and one still wrong after that is answered W on its own. With a TTL of 0 the proxy, like a client, sends only the primaries first and the plain form to the other owners once they have answered, for the keys a primary answered (see t). The proxy also holds the merged M reply to the same 16 MiB of values each node holds its own sub-reply to, so through the proxy a - can stand for a key its node did touch: a value past the merged budget is left out of the reply although the node applied the TTL (and, with a TTL of 0, expired the entry). The left untouched rule above holds for a node's own reply only.

y needs a node that knows it: an older node closes the connection (from 0.4.8 on it answers En\n first; before that it closed without a reply), so upgrade every node before clients start sending it.

> o 0 2 1 1 1 1\nkalb
< O 2 S S\n
> y 0 2 1 1 60\nkl
< Y 2 S S\n

m / o — batched get and set

m <namespace-length> <n> <key-length-1> ... <key-length-n>\n<namespace><key-1>...<key-n>

Gets n keys under one frame and, on the node that owns them, one round trip through the cache — instead of n independent g frames. Replies:

M <n> <result-1> ... <result-n>\n<values of the hits, concatenated in request order>

Each <result-i> is one of three tokens, in the same order as the request's keys: a decimal byte length (a hit — that many bytes of the trailing body belong to this key, in order), - (a miss), or W (this node doesn't own this particular key). The R status and the fatal E still apply to the whole frame, but W never does — a <key-length> being wrong for one key out of a thousand doesn't invalidate the other 999.

n is at least 1: an n of 0, or a header that does not list exactly n lengths, is a malformed request (likewise for o below and y above). There is no upper bound of its own beyond the request-size limit, which a frame of one-byte keys reaches at about 349,000 of them, and, for o and y, the namespace limit above; the SDKs send at most 400 keys in a frame (fewer for an o or y whose namespace is long, see above). The reply header carries one token per key, so it grows with n. A key may appear more than once in a frame; every occurrence is handled on its own, in order, and answered at its own position. m answers each occurrence of a hit with the value; with o the later occurrence overwrites the earlier, so the last value stays; y applies its TTL in order, once per occurrence.

The hit values in one M reply total at most 16 MiB (the reply may therefore be larger than the 1 MiB request limit). Hits past that budget are answered - (a miss), in place, so a batch that names the same large key many times, or many large keys, degrades to misses instead of growing the reply without bound.

o <namespace-length> <n> <key-length-1> <value-length-1> ... <key-length-n> <value-length-n> [<ttl-seconds>]\n<namespace><key-1><value-1>...<key-n><value-n>

Sets n keys the same way, and <ttl-seconds> means exactly what it means for S — omitted = no expiry, a literal 0 is a TTL that has already elapsed and every key in the batch expires immediately. One <ttl-seconds> for the whole batch, not per key — every real caller of a batched set (Django's set_many, cache-manager's mset) already passes one TTL per call, so a per-key TTL field would complicate the frame for no consumer that exists. Replies:

O <n> <result-1> ... <result-n>\n

No body — a set has no value to echo back, unlike m's hits. Each <result-i> is S (stored) or W (wrong node), same per-key independence as m. O is never confused with the On/OnT identity reply A gives: that only ever appears as the very first reply on a connection, right after its A frame, so no other request's reply is ever mistaken for it.

Both are always namespaced, same reasoning as INCR/ k/x: neither has a pre-namespace legacy form, so <namespace-length> is unconditionally present (0 = the default namespace). Both are also restricted to one namespace per frame — rendezvous hashing routes on (namespace, key), so a frame mixing namespaces couldn't route as a single unit anyway, and every real many-key caller already works within one namespace at a time (a cache-manager store, a Django key prefix, a Spring cache name).

A batch never fails as a whole. Every key's outcome — hit/miss/wrong-node for m, stored/wrong-node for o — is independent of every other key's. A client (or nanocached-proxy in proxy mode) that sees a W for some subset of a batch refreshes its routing table and re-issues just those keys, exactly as it would for a single-key G/ S's own W — the rest of the batch's answers stand.

Apart from that, an o (and a y) whose namespace length times its key count is over 64 MiB is refused whole, not per key (see the limits at the top of this page). Through nanocached-proxy the sub-frames sent to the nodes carry the client’s namespace and a subset of its keys, so they stay within it when the client’s frame does.

Through nanocached-proxy, a batch spanning multiple owners is split into one sub-frame per node and reassembled before answering the client — the one dispatch shape that turns a single client frame into more than one key's worth of backend traffic. A m sub-frame targets each key's primary only, same as a plain G; a o sub-frame reaches every owner a sub-batch's keys have (primary or replica — the same node can be primary for one key and a replica for another within one batch), and only the primary's outcome decides a key's client-facing status, replica outcomes being logged and swallowed exactly like a plain S's replication already is.

Two keys set in one frame, then three read — the third is missing:

> o 0 2 1 1 1 1\nkalb
< O 2 S S\n
> m 0 3 1 1 1\nklz
< M 3 1 1 -\nab

Response tags (tagged mode)

Pipelined clients match responses to requests by order alone — the frames above carry nothing to verify against. A connection whose A carried the T flag and was answered OnT runs in tagged mode: every G/S/D (and g/s/d/c/F/i/t/k/x/m/o/y) header must end with a client-chosen tag (a u32 in decimal), and the response echoes it as its own last field, so a client can verify the pairing before trusting the answer.

G <key-length> <tag>\n<key>
S <key-length> <value-length> <tag>\n<key><value>
S <key-length> <value-length> <ttl-seconds> <tag>\n<key><value>
D <key-length> <tag>\n<key>
g <namespace-length> <key-length> <tag>\n<namespace><key>
s <namespace-length> <key-length> <value-length> [<ttl-seconds>] <tag>\n<namespace><key><value>
d <namespace-length> <key-length> <tag>\n<namespace><key>
c <namespace-length> <tag>\n<namespace>
F <tag>\n
i <namespace-length> <key-length> <delta> <tag>\n<namespace><key>
t <namespace-length> <key-length> [<ttl-seconds>] [G] <tag>\n<namespace><key>
k <namespace-length> <key-length> <value-length> <cond> [<ttl-seconds>] <tag>\n<namespace><key><value>
x <namespace-length> <key-length> <cond> <tag>\n<namespace><key>
m <namespace-length> <n> <key-length-1> ... <key-length-n> <tag>\n<namespace><key-1>...<key-n>
o <namespace-length> <n> <key-length-1> <value-length-1> ... <key-length-n> <value-length-n> [<ttl-seconds>] <tag>\n<namespace><key-1><value-1>...<key-n><value-n>
y <namespace-length> <n> <key-length-1> ... <key-length-n> [<ttl-seconds>] [G] <tag>\n<namespace><key-1>...<key-n>

Responses become V <value-length> <tag>\n, S <tag>\n, D <tag>\n, N <tag>\n, W <tag>\n, and C <tag>\n; B\n stays bare (it is unsolicited, sent before authentication); a retryable error is R <tag>\n (see below). i's reply becomes I <value-length> [<ttl-seconds>] <tag>\n or T <tag>\n — the tag is always the last header field, after the optional TTL. k's reply is S <tag>\n or N <tag>\n; x's is D <tag>\n or N <tag>\n — both reuse the markers above, so tagging them needs no new shape. t's reply is S <tag>\n or N <tag>\n, and with G V <value-length> [<ttl-seconds>] <tag>\n (the tag after the optional TTL, as for i). m's reply is M <n> <result-1> ... <result-n> <tag>\n<hit values>, o's is O <n> <result-1> ... <result-n> <tag>\n, y's is Y <n> <result-1> ... <result-n> <tag>\n or, with G, m's M reply — same roster shape as the untagged form, with the tag appended as the header's own last field. Tagged mode is per connection and entirely opt-in — a plain A keeps every frame exactly as documented above.

A node’s refusal is the exception: a request it will not run (one it cannot parse, or one over the size limit) is answered with the bare En\n, untagged, even on a tagged connection, and the node then closes the connection. A node answers a connection’s requests one at a time, in order, so by the time it refuses a request every earlier reply has been written: the En is the answer to the oldest request still waiting, and whatever was sent behind it is never answered. nanocached-proxy reads it that way on its connections to the nodes, which is what lets it answer a refused expiring touch, i, k or x with R (to a client that asked for it) instead of the fatal E of an unknown outcome. To its own clients the proxy tags its errors, E <tag>\n, as the table below shows.

Every request ends its header with a tag and every reply echoes it:

> A 1 T\n\0
< OnT\n
> S 4 5 7\nnameAlice
< S 7\n
> G 4 8\nname
< V 5 8\nAlice
> G 7 9\nmissing
< N 9\n

What an untagged connection cannot promise

Without tags, an SDK pairs each response with the oldest request still waiting, by order alone. When a response of the wrong kind arrives (an S answering a G, say), the SDK knows the streams are misaligned, poisons the connection and redials — but the requests queued behind the mismatched one may already have been handed another request's reply, and a reply of the right kind for the wrong request is indistinguishable from a correct one. That window is inherent to matching by order; it is exactly what tagged mode removes, which is why every SDK asks for T and only falls back to an untagged connection against a server that predates tags (one that rejects the extended A). On such a server the window is the same one the pre-tags protocol always had.

Other statuses you may see

ReplyMeaning
B\nBusy. From a node: the connection limit is reached. From a discovery server: it is inside its startup grace after a restart, re-learning membership — retry shortly or try another replica. From nanocached-proxy: it has no member list yet (it has not reached discovery) — a client that declared the R capability below gets R instead.
W\nWrong node (clusters only): per the node's own view of membership it does not own this key, so the client's routing table is stale. SDKs respond by refreshing the node list and retrying once.
R\nRetryable error: this one request failed transiently — an upstream briefly unreachable behind nanocached-proxy, typically — and should be retried shortly on the same connection; nothing is torn down. Sent only to clients that declared the capability by appending R to their A frame (A <len> [T] R); for an upstream failure legacy clients keep getting the fatal E-and-close. A proxy never answers R for an i, k or x whose outcome it cannot know — its connection to the node failed after the command may have reached it — because a re-send could apply the command twice: that case is the fatal E, and the client reads the key to learn what happened. A t is idempotent, so a proxy that cannot reach its node answers it R. A proxy also answers R while it has no member list yet, and to a request it could not admit within 60 seconds because the in-flight byte budgets (per connection, per source IP and proxy-wide) stayed full, and at once to a request that would have to wait while the pool for waiting requests' frames (the budget's size again) is full; the request was not dispatched, so any command is safe to retry. A client that did not declare the capability gets a bare B\n in these cases instead, and its connection stays open. Tagged connections receive R <tag>\n. Today the proxy is the only emitter — nodes and discovery accept the capability token but have no transient per-request failure to report.
T\nNot numeric (i only): the key exists but its stored value isn't a decimal-ASCII i64, or applying <delta> would overflow one.
E\nError — En\n from a node, Ed\n from a discovery server: a failed A, a command sent before authentication, the request was malformed (an unknown command, a bad length or TTL, an empty key), or it was over the request-size limit. Usually followed by the server closing the connection. nanocached-proxy answers a failed A En\n like a node and every other such error with a bare E\n (E <tag>\n in tagged mode).

A request the server cannot parse, or a key of length 0:

> G 0\n
< En\n
= closed
> X\n
< En\n
= closed

L / Q — list nodes, list proxies

L\n
Q\n

These two are sent to a discovery server, not to a cache node (a node answers them En\n). They take no arguments: anything after the letter, or a lowercase l, is a malformed request, answered Ed\n with the connection closed. When the discovery server has an auth secret, send A first, as for any other command; without one, Ed\n.

L lists the cache nodes currently in the cluster:

N <count> <replication>\n
<name-length> <address-length>\n<name><address>\n    (<count> times)

Q lists the proxies registered with it, in the same shape without the replication field:

N <count>\n
<name-length> <address-length>\n<name><address>\n    (<count> times)

<replication> is the cluster’s replication factor, the number of nodes that hold each key. <name> is the node’s identity: an opaque string, chosen by the node, that the client hashes to place the node on the ring; it is not an address and carries no meaning of its own. <address> is host:port and is only for opening a connection. Both lengths are in bytes. Every entry ends with a \n that neither length counts, so the next entry’s header starts right after it. An empty list is N 0\n for Q. For example, one node and no proxies, with replication factor 2:

L
N 1 2
36 15
5b5e966a-925e-44ff-ba6e-1859cce25af6127.0.0.1:19456

Q
N 0

While a discovery server is inside its startup grace after a restart, L and Q are answered B\n and the connection is closed, because a partial list would read as a cluster that has fewer nodes (or no proxies) than it does; retry shortly. L is answered the same way while most of the nodes that reported a replication factor disagree with the discovery server’s own. Several requests can follow one another on a connection and are answered in order. The replies are never tagged: a connection that asked for tags is answered OdT to its A and still gets the plain N reply, which has no tag field.

Cluster-internal commands

Nodes, discovery servers, and SDKs additionally exchange membership and migration frames — L (node list, above), J/H/P (join, heartbeat, announce), M/X/C (migrate, cancel, complete), Y/Z/Q (proxy announce, deregister, and list — how nanocached-proxy registers with discovery and how SDKs in proxy mode find one), U/u/f/V (handoff store, handoff delete, handoff clear, and node leave — the frames a node uses to move its entries to another node, replay a client's clear there, and exit membership; see Deployment), and T (node roster: a registered node fetching the member list with every member's membership token, self-identifying the way a heartbeat does — the frames below need those tokens, and discovery hands them only to proven members, never to a client that merely holds the shared secret; see Architecture for what that trust model implies). M, U, u and f each carry the receiving node's membership token (U <ns-len> <key-len> <val-len> <token-len> [ttl] [A]; f <ns-len> <token-len> [A], where A clears every namespace), verified before the frame acts, since these are exactly the frames that skip the wrong-node check. A clear reaches another node as f rather than as the client's c/F so the receiver knows it was forwarded: it passes it on through its join targets as it would a client's clear, but through its own decommission's window (behind the writes queued for the entrants) only when it removed something, so a clear that changed nothing stops there, where a clear indistinguishable from a client's used to circle between a decommissioning node and the node forwarding to it until a window closed. U also carries an optional trailing A token: a survivor re-replicating a key after a membership change dropped a member (an eviction, or a leave it did not itself hand off for — see Architecture) sends it instead of a plain overwrite, so a re-replication racing a newer client write on the receiver can never regress it — the receiver stores the entry only if it was absent and no u removed it within the last two minutes (a relayed delete leaves a tombstone, since a sender may have read the key before that delete), and acks S\n either way, since a key already present there is a success for the sender, not a conflict. An ordinary decommission handoff never sets it. H takes an optional trailing <joiner-len> header field, with the joiner's name after the token in the body, naming a node whose handoff the sender completed but has not yet seen in any roster; discovery then follows its acknowledgment — the roster, or the bare A\n — with exactly one more line, K\n (still registered, or not known to be gone) or G\n (on record as having left, been evicted or abandoned, and not registered since), and on G the node abandons that handoff. A sender that put the field in must read that line. Two of these are part of what SDKs send and are frozen by the compatibility policy like the caching commands: L (the node list) and Q (the proxy list). The rest are versioned with the server and not part of the public caching API, so the compatibility policy does not freeze them; the server source documents them in detail.

A complete session

$ printf 'S 4 5\nnameAliceG 4\nname' | nc 127.0.0.1 8356
S
V 5
Alice

Two pipelined requests — a set and a get — answered in order on one connection.