Wire protocol
Each request starts with an ASCII header terminated by
\n; key and value bodies follow immediately, read by their
declared byte lengths. Bodies have no terminator and may contain
arbitrary bytes. One-byte commands keep overhead minimal while frames
stay readable during development.
From 1.0.0 the client-facing protocol on this page is frozen; the compatibility policy says what that covers.
The transcripts on this page show the bytes on the wire: a line
starting with > is sent, a line starting with
< is what the server answers, \n stands for a
newline and \0 for a zero byte, a line starting with
# is a comment, and = closed means the server then
closes the connection. A line ending carries no newline of its own.
Multiple requests per connection and pipelining are both supported. A
<key-length> of 0 is rejected for every
command; there is no dedicated key or value length limit beyond the
overall request-size limit: 1 MiB (1,048,576 bytes) per request on a node,
the header line and the bodies together, counted per frame (what is
pipelined behind a frame does not count towards it). A request over it,
like one the server cannot parse, is answered En\n and the
connection closed. The header line has no limit of its own: a header that
runs past the limit without its \n is refused the same way. Through
nanocached-proxy the limit is 11 bytes lower, 1,048,565: the
proxy puts a tag on the frame it passes to a node, and the node’s
limit still has to hold for the longer frame; the proxy answers a bare
E\n. An i through the proxy is further limited
to 1,048,512 bytes of namespace and key together, because the proxy sends
its result to the replicas as a set, which carries the new value and the
TTL that the i does not; a longer one closes the connection like
any other request the proxy cannot accept. An o or
y is also limited by its namespace: the namespace length times
the number of keys in the frame (all of them, whichever node they route to)
may not exceed 64 MiB (67,108,864). A node forwards each key of such a
frame as a frame of its own while a join, a decommission or an eviction
changes the cluster, and every one repeats the namespace, so without the limit
a frame of 1 MiB with a 500 KB namespace and about 90,000 one-byte
keys would make the node send tens of gigabytes. A frame over it is refused
from its header, before its body is read, like any request the server cannot
parse: a node answers En\n and closes the connection, the proxy
answers a bare E\n and closes the client’s connection
without sending anything to a node. The limit does not depend on whether the
cluster is changing. The SDKs send at most 400 keys in a frame and, for
o and y, no more than 64 MiB divided by the
namespace length (rounded down), so their frames stay under the limit with
any namespace; a namespace of 180 bytes or less cannot reach it at any frame
size. The frames only another node
sends (U, u, f) may exceed it by
4 KiB: the U that moves a value to another node carries the
receiver’s token on top of what the client’s S
carried, and a value stored at the limit must still be movable.
Numbers in headers
Every number in a header — a length, an entry count, a TTL, a tag, a
namespace length — is an unsigned decimal: one or more ASCII digits
0–9 and nothing else. A sign (+
or -), hex, an exponent, a non-ASCII digit, an empty field
and a number too large for its field are each a malformed request,
answered En\n with the connection closed. Fields are
separated by exactly one space, so a doubled or trailing space leaves an
empty field and is malformed too. A TTL (on S, s,
o, k, t and y) goes up to
264−1 and a tag up to 232−1; a length or count has no limit of its
own beyond the request-size limit above.
Leading zeros are accepted and mean the same number:
G 01 is G 1, a TTL of 007 is 7, and
the tag 007 is echoed back as 7. No SDK sends
them and no server does, but they are part of the frozen protocol: a
server keeps accepting them.
i’s <delta> is the one exception. It
is signed and is held to the grammar of a stored counter (see
i): an optional -, then digits with no
leading zero other than a lone 0, within the range of an
i64. -0 is accepted (it is 0); +3,
007 and -007 are malformed. The proxy and the
discovery server read numbers the same way (the proxy answers a bare
E\n where a node answers En\n).
> S 4 5\nnameAlice
< S\n
# leading zeros are the same number
> G 04\nname
< V 5\nAlice
# a delta takes none
> i 0 4 007\nname
< En\n
= closed
A — authenticate
A <secret-length>\n<secret>
Responds On\n on success or En\n followed by
the server closing the connection (Od\n/Ed\n
from a discovery server — the second byte always names the server type,
see the note below). If the server has no auth secret
configured, any well-formed A succeeds (an empty secret,
A 0, is malformed and answered En\n like any
other). A client that has no secret therefore sends a one-byte
placeholder: every SDK sends a single NUL byte, A 1 T R\n
followed by \0. If the server does have a secret, every other
command is rejected with En\n (and the connection closed)
until a matching A has been sent. Until then the node
accepts a frame of at most 4096 bytes, which bounds the A
frame itself (header and secret); a server refuses to start with a
secret too long to fit.
A failed A that asked for tags (a T field) is
answered EnT\n by a node and by the proxy and
EdT\n by a discovery server, echoing the capability as a success
does. Either way the connection is then closed.
Send A once, as the first frame. A node and a discovery
server accept a later A too and run it afresh: a matching one
is answered On/OnT (Od/OdT)
again, and the connection continues in the mode that latest
A asked for, tags or none; a failing one is answered
En and the connection is closed, even if an earlier
A had succeeded. nanocached-proxy accepts no
second A at all: it answers a bare E\n, even for
a correct secret, and closes the connection.
n/d —
node/discovery) is how a client discovers what kind of server it
reached and adapts, with no configuration; it is sent whether or not
the secret was accepted, and whatever form of A was used.
SDKs additionally append
a T field (A <secret-length> T\n) to
request response tags; a server that supports them
echoes the capability as OnT/OdT.The capability fields after the length are T
and R, in that order. Every other token is reserved: a node
answers En, a discovery server Ed and
nanocached-proxy a bare E, and the connection is
closed. That refusal is also how an SDK learns that a server does not know a
capability it asked for, and falls back to asking for less. A later release
may define a new token, only as an opt-in capability a client asks for, never
as a change to what a request without it does. V is set aside for
versioned writes (a version carried on write requests and replies, so that the
owners of a key can order the writes to it); no release defines it yet, and
nothing should send it.
A server with no secret, asked for tags with the placeholder secret:
> A 1 T R\n\0
< OnT\n
A server started with the secret hunter2:
> G 1\nk
< En\n
= closed
> A 7\nhunter2
< On\n
> G 1\nk
< N\n
> A 5\nwrong
< En\n
= closed
G — get
G <key-length>\n<key>
Responds with the value:
V <value-length>\n<value>
or N\n when the key is missing or expired.
S — set
S <key-length> <value-length>\n<key><value>
S <key-length> <value-length> <ttl-seconds>\n<key><value>
Responds S\n. <ttl-seconds> is an
unsigned 64-bit decimal (up to 18446744073709551615; a
larger number is a malformed request). Without a TTL the entry lives
until evicted (LRU under the memory bound) or deleted; with one it also
expires <ttl-seconds> after the write — a literal
0 is a TTL that has already elapsed, so the entry expires
immediately (the next access sees a miss).
A value may be empty (<value-length> of 0;
a <key-length> of 0 is not allowed). It is
stored like any other, by S or by a batched o:
G answers V 0\n — a hit with no bytes after the
header, not a miss — and m a result of 0.
A TTL of 0 has already elapsed, and an empty value is a hit:
> S 4 5 0\nnameAlice
< S\n
> G 4\nname
< N\n
> S 5 0\nempty
< S\n
> G 5\nempty
< V 0\n
D — delete
D <key-length>\n<key>
Responds D\n if the key existed, N\n
otherwise.
Set, read, delete, and read again:
> S 4 5\nnameAlice
< S\n
> G 4\nname
< V 5\nAlice
> D 4\nname
< D\n
> D 4\nname
< N\n
> G 4\nname
< N\n
g / s / d — namespaced get, set, delete
A namespace is a flat, opaque byte string that scopes a key:
the same key name in two namespaces is two independent entries. The
lowercase command letters are the namespaced counterparts of
G/S/D: the header gains one
leading <namespace-length> field and the namespace
bytes lead the body.
g <namespace-length> <key-length>\n<namespace><key>
s <namespace-length> <key-length> <value-length>\n<namespace><key><value>
s <namespace-length> <key-length> <value-length> <ttl-seconds>\n<namespace><key><value>
d <namespace-length> <key-length>\n<namespace><key>
Responses are exactly those of the uppercase commands. A
<namespace-length> of 0 addresses the default
namespace — the same one every G/S/D
addresses, so g 0 4\nname and G 4\nname are
the same request. <key-length> 0 is rejected as
always. The namespace is never interpreted: there is no delimiter,
no escaping, no hierarchy — it is sliced by its declared length like
every other body field, and may contain any bytes.
Namespaces also enter routing (clusters only). Each node's score is
fmix64(fnv1a(node-name) XOR key-hash), where the key hash
is
fnv1a(key)for the default namespace — byte-identical to the pre-namespace form, so existing keys keep their placement across an upgrade; andfnv1a(be32(namespace-length) || namespace || key)otherwise: the namespace length as a 4-byte big-endian integer, then the namespace bytes, then the key bytes, hashed as one stream.
Every SDK and the server pin the same test vectors for both forms. Namespaced frames need a server that understands them; a pre-namespace node (0.3.0 or earlier) closes the connection without a reply, so upgrade every node before clients start using namespaces.
The same key name in the default namespace and in users:
> s 5 4 5\nusersnameAlice
< S\n
> g 5 4\nusersname
< V 5\nAlice
> G 4\nname
< N\n
> d 5 4\nusersname
< D\n
c / F — clear a namespace, flush everything
c <namespace-length>\n<namespace>
F\n
Both respond C\n. c drops every entry in
one namespace — a <namespace-length> of 0 clears the
default namespace — and F drops every namespace, the
default one included. On the node this is a single sub-map removal (no
key scan); a map of 4096 entries or more is then freed on a thread of
its own (a smaller one inline, which takes a fraction of a
millisecond), so neither command stalls other requests the way a
scan-and-delete would. The memory is returned over the moments that follow, in
proportion to the entries dropped (about 0.3 µs per small entry), and
until it is the node can sit above its memory limit by the amount it
just dropped. Neither is
key-addressed, so W never applies: in a cluster a client
sends the command to every member node (a namespace's keys are
spread over all of them by rendezvous hashing) and each node clears its
own share. A node that is mid-handoff to a joining node replays the
clear there too, in order with the keys it is still sending, so a clear
racing a join can't leave stale entries on the joiner.
> s 5 4 5\nusersnameAlice
< S\n
> c 5\nusers
< C\n
> g 5 4\nusersname
< N\n
> F\n
< C\n
i — increment/decrement a counter
i <namespace-length> <key-length> <delta>\n<namespace><key>
Adds <delta> — a signed decimal i64; a
negative <delta> decrements, so there is no separate
decrement command — to the key's stored value, in place, and returns the
new value:
I <value-length> [<ttl-seconds>]\n<value>
or N\n if the key is missing or expired (INCR
never creates a key), or T\n if the key exists but its
stored value isn't a decimal-ASCII i64 — or applying
<delta> would overflow one. Values, and the <delta>, must match the same
grammar INCR itself writes back: an optional leading
-, then digits with no leading zero (other than a lone
0) — no +, no internal sign, no whitespace. A
leading zero is refused here although it is accepted in every other
number (see numbers in headers).
<namespace-length> is always present, unlike
G/S/D: INCR has no
pre-namespace legacy form, so a <namespace-length> of 0
addresses the default namespace like everywhere else. The optional
<ttl-seconds> field on a successful reply is the
entry's remaining TTL, rounded up to whole seconds, present
only when the entry has one — INCR never resets or clears an
existing TTL, unlike S. The TTL isn't cosmetic: a client or
nanocached-proxy replicating the result to a key's other
owners (see below) needs it to avoid silently making a TTL'd counter
immortal on every replica.
This is exactly as volatile as a plain SET.
LRU eviction and TTL expiry reclaim an incremented value the same as any
other entry — INCR makes an operation atomic against
concurrent requests on the node that owns the key, not durable or
resistant to memory pressure. It is a good fit for rate limiting and
approximate counters; it is not a fit for billing or inventory counts,
which need an update to survive.
In a cluster, replication is client-side (see
Architecture) and INCR is
not fanned out like a plain write: only the key's primary owner runs the
increment, and a node that is one of the key's owners but not its primary
answers W (unlike a plain write, which every owner accepts),
so a client whose ring is stale — after a join the old primary is still
an owner — is sent to refresh rather than served a second, independent
increment. Once it succeeds, the SDK (or nanocached-proxy in
proxy mode) writes the result — the new value and its TTL — to
the remaining owners as an ordinary S/s, never
by replaying i itself. Re-running the increment on a
replica could drift it from the primary (an eviction, or a replica leg
that missed an earlier write, leaves it starting from a different base),
so every replica always ends up with the exact bytes the primary
computed. One exposure remains: for at least a minute after a node joins
(60 s from the end of its handoff, plus 5 ms for each entry it sent),
an old owner forwards every write it receives to the joiner, the
replicated result of an increment included, and that unconditional
forward can land after a later increment and rewind the counter —
a counter incremented concurrently by several callers during that
window can lose a few increments; so can one caller whose increments
follow each other within a few milliseconds, while increments spaced well
above that are practically exact. Separately, two clients incrementing the
same key at the same time can leave a replica one or two increments behind
the primary, because each writes its result to the replica on its own and
the writes are not ordered.
See accepted
behaviour.
A counter is an ordinary entry holding decimal text; i never creates one:
> S 7 1\ncounter5
< S\n
> i 0 7 3\ncounter
< I 1\n8
> i 0 7 -10\ncounter
< I 2\n-2
> i 0 5 1\nother
< N\n
k / x — compare-and-set
k <namespace-length> <key-length> <value-length> <cond> [<ttl-seconds>]\n<namespace><key><value>
Stores <value> only if <cond> holds
against the key's current stored bytes; otherwise nothing changes.
<cond> is one of three bare tokens (not
length-prefixed — its own shape identifies it):
A— succeeds only if the key is absent (including lazily expired). This isadd/putIfAbsent.P— succeeds only if the key currently holds any (unexpired) value, whatever it is. This is the two-argumentreplace(key, value).- a 32-character lowercase hex digest — succeeds only if the key
holds an unexpired value whose digest (below) equals this exactly.
This is the three-argument
replace(key, old, new).
On success: S\n, the same acknowledgement a plain
S gives. On a condition mismatch: N\n, reusing
the same "nothing here to act on" status G/D
already use for a miss — k introduces no new response
marker. <ttl-seconds> means exactly what it means for
S — omitted = no expiry, a literal 0 is a TTL
that has already elapsed and the entry expires immediately; unlike
INCR, the new value is supplied whole by the caller, so
there is no old TTL to preserve.
x <namespace-length> <key-length> <cond> [<tag>]\n<namespace><key>
Both k and x are evaluated by the key's
primary owner alone, like INCR: an owner that is not the
primary answers W, so two clients with different rings
cannot both win the same condition on two owners.
Removes the key only if <cond> holds — this is the
two-argument remove(key, old). <cond> here
is always a digest: an absent- or present-only conditioned delete is
already the plain, unconditional D. On success:
D\n, the same acknowledgement a plain D gives
for a key that existed. On a mismatch or a missing key: N\n,
the same status D already gives when there was nothing to
delete.
Both are always namespaced, same as INCR: neither op has
a pre-namespace legacy form, so <namespace-length> is
unconditionally present (0 = the default namespace).
The digest. SHA-256 of the key's exact stored bytes —
the same bytes a V/I response body would carry
(for a compression-enabled client, that includes its marker byte, since
the server never decompresses) — truncated to the first 16 bytes (128
bits), lowercase hex-encoded (32 characters). Computed identically by the
server and every SDK; a fixed cross-language test vector pins the
agreement (SHA-256 of the UTF-8 bytes nanocached-cas-vector
truncates to 36287141940ca57acbd7695ccdde9d43).
A client obtains an expected digest from an ordinary GET
by hashing the response body itself — there is no separate wire
round-trip to fetch one, and GET's response shape is
unchanged. This is content-based CAS, so it is exactly as
sensitive to encoding as memcached's own value-based CAS: an expected
digest reconstructed by re-serializing/re-compressing a value a caller
already holds (rather than one taken from a real prior read) is only
correct if that reconstruction produces byte-identical output to what
the server actually stores — true within one client sharing one
serializer/compressor, not guaranteed across languages with client-side
compression enabled.
This is not a distributed lock. LRU eviction reclaims
a key exactly as it would after a plain SET, CAS or not: if
a key used as a lock (add to acquire, a TTL to eventually
release) is evicted under memory pressure, a second caller's
A-conditioned k succeeds while the first caller
still believes it holds the lock — a silent double-acquisition CAS
cannot detect. k/x are atomic against
concurrent requests on the node that currently owns the key, the same
guarantee INCR makes and no stronger.
Replication follows INCR's rule exactly: only the key's
primary owner evaluates <cond>. Once it succeeds, the
SDK (or nanocached-proxy in proxy mode) writes the
result to the remaining owners as an ordinary
S/s (for k) or
D/d (for x) — never by replaying
k/x itself. A replica evaluating the same
condition against its own possibly-different copy could reach a
different outcome than the primary just did.
Using the digest of the test vector above, and the digest of v4:
> S 1 21\nknanocached-cas-vector
< S\n
# the digest matches: replaced
> k 0 1 2 36287141940ca57acbd7695ccdde9d43\nkv2
< S\n
# the key now holds v2, so the same digest no longer matches
> k 0 1 2 36287141940ca57acbd7695ccdde9d43\nkv3
< N\n
# A: only if absent; P: only if present
> k 0 1 2 A\nkv4
< N\n
> k 0 1 2 P\nkv4
< S\n
> x 0 1 8e38a1ea5c681c8e9a08f1af465f1f07\nk
< D\n
t — refresh or remove an entry's TTL
t <namespace-length> <key-length> [<ttl-seconds>] [G]\n<namespace><key>
Replaces the TTL of an existing entry in place, without rewriting its
value. <ttl-seconds> means what it means for
S: counted from the moment the node receives the request,
and 0 expires the entry at once (the next read misses).
Omitting it removes the TTL, so the entry never expires.
Without G the reply is S\n, or N\n
if the key is missing or expired: t never creates a key, and
since it never changes the value either, it cannot overwrite a write that
lands at the same time. <namespace-length> is always
present, like i: a <namespace-length> of 0
addresses the default namespace. The node never interprets the value, so
a touch works on a compressed value as well.
With G (get and touch) the request is a read as well, and
the reply is that of a read:
V <value-length> [<ttl-seconds>]\n<value>
or N\n. The optional <ttl-seconds> is
the TTL now in effect, rounded up to whole seconds, present only when
the entry has one. A get and touch counts as a hit or a miss and makes the
entry the most recently used, as G does; a plain touch
changes only the expiry, so it counts as neither and does not protect the
entry from eviction.
In a cluster, replication is client-side, and a touch is not primary-only
like i, k and x: it is idempotent
(the new expiry does not depend on what the entry holds), so every owner
of the key accepts it, as it accepts an S, and a client sends
it to each owner rather than writing a result to the others — with
one ordering rule for a TTL of 0: a client (and
nanocached-proxy) sends an expiring touch to the primary alone
and the plain t to the other owners only once the primary has
answered. Sent together, a replica would expire its copy while the primary
leg was still failing (unreachable, or W from a stale ring),
and the retry over the refreshed ring would then ask that replica, now the
primary, for a key it no longer holds. Those deferred legs run outside the
request order a pipelined connection otherwise keeps. A node that
owns no copy of the key answers W. An owner that answers
N has no live copy; the touch does not create one and
nothing repairs it, while the owners that answered S keep the
new expiry. That includes a replica whose copy of the key is still on its
way: with fire-and-forget replica writes a client sends the replica legs of
a set and of a following touch of the same key
independently, so the touch can arrive first, find nothing, and the
set then land with its own TTL — none, for a plain
set — so that copy never expires. During a join’s or a decommission’s forwarding
window an old owner passes a touch on as the value it holds with the new
TTL, written like an S, not as t itself: the
joiner may not hold the key yet. As for every forwarded write, that
unconditional copy can land after a later write to the key and bring
back the earlier value — see
accepted behaviour.
t needs a node that knows it: an older node closes the
connection (from 0.4.8 on it answers En\n first; before that
it closed without a reply), so upgrade every node before clients start
sending it.
> S 4 5\nnameAlice
< S\n
> t 0 4 60\nname
< S\n
# with G: the value, and the TTL now in effect
> t 0 4 60 G\nname
< V 5 60\nAlice
# no TTL removes it; 0 expires the entry at once
> t 0 4\nname
< S\n
> t 0 4 0\nname
< S\n
> G 4\nname
< N\n
> t 0 4 60\nname
< N\n
y — batched touch
y <namespace-length> <n> <key-length-1> ... <key-length-n> [<ttl-seconds>] [G]\n<namespace><key-1>...<key-n>
t for n keys of one
namespace under one frame and, on the node that owns them, one round trip
through the cache. <ttl-seconds> means what it means for
t — counted from receipt, 0 expires every key
at once, omitted removes the TTL — and is one TTL for the
whole batch, as it is for o. A
key that is missing or expired is a miss and is never created.
<namespace-length> is always present, like
m.
Without G the reply carries no values:
Y <n> <result-1> ... <result-n>\n
Each <result-i> is S (touched),
N (no live entry) or W (this node doesn't own this
particular key), in the order of the request's keys.
With G the request is a read as well and the reply is
m's, for the keys it touched:
M <n> <result-1> ... <result-n>\n<values of the hits, concatenated in request order>
Each <result-i> is a decimal byte length, -
or W. Unlike a single t with G, there
is no per-key TTL in the reply: every key has the TTL the frame carried. The
values in one reply total at most 16 MiB, as for m, and a key
whose value does not fit is answered - and left
untouched, so a reply that leaves a value out never refreshed it on
that node. Only the node that was asked for values has this bound: the
plain form a client sends to a key’s other owners has none, so those
owners do refresh the key and its TTL can then differ between the
primary and a replica. Each
touched key counts as a hit and becomes the most recently used, as for
G; a miss counts as a miss. Without G a key counts
as neither and is not protected from eviction.
y is idempotent and not primary-only, like t:
every owner of a key accepts it, and a client sends it to each owner of the
keys in the batch. A client that wants the values sends the
G form to a key's primary and the plain form to its other
owners. During a join's or a decommission's forwarding window the old owner
passes each key it touched on as the value it holds with the new TTL, written
like an S, not as y itself. As for
m/o, a batch never fails as a whole: a
W for some keys leaves the answers for the others standing.
Through nanocached-proxy, a batch spanning several owners is
split into one sub-frame per node and the answers are put back in the
client's order, as for m/o. Every owner of a key is
sent it; with G, a key's primary is sent G for the
keys it is primary of (its answer is the client's) and every other owner the
plain form. A key whose primary answers W, or whose sub-frame
fails, is sent again once over a refreshed ring, and one still wrong after
that is answered W on its own. With a TTL of 0 the
proxy, like a client, sends only the primaries first and the plain form to
the other owners once they have answered, for the keys a primary answered
(see t). The proxy also holds the merged
M reply to the same 16 MiB of values each node holds its own
sub-reply to, so through the proxy a - can stand for a key its
node did touch: a value past the merged budget is left out of the reply
although the node applied the TTL (and, with a TTL of 0,
expired the entry). The left untouched rule above holds for a
node's own reply only.
y needs a node that knows it: an older node closes the
connection (from 0.4.8 on it answers En\n first; before that
it closed without a reply), so upgrade every node before clients start
sending it.
> o 0 2 1 1 1 1\nkalb
< O 2 S S\n
> y 0 2 1 1 60\nkl
< Y 2 S S\n
m / o — batched get and set
m <namespace-length> <n> <key-length-1> ... <key-length-n>\n<namespace><key-1>...<key-n>
Gets n keys under one frame and, on the node that owns
them, one round trip through the cache — instead of n
independent g frames. Replies:
M <n> <result-1> ... <result-n>\n<values of the hits, concatenated in request order>
Each <result-i> is one of three tokens, in the same
order as the request's keys: a decimal byte length (a hit — that many
bytes of the trailing body belong to this key, in order), -
(a miss), or W (this node doesn't own this particular key).
The R status and the fatal E still apply to
the whole frame, but W never does — a <key-length>
being wrong for one key out of a thousand doesn't invalidate the other
999.
n is at least 1: an n of 0, or a
header that does not list exactly n lengths, is a malformed
request (likewise for o below and y above). There is no
upper bound of its own beyond the request-size limit, which a frame of
one-byte keys reaches at about 349,000 of them, and, for o and
y, the namespace limit above; the SDKs send at most 400
keys in a frame (fewer for an o or y whose namespace
is long, see above). The reply header carries one token per key, so it grows
with n. A key may appear more than once in a frame; every
occurrence is handled on its own, in order, and answered at its own
position. m answers each occurrence of a hit with the value; with
o the later occurrence overwrites the earlier, so the last value
stays; y applies its TTL in order, once per occurrence.
The hit values in one M reply total at most 16 MiB (the
reply may therefore be larger than the 1 MiB request limit). Hits past
that budget are answered - (a miss), in place, so a batch
that names the same large key many times, or many large keys, degrades
to misses instead of growing the reply without bound.
o <namespace-length> <n> <key-length-1> <value-length-1> ... <key-length-n> <value-length-n> [<ttl-seconds>]\n<namespace><key-1><value-1>...<key-n><value-n>
Sets n keys the same way, and
<ttl-seconds> means exactly what it means for
S — omitted = no expiry, a literal 0 is a TTL
that has already elapsed and every key in the batch expires
immediately. One <ttl-seconds> for the whole
batch, not per key — every real caller of a batched set
(Django's set_many, cache-manager's mset)
already passes one TTL per call, so a per-key TTL field would
complicate the frame for no consumer that exists. Replies:
O <n> <result-1> ... <result-n>\n
No body — a set has no value to echo back, unlike m's
hits. Each <result-i> is S (stored) or
W (wrong node), same per-key independence as m.
O is never confused with the On/OnT
identity reply A gives: that only ever
appears as the very first reply on a connection, right after its
A frame, so no other request's reply is ever mistaken for
it.
Both are always namespaced, same reasoning as INCR/
k/x: neither has a pre-namespace legacy form,
so <namespace-length> is unconditionally present (0 =
the default namespace). Both are also restricted to one
namespace per frame — rendezvous hashing routes on
(namespace, key), so a frame mixing namespaces couldn't
route as a single unit anyway, and every real many-key caller already
works within one namespace at a time (a cache-manager store, a Django
key prefix, a Spring cache name).
A batch never fails as a whole. Every key's outcome
— hit/miss/wrong-node for m, stored/wrong-node for
o — is independent of every other key's. A client (or
nanocached-proxy in proxy mode) that sees a W
for some subset of a batch refreshes its routing table and re-issues
just those keys, exactly as it would for a single-key G/
S's own W — the rest of the batch's answers
stand.
Apart from that, an o (and a y) whose namespace
length times its key count is over 64 MiB is refused whole, not per key
(see the limits at the top of this page). Through
nanocached-proxy the sub-frames sent to the nodes carry the
client’s namespace and a subset of its keys, so they stay within it when
the client’s frame does.
Through nanocached-proxy, a batch spanning multiple
owners is split into one sub-frame per node and reassembled before
answering the client — the one dispatch shape that turns a single
client frame into more than one key's worth of backend traffic. A
m sub-frame targets each key's primary only, same as a
plain G; a o sub-frame reaches every owner a
sub-batch's keys have (primary or replica — the same node can be primary
for one key and a replica for another within one batch), and only the
primary's outcome decides a key's client-facing status, replica outcomes
being logged and swallowed exactly like a plain S's
replication already is.
Two keys set in one frame, then three read — the third is missing:
> o 0 2 1 1 1 1\nkalb
< O 2 S S\n
> m 0 3 1 1 1\nklz
< M 3 1 1 -\nab
Response tags (tagged mode)
Pipelined clients match responses to requests by order alone — the
frames above carry nothing to verify against. A connection whose
A carried the T flag and was answered
OnT runs in tagged mode: every
G/S/D (and
g/s/d/c/F/i/t/k/x/m/o/y)
header must end with a
client-chosen tag (a u32 in decimal), and the response echoes it as its
own last field, so a client can verify the pairing before trusting the
answer.
G <key-length> <tag>\n<key>
S <key-length> <value-length> <tag>\n<key><value>
S <key-length> <value-length> <ttl-seconds> <tag>\n<key><value>
D <key-length> <tag>\n<key>
g <namespace-length> <key-length> <tag>\n<namespace><key>
s <namespace-length> <key-length> <value-length> [<ttl-seconds>] <tag>\n<namespace><key><value>
d <namespace-length> <key-length> <tag>\n<namespace><key>
c <namespace-length> <tag>\n<namespace>
F <tag>\n
i <namespace-length> <key-length> <delta> <tag>\n<namespace><key>
t <namespace-length> <key-length> [<ttl-seconds>] [G] <tag>\n<namespace><key>
k <namespace-length> <key-length> <value-length> <cond> [<ttl-seconds>] <tag>\n<namespace><key><value>
x <namespace-length> <key-length> <cond> <tag>\n<namespace><key>
m <namespace-length> <n> <key-length-1> ... <key-length-n> <tag>\n<namespace><key-1>...<key-n>
o <namespace-length> <n> <key-length-1> <value-length-1> ... <key-length-n> <value-length-n> [<ttl-seconds>] <tag>\n<namespace><key-1><value-1>...<key-n><value-n>
y <namespace-length> <n> <key-length-1> ... <key-length-n> [<ttl-seconds>] [G] <tag>\n<namespace><key-1>...<key-n>
Responses become V <value-length> <tag>\n,
S <tag>\n, D <tag>\n,
N <tag>\n, W <tag>\n, and
C <tag>\n;
B\n stays bare (it is unsolicited, sent before
authentication); a retryable error is R <tag>\n (see
below). i's reply becomes
I <value-length> [<ttl-seconds>] <tag>\n or
T <tag>\n — the tag is always the last header field,
after the optional TTL. k's reply is S <tag>\n
or N <tag>\n; x's is D <tag>\n
or N <tag>\n — both reuse the markers above, so
tagging them needs no new shape. t's reply is
S <tag>\n or N <tag>\n, and with
G V <value-length> [<ttl-seconds>] <tag>\n
(the tag after the optional TTL, as for i). m's reply is
M <n> <result-1> ... <result-n> <tag>\n<hit values>,
o's is O <n> <result-1> ... <result-n> <tag>\n,
y's is Y <n> <result-1> ... <result-n> <tag>\n
or, with G, m's M reply —
same roster shape as the untagged form, with the tag appended as the
header's own last field. Tagged mode is per connection and
entirely opt-in — a plain A keeps every frame exactly as
documented above.
A node’s refusal is the exception: a request it will not run (one it
cannot parse, or one over the size limit) is answered with the bare
En\n, untagged, even on a tagged connection, and the node then
closes the connection. A node answers a connection’s requests one at a
time, in order, so by the time it refuses a request every earlier reply has
been written: the En is the answer to the oldest request still
waiting, and whatever was sent behind it is never answered.
nanocached-proxy reads it that way on its connections to the
nodes, which is what lets it answer a refused expiring touch, i,
k or x with R (to a client that asked
for it) instead of the fatal E of an unknown outcome. To its own
clients the proxy tags its errors, E <tag>\n, as the table
below shows.
Every request ends its header with a tag and every reply echoes it:
> A 1 T\n\0
< OnT\n
> S 4 5 7\nnameAlice
< S 7\n
> G 4 8\nname
< V 5 8\nAlice
> G 7 9\nmissing
< N 9\n
What an untagged connection cannot promise
Without tags, an SDK pairs each response with the oldest request
still waiting, by order alone. When a response of the wrong kind
arrives (an S answering a G, say), the SDK
knows the streams are misaligned, poisons the connection and redials —
but the requests queued behind the mismatched one may already
have been handed another request's reply, and a reply of the right kind
for the wrong request is indistinguishable from a correct one. That
window is inherent to matching by order; it is exactly what tagged mode
removes, which is why every SDK asks for T and only falls
back to an untagged connection against a server that predates tags
(one that rejects the extended A). On such a server the
window is the same one the pre-tags protocol always had.
Other statuses you may see
| Reply | Meaning |
|---|---|
B\n | Busy. From a node: the connection
limit is reached. From a discovery server: it is inside its startup
grace after a restart, re-learning membership — retry shortly or try
another replica. From nanocached-proxy: it has no
member list yet (it has not reached discovery) — a client that
declared the R capability below gets R
instead. |
W\n | Wrong node (clusters only): per the node's own view of membership it does not own this key, so the client's routing table is stale. SDKs respond by refreshing the node list and retrying once. |
R\n | Retryable error: this
one request failed transiently — an upstream briefly unreachable
behind nanocached-proxy, typically — and should be
retried shortly on the same connection; nothing is torn
down. Sent only to clients that declared the capability by
appending R to their A frame
(A <len> [T] R); for an upstream failure legacy clients
keep getting the fatal E-and-close. A proxy never answers R
for an i, k or x whose outcome
it cannot know — its connection to the node failed after the command
may have reached it — because a re-send could apply the command
twice: that case is the fatal E, and the client reads
the key to learn what happened. A t is idempotent, so a
proxy that cannot reach its node answers it R. A proxy
also answers R while it has no member list yet, and to a request it could not admit within
60 seconds because the in-flight byte budgets (per connection, per
source IP and proxy-wide) stayed full, and at once to a request that
would have to wait while the pool for waiting requests' frames (the
budget's size again) is full; the request was not dispatched, so any
command is safe to retry. A client that did not declare the capability
gets a bare B\n in these cases instead, and its connection
stays open. Tagged connections receive
R <tag>\n. Today the proxy is the only emitter —
nodes and discovery accept the capability token but have no
transient per-request failure to report. |
T\n | Not numeric (i
only): the key exists but its stored value isn't a decimal-ASCII
i64, or applying <delta> would
overflow one. |
E\n | Error — En\n from a
node, Ed\n from a discovery server: a failed
A, a command sent before authentication, the
request was malformed (an unknown command, a bad length or TTL, an
empty key), or it was over the request-size limit. Usually followed
by the server closing the connection. nanocached-proxy
answers a failed A En\n like a node and every
other such error with a bare E\n (E <tag>\n
in tagged mode). |
A request the server cannot parse, or a key of length 0:
> G 0\n
< En\n
= closed
> X\n
< En\n
= closed
L / Q — list nodes, list proxies
L\n
Q\n
These two are sent to a discovery server, not to a cache node
(a node answers them En\n). They take no arguments: anything
after the letter, or a lowercase l, is a malformed request,
answered Ed\n with the connection closed. When the discovery
server has an auth secret, send A first, as for any other
command; without one, Ed\n.
L lists the cache nodes currently in the cluster:
N <count> <replication>\n
<name-length> <address-length>\n<name><address>\n (<count> times)
Q lists the proxies registered with it, in the same shape
without the replication field:
N <count>\n
<name-length> <address-length>\n<name><address>\n (<count> times)
<replication> is the cluster’s replication factor, the
number of nodes that hold each key. <name> is the
node’s identity: an opaque string, chosen by the node, that the
client hashes to place the node on the ring; it is not an address and
carries no meaning of its own. <address> is
host:port and is only for opening a connection. Both lengths
are in bytes. Every entry ends with a \n that neither length
counts, so the next entry’s header starts right after it. An empty list
is N 0\n for Q. For example, one node and no
proxies, with replication factor 2:
L
N 1 2
36 15
5b5e966a-925e-44ff-ba6e-1859cce25af6127.0.0.1:19456
Q
N 0
While a discovery server is inside its startup grace after a restart,
L and Q are answered B\n and the
connection is closed, because a partial list would read as a cluster that
has fewer nodes (or no proxies) than it does; retry shortly. L
is answered the same way while most of the nodes that reported a
replication factor disagree with the discovery server’s own. Several
requests can follow one another on a connection and are answered in
order. The replies are never tagged: a connection that
asked for tags is answered OdT to its A and still
gets the plain N reply, which has no tag field.
Cluster-internal commands
Nodes, discovery servers, and SDKs additionally exchange membership
and migration frames — L (node list, above),
J/H/P (join, heartbeat,
announce), M/X/C (migrate,
cancel, complete), Y/Z/Q (proxy
announce, deregister, and list — how nanocached-proxy
registers with discovery and how SDKs in proxy mode find one),
U/u/f/V (handoff
store, handoff delete, handoff clear, and node leave — the frames a
node uses to move its entries to another node, replay a client's clear
there, and exit membership; see
Deployment), and T (node
roster: a registered node fetching the member list with every
member's membership token, self-identifying the way a heartbeat does —
the frames below need those tokens, and discovery hands them only to
proven members, never to a client that merely holds the shared secret;
see Architecture for what that trust
model implies). M, U, u and
f each carry the receiving node's membership token
(U <ns-len> <key-len> <val-len>
<token-len> [ttl] [A]; f <ns-len>
<token-len> [A], where A clears every
namespace), verified before the frame acts, since these are exactly
the frames that skip the wrong-node check. A clear reaches another
node as f rather than as the client's c/F
so the receiver knows it was forwarded: it passes it on through its
join targets as it would a client's clear, but through its own
decommission's window (behind the writes queued for the entrants) only
when it removed something, so a clear that changed nothing stops there,
where a clear indistinguishable from a client's used to circle between
a decommissioning node and the node forwarding to it until a window
closed.
U also carries an optional trailing A token:
a survivor re-replicating a key after a membership change dropped a
member (an eviction, or a leave it did not itself hand off for — see
Architecture) sends it instead of a
plain overwrite, so a re-replication racing a newer client write on the
receiver can never regress it — the receiver stores the entry only if it was absent
and no u removed it within the last two minutes (a relayed
delete leaves a tombstone, since a sender may have read the key before
that delete), and acks S\n either way, since a key already
present there is a success for the sender, not a conflict. An ordinary
decommission handoff never sets it. H takes an optional trailing
<joiner-len> header field, with the joiner's name
after the token in the body, naming a node whose handoff the sender
completed but has not yet seen in any roster; discovery then follows its
acknowledgment — the roster, or the bare A\n — with exactly
one more line, K\n (still registered, or not known to be
gone) or G\n (on record as having left, been evicted or
abandoned, and not registered since), and on G the node
abandons that handoff. A sender that put the field in must read that
line. Two of these are part of what SDKs send and are frozen by the
compatibility policy like the caching
commands: L (the node list) and Q (the proxy
list). The rest are versioned with the server and not part of the
public caching API, so the
compatibility policy does not
freeze them; the server source documents them in detail.
A complete session
$ printf 'S 4 5\nnameAliceG 4\nname' | nc 127.0.0.1 8356
S
V 5
Alice
Two pipelined requests — a set and a get — answered in order on one connection.