TLS
Three flags, the same on every binary, decide which segments of a deployment are encrypted. This page says what each one covers, why a node’s certificate must name its IP address, how to mint and distribute self-signed certificates, and how to rotate them.
What each flag encrypts
Every process accepts connections, and most of them also dial out.
--tls-cert/--tls-key (a PEM chain and its
key) turn on TLS for everything the process accepts, with no
plaintext fallback: a plaintext client gets a TLS alert and the
connection is closed. --tls-ca (a PEM file of one or more
CA certificates) turns on TLS for everything the process dials,
trusting only those certificates — not the system trust store,
since a private cluster’s certificates are not publicly issued.
The two are independent. A process that dials anything with only one
of them set warns at start-up that the other direction is plaintext
(a proxy or a discovery server always does; a node does when it has a
--discovery to register with — a standalone node with
only a certificate dials nothing and stays quiet; with only
--tls-ca, a node, a discovery server or a proxy warns that
what it accepts is plaintext, whether or not it dials anything). The SDKs take the
same two pieces as options: tls and, for anything that is
not publicly issued, ca. There are no client certificates:
a peer is authenticated by the shared secret, and TLS adds
confidentiality and the server’s identity on top.
| Segment | Server side | Dialling side | Verified against |
|---|---|---|---|
| Application → node (data) | node --tls-cert/--tls-key | SDK tls + ca | the node’s IP address, as configured (single node) or as discovery handed it out (cluster) |
| Application → discovery (node list) | discovery --tls-cert/--tls-key | SDK tls + ca | the address in the SDK’s configuration: a hostname or an IP |
| Node → discovery (registration, heartbeats) | discovery --tls-cert/--tls-key | node --tls-ca | the node’s --discovery address |
| Discovery → node (join and eviction orders) | node --tls-cert/--tls-key | discovery --tls-ca | the IP the node registered from |
| Node → node (handoffs) | node --tls-cert/--tls-key | node --tls-ca | the peer’s IP, from the roster |
| Application → proxy | proxy --tls-cert/--tls-key | SDK tls + ca | the proxy’s address: the IP it registered from, or a load balancer’s name |
| Proxy → node, proxy → discovery | as above | proxy --tls-ca | as above |
By shape: a single node has one segment, so the node
gets --tls-cert/--tls-key and nothing else. A
cluster has all of the first five, so discovery and
every node get all three flags. A proxy tier adds the
last two, and the proxy gets all three as well. The operations endpoint
(--metrics-port) is plain HTTP in every case; the
usual advice of keeping it
inside the private network is the whole of its protection (and a collector on the
same host, as in metrics
collection, never leaves it).
Names in certificates
Nodes are addressed by IP. Discovery records the source IP of a
node’s registration connection, hands that address to the SDKs
in the node list and to the other nodes in rosters, and dials it
itself for join and eviction orders. Every one of those connections
verifies the node’s certificate against that IP, so a
node’s certificate must carry its IP address in
subjectAltName. A certificate with only a DNS name fails
with certificate not valid for name "10.0.0.10" on
discovery and an IP address mismatch on the SDKs. The
same holds for a proxy that applications reach through the proxy list
discovery serves: that list is registration IPs too.
Discovery itself is different: nodes dial the address given as
--discovery and applications dial the address in their
configuration, so a discovery certificate names whatever that is
— a DNS name if you front discovery with one, an IP otherwise.
A proxy behind a load balancer, reached by the balancer’s name,
likewise names that name. Where discovery is reached by a DNS name,
its certificate carries that name beside its IP; on Kubernetes the
name is a Service, and the single label nodes dial (for example
disc) goes in the SAN as itself.
The IP has to be known when the certificate is minted, so give a node a fixed private address. Where an address is only assigned when the process starts — a container task — the certificate has to be minted then, from a CA whose certificate the applications already trust, by an init container; the distribution section below covers it.
Self-signed certificates
A self-signed certificate is a leaf that is its own trust anchor:
the same file is the node’s --tls-cert and every
dialler’s ca. It needs three extensions that
OpenSSL’s defaults do not give it — CA:FALSE
(a self-signed certificate is otherwise marked as a CA, which rustls
refuses to accept as a server’s leaf), a
subjectAltName with the IP, and serverAuth
— and the -subj is then only a label:
NODE_PRIVATE_IP=10.0.0.10
openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
-keyout node-key.pem -out node-cert.pem \
-subj "/CN=nanocached-node" \
-addext "subjectAltName=IP:$NODE_PRIVATE_IP" \
-addext "basicConstraints=critical,CA:FALSE" \
-addext "keyUsage=critical,digitalSignature,keyEncipherment" \
-addext "extendedKeyUsage=serverAuth"
The result is about 1.2 KB of certificate and 1.7 KB of
key, which matters for the distribution step. Because the system trust
store is never consulted (not by the servers, and not by an SDK once
ca is set), a self-signed deployment always carries the
certificate to the application tier explicitly; an SDK configured with
tls alone verifies against the platform store and
rejects the node with CERTIFICATE_VERIFY_FAILED, which
is the right outcome for a certificate nobody vouched for.
A cluster needs one certificate per host, each with its own IP, and
a trust anchor that does not change every time a host does. That is a
private CA: a self-signed CA certificate whose key stays with the
operator, and a leaf per host signed by it. The flags are the same;
--tls-ca and ca take the CA certificate
instead of the leaf, and a leaf can be reissued without the application
tier learning a new file. Minting the CA and one leaf:
# once: the CA. Its key never leaves the operator's machine.
openssl req -x509 -newkey rsa:2048 -nodes -days 3650 -keyout ca-key.pem -out ca.pem \
-subj "/CN=nanocached CA" \
-addext "basicConstraints=critical,CA:TRUE,pathlen:0" \
-addext "keyUsage=critical,keyCertSign,cRLSign"
# per host: a leaf for its address.
IP=10.0.0.11
openssl req -new -newkey rsa:2048 -nodes -keyout host-$IP-key.pem -out host-$IP.csr \
-subj "/CN=nanocached-$IP"
printf 'subjectAltName=IP:%s\nbasicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature,keyEncipherment\nextendedKeyUsage=serverAuth\nsubjectKeyIdentifier=hash\nauthorityKeyIdentifier=keyid\n' \
$IP > host-$IP.ext
openssl x509 -req -in host-$IP.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -days 365 \
-extfile host-$IP.ext -out host-$IP-cert.pem
The last two lines of the extension file are for the Python SDK:
Python 3.13 and later verify certificates strictly by default and refuse
a leaf with no authority key identifier
(CERTIFICATE_VERIFY_FAILED … Missing Authority Key
Identifier), and openssl x509 -req does not add one on
its own. The other five SDKs do not need it, and neither does a
self-signed leaf (the section above).
A host serves its leaf alone (no chain: the dialler trusts the CA directly), and a leaf from this CA and a self-signed leaf from the section above do not trust each other, so a deployment picks one model. A managed CA fits the same slot — it signs the leaves, its certificate is the trust anchor — and was not run for this page.
Distribution
Two files per process, with different sensitivity: the certificate is public, the key is a secret. Keep both in whatever secret store the platform offers — a parameter or secret store a host can read with its own identity, or a Kubernetes Secret — under one prefix, with the key marked as a secret. The sizes above fit comfortably in any of them. A third entry carries the certificates the application tier should trust, which is the same file today and becomes a bundle during rotation. With a private CA the layout gains a level: the CA certificate is the trust entry, and each host’s pair sits under its address, which the host reads from its own network configuration to pick its own.
The node fetches its two files before every start, so a replacement
host and a restart after rotation take the same path. Applications
fetch the trust entry the same way, and nothing on the application
side ever needs the key. Where the fetched files land matters once a
host is imaged: write them to /run, a tmpfs that is never
in a snapshot.
Where nothing is pre-minted — a container whose address is only assigned at start — the store carries the CA instead of leaves: the trust bundle, and the signing certificate and key an init container uses to mint the task’s own leaf at start. The signing certificate is an entry of its own rather than “the first certificate in the trust bundle” because during a CA rotation the trust bundle holds two certificates and the signing key matches only one of them. That puts the CA’s private key within reach of every init container, where with pre-minted leaves it never left the operator’s machine. It is the price of minting at start, and the init limits it: the key exists only inside a container that deletes its copy from the shared volume and exits before the node starts.
On Kubernetes the same three files are two Secrets — the trust bundle in one, the signing pair in the other — mounted read-only into the init container, and only the trust Secret is mounted by the applications. A Secret is readable by anything that can mount Secrets in its namespace, so the signing Secret stays in the cluster’s namespace and applications in another namespace get a copy of the trust Secret alone:
kubectl -n nanocached create secret generic nanocached-tls-ca --from-file=ca.pem=ca.pem
kubectl -n nanocached create secret generic nanocached-tls-signing --from-file=signing-cert.pem=ca.pem --from-file=signing-key.pem=ca-key.pem
Rotation
Certificate and key are read once, at start, so a rotation is a restart: replace the files (overwrite them where the fetch step reads them) and restart the process. With a self-signed leaf the trust anchor rotates with it, which forces an order on the steps: the application tier has to trust old and new before the node switches, or every connection fails the moment it does.
- Mint the new pair with the same IP. Publish old + new,
concatenated, as the trust file, and let the applications pick
it up (an application reads its
cafile when it constructs a client, so re-create the clients or restart the application). - Overwrite the node’s certificate and key with the new pair and restart the node; a standalone node comes back empty either way.
- Once no application still trusts only the old certificate, publish the new certificate alone as the trust file.
With the node still on the old certificate, both the bundle and the
old file alone connect; after the restart, the bundle connects and the
old file alone fails with CERTIFICATE_VERIFY_FAILED.
With a private CA there are two rotations. A leaf
rotates alone: overwrite that host’s two files and restart
its processes; nothing else changes, because every dialler trusts the
CA, not the leaf. The CA rotates in the three steps
above, applied to every process at once: publish old + new as the
trust file and restart every process (each reads
--tls-ca at start), re-fetch on the applications, then
reissue each host’s leaf under the new CA and restart it, host
by host; publishing the new CA alone, and the restart that picks it
up, can wait for the next routine restart.
Where an init container mints a fresh leaf at every start, a restart
is a rolling deployment and leaf rotation is that deployment and
nothing more; a leaf’s lifetime only matters for a task that
lives that long. CA rotation is the same three steps with the stored
entries: trust := both CAs and a rolling restart of discovery and
node; signing pair := the new CA and another rolling restart; trust :=
the new CA alone whenever the next restart happens. Each rolling
replacement of the node service is one in which every old task drains
and every new one joins, so the data survives it. On Kubernetes the
restart is kubectl rollout restart, and a changed Secret
is applied with kubectl create secret --dry-run=client -o yaml |
kubectl apply -f -; a pod that mounts the trust Secret sees the
new file within about a minute, but an application reads its
ca file when it constructs a client, so count on the
restart, not on the propagation.