TLS

Three flags, the same on every binary, decide which segments of a deployment are encrypted. This page says what each one covers, why a node’s certificate must name its IP address, how to mint and distribute self-signed certificates, and how to rotate them.

What each flag encrypts

Every process accepts connections, and most of them also dial out. --tls-cert/--tls-key (a PEM chain and its key) turn on TLS for everything the process accepts, with no plaintext fallback: a plaintext client gets a TLS alert and the connection is closed. --tls-ca (a PEM file of one or more CA certificates) turns on TLS for everything the process dials, trusting only those certificates — not the system trust store, since a private cluster’s certificates are not publicly issued. The two are independent. A process that dials anything with only one of them set warns at start-up that the other direction is plaintext (a proxy or a discovery server always does; a node does when it has a --discovery to register with — a standalone node with only a certificate dials nothing and stays quiet; with only --tls-ca, a node, a discovery server or a proxy warns that what it accepts is plaintext, whether or not it dials anything). The SDKs take the same two pieces as options: tls and, for anything that is not publicly issued, ca. There are no client certificates: a peer is authenticated by the shared secret, and TLS adds confidentiality and the server’s identity on top.

SegmentServer sideDialling sideVerified against
Application → node (data)node --tls-cert/--tls-keySDK tls + cathe node’s IP address, as configured (single node) or as discovery handed it out (cluster)
Application → discovery (node list)discovery --tls-cert/--tls-keySDK tls + cathe address in the SDK’s configuration: a hostname or an IP
Node → discovery (registration, heartbeats)discovery --tls-cert/--tls-keynode --tls-cathe node’s --discovery address
Discovery → node (join and eviction orders)node --tls-cert/--tls-keydiscovery --tls-cathe IP the node registered from
Node → node (handoffs)node --tls-cert/--tls-keynode --tls-cathe peer’s IP, from the roster
Application → proxyproxy --tls-cert/--tls-keySDK tls + cathe proxy’s address: the IP it registered from, or a load balancer’s name
Proxy → node, proxy → discoveryas aboveproxy --tls-caas above

By shape: a single node has one segment, so the node gets --tls-cert/--tls-key and nothing else. A cluster has all of the first five, so discovery and every node get all three flags. A proxy tier adds the last two, and the proxy gets all three as well. The operations endpoint (--metrics-port) is plain HTTP in every case; the usual advice of keeping it inside the private network is the whole of its protection (and a collector on the same host, as in metrics collection, never leaves it).

Names in certificates

Nodes are addressed by IP. Discovery records the source IP of a node’s registration connection, hands that address to the SDKs in the node list and to the other nodes in rosters, and dials it itself for join and eviction orders. Every one of those connections verifies the node’s certificate against that IP, so a node’s certificate must carry its IP address in subjectAltName. A certificate with only a DNS name fails with certificate not valid for name "10.0.0.10" on discovery and an IP address mismatch on the SDKs. The same holds for a proxy that applications reach through the proxy list discovery serves: that list is registration IPs too.

Discovery itself is different: nodes dial the address given as --discovery and applications dial the address in their configuration, so a discovery certificate names whatever that is — a DNS name if you front discovery with one, an IP otherwise. A proxy behind a load balancer, reached by the balancer’s name, likewise names that name. Where discovery is reached by a DNS name, its certificate carries that name beside its IP; on Kubernetes the name is a Service, and the single label nodes dial (for example disc) goes in the SAN as itself.

The IP has to be known when the certificate is minted, so give a node a fixed private address. Where an address is only assigned when the process starts — a container task — the certificate has to be minted then, from a CA whose certificate the applications already trust, by an init container; the distribution section below covers it.

Self-signed certificates

A self-signed certificate is a leaf that is its own trust anchor: the same file is the node’s --tls-cert and every dialler’s ca. It needs three extensions that OpenSSL’s defaults do not give it — CA:FALSE (a self-signed certificate is otherwise marked as a CA, which rustls refuses to accept as a server’s leaf), a subjectAltName with the IP, and serverAuth — and the -subj is then only a label:

NODE_PRIVATE_IP=10.0.0.10
openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
  -keyout node-key.pem -out node-cert.pem \
  -subj "/CN=nanocached-node" \
  -addext "subjectAltName=IP:$NODE_PRIVATE_IP" \
  -addext "basicConstraints=critical,CA:FALSE" \
  -addext "keyUsage=critical,digitalSignature,keyEncipherment" \
  -addext "extendedKeyUsage=serverAuth"

The result is about 1.2 KB of certificate and 1.7 KB of key, which matters for the distribution step. Because the system trust store is never consulted (not by the servers, and not by an SDK once ca is set), a self-signed deployment always carries the certificate to the application tier explicitly; an SDK configured with tls alone verifies against the platform store and rejects the node with CERTIFICATE_VERIFY_FAILED, which is the right outcome for a certificate nobody vouched for.

A cluster needs one certificate per host, each with its own IP, and a trust anchor that does not change every time a host does. That is a private CA: a self-signed CA certificate whose key stays with the operator, and a leaf per host signed by it. The flags are the same; --tls-ca and ca take the CA certificate instead of the leaf, and a leaf can be reissued without the application tier learning a new file. Minting the CA and one leaf:

# once: the CA. Its key never leaves the operator's machine.
openssl req -x509 -newkey rsa:2048 -nodes -days 3650 -keyout ca-key.pem -out ca.pem \
  -subj "/CN=nanocached CA" \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:0" \
  -addext "keyUsage=critical,keyCertSign,cRLSign"
# per host: a leaf for its address.
IP=10.0.0.11
openssl req -new -newkey rsa:2048 -nodes -keyout host-$IP-key.pem -out host-$IP.csr \
  -subj "/CN=nanocached-$IP"
printf 'subjectAltName=IP:%s\nbasicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature,keyEncipherment\nextendedKeyUsage=serverAuth\nsubjectKeyIdentifier=hash\nauthorityKeyIdentifier=keyid\n' \
  $IP > host-$IP.ext
openssl x509 -req -in host-$IP.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -days 365 \
  -extfile host-$IP.ext -out host-$IP-cert.pem

The last two lines of the extension file are for the Python SDK: Python 3.13 and later verify certificates strictly by default and refuse a leaf with no authority key identifier (CERTIFICATE_VERIFY_FAILED … Missing Authority Key Identifier), and openssl x509 -req does not add one on its own. The other five SDKs do not need it, and neither does a self-signed leaf (the section above).

A host serves its leaf alone (no chain: the dialler trusts the CA directly), and a leaf from this CA and a self-signed leaf from the section above do not trust each other, so a deployment picks one model. A managed CA fits the same slot — it signs the leaves, its certificate is the trust anchor — and was not run for this page.

Distribution

Two files per process, with different sensitivity: the certificate is public, the key is a secret. Keep both in whatever secret store the platform offers — a parameter or secret store a host can read with its own identity, or a Kubernetes Secret — under one prefix, with the key marked as a secret. The sizes above fit comfortably in any of them. A third entry carries the certificates the application tier should trust, which is the same file today and becomes a bundle during rotation. With a private CA the layout gains a level: the CA certificate is the trust entry, and each host’s pair sits under its address, which the host reads from its own network configuration to pick its own.

The node fetches its two files before every start, so a replacement host and a restart after rotation take the same path. Applications fetch the trust entry the same way, and nothing on the application side ever needs the key. Where the fetched files land matters once a host is imaged: write them to /run, a tmpfs that is never in a snapshot.

Where nothing is pre-minted — a container whose address is only assigned at start — the store carries the CA instead of leaves: the trust bundle, and the signing certificate and key an init container uses to mint the task’s own leaf at start. The signing certificate is an entry of its own rather than “the first certificate in the trust bundle” because during a CA rotation the trust bundle holds two certificates and the signing key matches only one of them. That puts the CA’s private key within reach of every init container, where with pre-minted leaves it never left the operator’s machine. It is the price of minting at start, and the init limits it: the key exists only inside a container that deletes its copy from the shared volume and exits before the node starts.

On Kubernetes the same three files are two Secrets — the trust bundle in one, the signing pair in the other — mounted read-only into the init container, and only the trust Secret is mounted by the applications. A Secret is readable by anything that can mount Secrets in its namespace, so the signing Secret stays in the cluster’s namespace and applications in another namespace get a copy of the trust Secret alone:

kubectl -n nanocached create secret generic nanocached-tls-ca      --from-file=ca.pem=ca.pem
kubectl -n nanocached create secret generic nanocached-tls-signing --from-file=signing-cert.pem=ca.pem --from-file=signing-key.pem=ca-key.pem

Rotation

Certificate and key are read once, at start, so a rotation is a restart: replace the files (overwrite them where the fetch step reads them) and restart the process. With a self-signed leaf the trust anchor rotates with it, which forces an order on the steps: the application tier has to trust old and new before the node switches, or every connection fails the moment it does.

  1. Mint the new pair with the same IP. Publish old + new, concatenated, as the trust file, and let the applications pick it up (an application reads its ca file when it constructs a client, so re-create the clients or restart the application).
  2. Overwrite the node’s certificate and key with the new pair and restart the node; a standalone node comes back empty either way.
  3. Once no application still trusts only the old certificate, publish the new certificate alone as the trust file.

With the node still on the old certificate, both the bundle and the old file alone connect; after the restart, the bundle connects and the old file alone fails with CERTIFICATE_VERIFY_FAILED.

With a private CA there are two rotations. A leaf rotates alone: overwrite that host’s two files and restart its processes; nothing else changes, because every dialler trusts the CA, not the leaf. The CA rotates in the three steps above, applied to every process at once: publish old + new as the trust file and restart every process (each reads --tls-ca at start), re-fetch on the applications, then reissue each host’s leaf under the new CA and restart it, host by host; publishing the new CA alone, and the restart that picks it up, can wait for the next routine restart.

Where an init container mints a fresh leaf at every start, a restart is a rolling deployment and leaf rotation is that deployment and nothing more; a leaf’s lifetime only matters for a task that lives that long. CA rotation is the same three steps with the stored entries: trust := both CAs and a rolling restart of discovery and node; signing pair := the new CA and another rolling restart; trust := the new CA alone whenever the next restart happens. Each rolling replacement of the node service is one in which every old task drains and every new one joins, so the data survives it. On Kubernetes the restart is kubectl rollout restart, and a changed Secret is applied with kubectl create secret --dry-run=client -o yaml | kubectl apply -f -; a pod that mounts the trust Secret sees the new file within about a minute, but an application reads its ca file when it constructs a client, so count on the restart, not on the propagation.