Metrics collection
Every binary serves Prometheus-format metrics on its operations port. Which collector picks them up depends on the monitoring stack you already run, so this page is separate from the operations page: it says what there is to collect and how the two kinds of metric behave once collected.
What there is to collect
GET /metrics on --metrics-port answers in
Prometheus text format (the endpoint, with its two probe paths, is
described under what every
deployment needs). A node exports eighteen series plus two per
namespace (three where a namespace has a budget), a discovery server six; the ones to scale on are listed under
metrics to scale on,
and curl -s http://127.0.0.1:9356/metrics | grep -v '^#'
prints the full set with its current values. Three things about them
decide how a collector should be configured.
Gauges and counters. The
_bytes, _entries and
connections series are gauges: the value at the moment of
the scrape. Everything ending in _total is a counter: a
running total since the process started, which drops back to zero when
it restarts. A Prometheus server stores counters as they are (an
instant query returns each process’s exact cumulative value) and
rate() or increase() does the rest,
extrapolating across the window you give it (an
increase() over five minutes can answer 2,369 on a process
that took exactly 2,100 writes inside the window, which is the
function working as designed, not a doubled write). A backend that
stores statistics per period leaves it to the collector what number it
hands over for a counter — the cumulative value, or the increase
since the previous scrape. With the increase, Sum over a
period is the number of
operations in that period, and after a restart the counter simply
starts again from zero (the period’s Sum is the
operations since the restart, never a negative number); a
gauge’s Sum is meaningless there
(four scrapes a minute of a 250-entry cache sum to 1,000), so gauges
are read with Maximum or Average. Whichever
backend you use, check which of the two it stores for a counter before
building a dashboard on it.
Labels. The per-namespace series
(nanocached_node_namespace_entries,
…_namespace_used_bytes, and …_namespace_budget_bytes
where a budget is set) carry a namespace label; nothing
else does. They cover the 256 largest namespaces by bytes;
nanocached_node_namespaces counts every live namespace and
…_namespaces_unlisted the ones beyond those 256, whose
entries and bytes are still in the node-wide gauges. A name longer than
256 bytes appears cut to 128 bytes with its length and a hash of the
whole name appended, so a scrape stays a few hundred bytes per listed
namespace whatever a client names its namespaces. The default namespace is the empty string. Prometheus
treats an empty label value as no label at all, so there the default
namespace is the series without a namespace
label, next to {namespace="orders"}; a collector that
turns labels into dimensions may leave out a series whose dimension
value is empty, and there the default namespace is read from the
node-wide nanocached_node_entries and
…_memory_used_bytes instead, with the labelled series
covering the named namespaces. Each
named namespace is one more series per metric, which is what a
per-metric price is charged on.
Plain HTTP, no authentication. The operations
endpoint is unauthenticated and never TLS, so bind it where only the
private network reaches it. The least exposure is a collector on the
same host or in the same task scraping 127.0.0.1; a
central scraper is the alternative, reaching the port through a
firewall rule that admits the operations port from inside the private
network and nothing wider.