Metrics collection

Every binary serves Prometheus-format metrics on its operations port. Which collector picks them up depends on the monitoring stack you already run, so this page is separate from the operations page: it says what there is to collect and how the two kinds of metric behave once collected.

What there is to collect

GET /metrics on --metrics-port answers in Prometheus text format (the endpoint, with its two probe paths, is described under what every deployment needs). A node exports eighteen series plus two per namespace (three where a namespace has a budget), a discovery server six; the ones to scale on are listed under metrics to scale on, and curl -s http://127.0.0.1:9356/metrics | grep -v '^#' prints the full set with its current values. Three things about them decide how a collector should be configured.

Gauges and counters. The _bytes, _entries and connections series are gauges: the value at the moment of the scrape. Everything ending in _total is a counter: a running total since the process started, which drops back to zero when it restarts. A Prometheus server stores counters as they are (an instant query returns each process’s exact cumulative value) and rate() or increase() does the rest, extrapolating across the window you give it (an increase() over five minutes can answer 2,369 on a process that took exactly 2,100 writes inside the window, which is the function working as designed, not a doubled write). A backend that stores statistics per period leaves it to the collector what number it hands over for a counter — the cumulative value, or the increase since the previous scrape. With the increase, Sum over a period is the number of operations in that period, and after a restart the counter simply starts again from zero (the period’s Sum is the operations since the restart, never a negative number); a gauge’s Sum is meaningless there (four scrapes a minute of a 250-entry cache sum to 1,000), so gauges are read with Maximum or Average. Whichever backend you use, check which of the two it stores for a counter before building a dashboard on it.

Labels. The per-namespace series (nanocached_node_namespace_entries, …_namespace_used_bytes, and …_namespace_budget_bytes where a budget is set) carry a namespace label; nothing else does. They cover the 256 largest namespaces by bytes; nanocached_node_namespaces counts every live namespace and …_namespaces_unlisted the ones beyond those 256, whose entries and bytes are still in the node-wide gauges. A name longer than 256 bytes appears cut to 128 bytes with its length and a hash of the whole name appended, so a scrape stays a few hundred bytes per listed namespace whatever a client names its namespaces. The default namespace is the empty string. Prometheus treats an empty label value as no label at all, so there the default namespace is the series without a namespace label, next to {namespace="orders"}; a collector that turns labels into dimensions may leave out a series whose dimension value is empty, and there the default namespace is read from the node-wide nanocached_node_entries and …_memory_used_bytes instead, with the labelled series covering the named namespaces. Each named namespace is one more series per metric, which is what a per-metric price is charged on.

Plain HTTP, no authentication. The operations endpoint is unauthenticated and never TLS, so bind it where only the private network reaches it. The least exposure is a collector on the same host or in the same task scraping 127.0.0.1; a central scraper is the alternative, reaching the port through a firewall rule that admits the operations port from inside the private network and nothing wider.