Prometheus and Grafana integration

The hotpath-prometheus feature exposes every profiling subsystem as Prometheus metrics on a dedicated GET /metrics endpoint. Point a Prometheus scraper at it and build Grafana dashboards on top of any of the performance signals measured by the library.

Configure prometheus metrics endpoint

Add hotpath-prometheus feature forwarding:

[dependencies]
hotpath = "0.25"

[features]
hotpath = ["hotpath/hotpath"]
hotpath-prometheus = ["hotpath/hotpath-prometheus"]

cargo run --features='hotpath,hotpath-prometheus'

The exporter starts automatically with the profiler on 127.0.0.1:6772 (customizable with HOTPATH_PROMETHEUS_PORT, HOTPATH_PROMETHEUS_HOST). Verify it with:

curl http://127.0.0.1:6772/metrics

Prometheus scrape config

Minimal prometheus.yml:

global:
  scrape_interval: 5s

scrape_configs:
  - job_name: hotpath
    scrape_native_histograms: true
    static_configs:
      - targets: ["127.0.0.1:6772"]

With scrape_native_histograms: true Prometheus negotiates the protobuf format and ingests high-resolution native histograms. Without it, the exporter serves the text format with coarse classic buckets.

The two representations are queried differently. A native histogram is a single series under the bare metric name, e.g. hotpath_function_duration_seconds. A classic histogram is a set of float series with the _bucket, _sum and _count suffixes. When Prometheus ingests the native part, it drops the classic part, so the suffixed series do not exist. Add always_scrape_classic_histograms: true to the scrape config to store both.

Authentication

Set HOTPATH_PROMETHEUS_AUTH_TOKEN and every request must carry it in the Authorization header, either bare or Bearer-prefixed. This matches Prometheus’ authorization scrape config:

global:
  scrape_interval: 5s

scrape_configs:
  - job_name: hotpath
    scrape_native_histograms: true
    authorization:
      credentials: your-secret-token
    static_configs:
      - targets: ["127.0.0.1:6772"]

The token travels in plaintext: it guards against other local processes and accidental exposure, not a substitute for TLS.

Grafana

Add Prometheus as a data source and query the metrics with PromQL. The histogram queries below assume native histograms (the scrape_native_histograms: true config above); the classic equivalents follow.

# p99 function duration
histogram_quantile(0.99, sum by (function) (rate(hotpath_function_duration_seconds[1m])))

# Average function duration
histogram_sum(rate(hotpath_function_duration_seconds[1m])) / histogram_count(rate(hotpath_function_duration_seconds[1m]))

# Calls per second per function
rate(hotpath_function_calls_total[1m])

# Allocation rate per function
rate(hotpath_function_alloc_bytes_total[1m])

# SQL queries per served request, per route
rate(hotpath_server_sql_calls_total[5m]) / rate(hotpath_server_scoped_requests_total[5m])

# I/O throughput that stays correct under time sampling
rate(hotpath_io_sampled_bytes_total[1m]) / histogram_sum(rate(hotpath_io_op_seconds[1m]))

# Average future poll duration
rate(hotpath_future_poll_seconds_total[1m]) / rate(hotpath_future_sampled_polls_total[1m])

Durations are exported in seconds. Counters are cumulative since profiling started, so use rate() / increase() in queries.

Classic histogram queries

Without scrape_native_histograms (or with always_scrape_classic_histograms: true) histograms are stored as _bucket, _sum and _count series. Quantiles need the le label and the _bucket suffix, and sums and counts are plain series instead of histogram_sum() / histogram_count():

# p99 function duration
histogram_quantile(0.99, sum by (function, le) (rate(hotpath_function_duration_seconds_bucket[1m])))

# Average function duration
rate(hotpath_function_duration_seconds_sum[1m]) / rate(hotpath_function_duration_seconds_count[1m])

# I/O throughput that stays correct under time sampling
rate(hotpath_io_sampled_bytes_total[1m]) / rate(hotpath_io_op_seconds_sum[1m])

Classic buckets are coarse, log-spaced 1-3 steps, so quantiles from them are rough estimates. Prefer native histograms when accuracy matters.

Time sampling

With time sampling enabled, *_total call counters still count every call, while duration histograms only contain the sampled ones. Their count (histogram_count(), or the classic _count series) is the number of timed calls, so averages derived from sum / count stay correct.

Available metrics

Families with no data are omitted from the scrape, e.g. lock metrics appear only when a mutex! or rw_lock! wrapper has been used.

Process

hotpath_build_info describes the binary that produced the scrape:

hotpath_build_info{hotpath_version="0.24.0",rustc_version="1.97.0",profile="release",os="linux-x86_64"} 1

• hotpath_version / rustc_version - crate and compiler versions the binary was built with.
• profile - the Cargo profile the binary was built with: debug, release, or a custom profile’s name such as profiling.
• os - OS-ARCH pair of the running process, e.g. linux-x86_64 or macos-aarch64.

Functions

Requires #[hotpath::measure] / #[hotpath::measure_all] instrumentation.

With the hotpath-alloc feature:

The two per-call histograms are omitted for async entries whose measurements carry no per-call totals.

SQL queries

Requires a SQL tracing integration. Series are keyed by normalized query text; queries longer than HOTPATH_MAX_LOG_LEN are truncated with a hash suffix so distinct queries never collapse into one series.

source is the innermost instrumented caller, route the axum route handling the request (see route scoping). Aggregate with sum by (query) for the per-query view.

HTTP client requests

Requires hotpath::http!(client).

HTTP server (axum)

Requires hotpath::axum!(router).

Divide the SQL and HTTP call counters by hotpath_server_scoped_requests_total for per-request rates.

Locks

Requires hotpath::mutex! / hotpath::rw_lock! wrappers. Call-site labels: source is the file:line:column of the wrapper macro, label the user-provided label (empty when unset), iter the instantiation index for call sites that create several instances.

Channels

Requires hotpath::channel! wrappers. type is the channel kind (bounded[N], unbounded, oneshot), payload the message type name.

Streams

Requires hotpath::stream! wrappers.

Subtract the closed counter from the created one for the number of instances currently alive at a call site.

Futures

Requires #[hotpath::measure(future = true)] on the async function. source is the function path.

I/O

Requires hotpath::io! wrappers. type is the wrapped type name, op one of read, write, flush, shutdown. Op kinds a wrapper never touched are not exported.

Threads

Requires the threads feature. Per-thread series cover live threads only; a thread’s series goes stale after it exits, while its allocations stay in the process-level totals.

Tokio runtime

Requires the tokio feature and hotpath::tokio_runtime!(). Some series are only available when Tokio’s unstable (RUSTFLAGS="--cfg tokio_unstable") metrics are enabled.

Gauges

Requires hotpath::gauge! entries.

Configuration

Example:

HOTPATH_PROMETHEUS_PORT=9100 HOTPATH_PROMETHEUS_AUTH_TOKEN=secret123 \
cargo run --features='hotpath,hotpath-prometheus'