Regression policy: decide what counts as a performance regression

The regression policy is a TOML file in your repository, hotpath/policy.toml, that decides which changes between a pull request and its baseline count as regressions. For every profiled resource it sets how large a change must be to matter, which metrics decide, and which entries to ignore. To set absolute limits that apply regardless of the baseline, add performance budgets to the same file.

Policy file

Creating one

hotpath cloud init writes a starting policy to hotpath/policy.toml:

cargo install hotpath --features cloud
hotpath cloud init

--benchmark NAME writes hotpath/NAME-policy.toml, a policy for that benchmark only. An existing file is never replaced unless you pass --force.

The starting file turns on the pull request check and lists the three function sections. To compare SQL queries, routes, locks or any other resource, add its section from Sections: a section the file leaves out is not compared.

Which file a run uses

A run picks its policy file in this order:

1. the file HOTPATH_POLICY_PATH names, relative to the working directory or absolute. When the variable is set, the two files below are never tried
2. hotpath/<benchmark>-policy.toml, a benchmark’s own policy
3. hotpath/policy.toml, shared by every benchmark without its own

An upload without a policy file is refused. So is one whose picked file cannot be sent: a HOTPATH_POLICY_PATH that names a missing file, or a file that is unreadable, blank, not valid UTF-8, larger than 65536 bytes or outside the repository. The lookup does not fall through to the next file in that case. A pull request is judged under the policy of its own branch, so review changes to hotpath/*.toml like changes to CI configuration.

Validating

Check a file before you push with hotpath cloud validate-policy. A refused file lists its problems with their line:

{
  "path": "hotpath/policy.toml",
  "valid": false,
  "problems": [
    {
      "line": 3,
      "message": "unknown field `min_chnage`, expected one of `judged`, `fail_check`, `min_percent_change`, `metrics`, `min_calls`"
    }
  ]
}

An upload with an invalid policy is refused, and the pull request comment says why.

Sections

A section is one table per resource and metric family, such as [functions.alloc]. Each block below shows the built-in values, which a key you leave out takes; the ignore patterns are examples. A section you leave out is hidden everywhere: it is not compared, not shown on the comparison page and not in the comment.

Functions timing

Execution time of instrumented functions. Timing on a shared CI runner varies from run to run, so it is listed without being judged.

[functions.timing]
judged = false
min_percent_change = 50
metrics = ["avg", "total"]
min_calls = 1

Metrics: avg and total time per function, percentiles such as p95.

Functions instructions

Instructions executed by instrumented functions, from benchmarks run under Callgrind with the hotpath-callgrind feature, alone or merged with a native run (the report’s functions_icount section). Instruction counts measure computational work, not elapsed time; a deterministic workload executes nearly the same instructions on every run, so a small change usually reflects a change in the code’s work.

[functions.icount]
judged = true
fail_check = false
min_percent_change = 2
metrics = ["avg", "total"]
min_calls = 1

Metrics: avg and total instructions per function. Callgrind counts have no percentiles.

Counts depend on the environment: when the two reports differ in Valgrind version, rustc or OS, or one side lacks the Valgrind version, the section is still compared and judged, but the comment and every view carry an environment mismatch alert and its regressions never fail the check, since the changes may come from the environment rather than the code. Pin all three in CI, see CI integration. Instruction counts of an ARM64 Linux report are only accepted from the patched Valgrind build, see ARM64 runners.

The built-in 2% suits deterministic single-threaded workloads. Thread interleaving moves the counts of concurrent code between runs of one commit by an amount that depends on the workload, so run the benchmark a few times, measure that spread and set min_percent_change above it; 3% is a conservative start. The drain benchmark policy shows these thresholds on a multithreaded benchmark.

In a merged report (native timing + Callgrind instruction counts in one report, the recommended CI mode), the instruction counts are judged here and the native timing comes with them:

• Timing stays listed under [functions.timing] with judged = false, as the default policy writes it: native timing on shared runners is too noisy to judge, but it shows next to the counts on the comparison page and under “Related metrics” in the pull request comment, where a finding shows its function’s row in every section the policy lists.
• Leaving [functions.timing] out stops timing being judged and also hides it from the comparison page and the comment. The report’s own page still shows every measured section.
• A timing budget on a merged report judges native timing measured on a shared runner, so give it headroom (a tail percentile such as p99.9 moves the most), or bound the work with the instruction counts instead.

Functions allocations

Memory allocated by instrumented functions, from builds with the hotpath-alloc feature. A fixed workload allocates the same amount on every run, so even small changes are real.

[functions]
ignore = ["my_crate::test_support::*"]
min_percent_total = 2

[functions.alloc]
judged = true
min_percent_change = 20
metrics = ["avg", "total"]
min_calls = 1

Metrics: avg and total allocated per function, percentiles such as p95. ignore matches the function name, and the [functions] keys apply to all three function sections.

SQL queries

Execution time of SQL queries, one row per query, route and source function.

[sql]
ignore = ["CREATE TABLE*"]
min_percent_total = 0.5

[sql.timing]
judged = false
min_percent_change = 50
metrics = ["avg", "total"]
min_calls = 0

Metrics: avg, total, percentiles. ignore matches query | route | source, so SELECT * FROM sessions* matches the query wherever it ran.

HTTP requests

Duration and errors of outbound HTTP requests, one row per endpoint, route and source function.

[http.timing]
judged = false
min_percent_change = 50
metrics = ["avg", "total"]
min_calls = 0

Metrics: avg, total, percentiles, errors.

Server timing

Response time, errors and downstream calls of each served route.

[server]
ignore = ["GET /health"]
min_percent_total = 0.5

[server.timing]
judged = true
min_percent_change = 50
metrics = ["avg", "total"]
min_calls = 0

Metrics: avg, total, percentiles, status_4xx, status_5xx, sql_per_request, http_per_request. ignore matches the route, e.g. GET /users/{id}.

Server allocations

Memory allocated while serving each route, from builds with the hotpath-alloc feature.

[server.alloc]
judged = true
min_percent_change = 2
metrics = ["avg", "total"]
min_calls = 0

Metrics: avg, total, percentiles, bytes_per_request, allocs_per_request.

Mutexes

Contention on mutexes: how long callers wait for the lock and how long it is held.

[mutexes.timing]
judged = false
min_percent_change = 50
metrics = ["wait_avg", "acquire_avg"]
min_calls = 0

Metrics: wait_avg, acquire_avg, percentiles such as wait_p95. ignore matches the lock’s label.

RwLocks

Contention on read-write locks, with reads and writes measured apart.

[rw_locks.timing]
judged = false
min_percent_change = 50
metrics = ["read_wait_avg", "write_wait_avg", "read_acquire_avg", "write_acquire_avg"]
min_calls = 0

Metrics: the four averages above, percentiles such as read_wait_p95. ignore matches the lock’s label.

Channels

Delay, backlog and throughput of channels.

[channels.flow]
judged = false
min_percent_change = 50
metrics = ["delay_avg", "max_queue_size"]
min_calls = 0

Metrics: delay_avg, percentiles such as delay_p95, max_queue_size, sent_per_sec, received_per_sec. For the two rates a drop is the regression. ignore matches the channel’s label.

I/O

Latency, throughput and errors of I/O streams, one row per stream and operation.

[io.flow]
judged = false
min_percent_change = 50
metrics = ["avg", "errors", "bytes_per_sec"]
min_calls = 0

Metrics: avg, total, percentiles, errors, bytes_per_sec (a drop is the regression). ignore matches label | operation | -, so stdout* matches every operation on it.

Threads

Memory allocated by named threads, from builds with the hotpath-alloc feature.

[threads]
names = ["db-writer", "rt-worker"]

[threads.alloc]
judged = false
min_percent_change = 20
metrics = ["total"]
min_calls = 0

Metrics: total. Threads sharing a name, such as a pool’s workers, are summed into one row; min_calls reads how many threads carry the name.

Only threads listed in names are compared, matched exactly. An unnamed thread shows up as thread_<id> with an id that changes every run, so name the ones you want compared:

std::thread::Builder::new()
    .name("db-writer".into())
    .spawn(move || run_writer(rx))?;

On Linux a thread name is cut to 15 bytes, so tokio-runtime-worker shows up as tokio-runtime-w. List the name as the report shows it.

Options

Section keys

Every section takes the same six keys:

Percentiles are written the way hotpath names them (p95, p99.9), and a report has only the ones it was profiled with (p95 by default, set with percentiles on #[hotpath::main]).

Resource keys

The resource tables ([functions], [sql], [http], [server], [mutexes], [rw_locks], [channels], [io]) take:

[threads] takes names instead of ignore, see Threads.

Ignoring an entry in one section

The resource’s ignore drops an entry from every section of the resource. To drop it from one section only, list it in that section’s ignore. For example, a function whose allocations depend on thread scheduling while its instruction counts are stable:

[functions.icount]
judged = true

[functions.alloc]
judged = true
min_percent_change = 2
ignore = ["my_crate::mailbox::recv"]

The patterns match the same names as the resource’s ignore and take up to 64 entries. Budgets still apply to an entry a section ignores, so a zero-allocation budget on it keeps working. [threads.alloc] does not take ignore: leave the thread out of names instead.

Pull request feedback

Pull request comment

By default every pull request upload posts or updates a comment with its result. To comment only when something changed, set pr_comment at the top of the file, before any table:

pr_comment = "on_change"

[functions]
min_percent_total = 2

With "on_change", a comment the pull request already has is still updated, so an earlier regression never stays up after a fix. An upload the server refuses, or a report it cannot read, always gets a comment explaining why.

Pull request check

A pull request upload can also post a hotpath / <benchmark> check on the pull request’s head commit. Set pr_check at the top of the file to choose when; the file hotpath cloud init writes sets "fail":

pr_check = "fail"

[functions]
min_percent_total = 2

[functions.alloc]
judged = true
fail_check = true

pr_check decides whether the check can fail; fail_check on each block decides what fails it. The check fails only when pr_check is "fail" or "failures_only" and a finding comes from a block with fail_check = true. A broken budget rule has fail_check = true unless it says otherwise; a section has fail_check = false unless it opts in, like [functions.alloc] above.

Where a check is posted, a clean result is a success, and nothing judged (no baseline yet and no budgets) is neutral. The comment marks each finding that fails the check. A "failures_only" check cannot be made required: a clean pull request never gets one, so GitHub would wait for it. A clean re-run on the same commit also leaves an earlier failure in place; a new push clears it.

An upload the server refuses gets no check. A re-run posts a new check, and GitHub shows the latest. Put the key in hotpath/<benchmark>-policy.toml to use the check for one benchmark only. To block merging, use "fail" and make the check required, see CI integration.