Regression policy: decide what counts as a performance regression

The regression policy is a TOML file in your repository, hotpath/policy.toml, that decides which changes between a pull request and its baseline count as regressions. For every profiled resource it sets how large a change must be to matter, which metrics decide, and which entries to ignore. For hard limits that hold whatever the baseline did, add performance budgets to the same file.

Policy file

Creating one

hotpath cloud init writes a starting policy to hotpath/policy.toml:

cargo install hotpath --features cloud
hotpath cloud init

The starting file turns on the pull request check and lists the two function sections. To compare SQL queries, routes, locks or any other resource, add its section from Sections: a section the file leaves out is not compared.

Which file a run uses

The upload takes the first policy file it finds:

1. the file HOTPATH_POLICY_PATH names
2. hotpath/<benchmark>-policy.toml, a benchmark’s own policy
3. hotpath/policy.toml, shared by every benchmark without its own

An upload without a policy file is refused. A pull request is judged under the policy of its own branch, so review changes to hotpath/*.toml like changes to CI configuration.

Validating

Check a file before you push with hotpath cloud validate-policy. A refused file lists its problems with their line:

{
  "path": "hotpath/policy.toml",
  "valid": false,
  "problems": [
    {
      "line": 3,
      "message": "unknown field `min_chnage`, expected one of `judged`, `fail_check`, `min_percent_change`, `metrics`, `min_calls`"
    }
  ]
}

An upload with an invalid policy is refused, and the pull request comment says why.

Sections

A section is one table per resource and metric family, such as [functions.alloc]. Each block below shows the built-in values, which a key you leave out takes; the ignore patterns are examples. A section you leave out is hidden everywhere: it is not compared, not shown on the comparison page and not in the comment.

Functions timing

Execution time of instrumented functions. Timing on a shared CI runner varies from run to run, so it is listed without being judged.

[functions.timing]
judged = false
min_percent_change = 50
metrics = ["avg", "total"]
min_calls = 1

Metrics: avg and total time per function, percentiles such as p95.

Functions allocations

Memory allocated by instrumented functions, from builds with the hotpath-alloc feature. A fixed workload allocates the same amount on every run, so even small changes are real.

[functions]
ignore = ["my_crate::test_support::*"]
min_percent_total = 2

[functions.alloc]
judged = true
min_percent_change = 20
metrics = ["avg", "total"]
min_calls = 1

Metrics: avg and total allocated per function, percentiles such as p95. ignore matches the function name, and the [functions] keys apply to both function sections.

SQL queries

Execution time of SQL queries, one row per query, route and source function.

[sql]
ignore = ["CREATE TABLE*"]
min_percent_total = 0.5

[sql.timing]
judged = false
min_percent_change = 50
metrics = ["avg", "total"]
min_calls = 0

Metrics: avg, total, percentiles. ignore matches query | route | source, so SELECT * FROM sessions* matches the query wherever it ran.

HTTP requests

Duration and errors of outbound HTTP requests, one row per endpoint, route and source function.

[http.timing]
judged = false
min_percent_change = 50
metrics = ["avg", "total"]
min_calls = 0

Metrics: avg, total, percentiles, errors.

Server timing

Response time, errors and downstream calls of each served route.

[server]
ignore = ["GET /health"]
min_percent_total = 0.5

[server.timing]
judged = true
min_percent_change = 50
metrics = ["avg", "total"]
min_calls = 0

Metrics: avg, total, percentiles, status_4xx, status_5xx, sql_per_request, http_per_request. ignore matches the route, e.g. GET /users/{id}.

Server allocations

Memory allocated while serving each route, from builds with the hotpath-alloc feature.

[server.alloc]
judged = true
min_percent_change = 2
metrics = ["avg", "total"]
min_calls = 0

Metrics: avg, total, percentiles, bytes_per_request, allocs_per_request.

Mutexes

Contention on mutexes: how long callers wait for the lock and how long it is held.

[mutexes.timing]
judged = false
min_percent_change = 50
metrics = ["wait_avg", "acquire_avg"]
min_calls = 0

Metrics: wait_avg, acquire_avg, percentiles such as wait_p95. ignore matches the lock’s label.

RwLocks

Contention on read-write locks, with reads and writes measured apart.

[rw_locks.timing]
judged = false
min_percent_change = 50
metrics = ["read_wait_avg", "write_wait_avg", "read_acquire_avg", "write_acquire_avg"]
min_calls = 0

Metrics: the four averages above, percentiles such as read_wait_p95. ignore matches the lock’s label.

Channels

Delay, backlog and throughput of channels.

[channels.flow]
judged = false
min_percent_change = 50
metrics = ["delay_avg", "max_queue_size"]
min_calls = 0

Metrics: delay_avg, percentiles such as delay_p95, max_queue_size, sent_per_sec, received_per_sec. For the two rates a drop is the regression. ignore matches the channel’s label.

I/O

Latency, throughput and errors of I/O streams, one row per stream and operation.

[io.flow]
judged = false
min_percent_change = 50
metrics = ["avg", "errors", "bytes_per_sec"]
min_calls = 0

Metrics: avg, total, percentiles, errors, bytes_per_sec (a drop is the regression). ignore matches label | operation | -, so stdout* matches every operation on it.

Options

Section keys

Every section takes the same five keys:

Percentiles are written the way hotpath names them (p95, p99.9), and a report has only the ones it was profiled with (p95 by default, set with percentiles on #[hotpath::main]).

Resource keys

The resource tables ([functions], [sql], [http], [server], [mutexes], [rw_locks], [channels], [io]) take:

Pull request feedback

Pull request comment

By default every pull request upload posts or updates a comment with its result. To comment only when something changed, set pr_comment at the top of the file, before any table:

pr_comment = "on_change"

[functions]
min_percent_total = 2

With "on_change", a comment the pull request already has is still updated, so an earlier regression never stays up after a fix. An upload the server refuses, or a report it cannot read, always gets a comment explaining why.

Pull request check

A pull request upload can also post a hotpath / <benchmark> check on the pull request’s head commit. Set pr_check at the top of the file to choose when; the file hotpath cloud init writes sets "fail":

pr_check = "fail"

[functions]
min_percent_total = 2

[functions.alloc]
judged = true
fail_check = true

pr_check decides whether the check can fail; fail_check on each block decides what fails it. The check fails only when pr_check is "fail" or "failures_only" and a finding comes from a block with fail_check = true. A broken budget rule has fail_check = true unless it says otherwise; a section has fail_check = false unless it opts in, like [functions.alloc] above.

Where a check is posted, a clean result is a success, and nothing judged (no baseline yet and no budgets) is neutral. The comment marks each finding that fails the check. A "failures_only" check cannot be made required: a clean pull request never gets one, so GitHub would wait for it. A clean re-run on the same commit also leaves an earlier failure in place; a new push clears it.

An upload the server refuses gets no check. A re-run posts a new check, and GitHub shows the latest. Put the key in hotpath/<benchmark>-policy.toml to use the check for one benchmark only. To block merging, use "fail" and make the check required, see CI integration.