# Read hotpath-cloud verdicts and fix performance regressions

Instructions for an AI coding agent. `hotpath cloud` is the command line client of hotpath.rs. CI uploads a hotpath benchmark report on every push to the default branch and every pull request. hotpath.rs compares each pull request report with its baseline (the report of the base branch), judges it under the repository's policy file and posts a comment on the pull request. The CLI returns the same judgement as JSON.

## Documentation

The hotpath docs are the reference for every command, flag, JSON field and policy key; this page only holds the workflow. Every page of https://hotpath.rs is served as markdown when requested with an `Accept: text/markdown` header:

```bash
curl -fsSL -H 'Accept: text/markdown' https://hotpath.rs/agents_cli
```

Fetch https://hotpath.rs/agents_cli before the first `hotpath cloud` command, and the other pages when the task reaches them, instead of relying on memory. Keep `-fsSL` and the header. If a fetch fails, stop and tell the user which URL could not be retrieved; do not continue from memory. A `#fragment` is not sent to the server: fetch the whole page and find the heading.

The docs describe the latest hotpath release, `v0.31.2` (the crate version is the tag without its `v`). Check the installed version in `Cargo.lock`:

- If it supports what the task needs, keep it; do not upgrade. A hotpath cargo feature that is not enabled on the dependency yet (`tokio`, an integration) only needs enabling, not an upgrade.
- If it lacks functionality the task needs (compare the docs with https://github.com/hotpath-rs/hotpath/blob/main/CHANGELOG.md), upgrade to the latest release without asking. Save the `Cargo.toml` files and `Cargo.lock` first. Raise the `hotpath` requirement (each crate's `Cargo.toml`, or `[workspace.dependencies]`) where it does not allow `"0.31"`, keeping its features, and run `cargo update -p hotpath`; change no other dependency. Build with and without the project's hotpath features (`cargo check`, and `cargo check --features` with the feature names the project defines) and fix breakage using the changelog.
- Tell the user about every upgrade: the old and new version, why it was needed and any code you changed for it.
- If the upgrade cannot be made to build, restore the saved `Cargo.toml` files and `Cargo.lock` (and revert only the code you changed for it), tell the user why it failed and which functionality stays unavailable.

Do not commit unless the user asks. If they do and you upgraded hotpath, keep the upgrade in a commit of its own, never part of a regression fix: a new hotpath release can change what the reports measure, so the next verdict compares across two versions. Tell the user so when the upgrade lands in a pull request that is being judged.

| Page | Covers |
|---|---|
| https://hotpath.rs/agents_cli | install and authenticate, `repos`, `benchmarks`, `report`, `reports` (history and filters), `diff` (flags, JSON answer, how to read it), `validate-policy`, `init`, output and exit codes |
| https://hotpath.rs/regression_policy | the policy file: which file a run uses, sections, keys, the pull request comment and check |
| https://hotpath.rs/performance_budgets | budget rules, units, broken budgets |
| https://hotpath.rs/ci_integration | the CI workflow that uploads reports, "Profiling modes" (merged, native, Callgrind only), troubleshooting uploads |
| https://hotpath.rs/callgrind | instruction counts (`icount`): merged reports, what is counted, reproducibility limits, comparing runs |

## Setup

1. Check the CLI: `hotpath cloud --help`. If it is missing or has no `cloud` command, install it as "Install and authenticate" describes.
2. Check the token: `hotpath cloud auth`. It reads `HOTPATH_API_TOKEN`. If it fails with `invalid_token` or the variable is unset, ask the user to create a token at https://hotpath.rs/app/tokens and export it. Never ask the user to paste the token into the conversation.
3. Find the repository (`OWNER/NAME`, from `git remote get-url origin`) and the benchmark name: `hotpath cloud benchmarks --repo OWNER/NAME`. The benchmark is also the `HOTPATH_BENCHMARK` value in the repository's `.github/workflows/*.yml`.

`validate-policy` and `init` need no token.

## Workflow: check or fix a change

Branch on the exit code, then read the JSON ("Output and exit codes"). Do not parse the human text of the pull request comment.

1. Push the commit. CI runs the benchmark and uploads the report, which takes as long as the benchmark job.
2. Wait for the report. Poll until it exits `0`, waiting about 30 seconds between attempts and giving up after about 30 minutes:

   ```bash
   hotpath cloud report --repo OWNER/NAME --benchmark NAME --commit $(git rev-parse HEAD) --no-payload
   ```

   `not_found` means the upload has not arrived yet. If it never arrives, check the CI run with `gh run list` / `gh run view` and the "Troubleshooting" table of https://hotpath.rs/ci_integration.
3. Read the judgement, as the "diff" section explains it:

   ```bash
   hotpath cloud diff --repo OWNER/NAME --benchmark NAME --commit $(git rev-parse HEAD)
   ```

4. Exit `0`: done, report the verdict and `dashboard_url` to the user. Exit `1`: read the regressions and broken budgets, fix the code, and repeat from step 1.

`--pr N` or `--id ID` select a report instead of `--commit`; `--advisory` and `--full` widen what `diff` lists. To find earlier reports (a benchmark's history on the default branch, recent regressions, every report of a pull request), page through `hotpath cloud reports`; `--metadata KEY=VALUE` narrows it to runs uploaded with that `user_metadata` pair, e.g. one runner or CPU, to compare the same benchmark across machines. `--select '{"functions":{"keys":["crate::module::fn"]}}'` on `report` or `reports` filters each payload to those exact entities, in the payload's usual shape, to follow them across runs without the rest of the report. On `diff` it keeps only those entities' rows: alone every row of them (unchanged and ignored ones included), with `--advisory` only their findings; the verdict and exit code stay those of the whole comparison. `--precise` on `report`, `reports` or `diff` returns values unrounded (durations in `ns`, sizes in `B`, `change_percent` unrounded) for benchmarking or statistics where rounding hides small differences; the verdict and exit code are the same.

Useful extracts:

```bash
# Regressions with the values that crossed the bar
hotpath cloud diff ... | jq '.result.sections[]? | .rows[] | select(.outcome == "regression")
  | {name, location, cells: [.cells[] | select(.crossed) | {column, base, head, change_percent}]}'

# Broken budgets with their messages
hotpath cloud diff ... | jq '.budgets.findings[] | {entity: .entity.name, actual, limit, message}'
```

## Fixing a regression

- Start from `location` of each regressed row and the `crossed` cells: an allocation regression points at new or larger allocations in that function, a `calls` / `count` change at extra work, `sql_per_request` at an N+1 query.
- Allocations are deterministic for a fixed workload, so an allocation change is real. Timing on shared CI runners is noisy; be careful attributing a small timing change to the code.
- A merged report (`meta.run` is `"merged"`: native timing + Callgrind instruction counts in one report) carries both. Its `icount` findings are the CPU verdict; the timing next to them is advisory, and on shared runners mostly noise unless the change is large and repeats across runs.
- An `icount` regression (Callgrind instruction counts) means the function executed more instructions: extra work, a different algorithm or code that is no longer inlined. To investigate one:
  - Read the changed code of the regressed function and its callees, and how far the count moved (`base`, `head`, `change_percent`).
  - Check the comment and `diff` for an `environment mismatch` alert (`compared anyway, changes may come from the environment and never fail the check`): the two reports differ in Valgrind version, `rustc` or OS, so the section was compared with that caveat and its regressions cannot fail the check. Its result means neither clean nor regressed: the differences may be environmental. Say so, and suggest pinning the environment (and a new baseline from the default branch) before trusting that section.
  - When possible, compare repeated runs of the same commit (re-run the job). How much counts vary depends on the workload, concurrent code most; a change within that spread may be noise.
  - More instructions is more work, not necessarily more elapsed time; say so when you report it, and do not claim a speed regression from the count alone.
  - Counts from a local run differ from CI's even for the same commit: compare a fix against a local run of the base commit, never against the CI numbers.
  - Raising `[functions.icount]` `min_percent_change` to make a result disappear is loosening the policy (see "Rules"). Raise it only with the user's agreement and from measured run-to-run variation.
- Profile locally to confirm before and after a fix: run the benchmark the CI workflow runs, with the same `--features`, and compare the printed hotpath report. For an `icount` regression that means running it under Valgrind as the workflow does, which needs x86_64 or ARM64 Linux (ARM64 with the patched Valgrind `3.28.0.hotpath-rs`, see https://hotpath.rs/callgrind; counts from the two architectures are not comparable, so use the CI runner's): the merged pair (native run, then the Callgrind run with `HOTPATH_MERGE_REPORT`) when you also want timing, or only the Callgrind run when the counts are enough. Elsewhere, push the fix and read the next verdict.
- A broken budget names its rule's `message`, which usually says what the limit protects.

## Rules

- Never loosen the policy (`hotpath/*.toml`) or a budget to make a regression or a broken budget go away. Fix the code, or explain the trade-off to the user and let them decide.
- If the user asks to change the policy, fetch https://hotpath.rs/regression_policy (and https://hotpath.rs/performance_budgets for budgets), edit `hotpath/policy.toml` (or `hotpath/<benchmark>-policy.toml`) and check it with `hotpath cloud validate-policy` (`--benchmark NAME` for a benchmark's own file) before pushing. An upload with an invalid policy is refused.
- `hotpath cloud init` writes the default policy when the repository has none. It never replaces an existing file without `--force`; do not pass `--force` unless the user asks.
- `HOTPATH_API_TOKEN` is a credential: never print it, echo it, log it or commit it.