CI integration: benchmark Rust pull requests in GitHub Actions

hotpath-cloud reviews the performance of every pull request. Your CI job runs a benchmark with hotpath profiling enabled and uploads the report. hotpath.rs compares it with the report of the base branch and posts the result as a comment on the pull request.

What counts as a regression is customizable with a regression policy and performance budgets.

Setup

Log in at hotpath.rs/app and install the hotpath-rs GitHub App on the repository from the dashboard. The app has no access to your code, only write access to pull requests to post comments, and to checks to report the result of each benchmark.

hotpath-cloud requires hotpath 0.30.0 or newer.

Add the hotpath-cloud feature next to the profiling features you use:

[dependencies]
hotpath = "0.31"

[features]
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cloud = ["hotpath", "hotpath/hotpath-cloud"]
hotpath-callgrind = ["hotpath", "hotpath/hotpath-callgrind"]

Add a policy file. Every benchmark needs one: its own hotpath/<benchmark>-policy.toml, or the shared hotpath/policy.toml when it has none. An upload without one is refused. hotpath cloud init writes the default policy to hotpath/policy.toml:

cargo install hotpath --features cloud
hotpath cloud init

Commit the file. The regression policy page explains what it contains and how to change it.

The benchmark is any program of yours instrumented with hotpath: an example, a binary or an integration test that runs a fixed workload. The report is uploaded at the end of the run.

There are two ways to wire the workflow, depending on where pull requests come from:

The fork setup also handles pull requests from the same repository, so a project that accepts both needs only that one.

Both setups run the benchmark in merged mode (native timing + Callgrind instruction counts in one report), the recommended mode on Linux runners, x86_64 and ARM64 (ARM64 is highly experimental and needs a patched Valgrind, see ARM64 runners). One build runs twice: natively, for timing, allocations and every other section, then under Valgrind’s Callgrind, which adds instruction counts and uploads one merged report. Timing on shared runners is too noisy to judge small changes, while instruction counts are reproducible for a deterministic workload, so the counts are judged and the timing is shown next to them. Where Valgrind cannot run the benchmark, use native mode; Profiling modes compares the modes.

Pull requests from the same repository

One workflow runs on pushes to the default branch and on pull requests. Both upload from the benchmark process itself.

name: Benchmark

on:
  push:
    branches: [main]
  pull_request:

jobs:
  benchmark:
    name: benchmark
    # Fork pull requests get no `id-token: write`, so there is nothing to upload.
    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository
    # Instruction counts depend on the Valgrind build, the Rust toolchain and
    # the OS, so the image, the package and the toolchain are pinned: a change
    # in any of them shows up as an environment-mismatch alert on the comment.
    runs-on: ubuntu-26.04
    timeout-minutes: 30
    permissions:
      contents: read
      id-token: write # the upload authenticates with the job's OIDC token
    # Shared by both runs, so their reports describe one benchmark. Static
    # values: job-level `env` cannot read the `runner` context.
    env:
      HOTPATH_BENCHMARK: my_benchmark
      HOTPATH_USER_METADATA: runner=Linux-X64,toolchain=1.99.0,profile=release,valgrind=3.26.0
      HOTPATH_OUTPUT_FORMAT: json
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@master
        with:
          toolchain: "1.99.0"
      - uses: Swatinem/rust-cache@v2
      - name: Install valgrind
        run: |
          sudo apt-get update
          sudo apt-get install -y valgrind=1:3.26.0-0ubuntu1
          valgrind --version
      - name: Build benchmark
        run: |
          cargo build -q --release --example my_benchmark \
            --features hotpath-alloc,hotpath-cloud,hotpath-callgrind
      - name: Run benchmark natively
        env:
          HOTPATH_OUTPUT_PATH: ${{ runner.temp }}/native.json
        run: target/release/examples/my_benchmark
      - name: Run benchmark under Callgrind and upload the merged report
        env:
          HOTPATH_UPLOAD: fail-on-error
          HOTPATH_UPLOAD_RESPONSE_PATH: ${{ github.workspace }}/upload.json
          HOTPATH_OUTPUT_PATH: ${{ runner.temp }}/merged.json
          HOTPATH_MERGE_REPORT: ${{ runner.temp }}/native.json
          HOTPATH_CALLGRIND_OUT_FILE: ${{ runner.temp }}/callgrind.out.%p
        run: |
          valgrind -q --tool=callgrind --instr-atstart=no \
            --callgrind-out-file="$HOTPATH_CALLGRIND_OUT_FILE" \
            target/release/examples/my_benchmark
          test -s "$HOTPATH_UPLOAD_RESPONSE_PATH"
          jq -e '.meta.run == "merged" and (.functions_icount.data | length > 0) and (.functions_timing.data | length > 0)' \
            "$HOTPATH_OUTPUT_PATH" > /dev/null

Pushes to the default branch upload the baseline that pull requests are compared against.

What each part is for:

• One build, two runs. The benchmark is built once and the same binary runs natively, then under Valgrind. Never valgrind cargo run, which would instrument Cargo. For a binary target the path is target/release/<name>.
• Shared settings in the job’s env. The benchmark name, the metadata and the JSON output must be the same for both runs, so they are set once for the job, with static values: job-level env cannot read the runner context, and the metadata describes the pinned environment anyway. HOTPATH_UPLOAD, HOTPATH_MERGE_REPORT and the output paths belong to their step. Set on the native step, HOTPATH_UPLOAD would upload a second, native-only report, and HOTPATH_MERGE_REPORT fails it, since that run is not under Valgrind.
• The merge. HOTPATH_MERGE_REPORT names the native run’s report. The Callgrind run takes it as the base, adds its own instruction counts and uploads the result. It checks that both runs come from the same build, checkout, benchmark and CI run first. A failed merge fails the step whatever HOTPATH_UPLOAD says, writes no JSON and uploads nothing. Timing and instruction counts in one report describes what a merged report carries.
• Run from the repository root, both runs from the same directory, so the git information and the policy file are found and the two reports describe the same checkout. The native report goes outside the checkout (runner.temp), to a different file than the merged one.
• Pinned environment. Instruction counts depend on the Valgrind version, the Rust toolchain and the OS (including the architecture). When the two reports differ in any of them, hotpath.rs still compares the counts, but the comment and every view carry an alert (functions_icount: environment mismatch (...); compared anyway, changes may come from the environment and never fail the check), and the section’s regressions never fail the pull request check. Such a comparison is not a clean verdict: the changes it shows may come from the environment, not the code. So pin the runner label, the Valgrind package and the toolchain instead of ubuntu-latest and stable, and bump them deliberately: the first push to the default branch after a bump uploads the new baseline, and pull requests compared with the old one show the alert until then. A runner label is not an immutable image, so system libraries can still change under it; the pins make reports comparable, not identical.
• --instr-atstart=no keeps Callgrind’s instrumentation off until hotpath turns it on at guard creation. That cuts the cost of startup, but the program still runs under Valgrind the whole time. It changes speed, not counts; Callgrind options explains it and the options not to pass.
• HOTPATH_CALLGRIND_OUT_FILE names the same file as --callgrind-out-file, here outside the checkout. Without it hotpath looks for callgrind.out.<pid> in the working directory.
• test -s and the jq check. The response file is written only when the server accepted the report, and removed when the upload was skipped or failed, so test -s fails the job when no report was stored. Keep its path absolute: a relative one resolves against the working directory of the benchmark program. Give every uploading program its own file. The jq check fails the job when the stored report is not a merged one with both instruction counts and timing.
• A longer timeout, for Valgrind, which runs the benchmark tens of times slower. The hotpath-callgrind feature also costs one extra call per measured invocation in the native run, which is visible in its timings.

The workload must run twice in one job doing the same work: fixed inputs and seeds, no network, and nothing the first run leaves behind (a database it fills, files it creates) changing the second run. Before relying on the verdicts, re-run the job a few times on one commit and set min_percent_change of [functions.icount] above the variation you see: how much concurrent code moves between runs depends on the workload.

Failing the job

HOTPATH_UPLOAD decides whether a failed upload fails the job:

A failed merge (HOTPATH_MERGE_REPORT set) always fails the job, whatever HOTPATH_UPLOAD says. A skipped upload (not in GitHub Actions, no OIDC token, invalid benchmark name, the program panicked) never fails the job. A regression or a broken budget never fails the job either: it is a warning in the job log. The pull request’s hotpath / <benchmark> check fails when the policy sets pr_check = "fail" and the finding comes from a block with fail_check = true: every budget rule by default, a policy section only when it opts in.

To block merging a pull request with a regression, make hotpath / <benchmark> a required check in the branch protection rules. A required check also blocks merges while hotpath.rs is unavailable, since no check is posted then.

Environment variables

See configuration for the variables that shape the report itself.

Pull requests from forks

A job triggered by a pull request from a fork runs the fork’s code, so GitHub gives it a read-only token and no OIDC token, so the benchmark job cannot upload. It writes the report to an artifact instead, and a second workflow, triggered when the first one completes, uploads it.

The benchmark workflow

Pushes to the default branch upload directly, as in the same-repository setup. Pull requests run the same two runs, write the merged report to a file and store it as an artifact.

name: Benchmark

on:
  push:
    branches: [main]
  pull_request:

jobs:
  upload:
    name: benchmark (push)
    if: github.event_name == 'push'
    runs-on: ubuntu-26.04
    timeout-minutes: 30
    permissions:
      contents: read
      id-token: write
    env:
      HOTPATH_BENCHMARK: my_benchmark
      HOTPATH_USER_METADATA: runner=Linux-X64,toolchain=1.99.0,profile=release,valgrind=3.26.0
      HOTPATH_OUTPUT_FORMAT: json
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@master
        with:
          toolchain: "1.99.0"
      - uses: Swatinem/rust-cache@v2
      - name: Install valgrind
        run: |
          sudo apt-get update
          sudo apt-get install -y valgrind=1:3.26.0-0ubuntu1
      - name: Build benchmark
        run: |
          cargo build -q --release --example my_benchmark \
            --features hotpath-alloc,hotpath-cloud,hotpath-callgrind
      - name: Run benchmark natively
        env:
          HOTPATH_OUTPUT_PATH: ${{ runner.temp }}/native.json
        run: target/release/examples/my_benchmark
      - name: Run benchmark under Callgrind and upload the merged report
        env:
          HOTPATH_UPLOAD: fail-on-error
          HOTPATH_UPLOAD_RESPONSE_PATH: ${{ github.workspace }}/upload.json
          HOTPATH_OUTPUT_PATH: ${{ runner.temp }}/merged.json
          HOTPATH_MERGE_REPORT: ${{ runner.temp }}/native.json
          HOTPATH_CALLGRIND_OUT_FILE: ${{ runner.temp }}/callgrind.out.%p
        run: |
          valgrind -q --tool=callgrind --instr-atstart=no \
            --callgrind-out-file="$HOTPATH_CALLGRIND_OUT_FILE" \
            target/release/examples/my_benchmark
          test -s "$HOTPATH_UPLOAD_RESPONSE_PATH"
          jq -e '.meta.run == "merged" and (.functions_icount.data | length > 0) and (.functions_timing.data | length > 0)' \
            "$HOTPATH_OUTPUT_PATH" > /dev/null

  report:
    name: benchmark (pull request)
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-26.04
    timeout-minutes: 30
    permissions:
      contents: read
    env:
      HOTPATH_BENCHMARK: my_benchmark
      HOTPATH_USER_METADATA: runner=Linux-X64,toolchain=1.99.0,profile=release,valgrind=3.26.0
      HOTPATH_OUTPUT_FORMAT: json
    steps:
      - uses: actions/checkout@v4
        with:
          # The pull request's head commit, not the default merge commit.
          ref: ${{ github.event.pull_request.head.sha }}
      - uses: dtolnay/rust-toolchain@master
        with:
          toolchain: "1.99.0"
      - uses: Swatinem/rust-cache@v2
      - name: Install valgrind
        run: |
          sudo apt-get update
          sudo apt-get install -y valgrind=1:3.26.0-0ubuntu1
      - name: Build benchmark
        run: |
          cargo build -q --release --example my_benchmark \
            --features hotpath-alloc,hotpath-cloud,hotpath-callgrind
      - name: Run benchmark natively
        env:
          HOTPATH_OUTPUT_PATH: ${{ runner.temp }}/native.json
        run: target/release/examples/my_benchmark
      - name: Run benchmark under Callgrind and write the merged report
        env:
          HOTPATH_OUTPUT_PATH: report.json
          HOTPATH_MERGE_REPORT: ${{ runner.temp }}/native.json
          HOTPATH_CALLGRIND_OUT_FILE: ${{ runner.temp }}/callgrind.out.%p
        run: |
          valgrind -q --tool=callgrind --instr-atstart=no \
            --callgrind-out-file="$HOTPATH_CALLGRIND_OUT_FILE" \
            target/release/examples/my_benchmark
          test -s report.json
          jq -e '.meta.run == "merged" and (.functions_icount.data | length > 0) and (.functions_timing.data | length > 0)' \
            report.json > /dev/null
      - uses: actions/upload-artifact@v4
        with:
          name: hotpath-report
          path: report.json
          retention-days: 1
          if-no-files-found: error

The pull request job still builds with the hotpath-cloud feature. The feature is what adds the commit, the pull request, the benchmark name and the policy file to the JSON report. HOTPATH_UPLOAD is not set in that job: it has no token to upload with. A failed merge fails it before the artifact upload, so the relay never posts a report that is not merged. The parts the two jobs share are explained in the same-repository setup.

The relay workflow

name: Benchmark relay

on:
  workflow_run:
    workflows: ["Benchmark"] # the `name:` of the benchmark workflow
    types: [completed]

permissions: {} # jobs get only what they declare

jobs:
  relay:
    # only successful pull request runs have a report; the action checks this too
    if: >-
      github.event.workflow_run.event == 'pull_request' &&
      github.event.workflow_run.conclusion == 'success'
    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      actions: read # download the artifact of the benchmark run
      id-token: write # authenticate the upload
      pull-requests: read # find the pull request the run belongs to
    concurrency: # a newer run of the same benchmark and branch supersedes an older one
      group: hotpath-relay-${{ github.event.workflow_run.name }}-${{ github.event.workflow_run.head_repository.full_name }}-${{ github.event.workflow_run.head_branch }}
      cancel-in-progress: true
    steps:
      - uses: hotpath-rs/[email protected]
        with:
          benchmark: my_benchmark
          artifact_name: hotpath-report
          report_file: report.json

hotpath-relay is a GitHub Action. It runs after a successful benchmark run of a pull request, downloads the artifact, uploads the report and prints the server’s response in the step summary; runs of other events, and failed runs, are skipped with a notice. A rejected upload fails the relay job, and a comment that could not be posted is a warning. The pull request the report belongs to is looked up from the benchmark run’s head commit, not read from the report, which the pull request’s own code wrote; that lookup is what pull-requests: read is for. The relay job never checks out or runs code from the pull request: keep the action its only step, since every step of the job could mint the upload token.

Pin the action to an exact version, such as v1.0.0. Its versions are independent of hotpath’s and published as immutable releases: a version tag can never be moved to another commit, so it is as fixed as a commit SHA. There is no moving v1 tag; upgrade by changing the version.

Things to know about workflow_run:

• It runs the copy of the relay workflow that is on the default branch. Changes to it do nothing until they are merged.
• It matches the benchmark workflow by name. Rename that workflow and the relay stops firing, without an error.
• Its jobs are not checks on the pull request. The hotpath / <benchmark> check the server posts is, so a regression shows among the pull request’s checks of fork pull requests too.

A pull request is judged under the policy file of its own branch, and that includes pull requests from forks. Review changes to hotpath/*.toml like changes to CI configuration.

Environment variables

In the pull request job, which writes the report:

The push job takes the variables of the same-repository setup.

Profiling modes

Merged mode needs:

• a Linux runner, x86_64 or ARM64; on ARM64 (highly experimental), the patched Valgrind and hotpath 0.31.0 or newer, see ARM64 runners;
• a CPU-bound workload: its time goes into computation, not into network, disk, sleeps, timers or a database service, and it is not dominated by many threads racing each other (Valgrind serializes threads, so their interleaving moves the counts);
• the code to judge in functions with #[hotpath::measure] (or measure_all): measure_block! blocks are not counted;
• a [profile.release] that keeps symbols (no strip = true or "symbols");
• a workload that finishes under Valgrind within the job timeout: one that takes more than about a minute natively should be made smaller or run natively;
• a workload that runs twice in one job doing the same work.

Merged

The workflows of both setups above. The judged instruction counts are configured under [functions.icount], and timing stays listed under [functions.timing] with judged = false, as the default policy writes it, so it shows next to the counts.

ARM64 runners

Warning: ARM64 support is highly experimental. Instruction counts on ARM64 come from hotpath’s own Valgrind fork, a proof of concept that is likely to return invalid results. Treat ARM64 counts as an experiment, not as a basis for decisions, and run Callgrind on x86_64 Linux for counts you rely on.

The workflows above run on x86_64 (ubuntu-26.04), where Ubuntu’s Valgrind package counts correctly. Merged mode also runs on ARM64 Linux runners, with these changes to every job that runs the benchmark (the same-repository job, and both the upload and report jobs of the fork setup):

    runs-on: ubuntu-26.04-arm
    env:
      HOTPATH_USER_METADATA: runner=Linux-ARM64,toolchain=1.99.0,profile=release,valgrind=3.28.0.hotpath-rs-2
    steps:
      # replaces the `Install valgrind` apt step
      - uses: hotpath-rs/[email protected]
        with:
          version: 3.28.0.hotpath-rs-2

Everything else stays as in the x86_64 workflows: one build, the native run, the Callgrind run with HOTPATH_MERGE_REPORT, the test -s and jq checks. Why the Valgrind step changes: because of a bug, Valgrind counts instructions incorrectly on ARM64, so Callgrind there needs the patched build from hotpath-rs/valgrind, installed by hotpath-rs/setup-valgrind. hotpath.rs refuses ARM64 instruction counts from any other build, so changing only runs-on gives a rejected upload (see troubleshooting).

x86_64 and ARM64 counts are not comparable: the same code executes a different number of instructions on each. Moving an existing benchmark from one architecture to the other starts over: the first pushes after the move are compared with the old baseline under an environment-mismatch alert (the reports differ in OS and Valgrind version) until the default branch has uploaded a new baseline, so consider giving the ARM64 benchmark a new HOTPATH_BENCHMARK name instead.

Native

Without Valgrind, the benchmark runs once and uploads its native report:

name: Benchmark

on:
  push:
    branches: [main]
  pull_request:

jobs:
  benchmark:
    name: benchmark
    # Fork pull requests get no `id-token: write`, so there is nothing to upload.
    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    timeout-minutes: 20
    permissions:
      contents: read
      id-token: write # the upload authenticates with the job's OIDC token
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@master
        with:
          toolchain: "1.99.0"
      - uses: Swatinem/rust-cache@v2
      - name: Run benchmark and upload report
        env:
          HOTPATH_UPLOAD: fail-on-error
          HOTPATH_UPLOAD_RESPONSE_PATH: ${{ github.workspace }}/upload.json
          HOTPATH_BENCHMARK: my_benchmark
          HOTPATH_USER_METADATA: runner=${{ runner.os }}-${{ runner.arch }},toolchain=1.99.0,profile=release
          HOTPATH_OUTPUT_FORMAT: none
        run: |
          cargo run -q --release --example my_benchmark \
            --features hotpath-alloc,hotpath-cloud
          test -s "$HOTPATH_UPLOAD_RESPONSE_PATH"

The build needs no hotpath-callgrind feature. In the fork setup, both jobs take the same shape: no Valgrind, no native step and no merge, one cargo run; the pull request job keeps writing report.json and storing it as the artifact.

Callgrind only

A benchmark that already runs only under Callgrind keeps working: its report carries instruction counts and allocations, and no timing, since durations under Valgrind are meaningless. It is the merged workflow without the native step and without HOTPATH_MERGE_REPORT, with the jq check reduced to .functions_icount.data | length > 0. New benchmarks should use merged mode, which adds timing at the cost of one native run.

Several benchmarks in one repository

Give every benchmark its own HOTPATH_BENCHMARK name. Each series has its own baseline and its own comment on the pull request. A series is judged under hotpath/<benchmark>-policy.toml when that file exists, and under the shared hotpath/policy.toml otherwise; without either, its uploads are refused.

In the same-repository setup, each benchmark can be its own workflow or a job in a shared one: every job uploads its own report.

In the fork setup, each benchmark needs its own workflow with a unique name:. The relay receives one workflow_run event per workflow run and relays one report from it, so a second benchmark job in the same workflow would never be uploaded (the matrix setup below is the exception). One relay workflow serves every benchmark workflow: list them all in workflows: and look up the series from the name of the workflow that triggered the run. Artifacts belong to one run, so every pull request job can keep the default hotpath-report artifact with report.json in it, without collisions:

name: Benchmark relay

on:
  workflow_run:
    workflows: ["Benchmark", "Parser benchmark"]
    types: [completed]

permissions: {}

jobs:
  relay:
    # only successful pull request runs have a report; the action checks this too
    if: >-
      github.event.workflow_run.event == 'pull_request' &&
      github.event.workflow_run.conclusion == 'success'
    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      actions: read
      id-token: write
      pull-requests: read
    concurrency: # a newer run of the same benchmark and branch supersedes an older one
      group: hotpath-relay-${{ github.event.workflow_run.name }}-${{ github.event.workflow_run.head_repository.full_name }}-${{ github.event.workflow_run.head_branch }}
      cancel-in-progress: true
    steps:
      - uses: hotpath-rs/[email protected]
        with:
          # benchmark workflow `name:` -> its `HOTPATH_BENCHMARK`
          benchmark: ${{ fromJSON('{"Benchmark":"my_benchmark","Parser benchmark":"parser"}')[github.event.workflow_run.name] }}

Keep workflows: and the lookup in sync: a workflow missing from workflows: is never relayed, and one missing from the lookup is relayed with an empty benchmark, which fails the relay.

When the benchmark workflows differ in artifact_name, report_file or source_root (for example a benchmark built in a nested Cargo workspace), look those up the same way, listing only the workflows that differ and falling back to the default for the rest:

          artifact_name: ${{ fromJSON('{"Parser benchmark":"parser-report"}')[github.event.workflow_run.name] || 'hotpath-report' }}
          source_root: ${{ fromJSON('{"Parser benchmark":"parser"}')[github.event.workflow_run.name] || '' }}

List each benchmark workflow in exactly one relay: a workflow named in two relays’ workflows: is uploaded twice.

hotpath’s own repository runs this setup: four benchmark workflows, cloud-random-benchmark.yml (native mode), cloud-drain-benchmark.yml and cloud-meta-benchmark.yml (merged mode) and cloud-drain-criterion-benchmark.yml (merged mode, a Criterion benchmark), relayed by one cloud-benchmark-relay.yml. The drain and meta benchmarks profile hotpath itself, so their variables are prefixed HOTPATH_META_ instead of HOTPATH_.

An alternative for several scenarios of one program is a single benchmark workflow that writes one report file per scenario into one artifact, relayed by a matrix over the scenarios with benchmark: ${{ matrix.scenario }} and report_file: ${{ matrix.scenario }}.json. Add the scenario to the relay’s concurrency group too, or every scenario job of a run shares one group and cancels the others:

    concurrency:
      group: hotpath-relay-${{ matrix.scenario }}-${{ github.event.workflow_run.head_repository.full_name }}-${{ github.event.workflow_run.head_branch }}
      cancel-in-progress: true

MapLibre Martin runs this setup: hotpath-benchmark.yml and hotpath-relay.yml.

Troubleshooting