Run the benchmarks

peryx-bench-core owns neutral measurements and reports. peryx-bench owns neutral execution and comparison. Ecosystem owners keep workloads, fixtures, command arguments, and result interpretation in their own crates.

The PyPI workload and server adapters live in crates/peryx-ecosystem-pypi/src/bench/. The OCI workload lives in crates/peryx-ecosystem-oci/src/bench/. Each owner exposes its workload through a peryx-bench-* binary.

Benchmark contracts

just test runs crate tests, then executes all workspace benchmark harnesses:

just benchmark

A benchmark target must fail when setup or the measured path fails. Keep behavior assertions in the crate's tests/ tree; measurement code belongs in benches/. A benchmark does not replace a unit or integration test.

Run one target

List targets without measuring them:

cargo bench --workspace --all-features --no-run

Execute one harness through the same Cargo mode used by the repository gate:

cargo test --locked -p PACKAGE --all-features --bench TARGET -- --help

Replace --help with the target's documented arguments. Keep the toolchain and Cargo.lock fixed between revisions. Record the host, power mode, scratch filesystem, and competing load with measured results. Wait for service readiness during setup. A pacing sleep is valid when elapsed time is part of the workload.

Ecosystem suites

Inspect an ecosystem benchmark command or run one target:

cargo run -p peryx-ecosystem-pypi --features bench --bin peryx-bench-pypi -- --help
cargo bench --locked -p peryx-ecosystem-pypi --bench parse
cargo run -p peryx-ecosystem-oci --features bench --bin peryx-bench-oci -- --help
cargo bench --locked -p peryx-ecosystem-oci --bench manifest_by_digest

The PyPI root-catalog fixture is crates/peryx-ecosystem-pypi/src/bench/packages.rs. Build its million-project example before collecting at least five rounds from one machine and power state:

CARGO_TARGET_DIR=.tox/target-catalog cargo build --release \
  -p peryx-ecosystem-pypi --example catalog_million

/usr/bin/time -l .tox/target-catalog/release/examples/catalog_million

CodSpeed

CodSpeed runs owner-selected benchmarks in simulation mode on standard GitHub-hosted runners:

just codspeed PACKAGE

PyPI parsing and serving benchmarks use separate builds so parser measurements exclude serving dependencies. CI runs those selections in parallel; the local recipe runs both.

Use simulation for in-process compute paths. Compare wall-time revisions under the same host conditions outside CI.