Benchmarks¶

This page documents how the 8.0.0 Rust CLI compares to the 7.0.1 Python CLI. Both versions share the same Rust analysis engine; the difference is the CLI pipeline: argument parsing, config resolution, path walking, and output rendering now run in Rust, and the Python interpreter no longer does any analysis work.

Methodology¶

New CLI (pre-release): built from the release branch with uv run maturin develop --release, invoked as .venv/bin/complexipy. It reports version 7.0.1 because the version bump happens at release time; the exact commit is recorded in the environment block below.
Baseline CLI: complexipy==7.0.1 installed from PyPI with uv add into an isolated project, invoked as its venv's bin/complexipy.
Invocation parity: both CLIs run as bare console scripts with byte-identical flags. The uv run wrapper is excluded from timing so neither side pays uv process overhead.
Corpus: real open-source repos, shallow-cloned at pinned commits, plus a single-file probe that isolates interpreter and CLI startup from tree analysis.
Metrics: wall time via hyperfine (warmup 3, 5 runs; probe warmup 5, 20 runs) with stdout discarded; peak RSS via /usr/bin/time -l, 3 runs, maximum reported.
Parity gate: before timing, both CLIs export JSON for every corpus repo. The exports are byte-identical and the exit codes match, so the comparison measures the same work on both sides.

The benchmark is repeatable:

bash benchmarks/benchmark-cli.sh

Environment¶

Machine: MacBookPro18,1 (arm64)
OS: macOS 26.5.1
CPU: Apple M1 Pro
RAM: 16 GB
New CLI: complexipy 7.0.1 (built from 1699696, python 3.12.12)
Baseline CLI: 7.0.1 (PyPI, python 3.12.12)
uv: uv 0.10.3 (c75a0c625 2026-02-16)
hyperfine: hyperfine 1.20.0
Runs: hyperfine warmup=3 runs=5 (probe warmup=5 runs=20); RSS via /usr/bin/time -l, 3 runs, maximum
Modes: default, --quiet, --failed, and render (full output written to a file)
Scaling fixture: 285 base functions, sizes 1x/2x/4x, 5 runs each
Date: 2026-08-31

Corpus (shallow clones at pinned commits):

requests @ 5460f467b02e49471c0fd6cfc9ca0adab6351f98 (37 files)
flask @ d318b683471101618febed18996405ad26462110 (83 files)
django @ 0b40210e4808937a7c0922e8b7502bff4752faa3 (2929 files)
single-file probe: requests/src/requests/init.py @ 5460f467b02e49471c0fd6cfc9ca0adab6351f98

Wall time¶

Workload	8.0.0 Rust CLI	7.0.1 Python CLI	Speedup
requests (37 files) [default]	0.067 s ± 0.001 s	0.373 s ± 0.002 s	5.56x
requests (37 files) [quiet]	0.064 s ± 0.002 s	0.331 s ± 0.001 s	5.16x
requests (37 files) [failed]	0.065 s ± 0.001 s	0.343 s ± 0.002 s	5.24x
requests (37 files) [render]	0.067 s ± 0.002 s	0.376 s ± 0.005 s	5.62x
flask (83 files) [default]	0.074 s ± 0.001 s	0.474 s ± 0.002 s	6.42x
flask (83 files) [quiet]	0.070 s ± 0.003 s	0.406 s ± 0.002 s	5.76x
flask (83 files) [failed]	0.073 s ± 0.001 s	0.424 s ± 0.004 s	5.80x
flask (83 files) [render]	0.076 s ± 0.002 s	0.477 s ± 0.002 s	6.32x
django (2929 files) [default]	0.445 s ± 0.002 s	8.069 s ± 0.022 s	18.12x
django (2929 files) [quiet]	0.272 s ± 0.003 s	6.486 s ± 0.011 s	23.82x
django (2929 files) [failed]	0.440 s ± 0.003 s	6.920 s ± 0.011 s	15.71x
django (2929 files) [render]	0.452 s ± 0.004 s	8.115 s ± 0.026 s	17.95x
single-file probe [default]	57.5 ms ± 1.0 ms	231.4 ms ± 1.9 ms	4.02x
single-file probe [quiet]	57.9 ms ± 1.3 ms	228.2 ms ± 1.1 ms	3.94x
single-file probe [failed]	57.6 ms ± 1.1 ms	234.8 ms ± 6.6 ms	4.08x
single-file probe [render]	59.9 ms ± 4.4 ms	234.0 ms ± 4.3 ms	3.91x

Scaling (synthetic fixture)¶

Size	Mean	Ratio	--quiet mean	Ratio	Peak RSS
1x (3995 lines)	62.0 ms	-	64.7 ms	-	69 MB
2x (7990 lines)	68.7 ms	1.11	67.7 ms	1.05	73 MB
4x (15980 lines)	82.0 ms	1.19	81.6 ms	1.21	82 MB

The fixture is generated deterministically by benchmarks/generate_scaling_fixture.py (285 base functions, sizes 1x/2x/4x, kept outside the repo under /Users/rhafid/.cache/complexipy-benchmarks/scaling; never committed). Ratios near 2 mean linear scoring; ratios near 4 mean quadratic behavior.

Peak RSS¶

Workload	8.0.0 Rust CLI	7.0.1 Python CLI
requests (37 files) [default]	70 MB	106 MB
requests (37 files) [quiet]	69 MB	105 MB
requests (37 files) [failed]	70 MB	106 MB
requests (37 files) [render]	69 MB	107 MB
flask (83 files) [default]	70 MB	106 MB
flask (83 files) [quiet]	70 MB	105 MB
flask (83 files) [failed]	70 MB	106 MB
flask (83 files) [render]	70 MB	106 MB
django (2929 files) [default]	106 MB	120 MB
django (2929 files) [quiet]	107 MB	119 MB
django (2929 files) [failed]	108 MB	120 MB
django (2929 files) [render]	107 MB	120 MB
single-file probe [default]	65 MB	104 MB
single-file probe [quiet]	65 MB	103 MB
single-file probe [failed]	65 MB	104 MB
single-file probe [render]	65 MB	104 MB

Notes: stdout is discarded during timing except the [render] mode, which writes the full output to a file; both CLIs run as bare console scripts with identical flags. The django workload exits 1 on both CLIs because its own test fixture (tests/test_runner_apps/tagged/tests_syntax_error.py) is intentionally unparseable.

What the numbers say¶

Startup-dominated workloads win the most. The single-file probe is about 4x faster, which is the Python interpreter + typer/rich import cost removed from the hot path.
The scaling table at the bottom of the results measures the Rust engine on a synthetic fixture at 1x, 2x, and 4x sizes. Ratios near 2 mean linear scoring; ratios near 4 mean quadratic behavior. The recorded ratios sit near 1.1-1.2.
Small and medium trees (requests, flask) are roughly 5-6.4x faster.
Large trees (django, ~2900 files) are roughly 16-24x faster: the Rust engine analyzes the tree in parallel across all cores, where the Python CLI processed it on one.
Output rendering is cheaper too. Rendering the full results table on django costs ~0.2 s on the Rust CLI against ~1.6 s on the Python CLI (the default minus --quiet delta), so the wall-clock gap widens on large trees when results are actually printed - the [render] rows measure that path end to end.
Peak memory is lower across the board, by roughly 15-40 MB, because the Python interpreter and its display libraries are gone from the process.