Benchmarks

Measured, not marketed.

Every number below was produced by a harness that ships in the repository, on a single developer machine. Read the limitations before quoting anything.

Sample repository (the built-in demo)

  • Size: 18 files / 17 production modules.
  • Indexing time: ~4 seconds, measured end-to-end (load + scan + graph build).
  • Context-retrieval latency: 1–30 ms per query after indexing. Results are deterministic lookups against the local index — there is no model call in the loop.

Retrieval quality (50-scenario suite)

The repository ships a benchmark suite of 50 ground-truth scenarios (feature additions, bug investigations, impact analyses) against a reference repository with known structure. Latest full run:

  • File recall: 0.95 (feature addition), 0.98 (bug investigation), 0.97 (impact analysis) — the files you actually need are almost always in the answer.
  • File precision: 0.80 (feature), 0.30 (bug investigation), 0.88 (impact) — bug investigations deliberately cast a wider net of hypotheses.
  • All 50 scenarios execute without errors.

Reproduce it: py -3 benchmarks/generate_suite.py then py -3 benchmarks/runner.py in the Atlas repository.

Memory use

We have not published a rigorous memory benchmark yet. Anecdotally the desktop app is a single local Python process; a proper measurement across repo sizes will be published here when it exists. We will not fabricate a number in the meantime.

Limitations — read this part

  • All numbers are self-measured on one Windows development machine; your hardware will differ.
  • The 50-scenario suite runs against a reference repository we built, not a random sample of open-source projects.
  • The sample demo repo is small by design. Indexing large monorepos is slower and is an active work item (see the roadmap).
  • Retrieval quality is not the same as end-to-end task success with a coding agent — that study has not been run yet.