# 3-tier tool-level benchmark — 2026-05-06

_System: Darwin arm64 · Python 3.12.13 · ripgrep 15.1.0 · warm-cache · 5 runs · median + p95_

## Methodology

- **N runs:** 5 per query (after a single warm-up call) — warm-cache median + p95 reported.
- **Files filtered:** `*.py *.ts *.tsx`, excluding `node_modules .venv .git dist __pycache__`.
- **Index build excluded** from Cosmos timings — measures lookup latency on a pre-indexed repo.
- Speed and coverage are reported separately. Concept queries are not a fair speed comparison because grep/rg may legitimately return 0 hits in microseconds.
- Reproduce: `python -m benchmarks.tier_benchmark`

## Speed — exact-symbol lookup

| Tier | Repo | Files | LOC | Cosmos (median / p95) | rg (median / p95) | grep (median / p95) | Cosmos vs rg | Cosmos vs grep |
|---|---|---:|---:|---:|---:|---:|---:|---:|
| small | click | 63 | 23,743 | 0.6 / 1.4 ms | 8.1 / 10.3 ms | 9.2 / 10.2 ms | 13.5× | 15.33× |
| medium | Cosmos | 167 | 23,003 | 0.7 / 1.7 ms | 9.8 / 14.6 ms | 285.4 / 300.6 ms | 14.0× | 407.71× |
| large | cline | 1448 | 192,893 | 0.5 / 0.7 ms | 24.1 / 24.7 ms | 120.2 / 156.3 ms | 48.2× | 240.4× |

## Cost — total tokens / $ to answer the question

Cosmos returns full symbol bodies + docstrings in one call. rg
returns a list of file:line excerpts; to get the same context
the AI Reads the top hits. Below: tokens AI consumes for one
query, plus dollar cost at Opus 4.7 input · $5 / 1M.

| Tier | Cosmos (1 call) | rg + Read top 1 | rg + Read top 3 | Cosmos $ | rg-k1 $ | rg-k3 $ |
|---|---:|---:|---:|---:|---:|---:|
| small | 3604 tok | 24614 tok | 36921 tok | $0.01802 | $0.12307 | $0.18460 |
| medium | 2199 tok | 4441 tok | 18761 tok | $0.01099 | $0.02220 | $0.09380 |
| large | 3039 tok | 958 tok | 2779 tok | $0.01520 | $0.00479 | $0.01389 |

## Coverage — concept queries

These queries don't appear as literal substrings in the source. Cosmos's FTS5 + tokenizer surfaces semantically related symbols; raw text search hits 0 or near-zero.

| Tier | Repo | Query | Cosmos | rg | grep |
|---|---|---|---:|---:|---:|
| small | click | `lazy import` | 5 hits | 0 hits | 0 hits |
| small | click | `deferred initialisation` | 1 hits | 0 hits | 0 hits |
| medium | Cosmos | `assistant scheduler` | 5 hits | 2 hits | 2 hits |
| medium | Cosmos | `telegram bot` | 5 hits | 2 hits | 2 hits |
| large | cline | `slash command` | 5 hits | 72 hits | 72 hits |
| large | cline | `task abort` | 5 hits | 2 hits | 2 hits |