Context selection · budgets · benchmarks
ContextForge
Open source · MITWhat it does
Picks which files an AI should read when it cannot read everything, then checks whether that choice really beat a plain keyword search.
Read the source on GitHubProof — try it
The same five files. Two ways to spend the budget.
Filling in rank order lets one 17 KB document eat almost everything, and the 947-byte file that actually fixes the bug never gets in. Serving the smallest first costs the big documents some length and saves the one that matters.
Budget · 24 576 B
docs/postmortems/symlinked-paths.md16 986 / 16 986 B emitteddocs/benchmarks/refuse-symlinked-paths.md7 590 / 16 925 B emitteddocs/design/refuse-symlinked-paths.mdnever emitteddocs/threat-model.mdnever emittedsrc/compiler.mjsnever emitted
The file that answers the task — never emitted
Real file sizes from the recorded run. This is the defect the benchmark found, and the allocation that fixed it.
Case study
- Two benchmarks, five comparison strategies
- Corpora reproducible byte for byte from a seed
- Negative results published, not only the favourable ones
- Problem
- Decide which repository files enter a context window, without exceeding a budget, without leaking a credential, and without having to take the result on trust.
- Approach
- Lexical selection, bounded compilation, pattern redaction — then a comparative benchmark against five naive strategies under the same 24,576-byte budget, on deterministic corpora of up to 2,400 files.
- Architecture
- An explicit plan, then a compilation that divides bytes by max-min fair share, hashes every file with SHA-256, counts redactions without ever storing the values, and refuses symlinks at every path segment.
- What went wrong
- On a documentation-heavy corpus, required-file recall fell to 0.00: a single 17 KB document monopolised the budget and the file that implemented the task was never emitted. A naive grep did better.
- Result
- The case was frozen into a deterministic corpus before any fix. Diagnosis: greedy byte allocation, not ranking. After switching to max-min fair share, recall went from 0.00 to 1.00.
- Limits
- The fix cost precision: 1.00 to 0.67 on one fixture. And a second defect is still open — useful-file recall remains 0.00 on three corpora, because lexical scoring cannot separate a document that describes a task from the code that implements it.