Methodology

PRIMER runs the same deterministic tasks through the same agent twice per evaluation — once without the context file (the without arm) and once with it (the with arm). Both arms use the same Docker image, the same agent flags, the same task suite, and the same number of repetitions. The only difference between arms is whether the context file is present in /work.

This controlled design means that any difference in success rate is attributable to the context file — not to the agent version, the model, the environment, or the task selection.

Each run executes inside a pinned Docker container identified by a sha256 digest. Pinning by digest (not tag) guarantees byte-for-byte reproducibility: the same digest in a future run is provably the same environment.

The base image digest is recorded in every evaluation export and shown in the Instrument Status Bar and the Methods panel.

CLI → docker pull image@sha256:<digest>
      docker run --rm [--network <mode>] \
        -v task_dir:/work \
        image@sha256:<digest>
        <agent command>

When network_mode is proxy-egress, a deny-by-default outbound proxy runs on every container. The proxy logs all outbound connection attempts and blocks everything not on an explicit allow-list. This prevents the agent from phoning home, fetching external resources, or exfiltrating data — making the measurement environment reproducible across runs.

When egress_enforced is false (live data currently shows egress open), the container uses an open bridge network. The status bar renders an amber indicator to make this visible without asserting enforcement.

Tasks are derived from the repository's own test suite. PRIMER supports two task types:

• stub_function — a function is replaced with a stub (e.g., raise NotImplementedError). The agent must implement it so the pytest suite passes.
• revert_reimplement — a working implementation is reverted to an older commit. The agent must re-implement it so tests pass.

The sole verdict for each task run is the pytest exit code: 0 = pass, non-zero = fail. No partial credit, no heuristics.

Each arm runs N repetitions per task. The variance across runs (success_stddev) captures natural run-to-run fluctuation. The noise envelope is defined as:

This formula sets a floor of 1 / n_tasks — the minimum observable delta for a single-task flip — and raises it if the measured variance is higher. A delta within this envelope is classified as within-noise, not positive or negative.

With 5 tasks, the floor is ±20 pp. This means a single task flipping from fail to pass or vice versa is not enough to declare a verdict — multiple tasks must agree.

PRIMER tracks two separate cost streams that are never combined:

• Eval cost (cost_without, cost_with) — the token/API cost incurred by the agent under test during the controlled runs.
• PRIMER overhead (primer_overhead_usd) — the cost of generating the context file using the Layer-1 provider (e.g., Anthropic). This is always shown on a separate line and never summed into the eval cost delta.

Cost confidence is exact when provider APIs report exact usage, estimated when costs are approximate, and free for locally-run models.

PRIMER issues a refused verdict and sets success_delta = null when:

• The two arms used different models or providers — a delta across different models cannot be attributed to the context file alone.
• Isolation settings changed between compared evaluations (shown as isolation mismatch warning).

Refusal is a first-class outcome. It means "we cannot give you an honest answer here" — not a failure.

PRIMER distinguishes between two roles:

• Layer 1 — generation provider (provider/model): the model that writes the CLAUDE.md file. Supports Anthropic, OpenAI, Gemini, OpenRouter, and Ollama.
• Layer 2 — agent under test (agent_adapter): the coding agent that runs inside Docker and attempts the tasks. Default is claude_code; the gemini adapter is experimental.

The generation provider's cost is always PRIMER overhead, not eval cost. The current live data uses the experimental Gemini adapter path — the UI reads this from agent_adapter in the export and never hardcodes "Claude Code."

Every evaluation export includes all the information needed to reproduce the measurement: base image digest, agent adapter, provider/model, task count, runs per config, egress mode, and the exact repo commit. The JSON exports are versioned (schema_version) and available on the Export page.