Skip to main content
PRIMER

Reference

Score guide

How to read a PRIMER result. All cards below are illustrative examples — not your data.

Example only — not your data

The four verdicts

HelpedExample only — not your data
+18.0 pp±10.0 pp noise

The delta (+18.0 pp) lies outside the noise envelope (±10.0 pp). The context file improved agent success rate beyond what variance can explain.

HurtExample only — not your data
−14.0 pp±10.0 pp noise

The delta (−14.0 pp) lies outside the noise envelope (±10.0 pp) in the negative direction. The context file reduced agent success rate beyond variance.

No measurable effectExample only — not your data
0.0 pp±20.0 pp noise

The delta (0.0 pp) is real but cannot be distinguished from natural variance at this task count. You're protected from acting on noise.

Not comparableExample only — not your data
N/A

PRIMER refuses to compute a delta when the two arms used different models or providers. Any difference couldn't be attributed to the context file alone.

Exact verdict rules

success_delta == null→Not comparable
|delta| ≤ noise_threshold→No measurable effect
delta > noise_threshold→Helped
delta < −noise_threshold→Hurt

Where noise_threshold = max(1 / n_tasks, success_stddev)

Delta definition

success_delta = success_rate_with − success_rate_without. Positive means the with-arm succeeded more often.

The delta is always shown in percentage points (pp) with one decimal place and its noise envelope: +5.0 pp ± 20.0 pp.

Why 5 tasks gives a ±20 pp noise floor

The noise floor is 1 / n_tasks. With 5 tasks, that's 1/5 = 20 pp. This is the minimum observable delta: a single task flipping from fail to pass changes the success rate by exactly 1/5 = 20 pp. A delta smaller than this cannot be reliably attributed to the context file with 5 tasks.

More tasks = lower floor = more sensitive measurement.

Flip states

FAIL_TO_PASS

Fixed by the file

Task failed without, passed with. Best case.

PASS_TO_FAIL

Broken by the file

Task passed without, failed with. Harmful.

PASS_TO_PASS

Passed with and without

Context file had no effect on this task.

FAIL_TO_FAIL

Failed with and without

Task unsolved regardless. Not attributable to the file.

Cost streams

Eval cost (cost_without, cost_with) is the agent's token/API cost during the runs. PRIMER overhead is the cost of generating the context file — always separate, never summed.

Cost confidence: exact = provider-reported; estimated = approximate; free = local model, no cost.

Limitations

  • Pass/fail is binary — no partial credit or output quality measurement.
  • Small n means wide noise envelopes. More tasks = more sensitive measurement.
  • Results are specific to this agent, this model version, and this commit.
  • A positive verdict is evidence, not proof — the tasks may not represent all real-world use.
  • Flaky tasks (inconsistent pass/fail across identical runs) reduce reliability; the flaky tasks warning flag alerts you when any task was flaky.