A bounded, self-improving agent-loop suite — plus the experiments that test what it actually does. Here, credibility comes from disclosed limits and visible retractions, not hype.
Score an agent's judgment against a rubric it follows, and you measure the model agreeing with itself. Across this project, two clean ρ≈1.0 results were retracted for exactly this; same-model "blind consensus" turned out to be a persona artifact; a self-review study hit a robust floor. The recurring lesson: a self-authored value signal proves nothing.
Fix a real bug in a real library; let its own pre-existing test suite be the judge. Held-out grading, reward-hack guards, worktree isolation. 1 / 3 real bugs fixed. The verdict is reality's, not mine — it cannot be demoted by a control, because it is the control.
Each result is reported straight — including the ones a control demoted. The retractions are the point.
Relative judgment is far lower-noise; the item that coin-flipped under absolute scoring ranked stably under pairwise. Stabilizes ordering — not the cut line.
The Total≥28 cutoff is unsubstantiated; alignment carries the gate, and it's a coin-flip at the boundary (2 retractions, disclosed after blinded re-runs). Same-context self-review showed no collusion on blatant hacks — a null, reported as a floor effect.
Both near-perfect correlations — both retracted. The oracle was my own test-allocation, and value was defined as the thing being measured. Circular.
A 6/6 unanimous-wrong committee looked damning — until a 6×opus control showed the unanimity was a persona/sampling artifact. The "diversity fixes it" follow-up collapsed too.
0 genuine wrong fixes in 54 — untestable. The one apparent exception was a bug in my own oracle, caught in analysis. The 5th self-authored-oracle error of the session.
SWE-bench-lite on real python-semver bugfix commits. The fixer never sees the grading test (held out, defeats the hardcode hack); a reward-hack guard fired on a live attempt when a fixer tried to edit a test. The library's own 333 tests are the judge.
A running, honest log — new milestones are appended with dates as we move from studying the loop to grounding it. Full detail in CHANGELOG.md.
Demand characteristics caught and corrected; stimuli re-run blinded.
A 6×opus baseline collapsed a tempting consensus story. Every result earns a control.
A demoted result is a deliverable, not an embarrassment — left on the page with its mechanism.
The headline question — can an agent learn a maintainer's judgment? — is not answered yet, by construction. It needs real human decisions and cannot be self-generated without becoming circular. That honesty is the front door.