Current benchmark · Core-19
Can this local model build, debug, and recover a real system?
Compare local model variants, quantizations, and inference runtimes across 19 short-, medium-, and long-horizon terminal tasks.
results explorer
Compare runs
Model runs
Select up to four columns for direct comparison.
Run comparison
Like-for-like scores on the same 19-task denominator.
Task matrix
Open any result to see attempts, failure classification, verifier evidence, and transcripts.
Methodology
How to read these results
Current suite: Core-19. It removes only build-pov-ray: the task passed for 3/9 finished models, produced 8 agent-timeout attempts, and had a median model runtime near 4.9 hours. Summed historical medians fall by about 33%; actual runtime varies.
Primary score: passed within attempts. A pass requires a final verifier reward of exactly 1. Pass@1 is the first-attempt view.
Short through long horizon. The suite retains quick diagnostics, sustained implementation work, and long-context anchors including mailman and fix-ocaml-gc.
Sequential local runs. Harbor and Terminus use one OpenAI-compatible endpoint by default, with a three-hour agent timeout per attempt.
Full run identity. Each result retains the model, quant, platform, engine, backend, profile, tokens, timings, attempts, and transcripts.