Can it RunAlfrada?

Short answer: a model, inside a fixed production harness, doing a real job. The model supplies judgment and reasoning. Alfrada supplies planning, nineteen live tools, isolated workspaces, code execution, file creation and delivery. Because the harness stayed fixed, the ranking shows how different models operate inside the same system. It is not a ranking of raw model intelligence in isolation.

This is also not a coding benchmark. The job is lateral, multidisciplinary synthesis — research, judgment, computation, communication and delivery. A good harness should narrow the gap between models: checks, tool contracts, workspaces and validation keep work complete, inspectable and on spec while the model still determines the quality of the reasoning.

Each model became the reasoning engine inside the same production Alfrada session: same tools, same prompts, temperature 0, fresh workspace and strict cold start. No model could read another run or quietly delegate difficult work to a stronger cousin. We then gave that model-plus-harness system two jobs that resemble work a strategy team might actually commission.

The judge panel is Kimi K3, GLM-5.2, Gemini 3.5 Flash, GPT-5.6 Sol, GPT-5.6 Luna, Claude Fable 5 and Claude Opus 4.8. They do not merely read the final answer. They list the workspace, open the memos, re-run Monte Carlos from saved seeds, recompute CAC and payback, and vision-inspect slides and charts. Each of the nine contestants is judged by all seven, producing 126 verdicts. The judge panel was fixed at seven and was not expanded after scoring began, so Qwen and Inkling are contestants but not judges. Inkling's own work also fell below the usability cutoff and did not establish the verification discipline required of an evaluator.

GPT-5.6 Sol wins on audited quality (8.84 raw; +0.97 calibrated z), and it isn't self-dealing. Its own verdict is excluded from its calibrated rank, as every self-judgment is. Fable and K3 follow. Luna remains the price-performance outlier. Every model passed the mechanical delivery gates; that did not make every delivered job usable.

The matrix is where this study earns its keep. Raw means reward generous judges and duplicate architecture families. The calibrated rank instead z-scores each judge within each case, averages Sol/Luna into one GPT-5.6 signal and Fable/Opus into one Claude signal, then gives K3 and GLM one signal each. Self-votes are excluded. Gemini remains fully visible but receives zero ranking weight because its near-uniform marks add almost no discrimination.

Strategy is difficult to score because recommendations, writing and presentation involve taste. Computation does not. A model either reproduced its own Monte Carlo from the saved seed or it did not. CAC arithmetic either reconciled or it did not. A cited screenshot either contained the claimed figure or it did not. Those binary checks give us a hard spine inside an otherwise subjective evaluation.

This is why the scores do not map to SWE-bench or an exam percentage. Coding benchmarks usually ask whether one bounded patch passes tests. Here the coding sandbox is only one tool inside a much wider job. The run must research, model, decide, explain and deliver several mutually consistent artifacts — then survive seven auditors opening the files.

The harness also raises the floor. Schemas, hard gates, file validation and repeatable tools keep output at a steady minimum quality, so model differences appear mostly above that floor. We did not run a “naked model” control, so this study does not quantify the harness uplift causally. What it does show is the difference between shipping and surviving inspection: all nine models cleared every hard gate, while audited quality occupied a 5.60–8.84 band.

Use 6.0 as a go/no-go floor. Below it, the work needs substantial reconstruction. From 6–7, treat it as a supervised draft. From 7–8, it is useful with targeted expert review. Above 8, the package survives more of the audit — but no score removes the need for accountable judgment on consequential decisions.

Defect counts are observations, not rates. “No catalogued load-bearing defect” means none was found in this pinned run; it does not imply a 0% underlying failure probability. The cards below are directional selection hypotheses to validate on your own workload, not procurement guarantees.

A load-bearing error sits underneath the recommendation. Remove or correct it and the economics, confidence or proposed action materially changes. A mislabeled column is annoying; a breakeven claim calculated on the wrong profit measure can send a board into the wrong market. This is the most useful distinction for managers because it separates polish problems from decision risk.

By that cut, Sol, Luna and K3 combine zero catalogued load-bearing errors with evidence the panel could meaningfully inspect. Qwen also shipped clean core arithmetic, but its evidence pack does not substantiate the load-bearing market claims, so it should not be treated as equally low-risk. Fable, Opus and GLM each carry one: Fable's "month-26 breakeven" is really the first positive month before two more negative ones; Opus asked the board to approve a partner-led pilot whose partner economics its model never contains; GLM told the board month 30–33 when its own JSON records breakeven: null. Gemini carries two, one in each case. Inkling carries three and fails the cutoff.

Directionally, the fleet agrees more than the scores suggest: eight of nine models reached the same case-1 recommendation — a conditional, gated GO — with one principled dissent: Kimi K3 alone concluded conditional no-go, on arithmetic its judges verified. And all six with verified models agree there is no cumulative breakeven within 24 months. Where they genuinely diverge is the forecast:

The practical result is a controllable quality floor, not guaranteed correctness. The harness made every model ship the required package. Eight produced useful drafts; one demonstrated that completeness checks cannot rescue broken reasoning. Lower-cost entrants can do valuable work inside strong controls, but the controls must preserve enough evidence for an expert to audit the chain.

The accountable expert moves up the stack. Analysts, scientists and reporters spend less time assembling the first package and more time deciding which assumptions matter, replaying the consequential calculations and resolving conflicting evidence. Management shifts from checking whether work exists to checking whether the recommendation is warranted.

For higher-stakes production work, Alfrada's “Beast” workflow adds an integrated critic and judge loop intended to push borderline work toward the high-7s and 8s. That uplift is not claimed as a result of this study because Beast mode was not enabled for the contestant runs.

A benchmark you can't audit is an opinion. Below, per model: every judge's score, the verdicts verbatim, the defect catalogue with its load-bearing tags, the charts each model drew, and a downloadable package — the memos, financial models, simulation outputs and evidence files exactly as each model left them in its workspace, plus the full judge record.

Prefer a portable copy? Download the Markdown export (includes absolute URLs for every chart).

Setup. Runs executed July 17–19, 2026 against a production Alfrada build on a single dev machine, one isolated bot user per model run (separate workspaces, budgets, and history), temperature 0, seed 42 in the case config, $15 budget cap per run. Sub-agent workers were pinned server-side to the model under test; the worker requests each model made are recorded and published in the packages. Judge sessions ran under a separate critic user with read-only tools plus code execution for replay.

Scoring. The 0–10 raw composite is the mean of all seven judge scores across both cases and remains visible for interpretability. Rank is calibrated separately: within each case, every judge is z-scored across the nine submissions; Sol/Luna are averaged into one GPT-5.6 family signal, Fable/Opus into one Claude signal, and K3 and GLM each contribute one signal. Self-judgments are excluded. Gemini's verdicts remain published but receive zero calibrated weight because their 9.2–10.0 range provides almost no discrimination. This correction changes the fairness of the aggregation, not the winner or model order.

Usage accounting. The study processed 129,765,534 input/output tokens across 18 contestant sessions and 126 judge sessions. Contestants cost $48.11; judging cost $133.58; total recorded model and tool spend was $181.70. Per-model leaderboard costs intentionally show contestant spend only so procurement comparisons are like-for-like.

Evidence base. The published table is nine attributable entries from a ten-candidate publication cycle; the invalid Qwen swarm result is disclosed below. It sits on hundreds of internal benchmark and development executions accumulated over roughly three months. Because prompts, models and the harness evolved during that period, those runs are not pooled into the displayed scores. They make the broad pattern less surprising; they do not turn this cross-section into a repeated-trials estimate.

Limitations, plainly. Each displayed estimate remains one pinned result per model per case — scores carry roughly ±0.3 of observed run-to-run noise, small gaps should be read as ties, and defect counts must not be read as failure rates. Provenance reconciles as follows: four model entries use July 17 outputs (Fable, Opus, Luna, Gemini), three use July 18 (Sol, GLM, Qwen), and two use July 19 (K3, Inkling). Provider load varies. Judges are themselves models: the panel's biases are measurable (Figure 2), one judge required an automated correction pass after falsely reporting an empty workspace, and rationales — however well-verified — inherit their authors' blind spots. Family averaging reduces duplicate architecture weight but cannot remove correlated blindness: related judges may miss the same error classes together, and z-scoring calibrates scoring severity, not truth. There was no model-without-harness control and no Beast-mode treatment, so neither uplift should be inferred from these scores. Two rows carry disclosures: Kimi K3's result pairs two adjacent runs (its case 1 and case 2 each completed and were fully judged in back-to-back runs on July 19, after we diagnosed why earlier attempts died — a harness bug, not the model: the stream reader ignored keepalive pings during K3's legitimately long tool-history turns and killed live requests as idle; both paired cases scored 8.4 with complete panels). And a swarm-mode Qwen run was excluded after it delegated every worker to GPT-5.6 Sol against explicit instructions, which made its otherwise excellent output unattributable.

Auditable, not externally reproducible. The public evidence packages let outsiders inspect each contestant's memos, models, simulation outputs and full judge record. Outsiders cannot independently rerun the study because the harness is our production system. We therefore claim auditability of the published artifacts, not external reproducibility. Questions or corrections: community@strategize.inc.