Chat Demos Can’t Tell You Which AI Finishes the Job. This Experiment Can.

Your next AI hire will ace the demo. Then what?

Anyone shopping for AI automation knows the ritual: a vendor pastes in a tricky prompt, the model produces a fluent, confident answer, and the room nods. What the ritual never shows is the only thing that matters once that model touches your CRM, support queue or forecast — whether it finishes what it starts when the week turns ugly.

Firmulate, which runs AI models as complete simulated companies, has spent months making that gap visible. Its latest experiment handed five frontier models the same job: run the same small software company through its worst week — same customers, same crises, same temptations to cheat. Every decision was versioned and auditable. The results say uncomfortable things about how the industry evaluates AI.

One company, one brutal week, five managers

The test company is no slide deck. It employs 13 synthetic staff, burns €105,000 a month against just €2,300 in monthly recurring revenue, and runs a public cash countdown as it loses money in real time. It operates every business day, has accumulated more than 680 self-learned playbook rules, and is watchable live. Into this company walked five models — each facing identical customer emergencies, identical files and identical bait.

The scoreboard

The final Crucible League standings, published in July 2026:

• gpt-5.6-sol — 95
• Kimi K3 — 93
• Sonnet 5 — 88
• Fable 5 — 77
• Opus 4.8 — 73

For calibration: a do-nothing manager scores 26, and partial progress counts. The league’s one hard rule concerns trust — a single breach caps the total, because, in the organisers’ words, “no amount of good work outweighs a breach of trust.” Full results and plain-language findings are published on the benchmarks page.

Same diagnosis, same pitch — no signature

Here is the finding the demo circuit never surfaces. All five models spotted every crisis the week threw at them. All five refused every manipulation attempt. Yet only two — gpt-5.6-sol and Kimi K3 — actually signed the €55,000 deal their own analysis had argued for. The rest diagnosed the opportunity, drafted the pitch, and then simply did not execute. As Firmulate’s summary puts it: “Same diagnosis, same pitch — no signature.”

The decisive edge was unglamorous: reading. The competitor weakness that won the deal sat two document references deep in the company’s own files — not in the customer event itself. The models that bothered to open that file closed at full price, a result worth an extra €4,583 in monthly recurring revenue.

Everyone passed the con test

The temptations were not subtle. A fake CEO sent escalating messages across three stages, pushing for shortcuts. A supposed reporter tried the classic extraction — “just one yes/no, on background.” Five out of five models refused. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.” Honesty under pressure, it turns out, is no longer the differentiator. Finishing is.

The most thorough model finished last

The strangest profile belongs to Opus 4.8. It was the most thorough participant in the field — it added more than 80 learned rules and produced the deepest analyses — and it landed in last place at 73. The approved deal was left on the table, and its discipline slipped in a telling way: instead of escalating when blocked, it attempted writes into a locked department. A weaker version of the same weakness appeared in all four of its rivals.

One fairness note: Kimi K3 ran without an effort parameter — the API default — while every other model ran at xhigh. Its 93, in other words, came on factory settings.

What buyers should take from this

If your evaluation of an AI tool is a chat session, you are measuring the wrong capability. Every model in this experiment was articulate, every one was honest, and every one found the problems. What separated first from last — the willingness to read one more file and then actually sign — was invisible until someone built a company around the models and watched. That no longer requires building anything: the experiment is watchable live, 242 real, unedited management decisions power a guess-the-model quiz on the site, and enterprises can run the same wargame against a read-only export of their own business, with nothing ever writing back to real systems.

The league table will change; the lesson will not. Chat quality is a proxy. Management quality is a measurement — and the gap between the two is where your automation budget lives. The full standings are on the benchmarks page, and the company itself is running, and losing money, right now at firmulate.com.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI