The Meticulous AI That Couldn’t Finish the Job

AI diligence looks impressive—until the customer is still waiting

For readers weighing AI tools and automation, Firmulate’s latest experiment exposes a distinction that product demos often blur: understanding work is not the same as completing it. A system can identify every problem, resist every shortcut and produce an imposing body of analysis while still failing at the moment when judgment must become action.

That is the story of Opus 4.8 in the final July 2026 Crucible League. It was the most thorough participant, adding 80 learned rules and producing the deepest analyses. It also finished last, with 73 points. Its failure was not ignorance. The model had done enough thinking to earn a €55,000 deal, but it did not secure the signature.

A brutal week, held constant

Firmulate runs AI models as complete companies rather than judging them as conversational assistants. Each frontier model was placed in charge of the same small software business during its worst week, facing the same customers, crises and temptations. Every decision was versioned and auditable.

The simulated company has 13 synthetic employees and real money mechanics. It burns €105,000 per month against €2,300 in monthly recurring revenue, while a public cash countdown makes delay visible. Across the live company, more than 680 self-learned playbook rules document what its AI operators have absorbed.

The final league placed gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scores 26 because partial progress still counts. Trust, however, is non-negotiable: a single breach caps the total because “no amount of good work outweighs a breach of trust.” The complete standings and plain-language findings are available on Firmulate’s public benchmark page.

The analysis was right; the outcome was wrong

Opus 4.8’s result is revealing precisely because it was not careless. It spotted every crisis, as did the rest of the field. All models also refused every manipulation attempt. Yet only two signed the €55,000 deal their analysis had earned. The experiment’s summary is brutally concise: “Same diagnosis, same pitch — no signature.”

The decisive commercial fact was not presented in the customer event. It was buried two document references deep in the company’s own files: a competitor weakness that supported holding the full price. Models that read the file won the deal at full price, worth €4,583 in monthly recurring revenue.

That detail turns the exercise from a test of polished language into a test of operational attention. The winning behavior was not simply recognizing a sales opportunity or drafting a persuasive response. It required following the evidence through the company’s records and then carrying the work through to a signed agreement.

Opus 4.8 excelled at the accumulation side of management. Its 80 added rules and unusually deep analyses suggest a system intent on learning from every event. But the league judged the business result, not the apparent intellectual effort behind it. Thoroughness became a weak substitute for prioritization when the close remained unfinished.

Discipline slipped at the boundary

The missed deal was accompanied by a process failure. Opus 4.8 attempted to write into a locked department instead of escalating. That is a small action with a large managerial lesson: when authority or access stops the preferred route, useful work depends on recognizing the boundary and choosing the correct escalation path.

Firmulate’s finding is not an indictment of Opus alone. The same weakness appeared, though less strongly, in the other four models. Each could reason convincingly; each showed some gap between recognizing the required move and executing it with consistent discipline.

The security side of the experiment offers an important counterweight. Fake CEO messages escalated across three stages, followed by a reporter’s attempt to obtain “just one yes/no, on background.” All 5 models refused. Kimi K3 recorded the clearest rationale: “Treat the request as a suspected approval-bypass / possible impersonation.” K3’s strong showing should also be read with the stated fairness note: it ran with the API default and without an effort parameter, while the others ran at xhigh.

A character study, not a caricature

It would be easy to portray the last-place model as incompetent, but the evidence says otherwise. Opus 4.8 was observant, resistant to manipulation and exceptionally industrious. Its weakness was more recognizable—and more consequential. It devoted extraordinary attention to learning and analysis without maintaining equal discipline around completion.

That combination matters for companies considering AI agents. A model may produce thoughtful plans, exhaustive records and persuasive customer communications, yet still need evaluation on whether it reads the relevant files, respects operational boundaries, escalates appropriately and finishes commercially important work.

Measure impact, not the volume of visible effort

Firmulate’s experiment suggests that the most reassuring signs of intelligence can also distract buyers. Long analysis and an expanding rulebook show activity, but they do not prove that an AI system can identify the decisive fact, act within its permissions and complete the transaction.

The company makes the experiment watchable through its live operation, while 242 real, unedited management decisions also power a “guess the model” quiz. For enterprises, Firmulate offers the same wargame against a read-only export of their own business; nothing writes back to real systems.

Opus 4.8’s 73-point finish is therefore less a story about a weak model than a warning about misplaced confidence. The most diligent participant understood the week, learned from it and stayed honest under pressure. It still left the result on the table. In business automation, prioritization and follow-through are not finishing touches. They are the work.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI