🔍 Read the full analysis: OpenAI Software Agent Training: The Ironclad Fine Print To Read on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI described training a frontier model in hosted copies of contract-management software Ironclad, using 11 selected legal, commercial and procurement tasks. GPT-6 Astra met an average 55% of task rubric criteria, while its estimated completion times were simulated and do not show measured customer productivity gains. OpenAI says human oversight remains necessary and is inviting a small number of software companies to explore similar work.
OpenAI said on October 6 that it trained its frontier model GPT-6 Astra on selected legal, commercial and procurement workflows inside hosted copies of Ironclad’s contract-management software, building on concerns about AI agents that skip the fine print. The results point to progress in teaching AI agents to work inside specialist business applications, but the model met an average 55% of evaluation criteria across 11 tasks, and its reported time estimates were simulated rather than measured customer savings.
OpenAI’s post, titled “Advancing computer use with Ironclad,” describes a collaboration with Ironclad, a contract-management software company. Ironclad staff and OpenAI employees familiar with the product selected 11 tasks, including setting up nondisclosure agreements, creating procurement approval processes and updating a reusable contract clause based on a requester’s jurisdiction. OpenAI estimated that an experienced user would take 30 to 40 minutes to complete each task.
Each task was assessed against a rubric of 8 to 50 criteria, depending on its complexity. OpenAI reported that GPT-6 Astra met an average 55.0% of those criteria, compared with 41.6% for GPT-5.6 Sol at high reasoning effort. An internal model used in Astra’s development reached 63.7%. Astra met about 94% of the criteria on one showcase task, but that single result is not the overall score.
OpenAI said Ironclad provided hosted product environments for model practice. It says the synthetic training tasks were built from publicly filed contracts in the U.S. Securities and Exchange Commission’s EDGAR database and filtered to remove personal information. OpenAI also said it did not use its customer data, its internal contracts or non-public Ironclad customer data. Those statements describe the data sources for this work; the post does not establish results across other software or workflows.
OpenAI is training agents inside your software. Read the fine print on Ironclad.
Several AI trackers guessed “Ironclad” was a hardened agent framework. It’s a contract-management software company — and the post describes OpenAI training its frontier model inside a vendor’s real product, then inviting other vendors to do the same.
legal, commercial & procurement — e.g. NDAs, approval flows, jurisdiction clauses
per task, experienced user (OpenAI estimate)
criteria per task — a rubric, not pass/fail
public SEC filings; no customer or non-public Ironclad data
The average share of rubric criteria met — not tasks completed. In contracting, partial credit isn’t partial value: a workflow that skips one required approval is the exact failure the system exists to prevent.
37.0 → 19.2 minutes are “simulated estimates … not measured customer time savings,” per OpenAI’s own footnote. Credit to OpenAI for saying so plainly.
Its hardest customer problems get built into the next frontier model; agents that work well in its product make the product more valuable.
Every improvement makes the model better at operating the vendor’s interface. Taken far enough, the agent becomes the interface.
Averages hide missed approvals.
Narrowest access; no self-escalation.
METR found agents spoofing tool-call records.
Measure the whole loop.
Public filings, not your contracts.
Modest numbers, significant method. A frontier lab is moving from general computer use to training inside specialised business software, with the vendor’s help — agents learning their trade the way people do. Today: just over half of a contracting workflow’s requirements, in simulated time, on 11 research tasks.Software vendors are becoming training grounds for the agents that may one day operate their products for them.
Partial Scores Leave Approval Risks
The reported 55% score is the average share of rubric criteria met, not the percentage of tasks completed successfully. That distinction matters in contract and procurement work, where one missed requirement can invalidate an otherwise polished result. A purchasing workflow might need Finance approval above a spending threshold, a Security review for certain requests and Legal review for nonstandard terms. Meeting two requirements but missing the third could route a purchase around a required control.
OpenAI’s post acknowledges that losing track of a business rule limits what a company can safely ask an agent to do. Ironclad CTO Sunita Verma likewise emphasized preserving the controls teams rely on. For businesses considering agents in systems that handle contracts, spending or customer records, the results suggest that task-specific evaluation and human review matter more than a broad average score. The findings are a research result, not evidence that these workflows can now be handed over without checks.
The collaboration also has implications for software companies. OpenAI is asking a small number of vendors to bring tasks that agents cannot yet complete reliably, subject-matter experts, secure test environments and data suitable for research. That could help models learn product-specific work. It could also make the model more capable of operating a vendor’s application, potentially changing how customers use its interface. Whether that shifts value away from screens and toward data, rules, audit trails and controls is an interpretation, not a reported outcome of this trial.
As an affiliate, we earn on qualifying purchases.
Inside the Ironclad Trial
The project focuses on training models to follow business rules, carry out multi-step work in specialized software and check completed work against the original request. The 11 selected tasks covered legal, commercial and procurement processes rather than general-purpose computer use. OpenAI says the evaluation criteria varied with task complexity, so the average combines tasks with different rubrics.
OpenAI reported estimated completion times of 19.2 minutes for Astra and 37.0 minutes for GPT-5.6 Sol. A footnote says these figures are simulated estimates based on assumed processing and generation speeds. They are not measured time savings for Ironclad customers, and OpenAI says they apply to the 11 research tasks rather than Ironclad workflows broadly. The comparison should not be read as a real-world productivity study.
The post frames the trial as an early example of software companies helping train agents on professional workflows. It also argues that a full contracting platform remains important. That reflects the practical need for business rules and records to remain in force even if users increasingly ask an agent to operate the software on their behalf.
AI legal workflow automation tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Reported Results
The post does not show whether the model’s results will carry over to live customer environments, other Ironclad workflows or products from different software companies. It also does not provide a full breakdown of which criteria Astra missed on each task, making it difficult to judge the specific risks behind the average score. The showcase result of about 94% applies to one task and should not be treated as a typical outcome.
OpenAI’s time figures are simulated, and no measured customer time savings or independent evaluation are described in the supplied material. The post also does not specify the names or timeline of prospective software partners, or the availability of the model for customer use in these workflows. OpenAI’s statements about data sources and exclusions are company-reported details; the post does not describe an outside audit of those practices.
AI-powered document review software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Partner Research and Validation
OpenAI says it is inviting a small number of software companies to work on tasks that current agents cannot reliably complete. Potential partners are asked to provide concrete failure examples, people with deep knowledge of the work, a secure testing environment and data that can safely be used for research.
The next evidence readers should look for is whether OpenAI or its partners publish task-level results, explain how missed criteria are handled, and report performance in settings that reflect real business use. Until those details and measured outcomes are available, the Ironclad results show progress on a bounded evaluation, not verified productivity gains or a basis for removing human review from consequential workflows.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is Ironclad in this announcement?
Ironclad is a contract-management software company. OpenAI’s post describes training work conducted in hosted copies of Ironclad’s product, not a new agent framework called Ironclad.
What does GPT-6 Astra’s 55% score measure?
It is the average share of evaluation rubric criteria met across 11 selected tasks. It is not the share of tasks completed, nor a direct measure of customer satisfaction or production reliability.
Did Astra cut customer task times in half?
No customer time reduction was reported. OpenAI’s figures of 19.2 minutes for Astra and 37.0 minutes for GPT-5.6 Sol are simulated estimates, not measured results from customer workflows.
What data did OpenAI say it used?
OpenAI said it created synthetic training tasks from publicly filed contracts in the SEC’s EDGAR database, filtered to remove personal information. It said it used no OpenAI customer data, OpenAI internal contracts or non-public Ironclad customer data.
Can companies rely on agents to run these workflows without review?
The reported results do not support that conclusion. Astra averaged 55% of rubric criteria, and OpenAI’s post says human oversight still matters when agents may lose track of business rules.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
