TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Z.ai says its GLM-5.3-FlashX-Legal model achieved a 19.2% all-criteria pass rate on Harvey’s Legal Agent Benchmark. Vals AI’s leaderboard records a 19.17% score, seventh among 81 models; the benchmark gives no partial credit when a task misses any required criterion. Z.ai also reports a 12.5 percentage-point gain over its base model and claims leading performance among open-source models, though the comparison set for that claim is unspecified.
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on Vals AI’s leaderboard for Harvey’s Legal Agent Benchmark, ranking seventh among 81 models. The score means the model passed every required criterion on roughly one in five tasks; a task with even one failed criterion does not count as complete.
Vals AI’s archived leaderboard lists the result at 19.17% and gives a test cost of $0.15. Z.ai’s documentation rounds the score to 19.2% and says the model improved by 12.5 percentage points over the base model. The same benchmark run shows the legal model met 92.5% of individual rubric criteria, according to the source material. These measures describe different things: the first is the share of whole tasks passed, while the second is the share of individual checks passed.
Z.ai describes GLM-5.3-FlashX-Legal as optimized for professional legal workflows, including document understanding, legal reasoning, research, contract review and drafting. Its documentation says the model underwent legal-domain mid-training and post-training, including reinforcement learning in tens of thousands of task environments modeled on legal workflows. Z.ai recommends using it as an agent connected to services for laws, cases and company data. The company’s pricing page lists $0.37 per million input tokens and $1.25 per million output tokens.
The model is built on GLM-5.3-Flash. Z.ai’s AutoClaw blog announced a beta by application on October 13, 2026; Vals lists the release date as October 12. Z.ai says the model’s result represents state-of-the-art performance among open-source models in this evaluation. The source material does not specify the comparison set behind that claim, and it reports that published weights for the legal model were not found.
A top-ranked legal AI passes 1 in 5 tasks
Z.ai’s GLM-5.3-FlashX-Legal ranked seventh on Harvey’s benchmark. Its score counts only tasks where every required criterion passes.
Two scores, different meanings
A high share of checks passed can coexist with a much lower share of complete tasks.
- 92.5% of rubric criteria met.
- Measures performance across separate requirements.
- 19.17% passed every required criterion.
- One missed requirement makes the task incomplete.
How the benchmark works
Agents produce legal work products for review against detailed, binary rubrics.
Each task pairs instructions with relevant and irrelevant client materials.
The agent prepares a work product for review.
A task counts as complete only when all rubric criteria pass.
What buyers should weigh
Treat outputs as drafts
The benchmark does not establish readiness for a firm’s matters or review process.
Check citations and details
A small spot check found citations that could not be located; it does not establish a general error rate.
Review handling controls
The supplied material leaves client-data processing and review safeguards unanswered.
What the Score Means for Legal Work
The result points to a gap between strong performance on individual checks and completing an entire legal assignment without a rubric failure. Harvey’s all-pass rule treats every criterion as necessary: a missed citation, deadline or other required detail can make the full work product fail, even when many other parts are correct. That makes the 19.17% figure a strict measure of complete task execution, not a claim that the model gets only one-fifth of its answers right.
For law firms and legal departments, the distinction matters when evaluating whether an AI system can produce work that is ready for review or use. Z.ai presents the model as a tool for professional workflows and initial drafts, but the benchmark result alone does not establish how it performs in a particular firm’s matters, with its own documents and review process. Buyers also need to consider how client data is handled, which services receive it, and how lawyers will verify citations and other required details before work leaves the organization.
The score is a benchmark result, not a guarantee of legal reliability. A small spot check described in the source material found rulings cited by the legal model that could not be located. In the same four-task check, two Claude models cited a real ruling but described its holding incorrectly, while the OpenAI and Claude models in that check invented no rulings. The check was small and is not a controlled benchmark, so it cannot establish how the systems compare overall.
Products Worth Considering
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
Bose S1 Pro+ All-in-One PA Speaker
DJI Osmo Mobile 7P Phone Gimbal with Mic Mini Transmitter, Infinity Black
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Agents
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. The public set contains 1,250 tasks across 24 legal practice areas, with more than 75,000 expert-written rubric criteria. Each task pairs an instruction with a client matter containing relevant and irrelevant materials, then asks an agent to produce a work product for review.
The rubric criteria are binary checks covering matters such as facts, conclusions, citations, severity ratings, recommendations, deadlines, dollar amounts and formatting. Harvey says a task counts as complete only when every criterion passes. Its example of a change-of-control task contains 57 criteria, illustrating how one missed requirement can determine the outcome for an otherwise extensive response.
Vals AI’s public leaderboard uses a private holdout set that it says mirrors the public benchmark mix. The source material describes that holdout as containing 120 tasks, with internet access disabled. Two language-model judges grade each submission, and the reported score averages their task pass rates. Z.ai’s announcement and developer documentation are the sources for its training and performance claims; the leaderboard is the independent result cited for the ranked score.
“On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%, an improvement of 12.5 percentage points over the base model, achieving state-of-the-art performance among open-source models in this evaluation.”
— Z.ai, in its developer documentation
Open Questions on Reliability and Access
The result does not show how GLM-5.3-FlashX-Legal performs across every kind of legal matter or in ordinary firm workflows. The source material does not provide a breakdown of the model’s score by practice area, nor does it specify the comparison group for Z.ai’s claim of state-of-the-art performance among open-source models. It also reports that no published weights for the legal model were found, leaving its availability for independent deployment unclear.
The reported four-task citation check is too small to establish a general error rate. It does raise a practical verification question, but it should not be read as a benchmark or broad comparison with other systems. The source material also says the Z.ai pages reviewed describe outputs as drafts for legal professionals and did not carry a “not legal advice” notice; it does not establish what notices or safeguards may appear in other product materials or deployments.
Details about client-data processing, applicable rules and the full review controls available to customers are not provided in the supplied material. Those details matter for organizations considering use with confidential files, but the cited benchmark and model announcement do not answer them.
What Buyers and Evaluators Need
The next useful evidence would include results broken down by legal practice area, clearer documentation of the open-source comparison set, and more information about how the model handles citations and other high-impact requirements. Buyers can also look for product details on data processing, retention, access controls and the review steps expected before a draft is used.
Z.ai’s beta was opened by application, according to its October 13 announcement. The available source material does not give a date for wider access or describe a planned follow-up benchmark. For now, the leaderboard provides a ranked result under Harvey’s specific all-pass grading rule; how that result translates to supervised legal work remains to be established.
Key Questions
What does GLM-5.3-FlashX-Legal’s 19.17% score measure?
It is the share of benchmark tasks on which the model passed every required rubric criterion, as listed on Vals AI’s leaderboard. A task that misses even one criterion does not count as a pass.
Why is the individual-criteria score much higher?
The source material reports that the model met 92.5% of individual criteria in the same run. That measures separate checks across the tasks; the 19.17% score counts only tasks where all checks passed.
How did the model rank?
Vals AI’s archived leaderboard lists GLM Legal at 19.17%, seventh among 81 models. The listed test cost is $0.15.
Does the result prove the model is reliable for legal work?
No. It is a result on a particular benchmark and does not establish performance across all legal matters or replace review by legal professionals. The source material describes a small citation spot check, but that check is not large enough to establish a general error rate.
What does Z.ai claim about the model’s performance?
Z.ai says the score improved by 12.5 percentage points over the base model and calls it state of the art among open-source models in this evaluation. The source material does not specify the comparison set for the latter claim.
Source: Thorsten Meyer AI
