🔍 Read the full analysis: Why Z.ai’s Legal AI Benchmark Rank Is Near The Top, Despite Limited Passes on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s Harvey Legal Agent Benchmark leaderboard, placing seventh among 81 models in the archived version. The score counts a task as passed only when every required criterion is met; Vals also reports that the model passed 92.5% of individual criteria. The ranking is a benchmark result, not evidence that the system can independently complete legal work reliably.
Z.ai’s GLM-5.3-FlashX-Legal ranked seventh among 81 models on an archived Vals AI leaderboard for Harvey’s Legal Agent Benchmark, scoring 19.17% on its all-criteria task pass rate. That means it met every rubric requirement on about one in five benchmark tasks, even as Vals reported it passed 92.5% of individual criteria—two different measures of performance.
The benchmark’s all-or-nothing rule explains the apparent mismatch. A task counts as complete only if the model satisfies every required criterion, which can include correct facts, legal conclusions, citations, deadlines, recommendations and formatting. Missing even one criterion makes the whole task a non-pass, regardless of how many other requirements were met. The 19.17% figure is the score shown for GLM Legal in Vals AI’s archived Version 1 leaderboard; Z.ai rounded it to 19.2% in its product documentation.
Vals AI says the leaderboard uses a 120-task private holdout set, with internet access disabled. Two language-model judges assess submissions, and the reported task score averages their pass rates. Vals reports the separate 92.5% criteria pass rate for GLM Legal, counting a criterion only when both judges passed it. That figure is not a measure of how many tasks were completed end to end.
The archived table places GLM Legal ahead of its base model, GLM-5.3-Flash, which scored 6.67% on the all-pass measure. The difference is 12.50 percentage points, matching Z.ai’s claimed improvement. The table lists GLM Legal’s benchmark cost at $0.15 per test; Z.ai’s pricing page separately lists token charges of $0.37 per million input tokens and $1.25 per million output tokens.
What the Ranking Says About Reliability
The result shows why a strong average across individual rubric checks does not necessarily translate into dependable completion of complex legal assignments. When a task contains many requirements, a model can satisfy most of them and still fail the benchmark’s end-to-end test because of one missing or incorrect item. For legal teams, an omitted deadline or unsupported citation may matter even if the rest of a draft is useful.
Rank also depends on what is being measured. Seventh place is a relative position on one leaderboard and one archived version, not a certification of legal competence or a finding that the system is safe to use without review. The benchmark offers a structured comparison, while the 19.17% all-pass rate highlights how rarely the model met every requirement across the tested tasks. Vals lists several higher-scoring models, including some with substantially higher per-test benchmark costs, though those figures alone do not establish which system is best for a particular firm.
That distinction matters for procurement. Firms evaluating the product need to consider output verification, data handling and the work required to connect the model to legal sources and company records. Z.ai describes the system as intended for professional workflows and recommends using it as an agent connected to services for laws, cases and company data. The benchmark does not by itself answer how well those integrations work in a firm’s own environment.
Products Worth Considering
FLEXISPOT 60 x 30-Inch Bamboo Electric Standing Desk with White Frame
HP Sprocket Portable 2×3 Instant Photo Printer, Lilac, with ZINK Paper Bundle
HP OfficeJet 250 Wireless Mobile All-in-One Inkjet Printer
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Agent Tasks
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. It contains 1,250 tasks across 24 legal practice areas. The tasks ask an agent to use materials in a client matter and produce work for review; more than 75,000 expert-written criteria define what a successful answer must contain. Harvey’s grading protocol calls the approach all-pass grading: every criterion must pass for the task to count as complete.
Harvey has argued that partial completion can be inadequate in legal work: its example is a deal report that identifies eight of ten risks but misses the others. In an earlier baseline published May 26, Harvey reported that the best overall all-pass rate among the frontier models it tested was 7.1%, and said the models completed fewer than 10% of tasks end to end in aggregate. That was a different evaluation and should not be treated as a direct comparison with the later Vals leaderboard.
Z.ai’s documentation describes GLM-5.3-FlashX-Legal as a model for document understanding, legal reasoning and task execution, including research, contract review and drafting initial drafts. Z.ai says it underwent legal-domain mid-training and post-training, including reinforcement learning in tens of thousands of task environments modeled on legal workflows. Its AutoClaw blog post announcing an application-based beta was dated October 13, 2026; Vals lists the release as October 12.
“A task is marked complete only if every criterion passes.”
— Harvey, describing its benchmark grading rule
Questions the Benchmark Cannot Resolve
The leaderboard establishes performance under its defined tasks and grading process, but it does not show how the model performs across every jurisdiction, practice area or live client matter. It also does not establish that drafts are legally sound without professional review. The source material describes Z.ai’s product as supporting legal workflows and producing initial drafts; the reported score should not be read as approval for autonomous legal advice.
Z.ai calls the result state of the art among open-source models in this evaluation, but its cited documentation does not specify the comparison set. The source material also says it could not find published weights for the legal model, leaving the basis for describing it as open source unclear. A separate small, four-task spot check reported by Thorsten Meyer found citations to rulings that could not be located in outputs from the Z.ai model; that sample is not a benchmark and cannot establish a general error rate. The available material does not settle how client data is processed, what controls apply to connected services, or what citation checks users must perform before relying on an output.
What Firms Should Verify Before Use
The next practical test is whether the model’s performance holds up in workflows that resemble a firm’s own matters, with lawyers checking factual accuracy, legal reasoning and every citation. Vals’ archived leaderboard provides a reproducible point of comparison, but it does not replace testing on a firm’s approved tasks and sources. Z.ai’s documentation recommends connecting the model to services for laws, cases and company information; firms considering that setup will need to establish what data is sent to each service and under which rules.
The source material does not identify a scheduled update to the benchmark or a further product milestone. The figures and ranking reported here refer to Vals AI’s archived Version 1 table, last updated October 18, 2026. Later leaderboard revisions or more detailed disclosures from Z.ai could clarify how the model compares with other systems, the status of its weights, and the safeguards available for handling client data.
Key Questions
What does GLM Legal’s 19.17% score mean?
It is the share of benchmark tasks on which the model met every required rubric criterion, as shown on Vals AI’s archived leaderboard. A task with even one missed criterion counts as a non-pass.
How can the model pass 92.5% of criteria but only 19.17% of tasks?
The two percentages use different denominators. The criteria figure counts individual checks, while the task score requires every check in a task to pass. A model can satisfy most checks across tasks but miss at least one in many complete assignments.
Was the 12.5-percentage-point improvement independently verified?
Vals AI’s archived table lists GLM Legal at 19.17% and the base GLM-5.3-Flash at 6.67%, a difference of 12.50 percentage points. Z.ai’s rounded claim matches that comparison on the table.
Does the ranking show that GLM Legal can replace a lawyer?
No. The result measures performance on a defined benchmark, not readiness to provide legal advice or work without professional review. The cited product description refers to legal workflows and initial drafts, and the benchmark does not establish performance in every real-world matter.
What should law firms check before using the model?
Firms should test outputs against their own work, verify legal authorities and citations, and determine how client data is handled when the model is connected to external services. The benchmark score does not answer those operational and data-governance questions.
Source: ThorstenMeyerAI.com
