TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Z.ai’s GLM-5.3-FlashX-Legal achieved a 19.17% all-criteria task pass rate on Vals AI’s Harvey Legal Agent Benchmark leaderboard, placing seventh among 81 models. The same run passed 92.5% of individual rubric criteria, but the benchmark counts a task as complete only when every required criterion passes.
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on the all-criteria pass measure in Vals AI’s public leaderboard for Harvey’s Legal Agent Benchmark, ranking seventh among 81 models. The result means the model completed every required criterion on roughly one in five benchmark tasks; its individual-criterion pass rate in the same run was 92.5%.
Z.ai describes GLM-5.3-FlashX-Legal as a model for professional legal workflows, including document understanding, legal reasoning, research, contract review and initial drafting. Its developer documentation says the model received legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of simulated legal-workflow environments. The company recommends using it as an agent connected to services for laws, cases and company data.
On its documentation page, Z.ai reports a 19.2% all-criteria pass rate on Harvey LAB, 12.5 percentage points above the base model, and calls the result state of the art among open-source models in the evaluation. Vals AI’s archived leaderboard lists the model at 19.17%, seventh of 81, with a listed test cost of $0.15. The comparison with the base model is supported by the same leaderboard, according to the source material. Z.ai does not specify the comparison set behind its open-source claim, and the source material says it found no published weights for this legal model.
The announcement appeared on Z.ai’s AutoClaw blog on October 13, 2026, alongside an application-based beta; Vals lists the release as October 12. Z.ai’s pricing page lists rates of $0.37 per million input tokens and $1.25 per million output tokens. Its documentation page does not carry a publication date.
Why Z.ai’s Legal AI Stands Near the Top With a Low Pass Rate
GLM-5.3-FlashX-Legal ranked seventh on Vals AI’s Harvey benchmark. Its score shows how often it completed every requirement in a task, not how many individual criteria it passed.
Why the scores differ
Harvey counts a task as complete only when every required criterion passes.
- 92.5% of individual rubric criteria passed
- Criteria include facts, citations, deadlines and formatting
- One missed requirement can fail the task
- Every criterion must pass for completion
What the benchmark measures
The leaderboard tests legal work products under a defined protocol.
Each task gives an instruction and client matter materials.
The agent produces a deliverable for review.
Two language-model judges grade each submission against expert criteria.
What firms should weigh
A leaderboard result is one signal; it does not establish performance across a firm’s own matters.
Built for legal workflows
Z.ai describes legal reasoning, research, contract review and initial drafting as target uses.
Firm-level performance
The supplied results do not show outcomes across firms, jurisdictions or their own files.
Data and verification
Firms need details on client-data handling, access controls and lawyer review.
What the Score Means for Legal Work
The gap between the 92.5% individual-criterion rate and the 19.17% task rate reflects how the benchmark defines completion, rather than a contradiction in the reported results. A task receives a pass only if every rubric criterion succeeds. A missed citation, deadline, recommendation or formatting requirement can therefore make an otherwise strong work product fail the task.
That distinction matters to law firms considering AI for work products that need to be reviewed as a whole. A high rate across separate criteria does not tell a firm how often the system produced a fully compliant deliverable. The benchmark offers one measure of performance under its test conditions; it does not establish how the model performs across every firm’s files, jurisdictions or review procedures.
For procurement, the result also leaves practical questions beyond benchmark ranking: how client data is handled, which rules govern its processing, and how lawyers verify citations and other details before drafts are used. Z.ai’s materials, as described in the source, present outputs as drafts for legal professionals. The source author reports that the Z.ai pages reviewed did not include a “not legal advice” notice.
Products Worth Considering
Bose TV Speaker (2nd Gen) Compact Soundbar
Apple MacBook Air 15-inch with M5 chip, 16GB Memory, 512GB SSD, and Liquid Retina Display
DJI Osmo Mobile 8 3-Axis Gimbal Stabilizer for Phones
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Agents
Harvey released the Legal Agent Benchmark as open source on May 6, 2026. Its public set contains 1,250 tasks across 24 legal practice areas. Each task pairs an instruction with a client matter containing relevant and irrelevant materials, then asks an agent to produce a work product for review.
Harvey says the tasks are graded against more than 75,000 expert-written pass/fail criteria. Those criteria can cover facts, legal conclusions, citations, severity ratings, recommendations, deadlines, amounts and formatting. Harvey’s example change-of-control task includes 57 criteria. The company says a task is complete only if every criterion passes, describing this as “all-pass grading.”
The public leaderboard uses a private holdout set designed to mirror the public task mix. Vals AI describes that holdout as 120 tasks, with internet access disabled. Two language-model judges, GPT-5.5 and Claude Sonnet 4.6, grade each submission; the final score averages their task pass rates. These conditions define what the reported ranking measures and limit how directly it can be generalized to other legal work.
“an all-criteria pass rate of 19.2%”
— Z.ai developer documentation
Limits of the Ranking and Claims
The leaderboard result is a benchmark score, not evidence that the model will pass the same share of tasks in a particular firm’s practice. The source material does not provide details on how performance varies by legal area, task type or jurisdiction, or on how often a lawyer would need to revise its outputs.
Z.ai’s “state-of-the-art among open-source models” wording remains difficult to assess from the information supplied: the company does not name the comparison set, and the source author reports finding no published weights for the legal model. The source author also says a small four-task spot check found citations to rulings that could not be located. That check is not a benchmark and cannot establish a general error rate. The author reports that two Claude models in the same spot check cited a real ruling for the opposite of what it held, underscoring the limits of such a small sample.
The benchmark itself uses a private 120-task holdout and two LLM judges. The source material does not report judge-by-judge results or explain how the leaderboard ranking might change under a different task mix or grading method. Z.ai’s announcement also does not establish how client data is handled when firms connect the model to their own systems.
What Firms and Z.ai May Clarify
Z.ai’s AutoClaw announcement opened a beta by application. The company’s documentation recommends connecting the model to legal and company-data services, so prospective users will need details on access controls, data handling and the review steps expected for generated work products.
Further leaderboard entries or published evaluation details could clarify how GLM-5.3-FlashX-Legal compares with other models under the same protocol. For now, the reported result establishes its position on Vals AI’s listed evaluation and the scoring rule behind it; it does not resolve how well the model will perform in any individual firm’s workflow.
Key Questions
What did GLM-5.3-FlashX-Legal score?
Vals AI’s archived leaderboard lists a 19.17% all-criteria task pass rate on Harvey LAB. Z.ai reports the result rounded to 19.2%.
Why is the task pass rate lower than 92.5%?
The 92.5% figure is the share of individual rubric criteria passed in the same run. Harvey LAB counts a task as complete only when every criterion for that task passes, so one missed requirement can fail the whole task.
Where did the model rank?
Vals AI’s archived leaderboard places GLM Legal seventh among 81 models. The ranking applies to that leaderboard and its evaluation protocol.
Does the score show how the model will perform at a law firm?
No. It reports results on a 120-task private holdout under specified benchmark conditions. The source material does not establish performance across firms, jurisdictions or their own document sets.
What remains uncertain about Z.ai’s open-source claim?
Z.ai calls the result state of the art among open-source models but does not name the comparison set in the material provided. The source author also reports finding no published weights for this legal model.
Source: Thorsten Meyer AI
