Zhipu AI Releases Open-Source Legal Model GLM-Legal, Tops Global Benchmarks

Wen Ting reporting from Ao Fei Si Quantum Bit | Official Account QbitAI

Wow, Zhipu has brought the vertical model competition to the legal industry!

Just now, Zhipu released a new open-source model specifically for legal use—GLM-Legal.

It doesn't just chat about legal provisions; it can handle specific tasks such as contract review, evidence correlation, and cross-border compliance research, delivering results that are ready for immediate use.

The performance is also impressive.

In Vals AI's latest legal benchmark evaluation, GLM-Legal topped both the Harvey LAB and LegalBench leaderboards for open-weight models, achieving a 19.2% full task pass rate and an 86.23% accuracy rate.

Alright, after dominating chat, office work, and coding, AI is now targeting the intricate details of legal contracts and case files.

Upon hearing the news, users immediately couldn't sit still:

Great, great, great. From now on, GLM-Legal is my legal team.

But wait, which country's laws does it actually study? Can it really help us Chinese users get the job done?

Let's dig deeper to find out.

Contract Review, Evidence Correlation, Compliance: GLM-Legal Handles It All

The good news is, it indeed adapts to Chinese law, avoiding any "culture shock."

Whether it's Chinese legal practice, English legal tasks, or Southeast Asian cross-border compliance challenges, it can handle them all.

However, simulated exams are only one aspect.

Real-world legal work is riddled with pitfalls—one misstep and you're silent.

To see if it can truly do the work, we need to put it through real legal working environments, checking if it can help teams review contracts and determine privacy compliance requirements before overseas launches.

Let's first look at three official demos (GLM-5.3-Flash vs. GLM-Legal: A Self-Battle):

1. Review Contracts and Clarify Handover Arrangements After Cooperation Ends

When switching software service providers, we often need to thoroughly review the termination and handover clauses in the contract.

Can code, data, and documents be retrieved smoothly? Can the new provider take over seamlessly?

These risks are often buried within the clauses.

Therefore, GLM-Legal suggested shortening the cooperation termination notice period from 60 days to 30 days and adding up to 90 days of transition support, facilitating the arrangement of code, document, and knowledge handovers.

Additionally, based on existing deliverable ownership agreements in the contract, it required the return of all work products (including those unfinished or unpaid).

Compared to GLM-5.3-Flash, GLM-Legal's advantages are obvious:

The former provided only general suggestions, while GLM-Legal further specified timelines, handover scope, and aligned the modification plan with existing contractual terms.

Well, this "breakup process" after ending cooperation was clearly written by it, very dignified indeed.

This makes me wonder, when I throw a pile of scattered evidence at it, how will GLM-Legal respond?

2. Analyze Evidence and Connect Scattered Clues

This is actually a very realistic problem.

When disputes arise in project cooperation, how exactly can we judge whether the other party's claims are valid amidst massive reports, documents, and chat records?

For example, consider this Case provided officially: External reports claimed the project was 60% complete, but internal records showed only about 41%.

In actual testing, both GLM-5.3-Flash and GLM-Legal could identify this contradiction;

However, the difference lies in the fact that GLM-5.3-Flash only gave suggestions, whereas GLM-Legal, building on those suggestions, further linked progress reports with earlier risk warnings and internal communications, translating them into concrete actions.

Only after clarifying this clue can we conveniently go back to the originals for item-by-item verification and figure out which data is correct and usable.

3. Corporate Overseas Compliance: Organizing Regulatory Requirements into Action Checklists

When preparing to launch websites on overseas platforms like Singapore, Malaysia, and Indonesia, we may face privacy compliance challenges involving user tracking, email marketing, overseas cloud storage, etc.

At this point, we might need an assistant to help clarify what preparations are needed before going live.

In actual testing, GLM-Legal broke down the compliance matters requiring attention into detailed steps that teams can implement and continue to verify, including publishing specific privacy statements, adding unsubscribe requests to suppression lists, etc., making everything clear at a glance.

This makes things much more convenient.

So, looking back along this line of reasoning, we can gain a more intuitive understanding of why Harvey LAB’s 19.2% full-pass rate is so difficult to achieve—and where its true value lies.

How Hard Is the Exam Behind the 19.2% Full-Pass Rate?

Harvey LAB’s evaluations are far from trivial.

It employs a rigorous “all-conditions-must-be-met” mechanism: a model only passes a task if it satisfies every single scoring criterion; missing even one results in failure.

Its task design directly mirrors real-world law firm workflows: give an Agent a requirement and a set of client materials, and ask it to produce a document ready for review.

The initial version of Harvey LAB alone features over 1,200 simulated law firm tasks, covering 24 legal practice areas including M&A, intellectual property, and banking & finance.

This means that from reading materials and invoking tools to delivering results according to strict protocols, the model cannot afford any slip-ups anywhere in the pipeline if it hopes to score well here.

Beyond the scores themselves, there is a striking contrast in testing costs worth noting.

With comparable performance, GLM-Legal was nearly 30 minutes slower than Gemini 4 Argon, yet Gemini’s cost per test was approximately 70 times higher than GLM-Legal’s:

This result perhaps reminds us that when choosing a legal AI assistant, we must consider performance, cost, and time together.

So, how exactly was GLM-Legal trained?

The answer may lie in Zhipu AI’s comprehensive development pathway.

Not Just Teaching AI to Read Law, But Sending It to “Intern” at a Law Firm

Zhipu AI’s growth roadmap for GLM-Legal can be summarized in one sentence:

First send it to “law school,” then bring it to a “simulated law firm” for hands-on practice.

Step 1: GLM-Legal undergoes mid-stage training on legal corpora based on the GLM-5.3-Flash foundation, learning legal concepts, statutes, and analytical methods to build a solid professional foundation.

Step 2: GLM-Legal engages in reinforcement learning across tens of thousands of legal task environments, continuously practicing material analysis, task progression, and deliverable generation.

Meanwhile, the team iteratively refines the Agent execution framework, improving material organization, tool integration, and feedback loops, allowing the model and its working environment to mature together.

Sound familiar? It should!

Isn’t this exactly the leveling-up journey law students go through?!

First, get into law school and grind through textbooks to build internal strength; then dive headfirst into a law firm as an intern during junior or senior year; finally, endure several years of hard work—or even further exams—to evolve from an obscure newcomer into a reliable lawyer.

AI is now walking the very path we once took!

However, GLM-Legal is currently still in its relatively green “intern” phase. We shouldn’t expect it to immediately reach partner-level competence.

After all, Zhipu’s “legal specialist” still has many aspects requiring validation. The 19.2% full-pass rate indicates that under strict scoring conditions, most tasks have not yet been fully mastered, leaving significant room for improvement.

This aligns with Zhipu’s current actions:

Inviting law firms, legal tech companies, corporate legal teams, and independent practitioners to apply for the GLM-Legal Beta, enabling professionals to trial the model, provide feedback, and jointly refine it;

Additionally, Zhipu plans to integrate these capabilities into AutoClaw to drive practical applications.

Wow, this sends a clear signal:

Zhipu is collaborating with the most knowledgeable legal experts to polish GLM-Legal, with a clear goal: to make it a truly powerful auxiliary tool for lawyers in the future.

Whether Zhipu can realize this vision remains to be seen, but it is certainly something worth anticipating.

One More Thing

In fact, Zhipu’s entry into legal models was not a spur-of-the-moment decision.

As early as 2023, Zhipu collaborated with PowerLaw Intelligence, a legal tech company focused on smart contracts, to co-develop and launch PowerLawGLM, a hundred-billion-parameter vertical legal model based on the ChatGLM-130B framework. This model covers scenarios such as legal consultation, contract drafting, and review.

Since then, Zhipu has continued to participate in PowerLaw Intelligence’s strategic financing rounds—including a recent Pre-B round of nearly 80 million RMB—as a long-standing shareholder.

Zhipu is not the only one sending general-purpose models to "study law."

As early as August this year, Harvey partnered with Fireworks to post-train its own legal model—Tenet—using Kimi K3 as the base.

According to Harvey, after specialized training, Tenet completed nearly twice as many Legal Agent Benchmark (LAB) held-out tasks compared to the base model, while keeping costs stable.

These explorations may offer new insights for the industry:

After selecting a strong general-purpose foundation model, continuing to train it on specialized tasks, industry data, and realistic work environments can still push model capabilities further.

In the future, whoever can teach models their "specialized courses" more effectively, build more practical "simulated law firms," and enable them to master skills applicable in real-world workflows will likely be the ones to truly win over users in this sector.

References: [1] https://x.com/Zai_org/status/2109678174468534511?s=20 [2] https://z.ai/blog/glm-legal [3] https://x.com/ValsAI/status/2109678921017168061 [4] https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark?utm_source=chatgpt.com [5] https://media.qimingpian.cn/2023/06/29/bd6b454d1bf2400827f682a6c5e7eacf.html?utm_source=chatgpt.com