We're Grading Ourselves on Harvey's Benchmark. Here's How We Keep From Cheating.

Our first run scored 72 out of 76. It failed.

The task was a synthetic tax opinion — a corporate spin-off under §355, drawn from Harvey AI's Legal Agent Benchmark. Jubal read the deal documents, worked through the structure, and produced a genuinely competent opinion letter. It also left six citation placeholders where authorities should have been, skipped an opinion paragraph a tax lawyer would expect to see, and addressed the letter to the general counsel instead of the board. Under the benchmark's scoring, any missed criterion fails the task. Seventy-two of seventy-six is an F.

That's the right kind of hard, and it's why we're building this benchmark into jubal.law.

Credit where it's due

Late last year, Harvey open-sourced LAB: roughly 1,660 legal tasks across 25 practice areas, each with a realistic document set — lawyer-reviewed and fully synthetic, no client data — and a rubric of pass/fail criteria written by lawyers. A typical task comes with eight documents and fifty-seven criteria. The whole thing is MIT-licensed.

This was a generous thing to do, and we want to say so plainly. Legal AI has a measurement problem: every vendor claims accuracy, and almost no buyer has a way to check. A law firm evaluating tools like ours shouldn't have to take anyone's word for it — including ours. Harvey publishing a serious, open yardstick makes the whole category more honest, and we think the right response is to use it, credit it, and be transparent about how we did.

So that's what we're doing. Our fork is public at github.com/jubal-inc/legal-benchmarks, pinned so a score means the same thing months from now. To be clear: Harvey has no affiliation with us and doesn't endorse jubal.law. We're just using what they published, under the license they published it with.

The benchmark runs inside the product

Most benchmark results are produced in a lab harness — a stripped-down script that feeds the model the documents and collects an answer. We decided not to do that, because a harness measures the model. Our customers don't buy a model; they buy the whole system.

In jubal.law, a benchmark case becomes a real matter. The documents go through the same ingestion pipeline as a client's documents — same extraction, same indexing, same everything. The same agent works the same way it would on real work, and the deliverable passes through the same validation gates before it ships. Then Harvey's own evaluation code grades it, unmodified. We didn't port their judge or "adapt" their scoring, because a benchmark you've reimplemented is a benchmark you've quietly changed.

This means a score regression might be an ingestion bug rather than a reasoning failure. Good. We want to catch those too. When our extraction code improves or a new model ships, re-running the benchmark measures the system our customers actually use that day.

One thing we were careful about: benchmark matters are walled off by design. They live under a designated demo client, they're visibly labeled everywhere they appear, and the isolation is enforced in the backend, not just the interface — a synthetic case can never write to firm-level memory, never link to a real matter, never touch anything a client would see. Synthetic data and privileged client work do not mix, full stop.

How we keep from cheating on our own test

Here's the uncomfortable truth about self-administered benchmarks: any team that optimizes against a rubric long enough will start learning the rubric instead of the work. It's not dishonesty; it's gravity. So we built the discipline in as architecture, not policy.

When a run fails, an analysis agent studies the transcript and the deliverable and proposes improvements to the product. That agent never sees the rubric. It gets a description of what went wrong — "citations left as placeholders," "missing a standard opinion section" — never the criterion text itself. You can't teach to a test you're not allowed to read.

We also split the cases. The improvement work only ever touches a development subset; a held-out set gets scored periodically and is never shown to anything that proposes fixes. If our development scores climb while the held-out scores stay flat, we're gaming, not improving — and we'll know.

And every proposed fix has to generalize. "Verify every citation against a real authority before the document ships" is a fix. "When the task mentions §355, add a §361(a) paragraph" is memorizing an answer, and it gets rejected.

What that first failure bought us

Those four misses on the tax opinion each became something real: a citation-resolution pass so the agent finishes the lookup instead of leaving a placeholder, verification against public authority databases so citations are checked rather than trusted, completeness checklists per document type, and a hard delivery gate that refuses to ship any document still containing a placeholder. None of those fixes knows anything about tax law or that specific rubric. All of them make every matter better.

That's the trade we're after. The benchmark isn't a marketing scoreboard for us — it's a machine for finding out where we're weak, in public, on someone else's terms.

What comes next

We're building toward a measured pilot across a sample of practice areas, and we expect to fail a lot of tasks. All-pass scoring is brutal by design — one missed criterion out of fifty-seven fails the task — and we'd rather share honest numbers with honest framing than impressive numbers with an asterisk. When we have results worth publishing, we'll publish them, along with what we changed because of them.

If you run a small or mid-size firm and you're evaluating AI tools: ask every vendor how they measure themselves, and ask what they do when they fail their own test. The answers are more revealing than any demo.

---

Harvey's Legal Agent Benchmark is available at github.com/harveyai/harvey-labs under the MIT license. Harvey AI is not affiliated with and does not endorse jubal.law.