How to choose models for legal document processing: From PDF fidelity to clause reasoning

This article is an English translation of Han (Andrew) Zheng’s original analysis, lightly edited for clarity and source currency.

The selection of a model for legal document processing often starts with a specific question: Is it possible to find an open-weight model that can replace Gemini 3.8 Flash to handle the understanding, conversion, and summarization of complex legal documents?

This question cannot be answered simply by comparing the total scores of a few models.

Converting PDFs to HTML, extracting case information, parsing legal citations, summarizing the reasoning of judgments, and analyzing conditions and exceptions in contracts are distinct tasks. Just because a model excels at one of these tasks does not mean it can replace the entire processing workflow.

We believe that when selecting models for legal documents, one should compare combinations of tasks, input methods, models, and verification mechanisms, rather than searching for a general 'model that understands law best'.

1. Define the task first: what exactly should the model process?

Before choosing a model, you should clearly define the acceptance criteria for the output.

For example, a conversion specification that requires 'preserving the original text character-by-character, restoring paragraphs, stitching cross-page footnotes, and parsing full and short citations' sets out a specific set of engineering requirements. It may show that the task is complex, but it cannot prove that the model accepting this prompt already possesses these capabilities.

This distinction directly affects model selection:

• If the goal is full-text faithful conversion, prioritize evaluating document parsing and long-output integrity

• If the goal is contract analysis, prioritize evaluating evidence retrieval, conditional reasoning, and legal text capabilities.

Adding 'legal reasoning' to the former task does not necessarily yield benefits; providing a seemingly clean OCR text for the latter task does not mean the problem has been solved.

2. Key challenges: verify the input before testing three core challenges

The three core difficulties of legal documents can be summarized as:

• Cross-page citation

• Logical reasoning between clauses

• Precise analysis of technical terminology

However, before discussing them, a basic premise must be verified:

Is the content provided to the model by the system complete, is the sequence reliable, and do the position markers correspond to the original document?

We cannot assume this premise is already established.

1. Cross-page citation: page-number tags do not prove an understanding of cross-page relationships

It is necessary to distinguish between three levels.

• The first level is location recording. A certain piece of text is marked by the tool as the 12th page of the PDF.

• The second level is structural recovery. Whether this text belongs to the main body, footnotes, or an appendix; and whether it continues from the previous page.

• The third level is semantic association. Where exactly does this "see aforementioned provisions" point to; and which case does this short citation refer to.

These three levels cannot replace one another.

Suppose page 12 of a document is printed with 'Page 8,' and contains body text, footnotes, and the start of an attachment. A model reiterating 'Page 12' only demonstrates that it used a label; it does not prove it correctly understood the numbering on the page, much less that it knows where the footnotes should be attached.

Public benchmarks are not entirely devoid of sequential involvement. OmniDocBench explicitly includes PDF intra-page parsing and reading order metrics; however, intra-page order evaluation cannot directly prove that page number mapping, cross-page footnote continuation, and citation disambiguation in entire legal documents have been reliably resolved. (GitHub)

Therefore, the following capabilities should be listed as separate test items when selecting legal documents:Whether the page position is correct, whether the content continuity is correct, and whether the reference objects are correct.

Here, it is especially necessary to avoid a misattribution: attributing the success entirely to the model just because the pre-processing program provided the correct association and the final answer was also correct.

2. Clause reasoning: finding relevant paragraphs does not mean selecting the correct rule

Below is afictional contract test case:

The main text stipulates that certain notices must be issued within 30 days of an event. The appendix stipulates that for events occurring after July 1, this period is changed to 60 days. Given an event occurred in August, and the notice was issued on the 45th day.

The model needs to process the main text, the appendix, the time conditions, and the specific facts simultaneously.

• Only finding the main text can lead to an incorrect answer.

• Finding the attachment but missing the applicable time frame can also lead to an incorrect answer.

• A more subtle error is: the model correctly determines that the notice does not exceed the agreed-upon period, but then proceeds to write 'therefore all compensation conditions have been met'.

Tasks of this kind should be checked individually for:whether the basis is complete, whether the rules are selected correctly, whether the conditions are satisfied, and whether the conclusion goes beyond the evidence.

The task design of ContractNLI provides a useful reference: it requires distinguishing whether a contract "supports," "refutes," or "does not mention" a proposition and annotating the basis for that decision; the research also lists negation through expressions of exceptions as a difficulty. This design is more suitable as a testing starting point than requiring models to forcibly answer "yes" or "no" to all questions. (Stanford NLP)

3. Terminology precision: definitions and scope matter as much as recognition

Precise parsing in legal documents involves at least three types of issues.

• First, there are characters and symbols.§ 8.3(a)(ii) and § 8.3(a)(iii), or "including" versus "excluding," are not negligible approximations.

• Next is the definition within the document. Assuming a contract explicitly classifies a certain type of subcontractor as a "service provider," the model should use this local definition instead of answering based on common parlance.

• Finally, there is the attribution of identity. "The plaintiff claims to have fulfilled their obligations" and "the court has determined that obligations have been fulfilled," even if they contain the same keywords, cannot be extracted as the same fact.

Therefore, terminology testing should not only ask "what does this word mean," but also:

At this point in the document, which definition is it using? Who is proposing it? Is it subject to any qualifying conditions?

4. Fidelity conversion has another independent risk: rewritten or truncated output

Full text conversion and summarization have opposing acceptance goals.

Summarization allows for compression, while fidelity-based conversion requires preservation. In a fidelity task, a more fluent piece of text might actually mean the model modified the original text without authorization.

Furthermore, input limits and output limits must be checked separately. Taking the current Gemini 3.8 Flash as an example, the officially listed input limit is 1,048,576 tokens, while the output limit is 65,536 tokens. Being able to receive very long materials does not mean that one can completely reconstruct a document of the same length in a single response. (Google AI)

Full-text conversion depends on final coverage and completion status, rather than just how the first few pages look.

3. Which model families are available?

The current model landscape can be divided into four groups: general hosted models, open-weight multimodal models, specialized document parsing models, and legal text models.

1. General-purpose hosted models: end-to-end baselines across cost tiers

The value of these models lies, first and foremost, in their ability to establish a clear external baseline.

2. Open-weight multimodal models: primary candidates for localization and replacement tests

The preferred subjects for experimentation here are not necessarily the models with the largest number of parameters.

3. Specialized document parsers: essential for full-text conversion

If the task focus is to restore content from PDFs, these candidates should be compared separately, rather than just testing chat models.

These tools may output page numbers, coordinates, block types, or reading order.The existence of output fields only indicates that the system provided this type of result; whether the fields are correct still requires verification against the original document.

For legal documents, tests for footnotes, citation-dense areas, adjacent case boundaries, and appendix numbering should be especially prioritized. Generic document parsing scores cannot cover all of these issues.

4. Legal text models: valuable, but not synonymous with PDF models

Tongyi Farui’s farui-plus is a typical example. The current official API documentation lists: text input, text output, a maximum input of 12,000 tokens, and a maximum output of 2,000 tokens; it does not support Function Calling, structured output, or web search. It can be used for legal text task comparisons, but it is not suitable as a direct replacement for full-text conversion of long raw PDFs. (Aliyun Help Center)

LegalOne provides a verifiable open-weight path: including 1.7B, 4B, and 8B versions, based on the Qwen3-Base series and trained in the Chinese legal domain, with CSHaitao/LegalOne-8B already having a public model card and usage instructions. It should be classified as a legal text model, rather than a scanned document visual parsing model.(Hugging Face)

What these two types of models are suitable for answering is:

Given that the text has been correctly extracted and the necessary materials have been provided, has domain training improved performance on specific legal tasks?

instead of:

As long as the name contains Legal or Law, it can understand complex PDFs.

4. What can public benchmarks actually tell us?

1. High document-Q&A scores do not imply reliable full-text conversion

The original task of DocVQA is to answer questions based on document images, where the answers come from text snippets within the image. It does not require the model to verbatim reconstruct the entire document.(DocVQA)

Therefore, a model can correctly answer 'What is the date of the contract' while missing other paragraphs. For that specific QA task, the answer may be correct; for full-text conversion, the task still fails.

Do not interpret DocVQA scores as full-text OCR accuracy, much less as the overall delivery rate of legal documents.

2. Specialized parsing models are worth testing, but metric names matter

The table below reproduces selected results from the official OmniDocBench v1.6_full snapshot, preserving the model names and versions shown in that release. The repository has since advanced, so these figures should be read as a versioned snapshot rather than a current leaderboard. (GitHub)

For page parsing tasks, specialized models deserve a place in the candidate pool; they should not be excluded simply because general-purpose models are larger.

3. Legal post-training can help, but the gains have task boundaries

The LegalOne paper reports that, in the authors' test setup, the 8B official Qwen3-Thinking baseline achieved a JEC-QA score of 44.61, while LegalOne-8B reached 62.36; for LexEval, the scores were 60.06 and 71.19, respectively. The paper also notes that model-based judges were used for some generation tasks. (arXiv)

These results provide a research basis for the idea that "domain training can improve the performance of smaller models on legal text tasks."

However, it does not prove that LegalOne can accurately recover scanned pages, handle cross-page footnotes, or outperform every model released since September 2026. Progress in legal text evaluation cannot be automatically translated into progress in end-to-end legal PDF processing.

5. How to design useful model evaluations

Model documentation can only form a candidate pool. To decide whether to replace an existing solution, a set of tests capable of pinpointing the causes of failure is still required.

1. Test the complete system and controlled inputs

It is recommended to set up at least three test routes.

• Route A: Actual production inputs. Feed the same raw PDF to each candidate interface and record the complete processing results for each. This answers "which solution is more suitable for the business."

• Route B: Standardized page inputs. Provide the same set of rendered pages to multimodal candidates, with fixed resolutions, page collections, and permitted tools. For interfaces that only accept files, note that full control over preprocessing is not possible. This is closer to comparing the visual models themselves.

• Route C: Human-verified text and essential structures. Eliminate obvious recognition errors, then test extraction and reasoning. This is used to determine if bottlenecks still exist on the model side.

The results from the three routes cannot be combined into a single overall ranking.

Assuming a model performs poorly on Route A but shows significant improvement on Route C, one can only infer that input recovery might be a major bottleneck; it cannot be said that its legal reasoning ability has been proven as a result.

Conversely, if the correct evidence has already been fully provided and the model still misapplies exceptions or broadens the conclusion, it more clearly points to a problem with rule application.

2. Use minimal-difference tests

A small-scale test set can be built from real, authorized business documents, and then controlled variants can be added. The following is a proposed test design, not the existing model capability.

This set of tests examines three properties:stable to irrelevant changes, sensitive to decisive changes, and retaining uncertainty for cases that cannot be determined.

Test labels should be generated from the original document and task specifications, rather than simply having another model freely generate 'standard answers'.

3. Do not track average accuracy alone

For legal documents, it is important to focus on the following types of metrics:

• Full-text fidelity indicators. Differences in characters outside of permitted transformations, paragraph overwriting, missing characters in key locations, and whether the full text is complete. Permitted operations such as line break joining and quotation mark conversion must be specified in advance; rewrites cannot be arbitrarily explained as formatting cleanup.

• Correlation indicators. Whether page number mapping, footnote binding, referenced objects, and document boundaries are correct. Citations that 'actually exist' and citations where the 'object is correct' must be treated separately.

• Reasoning indicators. Whether conditions and exceptions are handled, whether key justifications are complete, whether conclusions are supported, and whether judgment is correctly suspended when evidence is insufficient.

• Overall task pass rate. When a task requires preserving the full text, correctly handling footnotes, and outputting in a specified structure, all criteria must be met simultaneously, rather than averaging the three dimensions to produce a score that looks good.

• Automated processing coverage. A system might achieve high accuracy by answering only a small number of the easiest questions; however, this does not mean it has solved the majority of the business workload.

4. Fix the configuration, or you are comparing different experiments

For each evaluation, record at least the model ID or weight revision, inference framework version, precision and quantization, thinking configuration, input preprocessing, image slicing, maximum output, tool permissions, retry strategy, and call date.

This is not just a formal requirement. For example, MiniCPM-V-4.6 offers different downsampling methods and slicing settings, which change the number of visual tokens and how details are preserved. Recording only the model name makes it impossible to reproduce the test conditions. (Hugging Face)

Also, do not assume that higher reasoning investment is necessarily better for faithful translation. Treat it as an experimental variable to see whether it actually improves critical errors or merely increases latency and output.

6. Practical selection: which experiments should come first?

In the absence of unified empirical testing, the reasonable conclusion iswhich candidates to test first, why they are being tested, and what criteria are used for elimination, rather than declaring an overall winner.

Scenario A: Legal PDF → High-fidelity HTML / Markdown

It is recommended to first compare:

Existing Gemini 3.8 Flash path, Qwen3.8-27B path, and PaddleOCR-VL-1.6 or MinerU2.5-Pro parsing paths. These candidates provide three verifiable implementation foundations: hosted general-purpose models, deployable general-purpose VLMs, and specialized parsing systems. (Google AI)

The experiment focuses on source-text integrity, key characters, footnotes, and document boundaries, rather than whether the summary reads smoothly.

If you bring in a model for semantic tag annotation, this step should be scored separately. You cannot assume that subsequent tags and citation relationships are also accurate just because the parser’s character recognition is accurate.

Scenario B: after you already have trustworthy text, do extraction, summarization, and limited reasoning

A useful comparison can include Qwen3.8-27B, Claude Sonnet 5, and GPT-6.1 Terra across different deployment and service models. For specialized Chinese legal tasks, add LegalOne-8B or farui-plus. These are candidates for controlled evaluation, not a performance ranking.

The most important thing to observe at this time is whether the domain model truly reduces terminology and conditional errors, or if it merely produces phrasing that resembles legal opinions more closely.

English contracts, Chinese judgments, and materials from other jurisdictions should be reported in groups. The gains from Chinese legal training cannot be directly extrapolated to other languages and jurisdictions.

Scenario C: Complex clause reasoning, willing to pay higher invocation costs

For higher-cost reasoning tests, begin with GPT-6 Astra, Claude Opus 5 or Claude Fable 5, and Gemini 3.1 Pro Preview. Depending on provider availability and deployment constraints, extend the comparison to additional candidates such as Kimi-K3, GLM-5.3-Flash, and DeepSeek-V4.1-Flash. Verify current model IDs and access terms with each provider before testing. The purpose is to determine whether harder samples improve under greater reasoning investment, not to appoint a final authority for automated correction.

If critical information is missing from the input, more expensive inference calls cannot conjure real evidence out of thin air.

Scenario D: Strictly localized, resource-constrained

It is recommended to narrow the scope of the task first, rather than requiring a small model to handle an entire complex document from the start.

You can test specialized OCR, local field extraction, and smaller multimodal or legal text models separately.Passing a page-level task does not mean it can be directly scaled up to full document-level automated processing.

Department assessments should also verify weight and code licensing, actual runtime dependencies, external calls, and data storage methods. Open weights only mean that model parameters can be obtained; it does not automatically mean that the entire process satisfies localization requirements.

7. How should costs be calculated?

Legal document tasks are not suitable for comparison solely based on 'cost per million tokens'.

We recommend calculating:

Cost per accepted task = (parsing + inference + retries and recovery + human review + operations and storage) / number of tasks that pass acceptance.

At the same time, report the automated processing coverage to avoid creating a low error rate by relying heavily on human intervention.

For full-text conversion, we also need to measurethe time taken to deliver the complete result. The first token is fast, but subsequent output is long and frequently truncated, requiring multiple completions, which is not necessarily a more efficient solution.

For self-deployed models, VRAM, visual input budget, context length, concurrency, and quantization configurations should be tested together. Running low-concurrency, short-sample tests only proves it works under those conditions.

Model replacement is also best done gradually: first replace a single, independently verified component, observe the failure distribution, and then decide whether to expand the scope. You cannot infer which part the gains come from based solely on final average scores after simultaneously changing OCR, chunking, model, and prompts.

Conclusion: choose the system that passes acceptance testing—not the one with the strongest label

Currently, there are several model routes worth verifying for legal document processing: hosted multimodal models, self-deployable general VLMs, specialized document parsing models, and legal domain text models.There is no shortage of candidates, but they solve different problems.

A truly useful selection sequence should be:Define tasks first, then verify inputs; establish a baseline first, then locate errors; satisfy quality and coverage requirements first, then compare delivery costs.

Cross-page citations, clause reasoning, and professional terminology remain three key directions worth prioritizing for testing. However, they should not be framed as established capabilities that 'the model can already perform, but is just occasionally unstable.'

Being able to receive a PDF is a fact of the input interface; being able to output page numbers is an output behavior; being able to correctly restore structures, correlate citations, and apply relevant rules requires separate test evidence for each.

For legal document processing, the final answer to model selection is not 'which name is the most advanced,' but:

Under these types of documents, this set of input conditions, and this set of acceptance rules, which model or combination of models can deliver verifiable results with an acceptable error rate, coverage, and cost?

Frequently asked questions

Can one model replace an entire legal PDF workflow?

Usually not without separate evidence for each stage. PDF recovery, structural reconstruction, evidence retrieval, clause reasoning, and summarization have different failure modes and should be evaluated independently before any end-to-end replacement decision.

Should teams evaluate the parser or the language model first?

Start by identifying where the current errors occur. Compare raw-PDF results with standardized page inputs and human-verified text. If performance improves sharply on verified text, document recovery is likely a major bottleneck; if errors remain, model reasoning or task design deserves closer attention.

Why are document-Q&A benchmarks not enough?

A system can answer a question correctly while still omitting other paragraphs, footnotes, tables, or appendices. Question answering measures whether the requested answer can be found; full-document conversion requires coverage, fidelity, structure, and traceability across the entire document.

How should teams compare the cost of legal-document systems?

Use cost per accepted task rather than token price alone. Include parsing, inference, retries, recovery, human review, operations, storage, and the share of work that can be completed automatically under the agreed acceptance criteria.

Original article

Read Andrew Zheng’s original analysis.