Ant Group Releases PolyOCR: Unified 2B Model for Full-Scenario Document Parsing

One-Stop Intelligent Parsing for Card and Document OCR

Contributed by Ant Group AI Security Lab

QbitAI | Official Account: QbitAI

As multimodal large models (MLLMs) become increasingly powerful, the limitations of traditional OCR are becoming more apparent.

Models are no longer just processing a single image or a paragraph of text; they are handling ID cards, financial reports, and full-page contracts. The requirement is not merely text extraction, but understanding "which field corresponds to what value" and "where the numbers originate."

For such real-world business documents and card-based scenarios, OCR should be built as a foundational capability that enables large models to acquire visual textual evidence.

Characters, positions, layout structures, and field relationships collectively determine what the model reads and, consequently, what it can answer. Only by integrating these elements into a continuous understanding process can we accurately measure a model's mastery of business materials.

To address this, Ant Group AI Security Lab proposes PolyOCR. By employing the Capability-Guided Policy Optimization (CGPO) algorithm, which combines on-policy distillation with reinforcement learning, PolyOCR equips large models with the underlying ability to truly "read and understand documents."

Experiments demonstrate that PolyOCR achieves State-of-the-Art (SOTA) performance across multiple benchmarks, including OCRBenchV2.1, CC-OCR, and the In-house KIE Benchmark. It also delivers high performance on OmniDocBenchV1.6, a benchmark focused on layout analysis.

△ Cross-benchmark performance of PolyOCR. OmniDocBench uses a 9B parameter model with a layout-guided pipeline, while MDPBench uses a 2.7B multilingual document parsing model. Note that CC-OCR does not include multilingual directions.

Why Is Traditional OCR No Longer Sufficient?

Let's return to business scenarios to identify the problem. Imagine a typical enterprise workflow: an employee uploads a photo of an expense report. The system must automatically recognize the amount, date, and invoice number, while also verifying where these numbers appear in the image—after all, forged invoices edited via Photoshop are common.

Traditional OCR solutions typically follow a segmented approach: first, one model detects text regions; then, another model recognizes characters within each region; finally, rules or a third model perform field extraction. These three stages operate in series without a unified standard. If any stage fails, the entire result is incorrect.

In contrast, PolyOCR unifies text recognition and layout parsing: it allows the large model to read the entire page at once, simultaneously outputting three layers of information: "what character is this," "where is it located," and "what field does it belong to." These three tasks share the same visual encoder and learn using the same set of parameters, allowing them to correct each other. For instance, if the model perceives that "this character looks like an amount, but its position seems off," it will re-evaluate rather than forcing an answer.

Data Engineering: Efficiently Constructing 60 Million Supervised OCR Samples

If training relies solely on naturally collected data proportions, common printed text may dominate, leaving complex tables, field associations, and long-tail layouts underrepresented. Simply increasing the total sample volume does not necessarily bridge this gap. PolyOCR's data construction focuses on capability coverage, prioritizing tasks where public data is scarce or annotation costs are high.

Taking information extraction from cards and forms as an example, the team first reconstructs documents into editable HTML/CSS. They then replace key-value content while preserving field semantics and layout constraints, rendering the results back into images. This approach varies the content while maintaining the correspondence between images and field annotations. Images exhibiting issues such as text overflow, element overlap, abnormal line breaks, or field misalignment are filtered out during quality checks.

△ Task-oriented data engine. Materials from various sources are converted into supervised data with spatial, structural, and semantic constraints.

The critical aspect here is preserving the supervision relationship. For example, table parsing verification goes beyond checking if the HTML is parsable; it requires confirming that the content matches the original image and that no tables on the page are missed. To achieve this, the team employs multi-model cross-validation, structural checks, and manual spot-checks. For difficult cases, discrepancies between models serve as clues for further verification rather than labels to be blindly trusted.

Through capability balancing and sampling, approximately 60 million supervised fine-tuning samples cover text perception, document structure, understanding/reasoning, and general multimodal tasks. The scale of the data provides a foundation for joint learning, while quality validation ensures that different tasks establish correct supervision relationships based on the same visual evidence.

Training: Aligning Teacher Guidance with Student Progress

With data secured, how do we allocate learning opportunities? PolyOCR's training consists of two stages: first, establishing diverse OCR capabilities through Supervised Fine-Tuning (SFT), followed by targeted optimization using CGPO.

SFT adopts a staged curriculum: early phases focus on text perception and document structure, while the proportion of understanding and reasoning tasks gradually increases from 10% to 35%, with general multimodal data maintained at 15%. As training progresses, more complex semantic operations are introduced, while foundational tasks continue to participate in learning.

Post-training addresses a more nuanced issue. The paper provides an example: consider a financial document containing 1,000 characters where the amount should be "18650," but multiple candidate outputs incorrectly generate "18850" or "18350." Using conventional text similarity metrics (such as edit distance) for scoring, each answer differs by only one character, resulting in a normalized edit similarity of 0.999. While the overall page score appears high, the critical amount is wrong.

If advantage calculation relies on intra-group relative rewards as in traditional GRPO, and all generated answers in a group have high scores, the gradient signal received by the model is nearly zero, failing to provide reward-driven corrective gradients. On-Policy Distillation (OPD) thus serves as a complement: the teacher provides token-by-token guidance along the context actually generated by the student, allowing local errors to receive finer-grained feedback. However, this introduces a new problem: teachers can also make mistakes. If students mechanically imitate, these errors are inherited.

△ Two-stage training framework. Curriculum learning establishes foundational capabilities, while CGPO adjusts teacher guidance per sample and maintains continuous optimization via GRPO.

CGPO's solution transforms distillation intensity from a uniform training schedule into a sample-level capability assessment. It simultaneously evaluates the absolute quality and relative advantage of the teacher. Even if the teacher is stronger than the student, imitation might reinforce errors if both perform poorly. Conversely, if the teacher performs well but the student has already mastered the sample, guidance may be redundant. Both conditions jointly determine the distillation weight.

Teachers and students are scored using the same reference answers and task validators. Teacher capabilities are estimated and cached in advance, while student capabilities are updated online. Independently generated samples are used to calibrate teacher quality (probability of outperforming the student) and the student's relative level. Distillation is enabled only when the teacher performs well AND the student still has a gap. When the calibrated teacher win rate falls below 0.5, the distillation weight for that sample is set to zero, continuing optimization driven by outcome feedback. At the same training moment, if a student has already mastered common fields, it does not rely on the teacher for complex tables. CGPO expresses this differentiation through sample-level judgments.

Evaluation: Where Does the Improvement from CGPO Come From?

Evaluation requires answering two distinct questions: what tasks the model can handle, and whether performance gains can be attributed to the proposed method. The paper begins from the same supervised fine-tuning (SFT) starting point, comparing GRPO, OPD, a fixed-weight combination of both, a cosine-decay combination, and CGPO. This setup aims to disentangle the respective contributions of supervision signals and sample-level routing.

△ CGPO Ablation Study. The cosine-decay baseline uniformly adjusts distillation weights based on training steps, serving to test the effectiveness of sample-level capability assessment.

The cosine-decay baseline uniformly reduces distillation weights based on training steps. Compared to the cosine scheme that decays weights according to training steps, CGPO improves scores by 2.19 points on OCRBenchv2.1 English and 2.06 points on Chinese, 1.45 points on OmniDocBench, and 0.11 points on CC-OCR. Within the scope of this comparison, the results support assigning guidance per sample rather than relying solely on a predefined decay strategy.

Compared to the fixed-weight combination of GRPO and OPD, CGPO increases the English score from 75.42 to 80.42, and the Chinese score from 77.08 to 78.37. PolyOCR integrates result feedback, fine-grained guidance, and capability assessment into a single framework, forming the complete technical logic of this post-training method.

It should also be noted that CGPO’s routing granularity is at the sample level. A teacher model that performs well overall may still make errors on specific characters. Therefore, the experiments validate the effectiveness of this supervision allocation method but do not imply that critical fields are error-free or that interference between multi-tasks has been completely resolved.

Understanding Full Pages: Challenges Beyond Text Recognition

In the evaluation results, the 9B model achieves scores of 80.42 and 78.37 on OCRBenchv2.1 for English and Chinese, respectively; the average score across the three evaluated directions in CC-OCR is 82.43. The English score and CC-OCR average lead among the models compared in the paper. These figures collectively support the model's coverage of various text-related tasks but cannot replace validation on specific business materials.

Two modes were used to further configure the OmniDocBench test: when the 9B model directly processes full pages, the OmniDocBench score is 91.57; when PP-DocLayoutV3 is used to analyze layout and crop regions, followed by the same PolyOCR model parsing text, formulas, and tables, the score rises to 95.52.

The higher score in the latter case does not mean PolyOCR lacks end-to-end capabilities. On the contrary, this result precisely illustrates that PolyOCR is positioned as a "foundational capability." Unified capability describes which tasks the model can share learning for and process, while engineering systems can still leverage layout modules to improve input organization. External modules are responsible for organizing regions and reading order, while the model handles content recognition and structural recovery. This set of experiments also suggests that errors in complex pages need to be localized along the processing pipeline and cannot all be attributed to the model failing to recognize text.

2B or 9B? Choosing Based on Application Scenarios

Building OCR as a foundational capability requires considering its invocation frequency. When submitting credentials online, users expect timely field population; when processing complex reports, the system prioritizes structural integrity and information accuracy. While these two scenarios share overlapping requirements, they differ in trade-offs regarding inference latency, resources, and task complexity.

Card and certificate evaluations provide business-relevant evidence for this multi-scale design: across four types of materials—Xuexin.com certificates, ID cards, admission notices, and international passports—the overall field-level F1 scores for 2B and 9B are 93.49% and 93.60%, respectively.

△ Field-level F1 comparison for cards and certificates. The overall F1 scores for 2B and 9B are 93.49% and 93.60%.

Unexpectedly, the overall gap between the two on this test set is only 0.11 percentage points (93.49% vs. 93.60%). The 2B model achieves 95.53% on Xuexin.com certificates, while the 9B model reaches 88.35% on admission notices, representing the highest values in their respective columns in the table. These results indicate that model size and performance on specific materials do not have a simple monotonic relationship; model selection should return to actual material distributions.

The gap becomes more apparent in broader capability benchmarks: 2B scores 72.33 and 71.32 on OCRBenchv2.1 English and Chinese, respectively, and 80.53 on CC-OCR; 9B corresponds to 80.42, 78.37, and 82.43. This also indicates that similar performance on card tests does not imply equivalence across all document and reasoning tasks.

Based on evaluation data, PolyOCR-2B offers a lightweight candidate worth testing for latency-sensitive real-world businesses, while 9B handles more complex document understanding and extraction needs. Deployment choices must therefore be validated against hardware, resolution, output length, and concurrency. Before actual latency data is available, it is recommended to use 2B for gray-scale verification in card scenarios, and reserve 9B for complex report-type materials.

Final Thoughts

PolyOCR aims to make "reading documents" a foundational capability of large models, rather than an external OCR tool.

Traditional OCR is about "recognizing characters"—you give it an image, and it gives you a string of text. PolyOCR is about "understanding"—give it an image, and it tells you the fields, positions, meanings, and relationships between fields.

The primary technical goal of PolyOCR is: to enable large models to preserve textual facts, spatial relationships, and structural constraints as much as possible when reading business materials, and to receive feedback matched to current capabilities during training. For document and card business operations, reliable extraction and reasoning depend first on whether the model correctly reads the materials it relies upon.

Technical Report: https://arxiv.org/abs/2609.37712 Code: https://github.com/inclusionAI/PolyOCR-Venus

One-click Triple Combo: "Like", "Share", "Heart"

We welcome your thoughts in the comments!

— End —

[Academic Submission] Please send emails on weekdays to: [email protected], with "[Submission]" in the subject line, and tell us: Who you are, Where you come from, attach the Project/Homepage link for the submission content, and provide Contact Information.

🎓 We will try to reply promptly :)

🌟 Light Up the Star 🌟

Daily Updates on Cutting-edge Tech Progress

Read Original Article

Open in WeChat