Baidu Open-Sources Wenxin Unlimited OCR 2 with 96% KV Cache Reduction

The Era of OCR 3.0 Has Arrived

By Tingyu, from Ao Fei Si

QbitAI | Official Account: QbitAI

Baidu’s OCR has upgraded again—ERNIE Unlimited OCR 2 is officially released and open-sourced!

This time, it directly tackles the challenge of reading an entire novel series in one go.

Seven Harry Potter books, from The Philosopher's Stone to The Deathly Hallows, are continuously recognized and transcribed across 3,600+ pages, all the way to the end.

Similarly, it can read the complete Three-Body Problem trilogy in one pass—three books, over 1,200 pages, hundreds of thousands of words—without pausing for breath.

However, processing so many consecutive pages requires the model to first overcome a hurdle: VRAM (Video RAM).

The longer the document, the more intermediate information (KV Cache) needs to be retained during inference. Think of this as the model’s "cheat sheet" that grows thicker with every page added. A bulky cheat sheet not only takes up space but also slows down retrieval.

In its technical report, ERNIE Unlimited OCR 2 presented a comparison based on single-sequence tests involving documents spanning hundreds of pages:

Peak KV Cache usage dropped from 47GB (standard decoding, where visual caches for all pages are kept in VRAM simultaneously) to 1.77GB, representing a reduction of approximately 96%.

What does 47GB signify? It exceeds the total system memory of many laptops. And this figure accounts only for the KV Cache; other overheads such as model weights are calculated separately.

Its predecessor, Unlimited OCR, had already gained significant popularity in the open-source community due to its ability to continuously parse multi-page documents, accumulating over 10 million downloads and roughly 28,600 stars on GitHub.

Community efforts have continued to expand around it, ranging from creating online demos to supporting SGLang inference.

For the second generation, the upgrade focuses on reducing computational and caching burdens while maintaining the ability to process even longer documents continuously.

To achieve this, ERNIE Unlimited OCR 2 introduces the Sparse Reference Sliding Window Attention (SR-SWA) mechanism, allowing the model to selectively retrieve relevant pages and visual information on demand, thereby reducing redundant computations.

Combined with a continuous decoding mechanism, the model can process ultra-long documents sequentially without letting the per-step decoding cost escalate as the document length increases.

Currently, ERNIE Unlimited OCR 2 has been officially open-sourced on Hugging Face and GitHub.

Open Source Links:

• GitHub: https://github.com/baidu/Unlimited-OCR
• HuggingFace: https://huggingface.co/baidu/Unlimited-OCR-2

Dissecting Core Technologies: How SR-SWA Reduces Computation and Chunked Decoding Minimizes Storage

How exactly does ERNIE Unlimited OCR 2 manage to read millions of characters and over 3,600 pages of the complete Harry Potter series consecutively?

It draws inspiration from how humans copy books by hand.

What is the workflow when a person copies text manually? They look at a section, copy it, and keep their eyes primarily focused on the page currently being copied. Only when cross-referencing is necessary do they flip back to check related pages.

Humans figured out the efficiency of copying long before models did: no one reads the entire Analects simultaneously in their mind while copying just one part of it.

However, the previous generation of Unlimited OCR did not operate this way. Although R-SWA significantly reduced KV Cache usage during inference, all input pages’ visual information still participated in attention calculations during the review process.

As documents get longer, more information is processed, yet much of it is irrelevant to the specific lines currently being recognized—a classic case of "running alongside without contributing."

△ SR-SWA mimics the localized focus strategy used by humans when copying books

Following this human-inspired approach, ERNIE Unlimited OCR 2 introduces the Sparse Reference Sliding Window Attention (SR-SWA) mechanism.

Its filtering process occurs in two layers: first, it identifies which pages need to be referenced; then, within those selected pages, it extracts truly relevant local information, blocking out unnecessary computations.

From an architectural perspective, ERNIE Unlimited OCR 2 retains the foundational design of its predecessor.

△ Overall Architecture of ERNIE Unlimited OCR 2

Page images first enter the DeepEncoder, where they are converted into visual tokens that the model can process—essentially data cards recording page content and layout features.

Subsequently, the language model generates recognition results step-by-step based on these cards.

The key change occurs within the language model itself: the Reference Sliding Window Attention (R-SWA) mechanism used in the previous generation has been upgraded to SR-SWA in this version.

The lower-left portion of the architecture diagram illustrates the two-layer filtering process of SR-SWA.

The first layer is the Page Gate, which determines "which pages to look at."

Based on the model’s current parsing position, it narrows down candidate visual information to relevant page ranges, preventing irrelevant pages from interfering.

In the context of copying a book, this is akin to finding a few relevant pages when reading a specific section, rather than flipping through the entire book from start to finish.

The second layer is Token Sieve, which determines "which lines to focus on within a given page."

Selecting the right pages is only the first step: even among these pages, not all visual tokens need to participate in computation. Token Sieve performs an additional filtering pass, selecting visual information blocks from the page that are more relevant to the current content, thereby capturing long-range visual dependencies.

The architecture diagram on the right expands this process: it first calculates relevance using concise representations of each information block, then uses TopK to select the highest-scoring blocks with a fixed quantity.

After selection, the model still examines the specific visual details within these blocks, rather than relying solely on summaries.

The combination of these two mechanisms allows the model to both locate relevant pages in long documents and reduce the amount of visual information processed at each step, thereby improving long-document parsing efficiency.

△ Token Sieve Details

The filtered visual information also participates in parsing alongside the Local Window (local sliding window) shown in the diagram.

This window has a fixed size, holds recent context, and shifts forward along with the generation process, ensuring the model continues writing based on the preceding content.

Since the number of information blocks is fixed and the local window size is also fixed, the amount of information referenced at each step is bounded—no matter how long the document is, the workload per step remains constant. This prevents the scenario where "the thicker the book, the harder it becomes to write each word."

SR-SWA is responsible for reducing computation per step; however, how much information needs to be retained in VRAM is a separate issue.

Each page image, after encoding, becomes a set of visual tokens. The model references them when generating parsing results, so it stores the corresponding KV cache for reuse in subsequent calculations.

If the entire document is loaded at once and all pages' visual caches are kept in VRAM, memory usage increases significantly with the number of pages.

More troublesome is that even if these pages are not involved in the current calculation, they continue to occupy VRAM as long as their caches are not released. This is akin to spreading out every draft page while copying a book without discarding any—even a large desk would run out of space.

Therefore, reducing computation must be paired with reducing storage.

To address this, ERNIE Unlimited OCR 2 introduces the Unlimited-Horizon Decoding Paradigm, controlling cache usage through chunked processing.

△ Schematic of Chunked Decoding

This mechanism divides the document into manageable page chunks and parses them sequentially. Adjacent chunks overlap by one page: the last page of the previous chunk serves as the starting point for the next—a ready-made bookmark.

When entering the next chunk, the model directly reuses the visual KV cache of the overlapping page while retaining the text cache from the end of the previous output, ensuring continuity between segments. Visual caches for other pages that are no longer needed are promptly released to free up VRAM.

As a result, the visual cache retained in VRAM is constrained within the scale of the currently processed chunk. After finishing a chunk, the system keeps only the information necessary for bridging to the next part before starting the new chunk—there is no need to keep the visual cache of the entire document resident throughout the process.

SR-SWA reduces computation per step, while the Unlimited-Horizon Decoding Paradigm controls cache usage. Their combination enables the model to continuously parse ultra-long documents.

Reading 884 Pages of The Lord of the Rings in 42 Minutes, Reducing Cache to 1.77GB

Having discussed the technical details, we wanted to see the results firsthand: we fed it an 884-page English edition of The Lord of the Rings.

Approximately 42 minutes and 56 seconds later, parsing was complete. With 884 pages of input corresponding to 884 pages of output, the average time per page was about 2.9 seconds.

In roughly the time it takes to watch an episode of a TV series, the journey through "Middle-earth" was completed.

For such a long document, does recognition quality hold up towards the end?

We specifically spot-checked the body text on pages 600 and 750: the recognized text matched the original perfectly. Being able to reproduce page 750 verbatim indicates that the model did not "get carried away" earlier in the reading process.

Throughout the task, the system also marked 466 instances of cross-page merging. For example, sentences near the end of the book (pages 873–874) that were hard-truncated by page breaks were correctly rejoined in the output.

While the real-world test demonstrated continuous parsing of the entire book, the technical report addressed three additional questions: how much cache is required, how fast it can run, and what the recognition quality is.

On the hundred-page-level benchmark MarathonDocBench, the team compared standard decoding against Unlimited-Horizon Decoding.

During single-sequence inference, using chunked processing and boundary cache reuse reduced the peak KV cache usage from 47GB to 1.77GB, a decrease of approximately 96%.

Large amounts of page information that previously had to reside in VRAM simultaneously can now be processed in batches and released promptly. Note that the 1.77GB figure refers only to KV cache; other VRAM overheads such as model weights are calculated separately.

Beyond saving cache, the report also tested performance when processing multiple documents concurrently.

In another 16-way concurrency test, adopting this decoding method increased end-to-end throughput from 360 to 1315 tokens/second, approximately 3.65 times that of standard decoding.

△ Comparison of Decoding Methods: Single-Sequence Peak KV Cache and 16-Way Concurrent Throughput

For services requiring batch processing of annual reports or technical manuals, these improvements have practical significance: lower VRAM usage means more workloads can fit on the same GPU card; higher concurrency means more content is generated per unit of time.

Users do not need to worry about these details either. They simply submit the entire document as usual, while chunking and stitching are handled internally by the model, eliminating the hassle of manually splitting files, submitting them in batches, and concatenating the results.

Beyond speed and efficiency, parsing quality must hold up.

MPDocBench evaluates the parsing quality of multi-page documents: when facing documents spanning dozens of pages, models must accurately identify content while also reconstructing document structure.

In this test, ERNIE Unlimited OCR 2 achieved a composite score of 84.59.

To further test hundred-page-level documents, the team created MarathonDocBench. It includes 100 documents, each ranging from 103 to 300 pages. Many paragraphs and tables continue onto the next page, specifically testing whether the model can correctly connect content across pages.

On this longer and more challenging benchmark, ERNIE Unlimited OCR 2 achieved a composite score of 77.06. In both evaluations, its composite score was the highest among all reported participating models.

△ From the technical report: Multi-page document parsing evaluation results

Changes in long-document processing speed are equally significant. In the report's hundred-page tests, SR-SWA's single-step decoding speed reached 7.0 times that of Full Attention (full attention mechanism) and 5.6 times that of R-SWA used in the previous generation.

The curves make this clearer: as documents get longer, the single-step decoding latency for the first two approaches climbs steadily, while SR-SWA's single-step latency remains nearly flat.

This means the time required for the model to generate each Token does not increase significantly with the number of input pages—the book gets thicker, but the writing speed stays constant.

△ Single-step decoding latency at different document lengths

Compared to the previous generation, ERNIE Unlimited OCR 2's per-page composite score remained largely consistent.

The real gap emerges once documents become longer: it can process more consecutive pages, recognize cross-page content more accurately, and simultaneously reduce computational and cache overhead for long-document parsing.

As for how people use it...

You can input hundreds of pages of annual reports or thick technical manuals.

Especially for scanned documents: hand them over to the model for continuous parsing and text conversion, then retrieve information and verify content—saving you from much of the hassle of splitting pages and manual stitching.

But with such powerful OCR, who would actually use it for work...?

Of course, it's for reading novels!

The Era of OCR 3.0 Has Arrived

Of course, moving from a few pages of material to an entire book changes the problems OCR needs to solve.

In the technical report, the team summarized the evolution of document parsing into three stages: OCR 1.0, 2.0, and 3.0.

Early OCR resembled a pipeline with clearly defined roles: first locate text positions, then recognize content, and finally organize layout and output results. For multi-page parsing, it typically processed one page at a time and stitched the results together—this was OCR 1.0.

Different modules had their specific duties, but errors from earlier steps could propagate downstream—one module sneezes, and the rest catch a cold.

With OCR 2.0, vision-language models began generating parsing results directly from page images, reducing handoffs between multiple modules. The first-generation Unlimited OCR extended this end-to-end parsing to documents spanning dozens of pages.

△ AI-generated

When documents grow from dozens to hundreds of pages, continuous parsing must ensure seamless content connection while controlling computational and cache overhead.

ERNIE Unlimited OCR 2 uses on-demand reference and chunked decoding, allowing the model to process more pages without expanding the computational scope of each step proportionally to the page count, nor retaining visual caches for all pages indefinitely.

The team calls this step OCR 3.0: making continuous processing of entire documents the new technical goal—in one go, no pagination, infinite parsing, never crashing.

And by the way, this goal has an anime version too.

The name "Unlimited OCR Works" itself pays homage to "Unlimited Blade Works" from Fate/stay night.

△ Fate/stay night Anime Official Website Homepage

In the original lore, this is a type of magecraft known as a "Reality Marble." When activated, it manifests a special space filled with countless swords.

After the previous generation was released, Hugging Face user Snarcy spotted this naming reference and took the liberty of adapting the classic incantation into an OCR version.

For the second generation, the team actually included this rewritten passage in the introduction of their technical report, retaining Snarcy's credit.

So, when you open this report discussing attention mechanisms and VRAM overhead, the opening lines present this incantation:

Pixels are my body, and tokens are my blood.

I have parsed over a thousand pages.

Unknown to KV cache.

Nor known to context limits.

So, as I pray: Unlimited OCR Works.

With memory usage dropping from 47GB to 1.77GB, the line "Unknown to KV cache" wasn't just sung for nothing.

Quite the geeky delight.

Like, Share, and Favorite

We welcome your thoughts in the comments!

— End —

🌟 Star Us 🌟

Daily updates on cutting-edge tech progress

Open in WeChat