Apple and Duke Introduce LensVLM: Long-Context Image Compression with On-Demand Zooming for NeurIPS 2026

Enabling Models to Consult Original Text On-Demand While Reading Compressed Content

When Agents execute long-horizon tasks, code, tool outputs, and interaction logs accumulate continuously, causing the context window to grow.

Generating summaries or compressing context can reduce input size but may also delete details required for subsequent steps. A record that seemed irrelevant at one moment might become a critical clue in the next. If the model only sees the compressed result, it becomes difficult to retrieve that information later.

DeepSeek-OCR demonstrated an alternative approach: using images to carry text, representing textual information with fewer visual tokens. Compressed pages can cover the full document, but clarity becomes a new bottleneck. The smaller the characters, the harder they are for the model to recognize; if compression is too aggressive, the answer may be on the page, yet the model can no longer read it clearly.

For many models, compression schemes act as a one-way street: once compressed, the model cannot return to consult the original text on demand. What if we gave the model a "magnifying glass"?

Apple’s Foundation Model Team and Duke University introduced LensVLM, which allows the model to first browse compressed pages covering the entire document, then expand specific pages relevant to the query. Across seven text-based QA benchmarks, LensVLM achieved an average accuracy of 68.9% with 4.3× effective compression, approaching the 72.4% baseline of full-text reference.

This work was recently accepted by NeurIPS 2026 and has garnered significant attention on X (formerly Twitter), sparking discussions about "converting text to images for long-context understanding." Victor Mustar, Product Lead at Hugging Face, highlighted the research on X, with the post receiving approximately 280,000 views. A public LensVLM 9B Demo is now live, allowing users to experience the process of page selection, expansion, and answering.

• Paper: https://arxiv.org/abs/2605.07019
• Model: https://huggingface.co/apple/LensVLM-9B
• Code: https://github.com/apple-aiml-research/ml-lensvlm
• Demo: https://huggingface.co/spaces/hugging-apps/lensvlm-9b-demo

What Does Converting Text to Images Save?

Language models typically split text into text tokens; Vision-Language Models (VLMs) convert images into visual tokens via visual encoders. If a single image contains enough text, the number of visual tokens required to represent it may be lower than the number of text tokens needed to input the original text directly.

Font size, page dimensions, and text density can therefore be adjusted to tune the compression rate without redesigning the text vocabulary. Compared to retrieving small snippets before reading, compressed images can cover the entire document, ensuring that evidence scattered across different locations still has a chance to be found by the model.

The cost is direct: as characters are compressed smaller, they may exceed the recognition capabilities of the visual encoder, becoming indistinguishable even at the pixel level. In the paper, when the base model directly reads images compressed by 5×, the average accuracy drops to just 31.3%, significantly lower than the full-text baseline.

Rendering results of the same text under 5×, 10×, and 15× settings. Stronger compression reduces visual tokens but makes characters harder to recognize.

Select Pages First, Then Read in Detail

LensVLM separates evidence localization from evidence reading. The model first receives the question and all compressed images, selects pages worth reading in detail, and then invokes an expansion tool. For pages rendered from plain text, the tool returns the corresponding original text and adds it to the context. After reading, if the evidence is insufficient, the model can continue selecting the next page.

In the demo, consider this question: What government position did the woman who played Corliss Archer in the film "Kiss and Tell" hold? Facing 15 compressed pages, the model selected page 10, invoked read_page to retrieve the original text, and ultimately answered "Director of the U.S. Office of Protocol."

Here, the "magnifying glass" directly accesses the stored source content. The system must retain the original text or images: compressed pages allow the model to browse the full document, while expanded content restores the details necessary for answering.

Page Selection Requires Training

Connecting a model to tools does not mean it knows which page to read.

The study uses Qwen3.5-9B-Base as the foundation, first filtering out samples it cannot answer directly. Then, Qwen3.5-397B generates tool-use trajectories based on the question, ground-truth evidence, and standard answers. Through supervised fine-tuning (SFT), the model learns the reasoning, page selection, and answering processes demonstrated in these examples.

Subsequently, the study employs DAPO reinforcement learning to further optimize the trajectories generated by the model itself. The reward is based on answer correctness; only when the answer is correct does using the tool receive an additional bonus.

Under a 5x initial compression setting, SFT improved average accuracy from 31.3% to 63.7%, and reinforcement learning further increased it to 68.9%.

LensVLM's inference and training pipeline. The model alternates between reasoning and expanding pages, learning selective reading through supervised fine-tuning and reinforcement learning.

How much can be understood under higher compression?

The main evaluation covers NQ, HotpotQA, MuSiQue, HELMET, as well as RULER, Qasper, and LongBench (which were not used for training), including single-hop and multi-hop long-document QA.

Under a 15x initial compression setting, the base model's average accuracy when directly reading compressed images dropped to 18.1%, while LensVLM achieved 52.1%. Although there remains a gap compared to full-text references, selecting pages before expanding them for reading significantly mitigated the accuracy drop caused by high compression.

However, expanding the original text also consumes input budget, so one cannot simply calculate how small the initial image is. The paper therefore reports the "effective compression ratio": dividing the number of tokens in the original text by the total input tokens comprising the compressed image and the expanded content.

Accounting for expansion overhead, the effective compression ratios corresponding to the 5x, 10x, and 15x initial settings are 4.3x, 7.4x, and 10.1x, respectively.

Average accuracy and effective compression ratio across seven text QA benchmarks. The horizontal axis accounts for input after expansion, illustrating the trade-off between accuracy and input budget for different methods.

Speed and Cost in Long Contexts

In long contexts ranging from 32K–140K tokens with a 15x initial compression setting, end-to-end latency for full-text input was 3.62 seconds, while LensVLM achieved 2.83 seconds. After pre-rendering and caching page images offline, latency further decreased to 2.52 seconds, representing an approximately 30% reduction compared to full-text input.

Under the same settings, LensVLM's peak KV cache usage was about one-fourteenth that of full-text input. In saturated concurrency tests, throughput for the version with pre-cached pages was approximately 3.7 times that of full-text input. Estimating at $6 per GPU-hour, the GPU serving cost per thousand requests decreased by approximately 73%.

In short contexts, direct full-text input remains faster. The advantages of this approach are primarily evident in long documents and scenarios involving multiple queries where page images can be reused.

End-to-end efficiency and latency. The version with pre-cached pages uses images rendered offline in advance; throughput was measured under saturated concurrency.

What Should PDFs Be Expanded Into?

Pure text pages can directly return the original text, but native PDFs also contain tables, graphics, and layout. Extracting only text may miss visual information necessary for answering questions.

The study additionally trained two document understanding variants: one expands pages into OCR text, and the other returns high-resolution pages, evaluated on MMLongBench-Doc.

Under a setting where page pixel count is reduced by 15x, the accuracy of directly reading compressed pages was 31.1%, the OCR expansion variant reached 32.1%, and the high-resolution page expansion variant achieved 41.9%, corresponding to a 10.3x effective compression ratio.

In this benchmark, the charts and layout information preserved by high-resolution pages proved more helpful than OCR text, but also incurred greater input overhead. The form of expansion needs to be chosen according to content: pages rendered from pure text are suitable for returning the original text, while documents relying on visual information are better served by returning high-resolution pages.

Summary

The core of LensVLM is enabling the model to consult the original text on demand while reading compressed content. The model first browses the entire document, then allocates its reading budget to the pages required for the current question. As model context windows continue to grow, how models actively allocate reading budgets—deciding what to skim first, what requires close reading, and when to refer back—may become a research direction as important as expanding context window size.

Author Introduction

Roy Xie is a Ph.D. student in Computer Science at Duke University, primarily researching post-training of foundation models, long-horizon task reasoning, and tool use. He is a recipient of the National Science Foundation (NSF) Graduate Research Fellowship and the Apple AI/ML Ph.D. Fellowship. Previously, he interned on Apple's foundation model post-training team and is currently a research intern at Google DeepMind. Personal website: royxie.com

© THE END

Please contact this official account for authorization to reprint.

For submissions or media inquiries: [email protected]