OvisOCR2 – End-to-End Document Parsing Model Developed by Alibaba ATH-MaaS Team

1. What is OvisOCR2

OvisOCR2 is an end-to-end document parsing model developed and fully open-sourced by the Alibaba ATH-MaaS team. It is trained based on the Qwen3.5-0.8B base model and has a parameter scale of only 0.8B. The model achieved the top score of 96.58 on the OmniDocBench v1.6 evaluation benchmark, demonstrating high-precision recognition and structured extraction capabilities for complex layout documents. It supports parsing of multi-column layouts, tables, formulas, and other scenarios, making it suitable for document intelligence processing needs in academic research, archival digitization, and enterprise knowledge base construction. Its end-to-end architecture directly outputs structured Markdown text, significantly reducing the engineering complexity of traditional multi-stage pipelines.

Technical Positioning and Domain: OvisOCR2 belongs to the document intelligence (Document Intelligence) direction, which is an interdisciplinary field of computer vision and natural language processing. It focuses on end-to-end document parsing. Unlike traditional OCR + post-processing pipelines, it takes document images as input and directly outputs structured text (such as Markdown), offering unique advantages in understanding complex layouts. The model fills the gap in lightweight end-to-end document parsing models, providing developers with a high-cost-performance option.

Development Background: Developed by the Alibaba ATH-MaaS team, which has deep expertise in visual language models and multimodal learning, and has previously launched the Ovis series of multimodal models. The motivation behind OvisOCR2's development stems from the issues of error accumulation and high engineering maintenance costs in existing document parsing systems that rely on multi-stage pipelines. The team aimed to simplify the process and improve accuracy through an end-to-end architecture. Large-scale pre-training was conducted on a vast amount of scanned documents and PDF renderings, enabling the model to learn rich layout semantics.

Core Value: OvisOCR2 addresses the pain points of traditional document parsing systems that require multiple independent models to be connected in series for detection, recognition, and layout analysis. By using a single model to directly output structured text from images, it significantly reduces deployment and maintenance complexity. At the same time, its lightweight 0.8B parameter size allows smooth inference on consumer-grade GPUs, offering a cost-effective document parsing solution for small and medium-sized teams and individual developers. The model is fully open-sourced, further lowering the barrier to entry.

Technical Features: The model employs a visual language model architecture, integrating a visual encoder with the Qwen3.5 text decoder to achieve end-to-end mapping from image to text. Through large-scale pre-training on a vast amount of scanned documents and PDF renderings, the model has learned rich layout semantics. Its end-to-end optimization strategy avoids error accumulation in traditional pipelines, achieving extremely low inference latency while maintaining high accuracy. The model supports dynamic resolution, adapting to document images of various sizes.

2. Key Features

• End-to-end Document Parsing: The model eliminates the traditional multi-stage pipeline of detection, recognition, and post-processing, directly inputting document images and outputting structured text (Markdown or plain text). This design significantly reduces engineering maintenance costs, avoids error accumulation between modules, and improves overall recognition accuracy. Users no longer need to maintain multiple models; a single model can complete the entire process from image to text.

• Complex Layout Recognition: Supports precise parsing of complex document layouts such as multi-column formatting, image-text mixing, nested tables, and formulas. The model learns semantic features of various layouts through large-scale pre-training, accurately reconstructing document structures without the need for additional rule-based post-processing. Its ability to handle complex layouts has been thoroughly validated in the OmniDocBench evaluation.

• High-precision OCR Extraction: Achieved a score of 96.58, ranking first in the OmniDocBench v1.6 evaluation, with character and paragraph recognition accuracy leading the industry. This result confirms the model's robustness in real-world scenarios, particularly in handling low-quality scans and dense text. The model outperforms similar models in both character-level and paragraph-level metrics.

• Lightweight and Efficient Inference: Based on the Qwen3.5-0.8B base model, the model has only 0.8B parameters and can run smoothly on consumer-grade GPUs (e.g., RTX 3090). Its inference speed is significantly faster than larger models of comparable accuracy, making it suitable for localized deployment and real-time processing. The model supports FP16 inference, further reducing memory usage and accelerating generation.

• Structured Output Capability: The model directly outputs structured text in Markdown format, automatically preserving document structure information such as heading levels, lists, and tables. This facilitates direct indexing and retrieval by downstream applications (e.g., RAG systems), reducing post-processing workload. The output format is standardized and can be directly used for document rendering or knowledge base import.

• Open Source and Fine-tunable: The model's weights and code are fully open-sourced on HuggingFace, with no commercial licensing restrictions. Developers can fine-tune the model using their own domain-specific data to adapt it to particular layouts (e.g., medical forms, legal documents), further customizing and enhancing its accuracy. Fine-tuning scripts and example data are provided, supporting efficient methods such as LoRA.

3. How to Use

1. Environment Preparation: First, clone the HuggingFace repository (git clone (link to be updated after official release)), and ensure you have Transformers 3.10+ and PyTorch 2.0+ installed. For hardware, it is recommended to use a GPU with at least 8GB of VRAM (such as RTX 3070+). While the model can run on a CPU, it will be significantly slower, with each page of document requiring approximately 10 seconds.

2. Load the Model and Tokenizer: Use AutoModelForCausalLM to load the OvisOCR2 weights from HuggingFace, along with its corresponding tokenizer. Example code: from transformers import AutoModelForCausalLM, AutoTokenizer; model = AutoModelForCausalLM.from_pretrained("ATH-MaaS/OvisOCR2", torch_dtype=torch.float16); tokenizer = AutoTokenizer.from_pretrained("ATH-MaaS/OvisOCR2"). Note that the first load will download approximately 1.6GB of model files, so it is recommended to download them in advance.

3. Document Image Preprocessing: Convert scanned documents or screenshots into the tensor format required by the model. Typically, the image needs to be resized to a dimension supported by the model (e.g., 448x448) and normalized to [0,1]. It is recommended to use PIL or OpenCV for preprocessing to maintain image clarity. For multi-page documents, process each page individually and perform inference separately.

4. Perform Inference and Retrieve Results: Feed the preprocessed image tensor into the model and call model.generate() for forward inference. The model directly outputs structured parsing results in Markdown or plain text format. You can set parameters such as temperature (e.g., 0.2), top_p (e.g., 0.9), etc., to control the diversity of the generated output. Example: inputs = processor(images=image, return_tensors="pt").to(device); outputs = model.generate(**inputs); result = tokenizer.decode(outputs[0], skip_special_tokens=True).

5. Fine-tuning and Custom Deployment (Optional): Based on your own document data, continue training the model using efficient fine-tuning methods such as LoRA to adapt it to specific domain layouts. Fine-tuning scripts and example data can be found in the repository. During deployment, you can export the model to ONNX or use vLLM to accelerate inference, further improving throughput.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: If the primary task involves processing complex-layout English documents (e.g., multi-column papers, table-dense reports), OvisOCR2 is the best choice, as its end-to-end structured output and 0.8B lightweight parameters achieve a good balance between accuracy and efficiency. For general OCR needs (e.g., text extraction from scanned documents), GOT-OCR2.0 offers more versatile capabilities, with smaller parameters and faster inference speed. If the application context is primarily in Chinese, PP-OCR is specifically optimized for Chinese text recognition and supports multi-stage fine-grained control, making it suitable for Chinese document pipelines.

For digitizing academic literature, Nougat is designed specifically for academic PDFs and has unique advantages in formula recognition and document structure extraction. However, it may not generalize as well to other document types (e.g., table-dense corporate reports) as OvisOCR2. Developers can choose the appropriate tool based on document type and accuracy requirements, or combine OvisOCR2 with other models—for example, using PP-OCR for Chinese preprocessing and then OvisOCR2 for structured output.

6. Editor's Summary

OvisOCR2 has made significant innovations in the field of end-to-end document parsing, particularly by achieving state-of-the-art (SOTA) performance with only 0.8B parameters, demonstrating the great potential of vision-language models even with a small parameter size. Its core value lies in simplifying the traditional multi-stage OCR pipeline into a single model, directly outputting structured text. This is not only a simplification in technical architecture but also reduces the complexity of engineering deployment and maintenance. In terms of practical value, OvisOCR2 provides a high-cost-performance ratio solution for document digitization for small and medium teams and individual developers, especially its precise parsing capability for complex document layouts gives it a significant advantage in scenarios such as academic literature processing, enterprise archive management, and financial document recognition. The model is fully open-sourced with no commercial restrictions, further promoting the popularization of document intelligence technology.

The target users mainly include: researchers who need to process academic papers in bulk, libraries or archives that need to digitize historical documents, financial technology companies that require automated processing of invoices and contracts, and AI application developers building RAG knowledge bases. In terms of future development, OvisOCR2 can be extended to more languages (such as Chinese and Japanese) and more complex document types (such as handwritten documents and ancient texts). Combined with the progress of multimodal large models, it has the potential to achieve breakthroughs in document understanding (not just extraction). Overall, OvisOCR2 is a lightweight end-to-end solution worth paying attention to in the field of document parsing, with both its technological innovation and practical value leading in the industry.

7. Application Scenarios

• Academic Document Digitization: Batch process scanned PDF papers, automatically extract the main text, chart titles, and reference structures, and generate structured Markdown files, accelerating the organization of research materials and the writing of literature reviews. Researchers can quickly build personal document libraries, supporting full-text search and citation management, significantly improving research efficiency.

• Enterprise Document Management: Convert scanned historical paper archives into searchable structured databases, supporting multi-column layouts and table recognition, reducing manual data entry costs, and improving the efficiency of archive utilization. Enterprises can build digital archive systems, enabling fast document retrieval and access control, supporting digital transformation efforts.

• Financial Document Processing: Accurately identify key fields and nested tables in invoices and contracts, supporting automated financial review processes, reducing manual verification errors, and improving processing efficiency. Financial institutions can integrate the model into OCR pipelines to achieve automatic document classification and information extraction, reducing operational costs.

• Educational Material Organization: Parse complex multi-column layouts in textbooks and exam papers, generating structured electronic resources for easy online retrieval and editing, assisting in the digitization of teaching materials. Educational institutions can convert printed textbooks into editable electronic documents, supporting content development for online teaching platforms and enriching digital educational resources.

• RAG Knowledge Base Construction: Provide high-quality structured text input for document Q&A systems, improving the accuracy of retrieval-augmented generation responses. By using OvisOCR2 to convert unstructured documents such as PDFs and scanned files into Markdown format, and then importing them into a vector database, the retrieval quality of RAG systems can be significantly enhanced, suitable for enterprise knowledge bases and intelligent customer service scenarios.

8. FAQ

Q: What image formats does OvisOCR2 support?
A: The model supports common image formats such as PNG, JPEG, TIFF, etc. It is recommended that the input resolution be no less than 300 DPI to ensure recognition accuracy. Both color and grayscale images are acceptable, but the model will internally convert them to RGB for processing. For multi-page PDFs, they need to be converted to single-page images first and then processed page by page.

Q: What is the output format of the model? Can it be customized?
A: The default output is structured text in Markdown format, preserving document structures such as headings, lists, and tables. Users can also output plain text or JSON format by modifying generation parameters or post-processing scripts. For example, by setting return_dict_in_generate=True and customizing the decoding logic.

Q: How fast is the inference speed of OvisOCR2? What hardware is required?
A: On an RTX 3090, it takes approximately 1-2 seconds to process an A4 document image. The minimum hardware requirement is a GPU with at least 8GB of VRAM. It can also run on a CPU, but the speed will be slower (about 10 seconds per page). It is recommended to use a GPU that supports FP16 inference for optimal performance, with VRAM usage around 2GB.

Q: Does the model support parsing Chinese documents?
A: The base model Qwen3.5 is primarily optimized for English, but it also has some capability for Chinese documents. For purely Chinese documents, the recognition accuracy may be slightly lower than for English. It is recommended to use fine-tuning or pair it with a Chinese OCR model for preprocessing in Chinese-specific scenarios.

Q: How can OvisOCR2 be fine-tuned? How much data is needed?
A: You can fine-tune OvisOCR2 using the HuggingFace Transformers library, which supports parameter-efficient fine-tuning methods such as LoRA. It is recommended to prepare at least 100–500 pages of annotated data, with the exact amount depending on the complexity of the domain. Fine-tuning scripts and examples are provided in the repository, and image quality should be maintained during training.

Q: What advantages does OvisOCR2 have compared to other OCR tools?
A: The main advantage is its end-to-end architecture, which simplifies the deployment process and directly outputs structured text. It is also lightweight in parameters and can run on consumer-grade GPUs. For documents with complex layouts, its accuracy is higher than traditional multi-stage OCR systems. However, it may not perform as well as specialized models in extreme scenarios such as handwriting. It is recommended to choose based on actual needs.

9. Project Links

• HuggingFace Model Repository: https://huggingface.co/ATH-MaaS/OvisOCR2