HyOCR-1.5 – Tencent Hunyuan's Lightweight End-to-End OCR Expert Model Open Sourced

1. What is HyOCR-1.5

HyOCR-1.5 is a lightweight end-to-end Optical Character Recognition (OCR) expert large model introduced by the Tencent Hunyuan team. With only 1B parameters, it integrates full-stack capabilities including document parsing, text detection and recognition, information extraction, photo translation, chart parsing, ancient character recognition, video subtitle extraction, and multi-page document Q&A. Based on DFlash speculative decoding technology, the model achieves a maximum 6.37x speedup in long structured output scenarios. It also ranks first in end-to-end OCR on the OmniDocBench v1.6 benchmark with a score of 94.74, demonstrating excellent balance between accuracy and efficiency.

Technical Positioning and Domain: HyOCR-1.5 is a vertical application of multi-modal large models in the OCR field, utilizing an end-to-end architecture that deeply integrates visual encoders with language models to directly output structured text. It eliminates independent modules such as detection, recognition, and post-processing in traditional OCR pipelines. Its positioning is as a lightweight, full-stack OCR expert, suitable for various scenarios such as document digitization, information extraction, and translation.

Development Background: This model was developed by the Tencent Hunyuan team, leveraging their technical expertise in visual encoders (Hunyuan-ViT) and language models (Hunyuan-0.5B). The team has long focused on the practical application and democratization of OCR technology, aiming to reduce the application threshold through open-sourcing and promote industry advancement.

Core Value: The core value of HyOCR-1.5 lies in achieving leading OCR accuracy and full-stack capabilities with an extremely compact model size (1B parameters). It supports local deployment on both CPU and consumer-grade GPUs, significantly reducing hardware costs. Its DFlash speculative decoding technology greatly reduces decoding latency without altering the output distribution, shortening the processing time per page to just 1.4 seconds, making it ideal for large-scale document digitization scenarios.

Technical Features: The model employs 4K native resolution visual encoding and supports a 128K context window, enabling it to handle high-density documents and ultra-large tables. It incorporates DFlash speculative decoding (with a block-diffusion draft model of approximately 90.7M parameters), significantly lowering decoding latency without affecting output distribution. Through Agentic Data Flow and a three-stage training recipe (pre-training, SFT, RL), the model continuously optimizes its capabilities, particularly addressing long-tail scenarios (such as 331 languages and ancient characters) with data augmentation.

2. Key Features

• Document Parsing: Convert dense documents, multi-column layouts, tables, and formulas into structured text (Markdown/HTML/LaTeX) with one click. This feature leverages the model's understanding of layout structure to preserve the original reading order and hierarchical relationships, making it suitable for digitizing complex documents such as academic papers and contracts.

• Text Detection and Recognition: Extract all textual content from images along with their corresponding coordinates, covering scenarios such as documents, street scenes, handwriting, advertisements, games, and videos. The model uses an end-to-end approach, eliminating the need for separate detection modules, and can simultaneously output text and positional information.

• Information Extraction: Extract specified fields from invoices, credentials, and receipts and return them in JSON format. This capability utilizes the model's language comprehension and structured generation abilities to quickly extract key data from unstructured images, ideal for automating processes in finance and government sectors.

• Photo Translation: Extract text from images and translate it into multiple languages while preserving the document layout and formula formatting. The model translates based on its understanding of the original text layout, maintaining paragraph structure, table alignment, and mathematical formulas, making it suitable for cross-border document processing.

• Chart Parsing: Parse flowcharts and statistical charts into Mermaid or Markdown formats. This feature automatically identifies relationships between chart elements and data trends, generating editable structured descriptions that facilitate secondary document editing and knowledge management.

• Ancient Chinese Character Recognition: Supports recognition of seven Chinese script styles (oracle bone, bronze inscriptions, seal, clerical, regular, cursive, and running script). The model uses Agentic Data Flow to supplement ancient character training data, enabling accurate identification of variant characters and ambiguous strokes in historical documents, aiding in the digitization of cultural heritage.

• Video Subtitle Extraction: Accurately extract subtitle text from video frames. The model can process dynamic subtitles in video frames, including multi-line, scrolling, and different font styles, suitable for video content analysis and automatically generating subtitle files.

• Multi-page Document Q&A: Perform cross-page retrieval, comparison, and evidence aggregation for questions based on multi-page PDFs. The model uses a 128K context window to understand content across multiple pages, answer questions requiring integration of information from different pages, and provide referenced evidence.

3. How to Use

1. Environment Requirements: Install Python 3.12+, CUDA 12.9, PyTorch 2.7.1, and vLLM (≥0.12.0). It is recommended to use a Linux operating system and ensure that the NVIDIA GPU has at least 16GB of VRAM for a smooth inference experience.

2. Model Launch: Start the inference service by executing the command vllm serve tencent/HunyuanOCR, or load the weights directly via Hugging Face Transformers. After launching, the model can be called through an HTTP API or Python interface.

3. Image Input and Task Instructions: Use PIL to read the image and construct a messages dialog structure containing the image and text fields. Choose the corresponding Prompt template based on the scenario, for example, "Please convert the document in the image to Markdown format" for document parsing, or "Please output all text and their coordinates in the image" for text detection.

4. Inference Invocation and Result Cleaning: Call the generation interface via vLLM or Transformers, setting temperature=0 and max_tokens=16384 to ensure deterministic output. After generation, use the built-in deduplication function to clean up any potential duplicate substrings and obtain the final structured result.

5. Local Deployment (Optional): Run the model on a CPU or consumer-grade GPU using llama.cpp without requiring a server. This involves converting the model weights to GGUF format and loading them with llama-cli. This method is suitable for environments without a GPU, but the inference speed will be slower.

6. Notes: For very long documents, it is recommended to process them in pages to avoid context overflow; for information extraction tasks, provide a clear list of required fields; when translating from photos, specify the source and target languages to improve accuracy.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: If high-accuracy end-to-end document parsing is required and hardware budget is limited, HyOCR-1.5 is the optimal choice, offering a clear cost-performance advantage with its 1B parameters and DFlash acceleration. If extremely high accuracy is a must and server resources are available, DeepSeek-OCR-2 can be considered as an alternative, though it comes with more than three times the latency and deployment cost. For scenarios requiring only basic text recognition without the need for complex layout understanding, PaddleOCR or Tesseract are lighter and more mature options, but they cannot handle structured outputs such as tables or formulas.

6. Editor's Summary

HyOCR-1.5 stands out with notable technological innovations, particularly its DFlash speculative decoding mechanism. By utilizing a block-diffusion draft model with approximately 90.7M parameters, it enables full-block token prediction, significantly increasing the inference speed for long documents while maintaining the output distribution of the target model. This represents a significant improvement in the efficiency of autoregressive decoding.

Its Agentic Data Flow data closed-loop concept automatically converts model weaknesses into data requirements, driving production and providing a replicable framework for continuous model optimization—a feature relatively uncommon in open-source OCR projects.

In terms of practical value, the ability to deploy locally with just 1B parameters makes it highly suitable for small and medium teams as well as individual developers, granting professional-level OCR capabilities without the need for expensive servers. A score of 94.74 on OmniDocBench and a result of 93.67 on the Table TEDS benchmark demonstrate that it has reached industry-leading performance in complex document parsing. Additionally, its support for 331 languages and ancient scripts expands the application boundaries of OCR.

The target users include document digitization service providers, research institutions, multinational corporations, ancient script researchers, and AI application developers. The full-stack open-source nature of the model also facilitates academic research and secondary development.

In terms of future development potential, with ongoing community contributions and fine-tuning, the model may see deeper applications in various vertical fields such as medical invoices and legal documents. However, it still needs to continuously address issues related to output repetition and generalization in long-tail scenarios. Overall, HyOCR-1.5 is a high-cost-performance, technically solid open-source option in the OCR field, with the potential to promote the widespread adoption of end-to-end OCR.

7. Application Scenarios

• Dense Document Digitization: Utilize HyOCR-1.5's document parsing capabilities to convert contracts, papers, reports, and other high-density multi-column layouts into Markdown or LaTeX with a single click, preserving the original reading order and formatting. Ideal for libraries and archives to process bulk scanned documents, significantly improving data entry efficiency.

• Complex Table and Formula Parsing: For oversized tables and mathematical formulas in financial reports and academic papers, the model can directly output HTML tables and LaTeX formulas without requiring additional tools. Researchers and financial analysts can quickly convert physical or image-based materials into editable digital formats.

• Multilingual Cross-Border Document Processing: Supporting 331 languages, the model automatically identifies and parses mixed-language documents, enabling photo-based translation while maintaining the original layout. Foreign trade companies and multinational corporations can use it to process multilingual documents such as contracts, product manuals, and invoices, reducing manual translation costs.

• Ancient Script Research and Preservation: Recognizes seven major forms of Chinese ancient scripts, including oracle bone script, bronze inscriptions, seal script, clerical script, etc., assisting museums and archaeological institutions in digitizing and researching historical documents. Researchers can directly extract text from rubbings and photos of ancient texts, accelerating the organization of ancient script materials.

8. FAQ

Q: What languages does HyOCR-1.5 support?
A: The model supports 331 languages, including commonly used ones such as Chinese, English, French, German, Japanese, and Korean, as well as a large number of low-resource languages. For the specific language list, please refer to the official documentation.

Q: Can the model run completely offline?
A: Yes. After converting the weights using llama.cpp, the model can run completely offline on a CPU or consumer-grade GPU without requiring an internet connection. The vLLM method can also be deployed locally, without relying on external services.

Q: How can the model be fine-tuned?
A: The official team has open-sourced the complete training recipe and code, supporting parameter-efficient fine-tuning methods such as LoRA and QLoRA. Users can fine-tune the model on their own datasets using Hugging Face weights to adapt it to specific scenarios.

Q: What output formats are supported?
A: The model supports Markdown, HTML, LaTeX, plain text, and JSON formats (for information extraction tasks). Users can specify the desired output format via the Prompt, and the model will generate the corresponding structure based on the instruction.

Q: What are the minimum hardware requirements?
A: Using llama.cpp, the model can run on a GPU with 4GB of VRAM or a CPU with 8GB of RAM, although the speed will be slower. It is recommended to use an NVIDIA GPU with at least 16GB of VRAM (such as the RTX 4060 Ti) for a smooth experience. For the vLLM method, a minimum of 24GB VRAM is advised.

Q: Can the model process handwritten text?
A: Yes, but the accuracy depends on the neatness of the handwriting. The model performs well on standard handwriting, but for extremely messy or artistic handwriting, it is recommended to use the information extraction feature with field constraints or perform targeted fine-tuning.

9. Project Links

• GitHub repository: https://github.com/Tencent-Hunyuan/HunyuanOCR
• HuggingFace model library: https://huggingface.co/tencent/HunyuanOCR
• arXiv technical paper: https://arxiv.org/pdf/2607.04884