PixelRAG – Berkeley's Open-Source Vision-Native RAG Framework

1. What is PixelRAG

PixelRAG is an open-source vision-native RAG framework developed by Berkeley's SkyLab/BAIR. It moves beyond the traditional RAG paradigm of extracting text from web pages or PDFs for retrieval, instead using a headless Chromium to render documents into screenshot tiles. It then directly reads tabular data, charts, and layout information from the images using a Qwen3-VL visual vector retrieval model fine-tuned with LoRA, thereby addressing the pain points of information loss in traditional RAG when dealing with tables, charts, and complex layouts. The framework provides a complete toolchain from rendering and indexing to retrieval and generation, and comes with a pre-built index covering 8.28 million Wikipedia pages, supporting keyless API and local index construction.

Image source: Official article
Image source: official article

Technical positioning and domain: Belongs to the vision-native RAG category within the Retrieval-Augmented Generation (RAG) domain, distinct from traditional text-extraction-based RAG. Its unique positioning lies in treating documents as images for direct processing, preserving visual layout information, making it suitable for knowledge-intensive tasks that are sensitive to visual structures such as tables, charts, and multi-column layouts.

Development background: Jointly developed by the SkyLab and BAIR labs at the University of California, Berkeley, the research team has deep expertise in visual language models and efficient retrieval systems. The motivation for development stemmed from the fundamental flaws of traditional RAG in handling visually rich documents—the text extraction process disrupts table alignment, chart semantics, and page structure, leading to a decline in retrieval and generation quality.

Core value: Solves the pain point of traditional RAG, where "it seems like there's an answer, but after text extraction, it's gone," achieving lossless document retrieval. Through pixel-level representation, it not only retains complete visual information but also introduces a unique efficiency lever—reducing the screenshot resolution can lower token costs by up to three times without sacrificing accuracy. Experiments show that it outperforms text-based RAG baselines across multiple open-domain question-answering benchmarks, with a maximum improvement of 18.1%.

Technical features: The core innovation lies in the "representation is the image" pipeline design, using screenshots instead of text as a unified medium for retrieval and generation. Technically, it integrates headless browser rendering, visual vector embedding (Qwen3-VL + LoRA), and direct reading by a Vision Language Model (VLM), forming a pure visual channel from document to answer, eliminating the need for complex HTML parsing and text cleaning processes.

2. Key Features

• Page Rendering (pixelshot): Utilizes headless Chromium to render web pages or PDFs into screenshot tiles, fully preserving visual structures such as table row and column alignment, chart axes, and infographic layouts. Supports mixed input of URLs and local files. On Windows/macOS, it automatically uses the system-installed Chrome without requiring additional browser configuration.

• Visual Semantic Search: Encodes screenshots into vector embeddings, supporting two query modes: text-to-image and image-to-image. Locally, it defaults to FAISS for efficient nearest-neighbor search; for large-scale deployments, it can switch to Qdrant, which supports quantized compression, disk storage, and shared indexing across multiple services, adapting to various scale requirements.

• Pixel-Level Reading: Screenshot tiles retrieved by search are directly fed as images into a Vision-Language Model (VLM), allowing the model to read numerical data, text, and layout information directly from the image, without any intermediate text conversion. This design avoids error accumulation typically introduced by traditional OCR or text parsing methods.

• Hosted Search Service: Offers password-free API and a web-based demo, with a pre-built index of 8.28 million Wikipedia pages. Users can perform searches via POST requests to https://api.pixelrag.ai/search without any configuration, or experience it instantly on the pixelrag.ai website, reducing the barrier to entry.

• Custom Index Pipeline: Provides a complete command-line toolchain, including pixelrag index, embed, build-index, serve, and other commands. Users only need to write a YAML configuration file specifying document paths and embedding models to index their own PDFs or websites. Small-scale indexing can be run on Mac (Apple Silicon).

• Agent Integration: Offers the Claude Code plugin pixelbrowse, allowing Claude to "see" web pages through screenshots rather than parsing incomplete HTML, without requiring MCP dependency. After installing via claude plugin install pixelbrowse@pixelrag-plugins, the Agent can directly read the rendered page screenshots, improving the quality of web page understanding.

3. How to Use

1. Environment Requirements: Requires Python 3.10 or higher, as well as the pip package manager. It is recommended to use Linux or macOS systems; Windows can also run but requires the Chrome browser to be installed (for pixelshot rendering). For large-scale index construction, it is recommended to use an NVIDIA GPU (such as A100). Small-scale testing can be performed on Mac Apple Silicon.

2. Install Core Tools: Run pip install pixelrag to install PixelRAG and its dependencies. This command installs core components such as the pixelshot rendering tool, embedding pipeline, and retrieval service. If you want to use the Claude Code plugin, you also need to run uv tool install pixelrag.

3. Render Pages as Screenshots: Run pixelshot https://example.com -o ./tiles to render the target webpage or PDF as screenshot tiles, outputting them to the specified directory. You can pass in multiple URLs or local file paths at the same time. Rendering parameters (such as resolution and tile size) can be adjusted in the configuration file.

4. Experience Hosted Search: No local configuration is required. Simply send a POST request to https://api.pixelrag.ai/search (with the query text), and you can search through the 8.28 million Wikipedia page index. Alternatively, open pixelrag.ai in your browser and use the built-in search Demo for an interactive experience.

5. Build Your Own Index and Start the Service: Create a pixelrag.yaml configuration file, specifying the document path (such as a local PDF folder), embedding model (default is Qwen3-VL-Embedding-2B LoRA weights), and output directory. Run pixelrag index build to construct the index, then execute pixelrag serve --index-dir ./my_index --port 30001 to start the local search service. Mac Apple Silicon can handle small document sets, while the full Wikipedia index (approximately 217GB) requires GPU computing power.

6. Integration with Claude Code: Install the pixelbrowse plugin: claude plugin marketplace add StarTrail-org/PixelRAG, then claude plugin install pixelbrowse@pixelrag-plugins. After this, Claude will automatically use screenshots instead of HTML parsing when browsing web pages, improving its ability to understand complex pages.

7. Advanced Training: If you want to customize the embedding model, go to the train/ directory (a standalone uv project) and follow the README to fine-tune using the provided LoRA training data and official open-source weights. The training data is built using contrastive learning and can be optimized for retrieval accuracy in specific domains.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For documents primarily consisting of plain text with simple visual structures (e.g., news articles, blogs), traditional text RAG (BM25 or DPR) still offers cost advantages, with simple deployment and minimal storage overhead. However, if the documents contain a large number of tables, charts, multi-column layouts, or infographics, visual RAG solutions like PixelRAG and ColPali can significantly enhance retrieval quality. PixelRAG has a more comprehensive engineering ecosystem and full-stack toolchain, making it particularly suitable for teams needing to quickly build a complete pipeline from document rendering to retrieval and generation. ColPali excels in fine-grained matching for document-dense layouts but has a relatively weaker engineering support, making it more appropriate for R&D teams with strong self-construction capabilities. For scenarios involving large-scale knowledge bases (e.g., full Wikipedia) and aiming to reduce token costs, PixelRAG's pixel efficiency leverage and verified scalability make it a more practical choice. In terms of Agent integration, PixelRAG's Claude Code plugin provides out-of-the-box web reading capabilities, while traditional RAG and ColPali currently lack similar deep integration solutions.

6. Editor's Summary

PixelRAG makes a substantive innovation in the RAG technology stack by treating documents as images directly, fundamentally bypassing the issue of information loss that comes with text extraction. This design is not simply about "replacing parsing with OCR," but rather about building a pure visual pipeline that spans from rendering, embedding, to generation. The embedding layer is fine-tuned using LoRA on Qwen3-VL, enabling screenshots to be semantically searchable, while the generation layer directly leverages the pixel understanding capabilities of the VLM. Technically, the biggest highlight is the discovery of efficiency gains from pixel representation—reducing the resolution of screenshots can significantly cut down token costs without sacrificing accuracy, a feature that holds immense value for cost-sensitive large-scale deployments. Experimental data supports its performance advantages: it outperforms text-based RAG baselines across multiple open-domain question-answering benchmarks, with a maximum improvement of 18.1%, and also performs better in multimodal and agentic tasks.

In terms of practical value, PixelRAG directly addresses one of the most common pain points in enterprise-level RAG applications: materials such as financial reports, technical documents, and product manuals that contain a large number of tables and charts often yield unsatisfactory retrieval results in traditional RAG systems. The framework provides a straightforward "rendering-as-indexing" path, significantly reducing engineering complexity. Additionally, the hosted search service and the Claude Code plugin lower the barrier to entry, allowing individual developers and small teams to quickly experience the capabilities of visual RAG.

The target audience includes knowledge management teams dealing with visually rich documents, AI application developers aiming to enhance the web comprehension abilities of their agents, and academic researchers exploring visual retrieval and multimodal RAG. In terms of future potential, as VLM capabilities continue to improve and multimodal data explodes, visually native RAG is likely to become one of the mainstream paradigms in knowledge retrieval. PixelRAG has already validated its feasibility at scale (full Wikipedia), and if it continues to optimize in areas such as Chinese support, document completeness, and the stability of dynamic content rendering, it has the potential to become a benchmark project in this field.

7. Application Scenarios

• Financial Reports and Data Table Q&A: Directly read numerical tables from financial reports in PDF or web formats, avoiding row and column misalignment caused by text parsing. Users can ask, "What was the revenue in the fourth quarter of last year?" The system identifies the screenshot region containing the table, and the VLM directly reads the numbers from the image to provide an answer, achieving significantly higher accuracy than text extraction methods.

• Dashboard and Chart Retrieval: Index screenshots of dashboards containing KPI panels, line charts, and bar charts. Support retrieval via text description (e.g., "Find the user growth trend chart") or example images. The retrieved chart area is directly input into the VLM, enabling the model to interpret trends and anomalies.

• Infographics and Illustrated Documents: Content such as infographics, flowcharts, and technical architecture diagrams, which traditional text indexing cannot cover, is fully preserved with visual semantics through screenshot tiles. Users can search for "microservices architecture diagram" to find the corresponding screenshot, and the VLM can further explain the relationships between components in the diagram.

• Multi-column Product Pages: Complex layouts such as e-commerce product comparison pages and multi-column news articles are preserved in their original format using screenshot tiles for retrieval. When searching for "iPhone 16 Pro vs Samsung S25 parameter comparison," the system can accurately locate the screenshot region containing the comparison table.

• Agent Web Browsing: Using the pixelbrowse plugin from Claude Code, AI Agents use screenshots instead of HTML parsing when browsing web pages. This allows them to correctly understand table data, button states, and chart information, improving the reliability of automated tasks such as data collection and form filling.

8. FAQ

Q: What is the core difference between PixelRAG and traditional RAG?
A: Traditional RAG extracts plain text from documents (via HTML parsing or PDF text extraction), resulting in the loss of table alignment, chart semantics, and page layout structure. PixelRAG renders documents as screenshot tiles, enabling direct retrieval and generation in image form, preserving all visual information intact. As a result, it achieves better retrieval performance for visually dense content such as tables and charts.

Q: Does it support Chinese documents?
A: The base model Qwen3-VL has the capability to understand Chinese, and the framework itself has no language restrictions. However, there are currently no publicly available benchmark data for the rendering quality and retrieval accuracy of Chinese documents. It is recommended that users evaluate performance after conducting small-scale tests. For documents containing Chinese tables and charts, PixelRAG's visually native approach is theoretically more advantageous than text extraction.

Q: What hardware configuration is required?
A: For small-scale testing (e.g., a few hundred pages of documents), pixelshot rendering and FAISS retrieval can be run on Mac Apple Silicon or regular CPUs. To build a full Wikipedia index (approximately 217GB), GPU computing power (such as NVIDIA A100) is required. During the inference phase, the VLM needs a certain amount of GPU memory to process the screenshots; it is recommended to have at least 8GB of GPU memory for medium-scale applications.

Q: Is the hosted search service free? Are there any limitations?
A: The hosted search API (https://api.pixelrag.ai/search) and web demo are currently free to use, with an internal index of 8.28 million Wikipedia pages. There are no publicly disclosed usage frequency limits, but for large-scale commercial applications, it is recommended to build your own index to avoid reliance on external services.

Q: Can it process PDF files?
A: Yes. The pixelshot tool supports rendering PDFs page by page into screenshot tiles, following the same workflow as web rendering. Visual elements such as tables, charts, and formulas in PDFs are preserved intact. For scanned PDFs (non-electronic versions), the rendering quality depends on the PDF's original quality. It is recommended to use OCR enhancement for better results.

Q: How can PixelRAG be integrated into existing applications?
A: After building your own index, you can provide search services via REST API (pixelrag serve to start), supporting both text and image queries. The returned screenshot tiles can be directly used as input for the VLM. Alternatively, you can quickly prototype by directly calling the hosted API. The Claude Code plugin offers the simplest path for Agent integration.

Q: How long does it take to build an index?
A: It depends on the number of documents and the hardware. A small document set (a few hundred pages) can be indexed in a few minutes on a Mac with Apple Silicon. A full Wikipedia index (30 million screenshots) requires several hours to days on a GPU cluster, with the exact time to be officially announced.

9. Project Links

1. Product Website: https://pixelrag.ai/
2. Hosted Search API: https://api.pixelrag.ai/search
3. GitHub Repository: https://github.com/StarTrail-org/PixelRAG
4. Technical Paper: https://arxiv.org/pdf/2606.28344