HyperEyes – Xiaohongshu and Cambridge's Parallel Multimodal Search Agent

1. What Is HyperEyes

HyperEyes is a parallel multimodal search agent jointly launched by Xiaohongshu and the University of Cambridge. It introduces the UGS (Unified Grounded Search) paradigm, fusing visual grounding and retrieval into a single atomic action for single-round concurrent multi-entity search. Built on a dual-granularity efficiency-aware reinforcement learning framework (TRACE + OPD), the 30B version exceeds the strongest open-source model at the same scale by 9.9% accuracy on six benchmarks while reducing tool-call rounds by 5.3×, achieving Pareto optimality between accuracy and efficiency. HyperEyes marks a fundamental shift in multimodal search from serial to parallel paradigms, offering a new path for complex visual question answering.

Image source: Official article

Technical positioning and domain: HyperEyes sits at the intersection of multimodal AI and information retrieval, focused on concurrent localization and retrieval of multiple entities in complex images. Unlike traditional VQA systems, it deeply integrates visual grounding, entity recognition, and external knowledge retrieval into an end-to-end intelligent search agent.

R&D background: Developed jointly by Xiaohongshu's technical team and Cambridge's Computer Laboratory. Xiaohongshu's recommendation and content understanding expertise provides rich multimodal training experience; Cambridge's strengths in reinforcement learning and visual reasoning support algorithmic innovation. Motivation came from efficiency and accuracy bottlenecks in existing serial search paradigms.

Core value: HyperEyes solves the inefficiency and error cascading of traditional multi-entity "crop-search-merge" serial pipelines. It embeds visual grounding directly into retrieval action parameters, carrying multiple target boxes in one function call, eliminating contamination from upfront localization bias. This enables concurrent multi-entity search with qualitative leaps in accuracy and efficiency in complex scenes.

Technical characteristics: Core advantages include the dual-granularity efficiency-aware RL framework—TRACE dynamically tightens trajectory-level efficiency benchmarks while OPD injects token-level correction signals on failed trajectories, jointly optimizing policy network accuracy and efficiency. Its progressive rejection-sampling data synthesis pipeline generates zero-redundancy parallel search seed data for high-quality training.

2. Key Features

• Parallel multimodal search: Concurrently localizes and retrieves multiple entities in an image within a single interaction round, replacing traditional serial crop-search-merge flows. After uploading a complex multi-target image, HyperEyes obtains all entity locations and retrieval results in one call, significantly improving search efficiency.

• Unified Grounded Search (UGS): Embeds visual bounding boxes directly into retrieval action parameters, carrying multiple target boxes in one function call for physical single-round multi-target concurrent pathways. This eliminates delay and error accumulation from visual grounding as a separate upfront step.

• Progressive rejection sampling: Purifies efficient trajectories under increasing round budgets, building 30,000 zero-redundancy parallel seed data points. Multi-round filtering and optimization ensure training data quality and diversity, avoiding redundant samples interfering with training.

• Dual-granularity efficiency-aware RL (TRACE + OPD): TRACE dynamically tightens trajectory-level efficiency benchmarks based on current best trajectories; OPD injects token-level correction signals on failed trajectories via a 235B teacher model for dense supervision. Together they improve accuracy while greatly reducing tool-call rounds.

• IMEB benchmark evaluation: Releases a 300-instance multi-entity visual evaluation set with CAS (Cost-Aware Scoring), unifying accuracy, token consumption, and tool-call rounds into a single efficiency metric. This provides a standardized framework for evaluating multimodal search agents.

• Multi-tool coordination: Integrates image search and text search external tools, concurrently obtaining visual and textual evidence. During parallel search, the model can call Bing Image Search and Google Custom Search APIs simultaneously for comprehensive, diverse results.

3. How to Use

1. Environment requirements: HyperEyes is recommended on servers with NVIDIA A100 or H100 GPUs—at least 80GB VRAM (30B) or 320GB (235B). OS: Linux (Ubuntu 20.04+); Python 3.10+, PyTorch 2.0+, CUDA 11.8+. Apply for Bing Image Search and Google Custom Search API keys as external retrieval tools.

2. Get open-source resources: Visit the GitHub repo (https://github.com/DeepExperience/HyperEyes) and git clone the project. Includes model weights, training scripts, and inference configs. Use a virtual environment for dependency isolation.

3. Prepare base model: Download Qwen3-VL-30B or Qwen3-VL-235B as the vision-language backbone. Run pip install -r requirements.txt, then python download_model.py --model_size 30B to download weights. Enable model parallelism or quantization if VRAM is insufficient.

4. Configure external retrieval tools: Set Bing Image Search and Google Custom Search API keys and search engine IDs in config/api.yaml. Supports custom parameters like region limits, search type (image/text), and result count.

5. Input multi-entity query: Run python run_agent.py --image_path ./test.jpg --query "Identify the people in the image, their professions, and background information" to start parallel search. The model executes UGS unified grounded search, concurrently localizing and retrieving all target entities in one round.

6. View parallel search results: The model returns structured JSON with bounding box coordinates, confidence scores, visual evidence links, and text evidence summaries per entity. Use --output_format to switch to Markdown or HTML output.

7. Notes: Run python benchmark.py first to verify environment setup. For scenes with 10+ entities, adjust max_entities to control parallel search limits. Observe search engine API rate limits to avoid throttling.

4. Pros and Cons

5. Comparison with Similar Tools

Selection advice: For production environments prioritizing extreme efficiency and accuracy, HyperEyes-30B is the top choice—2.2 average tool rounds and 64.0% accuracy lead peers. Especially for real-time processing of large volumes of complex images (e-commerce content review, social media monitoring), parallel search significantly reduces latency and compute cost.

For academic research or hardware-budget-limited teams, VDR is an alternative—54.1% accuracy is slightly lower but smaller scale and lower deployment barrier. Note VDR's 11.6 tool rounds mean higher API cost and longer inference. DeepEyes-V2's serial architecture and higher error cascade risk suit only specific scenarios with fewer image entities.

6. Editor's Take

HyperEyes represents a milestone innovation in multimodal search. Its core breakthrough fuses visual grounding and retrieval into a single atomic action—a first in academia and industry. Technically, the dual-granularity efficiency-aware RL framework (TRACE + OPD) addresses traditional RL efficiency optimization pain points—TRACE's dynamic benchmark adapts to query difficulty; OPD's asymmetric distillation prevents sacrificing accuracy for efficiency.

From practical value, HyperEyes has strong potential in e-commerce, social media, and news review. For e-commerce product comparison, traditional methods crop, search, and merge each product over minutes; HyperEyes completes all localization and retrieval in one round—5×+ efficiency gain. For platforms processing millions of images daily, this means significant cost savings.

HyperEyes suits professional users handling complex multi-entity visual queries—AI researchers, content review teams, e-commerce operators, and news media workers. For ordinary consumers, the learning curve is steep and technical background helps unlock full capability.

Future potential is strong. Expect evolution toward more external tools (knowledge graphs, database queries), video multi-frame search, and deep Chinese optimization. Lightweight versions (7B or 13B) could lower deployment barriers and expand reach.

Rationale: Extremely high technical innovation (UGS paradigm, dual-granularity RL), significant practical value (5.3× efficiency), good open ecosystem (Apache 2.0). 0.5 star deducted for high hardware requirements and limited Chinese support, somewhat limiting adoption speed.

7. Application Scenarios

• Multi-person visual reasoning: In group photos or news images, concurrently identify each person's identity, profession, and background. Upload an international summit photo and retrieve structured profiles for each attendee—name, title, and related news events.

• E-commerce product comparison: Concurrently search complex scene images for each product's price, brand, reviews, and purchase links. Upload a shelf or outfit photo; HyperEyes auto-locates all products and returns detailed comparison tables.

• Cross-modal knowledge QA: Answer complex queries about relationships between multiple objects, landmarks, and artworks in images. Upload a museum exhibit photo; identify each piece's name, era, artist, and cultural background, and answer deep questions like chronological relationships.

• News fact-checking: Concurrently retrieve multiple people and scenes in news images to verify event authenticity and background. Newsrooms can quickly check whether each image element matches reporting, identifying misinformation and misleading content.

• Academic chart parsing: Parallel localization and content retrieval for paper screenshots with multiple charts, formulas, and citations. Researchers upload screenshots; HyperEyes extracts each chart's title, data source, and key conclusions for literature review and data analysis.

• Social media moderation: Concurrently search multiple entities in complex images to quickly detect violations or misleading content. Platforms can auto-review multi-element ad images, political cartoons, or malicious composites, improving moderation efficiency and accuracy.

8. FAQ

Q: How does HyperEyes differ from traditional VQA systems?

A: Traditional VQA typically answers about the whole image or a single entity; complex multi-entity queries need multiple interactions. HyperEyes achieves single-round concurrent multi-entity search via UGS—simultaneously localizing, identifying, and retrieving multiple targets with structured multi-entity output. HyperEyes also integrates external search tools for real-time information; VQA usually relies on static training data.

Q: What are HyperEyes' minimum hardware requirements?

A: 30B version needs at least 80GB VRAM GPU (e.g., NVIDIA A100 80GB); dual A100 or H100 recommended. If VRAM is insufficient, 4-bit or 8-bit quantization reduces needs to ~40GB with slower inference. 235B needs at least 320GB VRAM—8× A100 or H100 cluster recommended. CPU inference is unsupported due to large vision encoder and language model parameters.

Q: Does HyperEyes support Chinese input and Chinese search?

A: Mainly trained and optimized for English; Chinese input support is limited. The model can understand Chinese queries but accuracy and recall lag English on Chinese multi-entity tasks. For Chinese scenarios, use English query descriptions or wait for future Chinese optimization. External search tools (Bing/Google) support Chinese search but model parsing of Chinese results is weaker.

Q: How do I evaluate HyperEyes search efficiency?

A: HyperEyes provides CAS (Cost-Aware Scoring): Acc²×100/(N_tok+2N_tool+1), where Acc is accuracy, N_tok is token consumption, N_tool is tool-call rounds. This unifies accuracy, compute cost, and API cost into one metric for cross-configuration comparison. Use --evaluate after inference to auto-compute CAS.

Q: What is HyperEyes' open-source license? Can it be used commercially?

A: Apache 2.0—fully open source for commercial and academic use. Model weights are based on Qwen3-VL; comply with Alibaba Qwen series license. External search tools (Bing/Google API) require respective terms and may incur API fees.

Q: How to handle API throttling and network latency?

A: Configure multiple API keys in config/api.yaml for automatic rotation. Set request delay (e.g., request_delay: 0.5 seconds) and retry (e.g., max_retries: 3). For production, use enterprise API tiers for higher quotas. If network latency is high, consider deploying local search (e.g., Elasticsearch) instead of external APIs.

9. Project Links

• GitHub repository: https://github.com/DeepExperience/HyperEyes
• arXiv paper: https://arxiv.org/abs/2605.07177
• HuggingFace model weights: https://huggingface.co/DeepExperience/HyperEyes-30B