PerceptionBench – Moonshot AI's Open-Source Visual Perception Diagnostic Benchmark

1. What is PerceptionBench

PerceptionBench is an open-source visual perception diagnostic benchmark launched by Moonshot AI, focusing on fine-grained evaluation of atomic-level visual perception capabilities. Starting from the failure cases of leading models in 42 existing multimodal benchmarks, it identifies 10 atomic visual perception abilities and constructs 3,000 isolated verification questions. Each question tests only a single perception ability, without requiring reasoning or external knowledge. Test results show that none of the 16 mainstream multimodal large language models (MLLMs) achieved an accuracy rate above 60%, revealing that atomic-level visual perception remains an unresolved challenge in the industry.

Image source: Official article
Image source: official article

Technical positioning and domain: PerceptionBench belongs to the field of multimodal large model evaluation, but its positioning differs from traditional comprehensive understanding benchmarks (such as MMMU, MMBench). It focuses on stripping away reasoning and knowledge interference, measuring only the model's fundamental capabilities at the visual perception level, including 10 atomic dimensions such as visual relationships, counting, attributes, depth, localization, composition, fine-grained recognition, context, OCR, and hallucination. This positioning makes it a specialized diagnostic tool for the model's perception module, rather than a general ranking list of overall capabilities.

Development background: During the development of the Kimi multimodal model, the Moonshot AI team discovered that traditional benchmarks' "overall score rankings" obscured the true weaknesses of models at the perception level—two models with similar total scores could have significant differences in their atomic perception abilities. To fundamentally diagnose the issue of models "seeing but misunderstanding," the team systematically collected failure cases of leading models from 42 existing benchmarks, constructed an error classification tree from the bottom up, and ultimately distilled 10 atomic perception abilities. Based on this, they designed PerceptionBench.

Core value: The core value of PerceptionBench lies in its paradigm shift from "comprehensive scoring" to "precise diagnosis." It enables developers to accurately pinpoint where a model is failing in its perception pipeline—for example, whether the issue lies in insufficient counting ability or a misunderstanding of visual relationships—thereby guiding subsequent data optimization or architectural improvements. Moreover, it shifts the industry's focus from "score inflation" back to foundational perception skills, promoting more accurate and consistent visual understanding capabilities.

Technical features: PerceptionBench employs a bottom-up error classification method, growing its evaluation dimensions from real failure cases. Question design follows the principle of "perception-reasoning decoupling," ensuring that each question challenges only a single perception ability, avoiding interference from reasoning chains or domain knowledge. All answers use concise and unambiguous formats, reducing uncertainty in automated evaluation and making the benchmark results more reliable.

2. Key Features

• Atomic Capability Diagnosis: Breaks down visual perception into 10 atomic capabilities, including visual relationships, counting, attributes, depth, localization, composition, fine-grained recognition, context, OCR, and hallucination. Each capability corresponds to a set of specially designed questions, enabling precise identification of model breakdown points in specific perception stages and providing module-level optimization directions for development teams.

• Isolated Evaluation: Each question tests only a single atomic capability, strictly avoiding interference from reasoning and domain knowledge. Traditional benchmarks often involve questions that simultaneously test perception, reasoning, and knowledge, making it difficult to attribute low scores; PerceptionBench uses an isolated design to ensure scores truly reflect the model's performance at the pure perception level, rather than its overall capabilities.

• Failure Case Attribution: Based on failure cases from state-of-the-art models across 42 existing benchmarks, 1,800 atomic sub-questions have been decomposed. These questions are directly derived from real-world errors, making the evaluation dimensions closely aligned with actual model weaknesses rather than hypothetical "ideal scenarios," thereby enhancing the ecological validity of the benchmark.

• Comprehensive Capability Profiling: The Leaderboard displays detailed scores for each of the 10 capabilities, forming a radar-chart-like capability profile. Two models with identical total scores may have vastly different capability distributions; this profiling reveals hidden capability gaps masked by total scores, offering more granular insights for model selection.

• Short Answer Unambiguous Design: All questions require concise and clear answers, such as numbers, words, or phrases, avoiding the ambiguity that comes with open-ended responses. This design makes automated evaluation more reliable, reduces the cost of manual verification, and improves the comparability of scores across different models.

• Perception-Reasoning Decoupling: The question design strictly ensures that challenges stem from visual perception itself, rather than reasoning chains or domain knowledge. For example, counting questions only require identifying the number of objects, without needing to understand their functions; localization questions only require determining spatial relationships, without requiring logical inference. This decoupling allows the evaluation to isolate and measure the model's perception boundaries.

• Open Source and Reproducible: Provides complete code, Hugging Face datasets, and technical papers, supporting community reproduction and extension. Researchers can add new atomic capabilities or adjust question difficulty based on the existing framework, promoting the continuous evolution of visual perception evaluation standards.

3. How to Use

1. Environment Requirements and Prerequisites: To use PerceptionBench, you need a Python 3.8 or higher environment with PyTorch and related dependencies installed. The evaluation framework itself does not have special hardware requirements, but the multi-modal model to be evaluated needs corresponding GPU resources (recommended: NVIDIA A100 or equivalent computational power). The operating system supports Linux and macOS.

2. Clone the Repository: Pull the project code from GitHub:

   git clone https://github.com/MoonshotAI/PerceptionBench.git
   cd PerceptionBench
   

   Install the required dependencies:

   pip install -r requirements.txt
   

3. Download the Dataset: Obtain 3,000 validation questions and their annotated data from Hugging Face:

   python scripts/download_data.py
   

   The dataset is downloaded by default to the data/ directory and includes images, questions, answers, and capability labels.

4. Integrate the Model to be Evaluated: Package the multi-modal model to be evaluated into an interface that the evaluation framework can call. The framework provides an example script examples/run_model.py, and users need to implement the ModelAdapter class in the standard format, defining the generate_answer(image, question) method, which returns a short answer text.

5. Run the Evaluation: Execute the evaluation script to automatically calculate the model's accuracy on 10 atomic capabilities and overall performance:

   python run_benchmark.py --model your_model_adapter
   

   The evaluation results will generate a detailed report, including scores for each capability, error distribution, and sample failure cases.

6. View Capability Profile: Compare your model's performance with the official Leaderboard to identify its weaknesses. The Leaderboard provides radar charts and score tables for the 10 capabilities of current mainstream models. Users can compare their model's scores with public results to identify relatively weaker areas and guide subsequent training optimization.

Notes: Before evaluation, ensure that the model's output format matches the expected answer format (short and unambiguous). If the model outputs long text, it is recommended to use a post-processing function to extract key information. During evaluation, it is advisable to record the inference time for each question to analyze the trade-off between latency and accuracy.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For multimodal model development teams, PerceptionBench should be prioritized during the mid-training phase for fine-grained diagnosis of perception modules, enabling targeted optimization of data or architecture after identifying specific capability shortcomings. Before model release, use comprehensive benchmarks such as MMMU and MMBench to evaluate the model's overall capabilities, ensuring a balanced performance in perception, reasoning, and knowledge. For application scenarios primarily focused on hallucination issues (e.g., object fabrication in visual question answering), POPE provides a more direct hallucination rate metric, which can be used in conjunction with the hallucination sub-capability in PerceptionBench for a more comprehensive hallucination assessment. For academic researchers, PerceptionBench can serve as a specialized benchmark for publishing related papers, while MMMU and MMBench are suitable for comprehensive capability comparison experiments.

6. Editor's Summary

PerceptionBench holds significant value in technological innovation: it is the first to systematically decompose the visual perception capabilities of multimodal models into 10 atomic dimensions. It employs a bottom-up error classification approach, extracting evaluation dimensions from real failure cases, making this data-driven methodology more ecologically valid than traditional human-assumed dimensions. Its "perception-reasoning decoupling" design philosophy precisely addresses the issue of capability mixing in traditional benchmarks, which leads to attribution difficulties. This allows evaluation results to directly guide model optimization, rather than merely providing a general ranking. In terms of practical value, PerceptionBench fills a gap in the field of multimodal evaluation at the fundamental perception level, offering a module-level diagnostic tool for model development and helping to accelerate the iteration and improvement of perception capabilities. Currently, none of the 16 mainstream models have achieved an accuracy rate of 60%, indicating that atomic-level visual perception remains an industry bottleneck. The release of PerceptionBench is timely, capable of guiding the research community to focus on this foundational issue.

In terms of target users, PerceptionBench is most suitable for algorithm engineers and researchers in multimodal model development teams, for post-training capability diagnosis and comparative experiments. It is also appropriate for enterprise technical decision-makers who can use the capability profile to select the most suitable commercial models based on specific perception needs (such as OCR, counting). Academic researchers can use it as a standard evaluation tool to publish perception-related papers, promoting standardization in this field. In terms of future development potential, PerceptionBench has the potential to become a benchmark standard in the field of multimodal perception. As the community contributes and the version iterates, its atomic capability system may further expand (such as adding motion perception, material perception), and the question set's scale and diversity will continue to grow. Moonshot AI's decision to open-source PerceptionBench and publish a technical paper will help attract more researchers to participate in its development, fostering a healthy ecosystem. Overall, PerceptionBench is a clearly positioned, rigorously designed, and highly practical evaluation tool, playing an important role in driving the trend of multimodal models shifting from "score-chasing competitions" to "capability building."

7. Application Scenarios

• Model Development Diagnosis: After training, multimodal teams can use PerceptionBench to identify shortcomings in the model's 10 atomic perception capabilities. For example, if the model performs poorly in the "counting" capability, the team can specifically increase the amount of counting-related training data or adjust the attention mechanism of the visual encoder, enabling data-driven, precise optimization.

• Model Selection Reference: When enterprises choose commercial multimodal models, they can refer to the capability profile of PerceptionBench based on their business needs. For instance, in document processing scenarios requiring high-precision OCR, models with higher scores in the OCR sub-capability can be selected; for quality inspection scenarios requiring fine-grained object recognition, models with strong fine-grained recognition capabilities should be prioritized, avoiding being misled by overall scores.

• Academic Research Benchmark: Researchers can use PerceptionBench as a standard evaluation tool for publishing papers on multimodal perception. Its open-source code and dataset ensure the reproducibility of experiments, and the scores for the 10 atomic capabilities can serve as quantitative metrics for ablation studies, verifying the improvement effects of new architectures or training methods at the perception level.

• Industry Capability Standards: Industry associations or standardization organizations can establish fine-grained industry standards for the visual perception capabilities of multimodal models based on PerceptionBench, replacing the traditional "overall score ranking" culture. By setting minimum thresholds for each atomic capability, the standards can drive models toward more accurate and consistent visual understanding.

• Training Data Filtering: Data teams can leverage the question design principles of PerceptionBench to automatically generate or filter training samples targeting specific perception capabilities. For example, if the model is found to be weak in the "visual relationship" capability, images containing spatial relationship annotations can be extracted from public datasets to build a specialized training set for reinforcement learning.

8. FAQ

Q: What is the fundamental difference between PerceptionBench and other multimodal benchmarks (e.g., MMMU)?
A: PerceptionBench focuses on diagnosing atomic-level visual perception capabilities, with each question testing only a single perception ability to avoid interference from reasoning and knowledge. Comprehensive benchmarks like MMMU evaluate perception, reasoning, and domain knowledge together, making it difficult to attribute scores to specific abilities. PerceptionBench provides a "capability profile" rather than a single overall score.

Q: What hardware configuration is required to use PerceptionBench?
A: The PerceptionBench evaluation framework itself has no special hardware requirements and can run evaluation scripts on a standard CPU. However, the multimodal model being evaluated will require corresponding GPU resources, and it is recommended to use an NVIDIA A100 or a GPU with equivalent computational power. If only using the publicly available Leaderboard results, no additional hardware is needed.

Q: How were the 10 atomic capabilities in PerceptionBench determined?
A: The team analyzed failure cases of state-of-the-art models across 42 existing benchmarks and built an error classification tree from the bottom up. The 10 atomic visual perception capabilities were extracted from the earliest failure points. This process ensures that the dimensions are derived from real model weaknesses, not theoretical assumptions.

Q: How should the results from PerceptionBench be interpreted?
A: The results include the accuracy of each of the 10 atomic capabilities and the overall average accuracy. It is recommended to focus on the capabilities where the model scores below 50%, as these represent the main weaknesses in the model's perception layer. Comparing your model's capability profile with similar models on the Leaderboard can help identify relatively weaker areas.

Q: Does PerceptionBench support Chinese multimodal models?
A: The questions in PerceptionBench are primarily in English, but visual perception tasks themselves have low language dependency (e.g., counting, localization, attribute judgment). Chinese models only need basic English visual question-answering capabilities to participate in the evaluation. For purely Chinese models, question language adaptation may be required, and the community can extend the Chinese version based on the open-source framework.

Q: Will the questions in PerceptionBench be updated regularly?
A: The current version of PerceptionBench is v1.0, which includes 3,000 validated questions. After the project is open-sourced, the community can contribute new questions or new capability dimensions. The official team plans to update the dataset and Leaderboard periodically based on model evolution, ensuring the benchmark remains up-to-date.

9. Project Links

• Project Website (Blog Introduction): https://www.kimi.com/blog/perception-bench
• GitHub Repository: https://github.com/MoonshotAI/PerceptionBench
• Hugging Face Dataset: https://huggingface.co/datasets/moonshotai/PerceptionBench
• Technical Paper (PDF): https://github.com/MoonshotAI/PerceptionBench/blob/master/paper/PerceptionBench.pdf