HarnessEval – The Evaluation System Jointly Launched by MirroS, Tsinghua University, Peking University, and NVIDIA

1. What is HarnessEval

HarnessEval is an agentic evaluation system jointly developed by MirroS with top institutions such as Tsinghua University, Peking University, UC Berkeley, MIT, and NVIDIA. This system agentifies the evaluation process itself, dynamically planning evaluation paths for each case, selecting applicable skills from the Skill Library, decomposing abstract problems into verifiable sub-problems, collecting multi-dimensional evidence through tool calling, and ultimately outputting a structured Evidence Tree. HarnessEval breaks through the limitations of traditional static benchmarks, transforming evaluation from a single black-box score into an interpretable, reproducible intelligent workflow. Its first implementation, HarnessEval-W, has been specifically designed for evaluating interactive world models.

Image source: Official article
Image source: official article

Technical positioning and domain: HarnessEval belongs to the field of AI evaluation and assessment technologies, focusing on the capability evaluation of interactive world models (World Models) and embodied intelligent systems. Unlike traditional video generation evaluation, it emphasizes the verification of dynamic characteristics such as action input, state transition, physical causality, and world continuity, filling the gap in existing evaluation systems for interactive scenarios.

Development background: Led by the MirroS laboratory, this system was jointly developed with Tsinghua University, Peking University, UC Berkeley, MIT, and NVIDIA. The team has deep expertise in agent systems, world models, and automated evaluation, aiming to address core pain points in current evaluation systems, such as the inability to adapt to the rapid evolution of world models, lack of interpretability in evaluation results, and difficulty in reproduction.

Core value: HarnessEval elevates evaluation from mechanical metric calculations to an intelligent analytical process, enabling evaluation conclusions to have complete reasoning chains and visual evidence, making them traceable, auditable, and reproducible. It reduces the subjectivity and cost of manual evaluation while providing model developers with precise failure mode identification, accelerating model iteration.

Technical features: It employs a four-layer agentic workflow (Plan-Router-Decompose-Verify), where the main agent oversees planning and verification, while each Skill Agent is responsible for specialized evaluations, and sub-agents handle specific subtasks, forming a hierarchical collaborative architecture. It includes an expandable Skill Library (initially featuring 9 core skill modules), supporting the dynamic addition of new skills to continuously expand evaluation capabilities as models evolve. Finally, it outputs an Evidence Tree, fully recording the test content, tool calls, visual evidence, and reasoning chain.

2. Key Features

• Dynamic Evaluation Planning: Instead of applying a fixed process to every case, the system automatically generates a customized evaluation plan based on the context and objectives of each case. During the Plan phase, the system understands the case background and outlines the evaluation path, ensuring the evaluation strategy is highly aligned with the specific scenario and avoiding resource waste on irrelevant metrics.

• Intelligent Skill Routing: Skills are selected from the Skill Library on an as-needed basis, rather than applying the same fixed set of metrics to all cases. During the Route phase, the system dynamically routes to the corresponding Skill Agent based on the evaluation plan, enabling adaptive combinations of evaluation dimensions and improving evaluation efficiency and relevance.

• Hierarchical Problem Decomposition: Abstract high-level evaluation questions (e.g., "physical plausibility") are broken down into measurable and verifiable sub-questions (e.g., "object collision response," "consistency of gravitational acceleration"). During the Decompose phase, these sub-questions are delegated to the corresponding sub-agents, achieving fine-grained decomposition and parallel processing of complex tasks.

• Multi-Angle Evidence Collection: Sub-agents and diagnostic tools are called to collect visual and logical evidence from different perspectives. For example, object permanence is verified through object detection, motion coherence is assessed using optical flow analysis, and action consistency is checked via state comparison, ensuring comprehensive evaluation basis.

• Evidence Verification and Logical Integration: The main agent verifies the quality and logical relationships of the evidence from each branch, eliminating contradictory or low-confidence evidence, and integrates multi-source evidence into a coherent reasoning chain. The Verify phase ensures the reliability and self-consistency of the final conclusion, preventing misleading results from single metrics.

• Evidence Tree Output: Evaluation results are presented in a hierarchical evidence tree format, fully documenting the test content, tool calls, visual evidence, and reasoning chain. Each score node is accompanied by traceable evidence, making the evaluation conclusion verifiable, reproducible, and accountable, rather than just outputting a single scalar score.

• Expandable Skill Library: The system includes 9 core skill modules by default, covering dimensions such as rendering quality, physical plausibility, state change verification, drift analysis, revisit consistency, and off-screen evolution. It supports dynamically adding new skills as model capabilities evolve, continuously expanding the evaluation dimensions through a recursive self-improvement mechanism.

• Automatic Evaluation Case Construction: Based on scene taxonomy, the system automatically generates initial world states, action sequences, and validation criteria, ensuring the quality and diversity of evaluation cases. The system can generate various interactive scenarios automatically, reducing the workload of manually constructing cases and improving evaluation coverage.

3. How to Use

1. Environment Requirements and Prerequisites: Requires Linux or macOS system, Python 3.8 or higher, and it is recommended to have an NVIDIA GPU (at least 8GB VRAM) to accelerate inference. Ensure Git and Conda package manager are installed.

2. Clone the Repository: Pull the HarnessEval-W codebase from GitHub. Run the command git clone https://github.com/mirros-lab/harnesseval-w.git and enter the project directory.

3. Create Isolated Environments: Create three separate conda environments to isolate different dependencies:

   • conda create -n harnesseval-main python=3.9 (for the main framework)
   • conda create -n harnesseval-metrics python=3.9 (for metric calculations)
   • conda create -n harnesseval-pavrm python=3.9 (for the PavRM model)
     Each environment should install the corresponding dependencies based on the provided environment.yml file in the project.

4. Configure Credentials and Variables: Copy config/example.env to harnesseval.env and modify the API key, model paths, and other parameters according to your actual environment. Load the environment variables using source harnesseval.env to ensure the evaluation framework can correctly access the required resources.

5. Prepare Evaluation Data: Place the video results generated by the model under evaluation in the designated directory (e.g., data/results/), and prepare the manifest.json manifest file, which should include metadata such as the action sequence and initial state description for each case. Refer to the project example for the manifest format.

6. Run Evaluation: Execute the harnesseval eval command, specifying the model result path, evaluation plan (optional default plan), and output directory. The system will automatically launch the Plan-Router-Decompose-Verify workflow to dynamically evaluate each case. A custom evaluation plan can be specified using the --plan parameter.

7. Verify and View Reports: Run harnesseval verify run to check whether the evaluation output is complete and scores are generated correctly. Finally, view the summary.json (summary scores), leaderboard_latest.csv (ranking table), and LEADERBOARD.md (readable report) in the output directory to obtain detailed scoring and reasoning chains.

Notes: It is recommended to read the docs/quickstart.md document before the first run; if evaluating a custom model, ensure the output video format and resolution meet the requirements; extension of the Skill Library must follow the plugin interface specifications, see docs/skill_development.md for details.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: If the evaluation target is an interactive world model (e.g., video generation models that support action input and state transitions), HarnessEval is currently the only system that provides dynamic planning, interpretable evidence chains, and physical causality evaluation. It is especially suitable for research teams and enterprises requiring in-depth diagnostic analysis of model failure modes. If the evaluation target is a traditional video generation model (e.g., text-to-video, image-to-video), and only quick acquisition of visual quality scores is needed, VBench or the FVD+CLIP combination is more lightweight, mature, and has lower deployment costs. For scenarios requiring highly customized evaluation dimensions, HarnessEval's scalable Skill Library offers greater flexibility, though it requires a certain level of development investment.

6. Editor's Summary

HarnessEval has made significant innovations in evaluation methodology by introducing agent technology into the assessment process, achieving a paradigm shift from static metrics to dynamic reasoning. Its four-layer Agentic workflow and hierarchical agent architecture make evaluation not just a simple score calculation, but an interpretable and reproducible intelligent analysis process. Particularly, the output of the Evidence Tree allows each score to be traceable, providing practical value for model developers to identify specific failure modes (such as physical violations or goal drift). In terms of practical value, HarnessEval-W has been applied to the evaluation of interactive world models, filling the gap of lacking systematic assessment tools in this field. For research teams working in strong interactive scenarios such as embodied intelligence, autonomous driving, and game generation, this tool can significantly reduce the cost of manual evaluation and improve iteration efficiency. The primary users are AI researchers, algorithm engineers, and evaluation engineers, especially teams working on world models, video generation, and robotic simulations. In terms of future development potential, as the Skill Library expands and community contributions increase, HarnessEval has the potential to become the standard evaluation platform for interactive AI systems. However, the current version still has room for improvement in terms of usability, documentation completeness, and hardware compatibility. We look forward to future versions that can lower the usage threshold and extend support to more model types.

7. Application Scenarios

• Interactive World Model Evaluation: Assess the correctness of state transitions, physical plausibility, and world consistency in video generation models under action conditions. Researchers can input action sequences and initial states, and HarnessEval automatically plans the evaluation path, generating detailed reports with evidence trees to identify specific issues in dimensions such as object interaction and gravity response.

• Embodied Intelligence and Robotics Simulation: Evaluate the control capabilities, physical interaction causality, and long-term state consistency of robots in virtual environments. For example, assess whether a robot's grasping action results in the correct displacement of objects and whether the environment maintains its state after the robot leaves. The physical causality skills of HarnessEval can specifically detect such behaviors.

• Autonomous Driving World Models: Verify the adherence to camera trajectories, object permanence, off-screen evolution, and physical dynamics in generated scenarios. Autonomous driving teams can use HarnessEval to perform causality checks on the continuous frames generated by world models, ensuring that the movement of elements such as vehicles and pedestrians follows physical laws.

• Gaming and Virtual World Generation: Evaluate scene consistency, NPC behavior causality, revisit consistency, and visual quality in open-world games. Game developers can use HarnessEval to automatically detect whether objects in the scene undergo unreasonable changes after the player leaves, or whether NPC behavior follows predefined logic.

• AI Video Creation Platforms: Provide interpretable quality diagnostics for models such as Seedance, Kling, and Sora, identifying specific failure modes such as extraneous events, target drift, and physical violations. Platform operators can integrate HarnessEval into their quality inspection processes to automatically filter out unqualified generated content and provide modification recommendations.

8. FAQ

Q: What are the main differences between HarnessEval and VBench?
A: HarnessEval is specifically designed for interactive world models, supporting action input and state transition validation, and using an Agentic workflow to dynamically plan evaluation paths, outputting traceable evidence trees. VBench, on the other hand, targets traditional video generation models, employing a fixed metric pipeline to output scalar scores. HarnessEval emphasizes interpretability and physical causality assessment, while VBench is more lightweight and mature.

Q: How do I install HarnessEval? What hardware configurations are required?
A: Clone the code from the GitHub repository, create three conda environments, and install dependencies. Recommended hardware includes an NVIDIA GPU (with at least 8GB VRAM). The system requires Linux/macOS and Python 3.8+. For detailed steps, please refer to the project README and docs/quickstart.md.

Q: Does HarnessEval support custom evaluation skills? How can I expand the Skill Library?
A: Yes, it does. The Skill Library is designed as an extensible architecture. Users can develop new skill modules according to the plugin interfaces provided by the project and register them within the framework. Development documentation is located in docs/skill_development.md, and example skills can be referenced from the built-in 9 core modules.

Q: How should I interpret the Evidence Tree in the evaluation results?
A: The Evidence Tree is a hierarchical structure, with the root node representing the final score, child nodes representing the evaluation results of each skill module, and leaf nodes representing specific tool calls and visual evidence. Each node contains an inference chain and confidence level, allowing users to expand layer by layer to view the basis for the scores, which helps in identifying specific failure modes of the model.

Q: Is HarnessEval available for commercial use? What is the license?
A: The project uses an open-source license (please check the LICENSE file in the repository for details), supporting both academic and commercial use. However, note that some dependent components may have different licenses; ensure compliance before use.

Q: What should I do if errors or missing scores occur during evaluation?
A: First, run harnesseval verify run to check the completeness of the output. Common causes include: data format not meeting requirements, missing fields in manifest.json, and GPU VRAM insufficiency leading to task timeouts. You can locate specific errors by reviewing the run logs in the logs directory and adjust configurations or data accordingly based on the error messages.

9. Project Links

• Product Website: https://mirros.ai/blog/harnesseval
• GitHub Repository: https://github.com/mirros-lab/harnesseval-w