MineExplorer – Meituan's Open-World Minute-Level Long-Horizon Task Evaluation Benchmark

1. What is MineExplorer

MineExplorer is the first open-world minute-level long-horizon task evaluation benchmark introduced by Meituan's LongCat team, based on Minecraft. This benchmark includes 813 manually verified instances with 1–4 hops. Task design incorporates hidden prerequisites, and instructions do not explicitly enumerate the dependency graph, requiring the model to autonomously infer subtasks. MineExplorer actively strips away game-specific knowledge, testing only general dynamic exploration capabilities, and is accompanied by a five-agent collaborative data synthesis framework and a rule-based milestone automatic evaluation system. The code and dataset have been open-sourced.

Technical Positioning and Domain: MineExplorer belongs to the field of multi-modal large model evaluation, focusing on the assessment of open-world long-horizon task planning and reasoning capabilities. Unlike existing static question-answering or single-step evaluation benchmarks, it emphasizes continuous decision-making in dynamic interactive environments, filling the gap in minute-level long-horizon task evaluation.

Development Background: Developed by Meituan's LongCat team, it aims to address the lack of minute-level long-horizon tasks in current evaluation benchmarks, over-reliance on game-specific knowledge, and the need for manual intervention during evaluation. The team has deep expertise in embodied intelligence and evaluation, and this benchmark represents a significant achievement in promoting the assessment of general-purpose Agent capabilities.

Core Value: Provides a standardized, reproducible evaluation of open-world exploration capabilities for multi-modal large models. Through the design of hidden prerequisites and knowledge decoupling, it truly tests the model's general dynamic exploration ability, driving forward cutting-edge research in long-horizon reasoning, autonomous planning, and implicit task decomposition.

Technical Features: Based on a real-time 3D sandbox environment, each task runs for 1800 environment steps; employs implicit DAG task modeling, where instructions do not explicitly list the dependency graph; generates tasks through five-agent collaboration, achieving an efficiency improvement of approximately 30%; uses a rule-based milestone checker for automatic scoring, eliminating the need for manual annotation and ensuring reproducible results.

2. Key Features

• Minute-level Open World Evaluation: Based on the real-time 3D sandbox of Minecraft, each task runs for 1800 environment steps (approximately 3 minutes). The model must continuously adjust its strategy based on dynamic visuals, rather than providing step-by-step responses, truly testing its long-term planning and adaptive capabilities. This creates an essential distinction from static evaluations.

• Hidden Precondition Multi-hop Tasks: Provides 813 manually verified instances with 1–4 hops. The instruction only gives the final goal, without enumerating the dependency graph. The model must independently infer the hidden subtasks and their dependencies, completing implicit task decomposition and execution, effectively testing high-level planning and reasoning capabilities.

• Knowledge-Decoupled Evaluation: Proactively removes Minecraft-specific mechanics, retaining only tasks that rely on general world knowledge. By using an LLM judge to filter domain-specific knowledge, the evaluation focuses on general dynamic exploration capabilities, avoiding biases from game-specific knowledge and enhancing the generalizability of the results.

• Rule-based Automated Evaluation: Converts subtasks into automatically executable milestone checkers, calculating task success rate (TSR) and milestone success rate (MSR) based on quantifiable metrics such as inventory items and coordinate regions. No manual annotation is required, ensuring reproducible results and enabling cross-model comparisons.

• Multi-agent Data Synthesis: Five Agent collaboration automatically generates tasks, covering the full process from task selection, scenario design, milestone design, to expert review and validation by Minecraft specialists. Compared to single-agent approaches, this method improves efficiency by approximately 30%, supports custom extensions from 1–12 hops, and significantly reduces the cost of manual task creation.

• Fine-grained Capability Decomposition Assessment: Drawing on the ReAct paradigm, the open-world exploration capability is systematically decomposed into three dimensions—perception, reasoning, and action—comprising 14 fine-grained metrics. It provides a comprehensive capability profile, helping researchers precisely identify shortcomings in areas such as object recognition, path planning, and action execution.

3. How to Use

1. Environment Requirements and Preparation: Clone the GitHub repository to your local machine and ensure that Python 3.9+ and the relevant Minecraft runtime environment (Java 17+) are installed. It is recommended to use a Linux system with at least 16GB of memory and an NVIDIA GPU (RTX 3090 or higher is recommended) for smooth operation of the sandbox environment.

2. Launch Evaluation Environment: Configure and start the Minecraft sandbox server, then execute the launch script (e.g., python launch_server.py). Ensure the evaluation environment can run in real-time and receive model action instructions. After verifying the environment connectivity, load the evaluation instances provided by MineExplorer.

3. Run Benchmark Evaluation: Connect the multi-modal model to be tested into the environment, where the model outputs action sequences based on real-time visuals. Run the main evaluation program (e.g., python run_evaluation.py --model your_model), and the system will automatically load 813 validation instances and execute them, with each instance running for 1800 steps.

4. Automatic Scoring and Result Review: After the evaluation is complete, the system automatically invokes the rule-based milestone checker, which calculates the TSR and MSR scores based on quantifiable metrics such as inventory items and coordinate regions. The evaluation report automatically outputs scores across various dimensions, supporting layered comparisons by model and difficulty for easier capability analysis.

5. Custom Task Generation: Run the multi-agent data synthesis script (e.g., python generate_tasks.py --hops 4), and configure the number of task hops (1–12 hops). Five collaborative agents automatically generate new task instances with hidden preconditions. It is recommended to manually verify the quality of the generated tasks.

6. Training and Iteration: Input the generated tasks as training data into the model and repeatedly perform exploration-feedback loops within the Minecraft sandbox. The system supports both online learning and offline training modes, and can be combined with reinforcement learning frameworks to enhance the model's long-term planning capabilities in open-world scenarios.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For researchers aiming to evaluate the long-horizon planning and dynamic adaptation capabilities of multimodal large models in open-world environments, MineExplorer offers a more realistic evaluation setting, with its hidden preconditions and knowledge decoupling mechanisms providing greater challenges and generalization significance. If the research focus is on completing complex tasks by leveraging internet knowledge, MineDojo's large-scale knowledge integration and creative task evaluation are more appropriate. For studies on embodied tasks in structured indoor environments, ALFRED and Habitat provide mature, standardized benchmarks, particularly with deep expertise in navigation and object interaction. In practice, selection should be based on evaluation objectives, environmental complexity, and knowledge dependency requirements.

6. Editor's Summary

MineExplorer demonstrates significant technological innovation: it introduces a novel minute-level open-world evaluation, expanding task duration from a single step to 1800 steps of dynamic interaction, thereby filling the gap in long-term planning evaluations; the implicit DAG task modeling forces the model to autonomously infer subtasks, increasing the challenge and realism of the evaluation; the knowledge decoupling design filters domain-specific knowledge through an LLM judge, ensuring the evaluation focuses on general capabilities and reduces bias caused by game-related knowledge. The multi-agent collaborative question generation and rule-based automated evaluation framework significantly reduces human labor costs and enhances the reproducibility of evaluations. In terms of practical value, this benchmark provides a standardized and scalable platform for assessing open-world exploration capabilities in multimodal large models, helping to advance research on planning and decision-making for agents in dynamic environments. Its primary users include researchers in multimodal AI, embodied intelligence developers, and model selection engineers. In the future, MineExplorer can be further expanded to include more jumps, more environments (such as indoor and outdoor simulators), and enriched evaluation dimensions, potentially becoming an important reference standard for assessing general dynamic exploration capabilities.

7. Application Scenarios

• Multimodal Large Model Evaluation: Acts as a standardized testing ground for top-tier MLLMs such as Claude, GPT, and Gemini, evaluating their dynamic open-world long-term planning and reasoning capabilities. Through 1800 steps of interaction and hidden prerequisite tasks, it comprehensively assesses the model's perception, reasoning, and action capabilities, outputting 14 fine-grained metrics.

• Agent Training Environment: Within the Minecraft sandbox, the model enhances its perception-reasoning-action closed-loop capabilities through repeated exploration-feedback cycles. It supports online learning and offline training, with customizable task difficulty and jump counts to accelerate Agent iteration.

• Model Selection Reference: Provides an open-world exploration capability leaderboard for the industry, aiding in model selection and boundary assessment. Through fine-grained metrics and hierarchical comparisons, it helps developers choose the most suitable model for specific scenarios.

• Academic Research Platform: Promotes research in cutting-edge areas such as long-term reasoning, implicit task decomposition, visual grounding, and action alignment. Open-sourced code and datasets lower the research barrier, supporting custom extensions and fostering community collaboration.

• Embodied Intelligence Pre-research Validation: Offers a low-cost sandbox for validating capabilities in real-world physical applications such as robotics and autonomous driving. It enables testing of planning and adaptation abilities in controlled environments, accelerating the transition of technology from simulation to reality.

8. FAQ

Q: What hardware requirements does MineExplorer have?
A: It is recommended to use a Linux system with at least 16GB of memory and an NVIDIA RTX 3090 or higher GPU to run the Minecraft sandbox and evaluation framework. A CPU with at least 4 cores and a Java 17+ environment is also recommended. Lower configurations may result in slow sandbox performance, affecting evaluation efficiency.

Q: How can I integrate my model with MineExplorer evaluation?
A: The model must implement a standard interface to receive real-time visuals and output action sequences. The evaluation framework provides example code and API documentation (see the examples directory in the GitHub repository). Follow the guide to configure your model for integration.

Q: What specific evaluation metrics does MineExplorer use?
A: The main metrics include Task Success Rate (TSR) and Milestone Success Rate (MSR), as well as 14 fine-grained indicators across three dimensions: perception, reasoning, and action. Examples include object recognition accuracy, path planning efficiency, and action execution success rate.

Q: What are the main differences between MineExplorer and MineDojo?
A: MineExplorer focuses on general dynamic exploration capabilities, conceals prerequisite tasks, and removes game-specific knowledge. MineDojo emphasizes using internet knowledge to complete tasks with clear instructions. The two have different evaluation goals and can complement each other.

Q: How can I generate custom tasks?
A: Run the multi-agent data synthesis script (python generate_tasks.py --hops N) and configure the number of task hops (1–12). Five collaborative agents automatically generate new tasks with hidden prerequisites. It is recommended to manually verify the quality and difficulty of the generated tasks.

Q: Are the evaluation results reproducible?
A: Yes. The rule-based milestone checker automatically scores based on quantifiable metrics such as inventory items and coordinate regions, ensuring consistent evaluation results each time. It supports multi-model comparisons, making the results reproducible and suitable for academic research and industrial model selection.

9. Project Links

• GitHub repository: https://github.com/meituan-longcat/MineExplorer
• arXiv technical paper: https://arxiv.org/pdf/2605.30931
• Meituan LongCat team GitHub organization: https://github.com/meituan-longcat