E-Bench – A Benchmark for Evaluating AI Agents Introduced by Tencent HunYuan and Others

1. What is E-Bench

E-Bench is a multi-step tool usage evaluation benchmark introduced jointly by the Tencent HunYuan team, along with the AI Research Institute (AIR) at Tsinghua University and Southeast University. It is based on three real-world products—Honor of Kings, QQ Music, and Tencent Meeting—and constructs a fully synthetic virtual environment. This environment includes 323 state change tasks, 41 database tables, and over 76,000 data records. Through a dual asymmetric design involving information and tool gaps, E-Bench forces AI agents to actively gather hidden information and orchestrate multi-step tool calls. The final evaluation is based on a deterministic database state diff score, revealing the performance boundaries of AI agents in real product scenarios.

Technical Positioning and Domain: E-Bench belongs to the AI agent evaluation domain and focuses on assessing the comprehensive capabilities of Large Language Model (LLM) Agents in multi-step tool usage, information gathering, and state changes. Compared to existing benchmarks, E-Bench emphasizes in-depth evaluation in real product-level scenarios, addressing the gap in achieving both environmental realism and evaluation determinism in tool usage assessments. It provides a more refined quantitative method for diagnosing agent capabilities.

Development Background: Developed primarily by the Tencent HunYuan team, in collaboration with the AI Research Institute (AIR) at Tsinghua University and Southeast University. The team has extensive experience in LLM Agents and agent evaluation, aiming to solve issues such as the limited task types in existing benchmarks and the high risk of contamination caused by reliance on real codebases. E-Bench offers a more reliable and scalable evaluation solution for the industry, promoting standardized assessment of agent technologies.

Core Value: The core value of E-Bench lies in its provision of a completely synthetic yet highly realistic virtual product environment. Through its dual gap design, it compels agents to actively explore and reason rather than relying on training memory. The deterministic state diff scoring eliminates subjective bias, ensuring stable and reproducible evaluation results. This provides a reliable basis for model selection and capability diagnosis, while also reducing benchmark maintenance costs.

Technical Features: It employs graph-guided database population technology to generate a self-consistent virtual world, ensuring cross-table consistency and data scale. Task generation is achieved through intent rewriting and rule substitution, eliminating the possibility of accidental correctness due to environmental sparsity. Additionally, it supports joint cost-performance analysis, helping assess the cost-effectiveness of models and providing quantitative support for productization decisions, offering both evaluation depth and flexible scalability.

2. Key Features

• Fully Synthetic Product Environment Construction: Based on graph-guided database population technology, it generates three virtual product worlds with no isolated records and cross-table consistency. Each database is populated with 76,000+ rows and 600,000+ data units, with the target entity surrounded by a large number of similar records. This ensures that the model cannot guess correctly based on sparse environmental cues, guaranteeing the discriminability and challenge of the evaluation.

• Dual Gap Task Generation: The question generation model has full access to the database, while the tested model can only interact through restricted business tools, creating both information and tool asymmetry. This forces the agent to actively search for hidden information and orchestrate multi-step tool calls, simulating real-world product interactions with incomplete information and tool limitations, and reproducing the core challenges of actual deployment.

• Deterministic State Evaluation: The state difference before and after database operations is used as the sole evaluation criterion. Only exact matches are counted as successful, eliminating partial credits and evaluator variance. The evaluation results are stable and reproducible, unaffected by subjective semantic judgments, ensuring fair comparisons between models and enhancing the reliability of the benchmark.

• Cost-Performance Joint Analysis: Accurately record the API cost for each task and plot the Pareto frontier of accuracy and cost, evaluating the model's cost-effectiveness. This helps enterprises identify high-value models, make trade-offs between accuracy and API cost, optimize deployment expenses, and provide data support for productization decisions.

• Code Execution Extension (E-Bench-Code): The exec_code tool is open-sourced, allowing the separation of two types of difficulty: orchestration mechanisms and information retrieval. It supports research into the benefit boundaries of code execution for different task types (computation-intensive vs. reasoning and decision-making), providing a reference for Agent architecture design and exploring optimal code offloading strategies.

• Task Generation and Cross-Validation: Each task is generated through a productization cycle of viewing data → identifying the target → modifying the state → summarizing the intent. Three different strong Agent models perform cross-validation, and only tasks that are consistently reproduced by at least two models are adopted. This ensures task solvability and annotation accuracy, avoiding ambiguous tasks and improving benchmark quality.

• Decoupled and Reusable Environment and Tasks: A single self-consistent product environment can support the generation of approximately hundreds of tasks, with extremely low marginal cost. The environment layer is controllable, and the task layer is extensible, making it easy for researchers to customize tasks, promoting community collaboration, and reducing the threshold for benchmark expansion.

3. How to Use

1. Environment Preparation and Loading: Obtain the fully synthetic virtual environment E-Bench from the accompanying resources of the paper, which includes database templates covering three product domains: Honor of Kings, QQ Music, and Tencent Meeting, along with 323 evaluation tasks. You will need a GPU environment capable of running an LLM Agent (recommended: NVIDIA A100 or equivalent computational power, with at least 16GB of VRAM) and a Python 3.8+ runtime environment. Install the required dependencies for the evaluation framework to ensure the environment is isolated and reproducible.

2. Version Selection and Configuration: Choose between the base version of E-Bench or E-Bench-Code based on your research objectives. The base version focuses on multi-step orchestration and information retrieval, while the Code version additionally tests code execution capabilities. Download the corresponding evaluation framework code, configure the database connection parameters (such as database address, port, username, and password), and set appropriate concurrency levels and timeout durations to avoid resource contention.

3. Model Integration: Integrate the target LLM Agent into the evaluation framework, configuring it to interact with the environment only through restricted domain-specific tools, and prohibiting direct queries to the underlying database. Implement the Agent's interface adaptation to ensure compatibility with the tool calling format of the framework, and set up reasonable retry strategies and error handling mechanisms to maintain the stability of the evaluation process.

4. Task Configuration and Execution: Define the number of independent runs per task (recommended to be at least 3 to assess stability) and the conditions for initiating a new database instance. Enable API call cost tracking. Launch the automated evaluation, during which the Agent receives natural language instructions and interacts with the virtual environment through multi-step tool calls to modify its state. The system automatically records the complete operational trajectory and Token consumption, while also monitoring for anomalies.

5. Result Analysis and Cost Evaluation: After the evaluation is complete, the system outputs the Pass@1 and Pass³ accuracy based on the deterministic database state differences. A joint cost-performance analysis report is generated. You can plot a Pareto front to evaluate the cost-effectiveness of different models and support productization decisions. When comparing multiple models, it is recommended to keep the number of runs and configurations consistent to ensure a fair comparison.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations:

If the research goal is to evaluate an Agent's multi-step tool usage and state change capabilities in real product scenarios, and a highly controllable and reproducible evaluation environment is required, E-Bench is an ideal choice. Its fully synthetic environment avoids pollution issues, and the joint cost-performance analysis helps with model selection and productization decisions, making it especially suitable for teams requiring fine-grained capability diagnostics of Agents. However, for code repair tasks, SWE-bench is more direct, as its evaluation based on real code repositories better reflects actual engineering capabilities; if the focus is on the comprehensive capabilities of a general AI assistant in open-ended questions, GAIA offers a broader range of reasoning and knowledge application evaluations.

If the evaluation focus is on an Agent's performance in web interaction scenarios, WebArena provides a more user-operations-oriented environment. E-Bench has advantages in depth and controllability within specific product domains, but its domain coverage is limited. It is recommended to combine multiple benchmarks for a comprehensive evaluation of Agent capabilities to obtain a more complete capability profile. For enterprise-level applications, E-Bench's joint cost-performance analysis feature is particularly valuable, helping to balance accuracy and API overhead, thereby reducing productization risks.

6. Editor's Summary

E-Bench stands out in terms of technological innovation, with its dual-gap design and graph-guided database population method addressing the core contradiction in existing evaluation benchmarks: the difficulty of achieving both environmental realism and evaluation determinism. By introducing an information gap, E-Bench forces the Agent to actively gather hidden information, while the tool gap restricts internal computational tools, replicating real-world product interaction scenarios where information is incomplete and tools are limited. This makes the evaluation tasks more aligned with practical applications. The deterministic state diff scoring mechanism eliminates subjective bias, ensuring high reproducibility of evaluation results and providing a reliable foundation for comparing model capabilities, marking a significant advancement in the field of intelligent agent evaluation.

In terms of practical value, E-Bench offers a standardized and quantifiable evaluation tool for assessing Agent capabilities, benefiting both enterprises and researchers. Its cost-performance joint analysis feature helps identify models with high cost-effectiveness, directly supporting productization decisions and reducing uncertainty in model selection. The environment-task decoupling design lowers the cost of expansion and supports custom task generation, enhancing the long-term usability of the benchmark and enabling researchers to quickly build new evaluation tasks tailored to their specific needs.

E-Bench is primarily suitable for LLM Agent developers, AI product managers, and researchers, especially those who need to evaluate an Agent's actual performance in complex product scenarios. For use cases such as model selection, capability diagnosis, and training data synthesis, E-Bench provides robust evaluation support, helping teams identify shortcomings in areas like information gathering and tool orchestration.

In terms of future development potential, E-Bench's methodology demonstrates strong transferability and can be extended to more product backends and CLI scenarios, potentially driving the establishment of universal evaluation standards for intelligent agents. The code execution extension version introduces a new experimental dimension for Agent architecture research, aiding in the exploration of the benefits and limits of code offloading. As more teams contribute to its development and expand its application domains, E-Bench is expected to become one of the important benchmarks in the intelligent agent evaluation field, contributing to the standardization of Agent technology assessments.

7. Application Scenarios

• AI Agent Capability Evaluation: Provides standardized, quantifiable multi-step tool usage capability assessment for LLM Agents, covering core dimensions such as information retrieval, state change, and cross-entity orchestration. Researchers can quickly identify shortcomings in the model's information gathering and tool orchestration capabilities using E-Bench, guiding model optimization and improving Agent performance in real-world product scenarios.

• Model Selection and Productization Decision-Making: Through a cost-performance Pareto chart, helps enterprises balance accuracy and API costs, identifying high-cost-performance models. Applicable for cloud service procurement and evaluation of model deployment strategies, reducing productization risks and ensuring decisions are data-driven.

• Agent Training Data Synthesis: Acts as a controllable and scalable source of training data to continuously enhance Agent capabilities in multi-step tool calling and reliability. It can generate a large volume of diverse tasks by decoupling environment tasks, enriching the training data distribution and improving the Agent's generalization ability.

• Code-Enhanced Agent Research: E-Bench-Code supports research into the performance gains of code execution for different types of tasks (computation-intensive vs. reasoning and decision-making), helping design more efficient Agent architectures and determining when computational tasks should be offloaded to code execution to optimize reasoning efficiency.

• Cross-Domain Benchmark Expansion: The methodology can be applied to more product backends and CLI scenarios, promoting the establishment of general-purpose Agent evaluation standards. Researchers can build custom evaluation environments based on the E-Bench framework to test Agent capabilities across different domains, fostering a diverse evaluation ecosystem.

8. FAQ

Q: What is the main difference between E-Bench and SWE-bench?
A: E-Bench focuses on multi-step tool usage and state change tasks, operating in a fully synthetic virtual product environment, with scoring based on database state diffs; SWE-bench focuses on code repair, based on real GitHub Issues, with scoring based on unit test pass rates. E-Bench emphasizes the design of information and tool gaps, forcing the Agent to actively explore, while SWE-bench focuses on code understanding and patch generation. The two benchmarks evaluate different dimensions.

Q: What hardware requirements are needed to participate in E-Bench evaluation?
A: A GPU environment capable of running LLM Agents is required, with NVIDIA A100 or equivalent GPU computing power recommended, and at least 16GB of VRAM. Additionally, a Python 3.8+ environment is needed, along with the dependency libraries required by the evaluation framework. Specific hardware requirements depend on the size of the model being tested and the inference framework. It is advised to reserve sufficient resources to ensure evaluation efficiency.

Q: Are E-Bench tasks reproducible?
A: Yes, E-Bench uses a fully synthetic environment and deterministic database state diff scoring, allowing for precise reproduction of evaluation results each time. It is recommended to run each task multiple times (e.g., three times) independently to assess stability, ensuring a new database copy is used each time to avoid residual states. Additionally, record the random seed to guarantee reproducibility.

Q: How can E-Bench be extended to new product domains?
A: E-Bench's decoupled environment-task design makes extension relatively straightforward. Researchers can build new virtual product environments using graph-guided database population techniques, define business tool interfaces, and create evaluation tasks following the task generation process. The framework provides relevant tools and scripts for support, but some development work is required. It is recommended to refer to the methodology described in the paper.

Q: Why does E-Bench choose database state diff as its scoring mechanism?
A: Database state diff provides a deterministic and objective scoring standard, avoiding subjective bias from semantic judges and the ambiguity of partial credit. Only exact matches are counted as passes, ensuring strict and reproducible evaluation results. This promotes fair comparisons between models and reduces variance across different judges, thereby enhancing the credibility of the benchmark.

9. Project Links

• arXiv technical paper: https://arxiv.org/pdf/2607.23722