WorkBuddy Bench – Tencent's Open-Source Code Agent Evaluation Suite

1. What is WorkBuddy Bench

WorkBuddy Bench is an open-source evaluation suite for code agents developed by Tencent, designed to provide standardized capability assessments for AI agents in four real-world scenarios: code, web development, office work, and security. This suite includes 260 tasks, all constructed by reverse-engineering real code commits, Pull Requests, and business scenarios, and rewritten as conversational role-play prompts to resist data contamination. The evaluation is conducted in an isolated Docker sandbox using multiple scoring mechanisms, including hidden testing, rule checking, and LLM evaluation, and cross-validated in a dual-agent framework, achieving an end-to-end evaluation that is open-source, reproducible, and auditable.

Image source: Official article
Image source: official article

Technical positioning and domain: Belongs to the AI evaluation and benchmarking domain, focusing on the capability assessment of code agents (Code Agent), covering four directions: software engineering, front-end development, office automation, and network security. Unlike existing benchmarks that focus solely on code repair (such as SWE-bench), WorkBuddy Bench expands the evaluation scope to real-world workflows involving multiple domains, multiple files, and multiple roles.

Development background: Developed by Tencent AI Lab, based on internal practical use cases and requirements, aiming to address issues with existing benchmarks such as limited coverage, susceptibility to data contamination, and lack of reproducibility and auditability. The team has long-term experience in agent frameworks (CodeBuddy) and evaluation methodologies, and this suite is the open-source output of internal practices.

Core value: Provides the first multi-domain, fully open-source, and auditable evaluation standard for code agents. It helps enterprises, researchers, and developers objectively measure agent performance in real workflows, supporting model selection, product iteration, and academic comparisons, and promoting transparency and reproducibility in the field of benchmarks.

Technical features: Utilizes innovative mechanisms such as reverse-engineering task construction, conversational prompts, deliberate information insufficiency, post-turn evaluation isolation, and dual-framework cross-validation. Tasks are reverse-engineered from real commits/PRs/CVEs and rewritten as concise conversational requests, making it impossible to reconstruct the prompts via search engines; agents cannot access scoring assets during problem-solving to prevent information leakage; all tasks are executed separately in two frameworks: CodeBuddy Code and Claude Code, eliminating systematic bias from a single framework.

2. Key Features

• Code Task Evaluation: Covers 80 repository-level software engineering tasks, including bug fixing, feature development, interface adjustment, algorithm implementation, and 18 other categories. Tasks are initiated by five different roles: developers, algorithm engineers, product managers, QA, and operations. This evaluates the agent's code comprehension and modification capabilities from different perspectives, assessing its context awareness and precision in task localization.

• Web Frontend Evaluation: Includes 70 frontend tasks across seven categories: page implementation, interactive design, data visualization, visual design, code testing, and document conversion. The agent is required to deliver runnable frontend artifacts (HTML/CSS/JS) rather than conversational text. Evaluation is conducted through three methods: rule checking, semantic matching, and VLM interaction, ensuring output quality and functional completeness.

• Office Workflow Evaluation: Involves 50 multi-file business process tasks. The agent must handle mixed-format files such as spreadsheets, documents, PDFs, and JSONs, completing handover-type tasks like data verification, report generation, and status updates. A rule-checking system and evidence-driven LLM evaluator are used to ensure consistency across files in the context.

• Security Capability Evaluation: Includes 60 red-blue team security tasks, such as vulnerability detection and exploitation, malware analysis, security operations, and AI agent security evaluation. A deterministic scoring script (scoring.py) is used, with no involvement of the LLM evaluator, ensuring absolute reproducibility and objectivity of the scoring results.

• Multi-dimensional Scoring System: The code subset is scored using hidden unit tests, the web subset is scored using a combination of rules, LLM/VLM evaluators, and agent evaluators, the office subset is scored using rule checking and evidence-driven LLM evaluators, and the security subset is scored using deterministic scripts. Each subset's scoring tool is independently designed to avoid bias from a single evaluator.

• Anti-Pollution Mechanism: Task construction blocks searchable prompts by enclosing them in a closed, concise, and colloquial role-play format, making it impossible to reconstruct the original question via search engines. Combined with regular dataset version management refreshes, this replaces traditional confidential anti-pollution methods, achieving sustainable anti-pollution capabilities.

• Dual Framework Cross-validation: All tasks are executed separately on two agent frameworks: CodeBuddy Code and Claude Code, eliminating systematic bias from a single framework. The leaderboard simultaneously reports token consumption and round efficiency, helping distinguish the actual contributions of efficiency and capability.

3. How to Use

1. Environment Preparation and Task Acquisition: Visit the GitHub repository (github.com/Tencent/workbuddy-bench) to download the standardized task directory. Each task includes instruction.md (task instruction), task.toml (metadata configuration), environment/ (Docker environment image), and tests/ (evaluation assets). You need to have Docker installed locally and ensure that your network can pull images.

2. Agent Integration and Configuration: Integrate the coding agent to be evaluated into the Harbor-style task directory. The agent must be capable of reading natural language requests (instruction.md) and autonomously exploring files and context within the workspace (environment/). It is recommended to test the agent's file read/write, command execution, and network access capabilities in advance.

3. Run Evaluation: Execute the evaluation command within an isolated Docker sandbox. The agent performs code modifications, frontend generation, office tasks, or security operations based on colloquial role-play instructions within a limited number of rounds. Upon task completion, the scorer is automatically triggered, with each subset's scoring script running independently.

4. View Results and Audit: After scoring is complete, review metrics such as subset scores, token consumption, and round efficiency. The leaderboard displays both capability scores and cost efficiency, enabling comprehensive analysis. Third parties can rerun individual tasks offline and directly inspect their content, ensuring end-to-end reproducibility and auditability. All scoring assets (tests, scoring criteria, reference answers) are publicly available, supporting independent verification.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: If the team primarily focuses on code defect fixing capabilities, SWE-bench is a well-validated classic benchmark, but it has limited domain coverage and is susceptible to data contamination. If a comprehensive evaluation of an agent's performance across multiple real-world working scenarios (code, frontend, office, security) is required, WorkBuddy Bench is a more complete choice, and it is fully open-sourced for auditability. CursorBench offers convenient online evaluation, but its closed dataset makes it unsuitable for in-depth analysis and reproduction in research contexts. AgentBench covers a broader range of task types, but it lacks the depth of real workflow simulation and anti-pollution mechanisms compared to WorkBuddy Bench.

For enterprise model selection, it is recommended to use WorkBuddy Bench as the primary benchmark, supplemented by SWE-bench for specialized code comparisons; for academic research, the full open-sourcing of WorkBuddy Bench makes it suitable for reproduction and improvement; security teams should focus on the security subset and combine it with internal penetration testing for additional validation.

6. Editor's Summary

WorkBuddy Bench demonstrates significant technological innovation in the evaluation of coding agents. Its core breakthrough lies in expanding assessment from single code fixes to four real-world scenarios: code, web pages, office tasks, and security. By employing mechanisms such as reverse engineering construction, colloquial requests, and deliberate information insufficiency, it effectively resists data contamination, addressing the widespread "benchmark gaming" issue in existing benchmarks. The dual-framework cross-validation design further eliminates systematic biases inherent in single frameworks, making the evaluation results more compelling. In terms of practical value, this toolkit is fully open-source and auditable, with all task sets, environment images, and scoring code made public, providing the industry with a reproducible, standardized evaluation platform that promotes transparent competition in coding agent technology. The target audience includes AI agent development teams, decision-makers selecting large models, security researchers, and academics in the field of agent evaluation within academia. In terms of future development potential, with the implementation of task set version management and regular updates, along with the establishment of a community contribution mechanism, WorkBuddy Bench has the potential to become an authoritative benchmark in the coding agent domain, facilitating the practical deployment and optimization of agents within real workflows. It is recommended that the team continue to expand the types and quantity of tasks, and introduce more cross-validation results from additional agent frameworks to maintain the evaluation's relevance and representativeness.

7. Application Scenarios

• Agent Development Benchmark: Coding agent development teams can use WorkBuddy Bench as a standardized capability benchmark to evaluate the real-world performance of agents across four dimensions: code, web, office, and security. This helps identify capability gaps and optimize them accordingly. For example, by analyzing scores in the web subset, teams can detect weaknesses in front-end state management and improve the agent's context modeling.

• Model Selection Reference: When enterprises are selecting large models, they can refer to the WorkBuddy Bench leaderboard to compare the scores and token efficiency of mainstream models such as Claude Opus 4.8, GPT-5.5, GLM-5.2, and DeepSeek-V4 across the four subsets. This allows them to choose the model with the best cost-performance ratio for their specific business scenarios.

• Academic Research and Comparison: Researchers can leverage the fully open-source task set and scoring protocol to perform third-party replication, auditing, and improvements, promoting the iterative development of domain benchmarks. For example, new anti-pollution mechanisms or scoring methods can be designed based on the task catalog and compared with existing results.

• Product Iteration and Optimization: By analyzing the detailed subset scores and token efficiency data, teams can pinpoint specific shortcomings in agents' performance in areas such as code navigation, front-end state management, and cross-file consistency. Product teams can use this information to set iteration priorities and improve the agent's completion rate in real-world workflows.

• Security Capability Evaluation: Security teams can independently run the security subset to quantitatively evaluate an AI agent's ability in vulnerability detection, exploitation, and malware analysis. Deterministic scoring scripts ensure reproducible results, making it a suitable supplementary tool for red-blue team exercises.

8. FAQ

Q: What is the core difference between WorkBuddy Bench and other coding agent benchmarks (such as SWE-bench)?
A: WorkBuddy Bench covers four major domains: code, web, office, and security, whereas SWE-bench focuses solely on code defect fixing. In terms of task construction, WorkBuddy Bench reverse-engineers tasks from real commits/PRs/business scenarios and rewrites them into conversational role-play requests, offering stronger resistance to contamination. Additionally, WorkBuddy Bench is fully open-source and auditable, supporting third-party reproduction.

Q: How is fairness ensured in the evaluation to prevent agents from achieving high scores by memorizing answers?
A: Multiple anti-contamination mechanisms are employed: task requests are rewritten in a concise, conversational format, making it impossible to reconstruct them via search engines; key details such as target files and precise interfaces are intentionally omitted, requiring the agent to recover context independently from the workspace; evaluation assets are introduced into the sandbox only after the agent completes the problem-solving, preventing information leakage; and the dataset version is regularly refreshed through version management.

Q: What hardware environment is required to run WorkBuddy Bench?
A: A Docker environment is required, with at least 16GB of memory and 50GB of disk space recommended. Some tasks (such as web rendering and security penetration testing) may require GPU support, but most tasks can be completed on a CPU. Specific image requirements can be found in the Dockerfile within each task directory.

Q: How can I contribute new tasks or improve existing ones?
A: WorkBuddy Bench is an open-source project, and the community is welcome to submit Pull Requests via GitHub. New tasks must follow the Harbor-style task directory structure and provide instruction.md, task.toml, environment images, and evaluation scripts. It is recommended to first read the technical paper and existing task examples to ensure task quality and consistency.

9. Project Links

• Project Website: https://workbuddybench.com/
• GitHub Repository: https://github.com/Tencent/workbuddy-bench
• HuggingFace Dataset: https://huggingface.co/datasets/tencent/workbuddy-bench
• Technical Paper: https://workbuddybench.com/report/main.pdf