EdgeBench – ByteDance's AI Learning Capability Benchmark Framework

1. What is EdgeBench

EdgeBench is a benchmark framework developed by ByteDance's Seed team, specifically designed to evaluate the long-term learning capabilities of autonomous AI Agents in real-world environments. The framework includes 134 real-world tasks covering six major domains such as scientific computing, software engineering, and combinatorial optimization. It tracks the Agent's try-observe-absorb-improve learning curve through continuous 12–72 hour runs, revealing predictable scaling laws of AI learning from the environment. The framework not only provides completion benchmarks from human experts (average 57.2 hours) but also uses 83 held-out tasks to effectively prevent benchmark contamination, offering quantitative standards for general intelligence research, Agent capability iteration, and model selection.

Image source: Official article
Image source: official article

Technical Positioning and Domain: EdgeBench belongs to the AI evaluation and benchmarking domain, distinguished by shifting from static Q&A evaluation to assessing long-term learning capabilities in dynamic environments. It focuses on the process of Agents improving performance through repeated interaction with real-world tasks, rather than single-inference accuracy, filling the gap in existing benchmarks along the "learning process" dimension.

R&D Background: Developed by ByteDance's Seed team, which has long been dedicated to general intelligence ("Seed Edge") research. With the widespread application of AI Agents in programming, scientific computing, and other scenarios, the industry urgently needs an evaluation tool that can quantify the speed of autonomous learning progress. EdgeBench was created to provide repeatable, predictable metrics for model iteration and general intelligence advancement.

Core Value: Addresses the inability of traditional benchmarks to capture the long-term learning dynamics of Agents. Through continuous 12–72 hour tracking, EdgeBench reveals that AI learning curves follow highly predictable log-sigmoid scaling laws (R² = 0.998), and finds that learning speed approximately doubles every three months. This discovery provides a mathematical foundation for predicting AI capability growth trends, while offering quantitative basis for developers to optimize Agent continuous learning strategies.

Technical Features: Adopts an environment-interaction learning loop (try-observe-absorb-improve), dividing long runs into multiple stages and continuously recording performance scores to form quantifiable learning curves. Cross-domain task modeling uses a unified evaluation protocol, covering six types of tasks with varying cognitive difficulty, ensuring diversity and authenticity of evaluation. The anti-contamination mechanism uses 83 held-out tasks to prevent models from overfitting to the benchmark, ensuring reliability of evaluation results.

2. Key Features

• Long-term Learning Assessment: Simulates an Agent operating continuously in a real-world environment for 12–72 hours, tracking the complete learning curve. Unlike single-inference evaluations, this feature records the Agent's performance score at each time point, revealing the full improvement trajectory from initial unfamiliarity to gradual proficiency, providing first-hand data for understanding AI's continuous learning capabilities.

• Multi-domain Task Coverage: Includes 134 real-world tasks spanning 6 major categories: scientific computing, software engineering, combinatorial optimization, formal mathematics, data analysis, and system design. Each task originates from actual work scenarios, with clear evaluation metrics and environment configurations, ensuring that assessment results reflect the Agent's generalization ability across diverse challenges.

• Anti-contamination Design: Among the 134 tasks, 51 are publicly available as examples, while 83 reserved tasks are used solely for internal evaluation and are not disclosed. This design effectively prevents models from over-optimizing for the benchmark (i.e., benchmark contamination), ensuring that evaluation results genuinely reflect the Agent's learning ability on unseen tasks and avoiding the "leaderboard gaming" phenomenon.

• Learning Law Quantification: By analyzing the learning curves of numerous Agents, it was discovered that performance improvement follows a log-sigmoid scaling law, with learning speed approximately doubling every 3 months. This feature provides researchers with a predictable scaling relationship (R² = 0.998), which can be used to estimate the upper bound of future model performance on long-duration tasks, guiding resource allocation and model selection.

• Human Baseline Comparison: Provides an expert human completion baseline averaging 57.2 hours, covering tasks across all domains. Developers can directly compare the Agent's learning curve with human performance, quantify the human-machine gap, and identify areas where current AI has approached human levels and where breakthroughs are still needed.

• Frontier Model Coverage: Has evaluated top models including Claude Opus 4.8, GPT‑5.5, Gemini 2.5 Pro, o3, and DeepSeek‑V4‑Pro, and published a leaderboard. Developers can directly compare their own Agent results with these frontier models horizontally, quickly positioning their model's relative standing.

3. How to Use

1. Environment Requirements: A runtime environment supporting Python 3.8+ is required, with Linux systems (Ubuntu 20.04 or later) recommended. For hardware, a GPU with at least 16GB of VRAM (such as NVIDIA A100 or equivalent computing power) is advised, as some scientific computing tasks involve large-scale matrix operations. Disk space of 50GB or more should be reserved for storing task data and Agent outputs.

2. Obtain the Repository and Task List: Visit the GitHub repository ByteDance-Seed/EdgeBench and clone the codebase. Review the public task description files (JSON/YAML format) in the tasks/ directory to understand each task's evaluation metrics, input/output formats, and environment configuration. It is recommended to first read the quick start guide in README.md.

3. Select a Task and Deploy the Agent: Choose a target domain task (such as scientific computing or software engineering) from the 51 public tasks, and configure the Agent's interface according to the task documentation. The Agent must implement a standardized interaction protocol: receive task input, execute actions, receive environment feedback, and update strategies. When deploying, ensure the Agent can run continuously for over 12 hours. Docker containerization is recommended for deployment to ensure environment consistency.

4. Run Evaluation and Collect Data: Launch the evaluation script. EdgeBench will automatically record the Agent's performance scores at various time stages (e.g., success rate, completion time, quality score). The learning curve can be viewed in real time during evaluation. It is recommended to run the evaluation multiple times (at least 3) to reduce the impact of randomness, and use the average as the final result.

5. Submit Results and Analyze: Compare the generated evaluation data (in CSV or JSON format) with the frontier models on the leaderboard. Results can be submitted via the EdgeBench official website (registration required), or you can independently analyze the log-sigmoid fitting parameters of the learning curve. Note that evaluation for retained tasks must be applied for through official channels to obtain accurate rankings.

4. Pros and Cons Analysis

5. Comparison of Similar Tools

Selection Recommendations: If the research goal is to evaluate an AI Agent's continuous learning ability and improvement speed over long-duration tasks, EdgeBench is the only framework that provides a complete learning curve and time dimension, making it especially suitable for general intelligence research and Agent capability iteration. For teams focused on software engineering code repair, SWE-bench is closer to real-world development scenarios and has lower evaluation costs. If you need to quickly test an AI assistant's overall performance across diverse daily tasks, GAIA or AgentBench are more appropriate, as they cover a wide range of tasks with short run times. It is recommended to combine these benchmarks at different stages of development: use GAIA/AgentBench in the early stage for rapid validation of basic capabilities, SWE-bench in the middle stage to evaluate code repair expertise, and EdgeBench in the later stage for in-depth analysis of long-term learning potential.

6. Editor's Take

The launch of EdgeBench marks a significant shift in AI evaluation from "static capability testing" to "dynamic learning process assessment." Its technological innovations are reflected in three aspects: first, it establishes a closed-loop evaluation framework of try‑observe‑absorb‑improve, simulating the progressive learning of humans in complex tasks; second, through time-segmented performance tracking, it introduces the learning curve as a core metric for the first time, making AI progress quantifiable and predictable; third, it discovers the log‑sigmoid scaling law (R² = 0.998), providing a solid mathematical foundation for predicting model capability growth. These innovations not only enhance the authenticity of evaluation but also offer researchers a window into understanding AI learning mechanisms.

From a practical value perspective, EdgeBench provides a unified quantitative standard for general intelligence research (e.g., ByteDance's "Seed Edge" initiative), helping developers identify bottlenecks in long-duration Agent tasks (such as insufficient early exploration, mid-term strategy fixation, etc.), thereby guiding model optimization directions. Additionally, the leaderboard already covers cutting-edge models like Claude Opus 4.8 and GPT‑5.5, allowing developers to directly compare their own Agents' long-term learning performance and avoid blind investment. However, the 12‑72 hour evaluation duration and high hardware requirements limit its usability as a daily testing tool, making it more suitable as a periodic deep evaluation platform.

Target users primarily include: general intelligence research teams, AI Agent developers, model selection decision-makers, and continuous learning algorithm researchers. For product teams pursuing rapid iteration, it is recommended to use EdgeBench as a quarterly or milestone evaluation tool rather than for daily regression testing.

Future development potential is immense: as the task library expands to more domains (e.g., healthcare, law, creative design) and the number of retained tasks increases, EdgeBench is poised to become an industry standard for measuring AI autonomous evolution capabilities. If a more lightweight evaluation mode (e.g., a 2‑4 hour quick scan version) becomes available in the future, it will significantly lower the barrier to entry.

7. Application Scenarios

• General Intelligence Research: Provides quantitative evaluation standards for long-term learning capabilities in general intelligence initiatives like "Seed Edge." Researchers can regularly test Agent learning curves on EdgeBench, observing whether they follow the log‑sigmoid pattern, thereby validating the impact of new algorithms (e.g., meta-learning, reinforcement learning) on long-term learning speed.

• Agent Capability Iteration: Helps developers identify bottlenecks in Agent performance during extended tasks. For example, if the learning curve rises rapidly early on but then plateaus, it indicates deficiencies in the Agent's exploration strategy or memory mechanism, allowing targeted optimization of feedback mechanisms or integration of external knowledge bases.

• Model Selection Reference: Compares the long-term learning performance of models such as Claude, GPT, and Gemini across various domains via leaderboards. When selecting underlying models, enterprises can refer to EdgeBench's final scores and learning speeds after 72 hours, rather than relying solely on single-inference accuracy.

• Human-AI Benchmarking: Uses the expert human baseline of 57.2 hours as a reference to measure AI's progress toward human-level performance. For instance, in scientific computing tasks, if an Agent outperforms the human average within 48 hours, the domain can be considered to have achieved "superhuman" capability, providing a basis for automated decision-making.

• Educational Training Design: Offers a standardized evaluation environment for research on AI autonomous learning and continuous improvement algorithms. Universities and research institutions can use EdgeBench as a course project, allowing students to design Agents and analyze their learning curves, deepening their understanding of concepts such as continual learning and the exploration-exploitation trade-off.

8. FAQ

Q: What is the core difference between EdgeBench and other benchmarks (e.g., SWE-bench)?
A: EdgeBench evaluates an Agent's continuous learning ability over long periods (12–72 hours), tracking the complete learning curve, whereas SWE-bench only assesses the final result of a single code fix without a time dimension. EdgeBench also provides human baselines and predictable scaling laws, making it more suitable for studying the autonomous learning progress of AI.

Q: How can I submit my Agent results to the EdgeBench leaderboard?
A: First, select a target task from the public tasks, deploy the Agent according to the standard protocol, run the evaluation, and generate learning curve data. Then, visit the EdgeBench official website (edge-bench.org), register an account, and upload the results following the guidelines. Note that evaluation of held-out tasks must be requested through official channels to ensure data security.

Q: How do EdgeBench's held-out tasks prevent benchmark contamination?
A: The task descriptions, evaluation metrics, and environment configurations of the 83 held-out tasks are not publicly disclosed. Models cannot access these tasks during training, preventing over-optimization for them. Results for held-out tasks can only be obtained through the official evaluation process, ensuring the authenticity and fairness of the assessment.

Q: What is the practical significance of the log-sigmoid scaling law followed by learning curves?
A: This law indicates that Agent performance improves rapidly in the early stages, then gradually approaches an upper limit (sigmoid shape), with the learning speed (curve slope) approximately doubling every three months. This means researchers can estimate the final performance of a current model after longer runtime, helping decide whether to increase runtime or optimize algorithms.

Q: What are the minimum hardware requirements for EdgeBench?
A: It is recommended to use a GPU with at least 16GB of VRAM (e.g., NVIDIA A100, RTX 4090), 32GB or more of RAM, and 50GB or more of disk space. Some scientific computing tasks require high-precision floating-point operations and run extremely slowly in CPU mode. If only testing simple tasks (e.g., combinatorial optimization), a GPU with 8GB VRAM may suffice, but task duration should be appropriately shortened.

Q: Are custom tasks supported?
A: Currently, EdgeBench does not officially provide a custom task interface, but developers can extend based on public task templates. If you need to include specific business scenarios in the evaluation, contact the Seed team to apply for collaboration. A task registration mechanism may be provided in the future.

Q: Are EdgeBench results reproducible?
A: Due to randomness in task environments (random seeds) and Agent strategies, single-run results may fluctuate. It is recommended to run at least three times, take the average, and record the standard deviation. The official standardized Docker image ensures environmental consistency, but different GPU models may cause minor performance differences.

9. Project Address

• Project Website: https://edge-bench.org/
• GitHub Repository: https://github.com/ByteDance-Seed/EdgeBench
• HuggingFace Dataset: https://huggingface.co/datasets/ByteDance-Seed/EdgeBench
• Technical Paper: https://edge-bench.org/paper.pdf