PAST-Bench – Princeton's Benchmark for Performance Attribution in Personal Agents

1. What is PAST-Bench

PAST-Bench is a benchmark introduced by Princeton University's team led by Mengdi Wang, specifically designed to evaluate the recursive self-improvement capabilities of personal AI agents. By comparing task performance under conditions with and without persistent memory, this benchmark precisely determines whether the agent's retained cross-session experiences genuinely enhance subsequent behavior. It covers four core capabilities: memory, process reuse, information gathering, and state updating, and includes 26 task scenarios and 204 task rounds, offering a standardized quantitative assessment of an agent's cross-session learning abilities.

Technical Positioning and Domain: PAST-Bench belongs to the field of AI agent evaluation and benchmarking, focusing on the attribution analysis of recursive self-improvement and persistent capabilities. Unlike existing benchmarks (such as SWE-Bench), which only assess task completion within a single session, PAST-Bench expands the evaluation focus to cross-session experience accumulation and reuse, filling a gap in standardized assessment within this domain. Its methodology draws on the concept of controlled experiments in causal inference, achieving precise attribution of the source of improvements for the first time in agent evaluation.

Development Background: Developed by Princeton University's Mengdi Wang team (Gen-Verse), this benchmark aims to address the widespread neglect of "attribution of improvement sources" in current AI agent evaluations. As personal AI agents increasingly gain memory and skill persistence capabilities, verifying whether these features genuinely lead to performance improvements has become critical. PAST-Bench was created to provide a rigorous causal comparison methodology, distinguishing the contributions of experience accumulation from other factors (such as model upgrades or prompt optimization). The team has previously accumulated deep expertise in reinforcement learning and multi-agent systems, providing theoretical support for the design of this benchmark.

Core Value: The core value of PAST-Bench lies in "attribution" rather than "scoring." It employs a controlled experiment design, running both persistent and non-persistent versions of each task family simultaneously, and directly calculates the self-evolution difference Δ (Delta) to accurately measure the net benefit brought by experience accumulation. Additionally, it introduces the Mechanism-Evidence Score, which tracks the complete path of "storage → retrieval → application," preventing accidental correctness from being misjudged as genuine improvement. This attribution capability provides clear guidance for optimizing an agent's persistent features, enabling developers to distinguish between "effective improvements" and "false enhancements."

Technical Features: PAST-Bench employs four key technologies: task family sequence design (simulating continuous use across multiple rounds of conversation), matched controlled experiments (with memory vs. without memory), mechanism evidence tracking (recording runtime telemetry data), and persistent artifact auditing (checking the actual content saved). These technologies collectively enable a comprehensive diagnosis from final outcomes to internal processes, making the evaluation not only about what happened, but also why it happened. Compared to existing benchmarks, PAST-Bench achieves attribution at the mechanism level, capable of identifying whether the agent truly follows the expected experience reuse path.

2. Key Features

• Recursive Self-Improvement Evaluation: The core functionality is to quantify the effectiveness of an Agent's cross-session experience reuse. By comparing the performance of a version with persistent memory against a version without persistent memory on the same task family, the Δ value (score with memory minus score without memory) is calculated, directly reflecting the performance gains from experience accumulation. The larger the Δ value, the stronger the Agent's self-improvement capability. This feature provides a quantifiable metric for the Agent's long-term learning ability.

• Experience Accumulation Effect Isolation: A matched control experiment design is employed to ensure that all conditions (tasks, models, prompts, environment) are identical between the "with memory" and "without memory" groups, except for the persistence capability. This eliminates interference from external factors such as base model upgrades, prompt adjustments, or task difficulty fluctuations, allowing the Δ value to be purely attributed to experience accumulation. This isolation design is unique among existing Agent benchmarks, ensuring the scientific validity of the evaluation results.

• Four-Dimensional Capability Diagnosis: The vague concept of "self-improvement" is decomposed into four specific dimensions: Memory, Workflow Reuse, Information Gathering, and State Update. Each task family corresponds to one of these capabilities, and the evaluation results can precisely identify where an Agent has deficiencies in cross-session abilities, providing clear guidance for improvement. For example, if an Agent has a low Δ value in the "Workflow Reuse" dimension, it indicates issues with its skill abstraction and reuse mechanisms.

• Mechanism Attribution Analysis: The Mechanism-Evidence Score (Mechanism-Evidence Score) is introduced, which not only examines whether the final answer is correct but also traces whether the Agent actually executed the complete path of "storing experience → retrieving relevant memory → applying experience" during runtime. This effectively distinguishes between "true experience reuse" and "accidental correctness," avoiding attribution bias. The Mechanism-Evidence Score is calculated by analyzing runtime telemetry data (such as memory writes, skill calls, and session retrievals), and is therefore auditable.

• Persistent Artifact Auditing: The evaluation process records the actual content saved by the Agent, including memory entries, skill files, and session indices. Audit dimensions include data structure rationality, timeliness (whether updates are timely), and reusability (retrieval efficiency). This feature helps developers identify invalid persistence issues such as "stored but not found" or "stored but not used," directly guiding optimization of the memory system design.

• Task Family Sequence Design: Each task family includes multiple rounds of sessions (early sessions for experience accumulation, later sessions for testing reuse), simulating the continuous usage process of a personal Agent in real-world scenarios. Task families cover various scenarios such as daily assistant tasks, tool usage, and information retrieval, with a total of 26 task families and 204 task rounds, ensuring comprehensive and representative evaluation. This sequence design enables the evaluation to capture the dynamic process of experience accumulation.

• Custom Agent Adapter: A standard adapter interface is provided, allowing users to evaluate their own Agents. By implementing the necessary interfaces (such as persist, retrieve, and act) in the agents/ directory and ensuring support for the --compare-no-persistence switch, PAST-Bench can be integrated into existing development workflows for continuous evaluation. The adapter design follows the principle of minimal intrusion, reducing integration costs.

• Result Comparison and Visualization: After evaluation, a sequence_comparison.json file is generated, containing structured data such as the Δ value for each task family, the Mechanism-Evidence Score, and detailed scores for each round. Developers can use this data to plot improvement curves and visually observe the trend of experience accumulation effects. Additionally, this file supports comparison with historical evaluation results, facilitating the tracking of Agent architecture iteration outcomes.

3. How to Use

1. Environment Preparation and Installation: First, clone the PAST-Bench repository from GitHub (git clone https://github.com/Gen-Verse/PAST-Bench.git), then install the core dependencies: pip install -r requirements.txt. Additionally, you need to install the Agent adapter (select the appropriate adapter based on the Agent being tested). The system must have Docker pre-installed (version 20.10+), as the evaluation needs to run in a sandbox environment to ensure task isolation and reproducibility. It is recommended to use a Linux operating system for the best compatibility.

2. Model and API Configuration: PAST-Bench itself does not bind to a specific model. The language model API key used is configured via environment variables. For example, set OPENAI_API_KEY or ANTHROPIC_API_KEY, etc. Ensure the model supports the required tool calling and long context capabilities, as multi-turn conversations may involve extended historical records (at least 8K context is recommended). If using a local model, ensure it is exposed through a compatible API service.

3. Sandbox Image Building: Execute the command past-bench build-image --kind sandbox to build the required Docker sandbox image for evaluation. This image contains an isolated task environment, ensuring that each task family runs in a clean environment and avoiding state contamination. The build process may take several minutes, depending on network and system configuration. After the build is complete, verify that the image was successfully created using docker images.

4. Quick Validation (Smoke Test): Run a single task family for quick validation using the command past-bench evolve --task-family <task_family_name> --compare-no-persistence. The --compare-no-persistence parameter runs both the persistent and non-persistent versions simultaneously, outputting preliminary Δ values. This is used to verify whether the environment is correctly configured and whether the Agent can complete basic tasks. It is recommended to choose the smallest task family (e.g., memory_basic) for the first test.

5. Full Evaluation: After confirming that the quick validation is successful, run the full evaluation with the command past-bench evolve --all --compare-no-persistence. This command will iterate through all 26 task families, automatically executing two sets of experiments for each: one with persistence and one without. The evaluation process may take a long time (depending on the number of tasks and model response speed; a full evaluation may take several hours). It is recommended to use a high-performance GPU or API service to speed up the process. During the evaluation, you can use the --verbose parameter to view real-time progress.

6. Result Analysis: After the evaluation is complete, check the sequence_comparison.json file in the directory specified by --trace-dir. This file contains detailed results for each task family: Δ values (self-evolution difference), Mechanism-Evidence Score, scores per round, and audit reports on persistent artifacts. Developers can use this to analyze the Agent's cross-session performance, with a focus on task families with Δ values close to or below zero, which typically indicate capability shortcomings.

7. Custom Agent Integration: If you want to evaluate your own Agent, create an adapter class in the agents/ directory that inherits from the BaseAgent interface. The key methods that must be implemented include: act() (execute actions), persist() (save experiences), and retrieve() (retrieve relevant memories). The adapter must also support the --compare-no-persistence switch, disabling all save and retrieve operations in non-persistent mode. Once the adapter is completed, specify it using the --agent your_agent_name parameter to run the evaluation. For detailed instructions, refer to the agents/README.md file in the GitHub repository.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations:

If your research or development focus is on validating the cross-session experience accumulation of a personal Agent, such as testing different memory architectures or skill reuse strategies, PAST-Bench is the only benchmark that provides precise attribution capabilities. It can tell you whether "improvements are genuinely from experience" rather than from the model becoming stronger or the task becoming easier. For teams developing Agent persistence features, PAST-Bench can serve as a core evaluation tool, complemented by its four-dimensional diagnostic results to optimize Agent design. Additionally, its mechanism-evidence score can distinguish between "true reuse" and "accidental correctness," preventing misleading conclusions from false improvements.

If your goal is to evaluate the single-task capability of an Agent in a specific domain (e.g., code repair), SWE-Bench is more appropriate. It focuses on the correctness of code patches and features a large number of real GitHub issues, making it the standard benchmark in the code Agent domain. GAIA and AgentBench, on the other hand, are better suited for evaluating the performance of general-purpose Agents in multi-step reasoning and environment interaction, but they do not focus on cross-session experience reuse. Therefore, in practical development, you can combine PAST-Bench with other benchmarks: use PAST-Bench to evaluate long-term learning capabilities, and use SWE-Bench or AgentBench to assess single-task execution capabilities, forming a complementary evaluation system. For example, when developing a personal assistant Agent, first use PAST-Bench to verify whether its memory system is effective, then use GAIA to test its reasoning ability in complex dialogues.

6. Editor's Summary

The emergence of PAST-Bench fills a critical gap in the AI Agent evaluation domain: recursive self-improvement attribution analysis. Current mainstream benchmarks (such as SWE-Bench, GAIA, and AgentBench) all focus on single-session task completion capabilities, while overlooking the key characteristic of Agents improving their subsequent behaviors by accumulating experience through continuous use. PAST-Bench, through its innovative task family sequence design and matched control experiments, has for the first time achieved precise quantification of cross-session experience accumulation effects. Its proposed Mechanism-Evidence Score reaches a granularity level at the mechanism level, far surpassing traditional result-level evaluations.

From a technical perspective, the design of PAST-Bench reflects rigorous causal inference principles. By constructing "memory-enabled" and "memory-disabled" control versions for each task family, it can cleanly isolate the contribution of experience accumulation, akin to the method of controlling variables in scientific experiments. This design not only enhances the scientific rigor of the evaluation but also provides clear signals for optimizing the persistence capabilities of Agents—developers can target specific dimensions with low Δ values or insufficient mechanism evidence scores for improvement. In fact, this team has already developed the Hermes+ framework based on issues exposed by PAST-Bench, demonstrating its practical value as a diagnostic engine. This "evaluation-diagnosis-improvement" closed-loop model provides a quantifiable iterative path for Agent development.

In terms of practical value, PAST-Bench is particularly important for three groups of people: academic researchers can obtain a standardized, reproducible evaluation environment to compare the self-improvement capabilities of different Agent architectures; framework developers can use the four-dimensional diagnostic results to precisely identify weaknesses and guide architectural iterations; model selection teams can compare the cross-session experience reuse performance of different base models under a fixed framework, identifying model capability biases. Additionally, product teams can use PAST-Bench to validate the real effectiveness of persistence features in new versions, avoiding meaningless development driven solely by the pursuit of persistence. For individual Agent developers, PAST-Bench provides an objective means to test whether their Agent truly becomes smarter with continued use.

Of course, PAST-Bench is still in its early stages and has room for improvement in terms of task scenario coverage, usability, and documentation completeness. However, its core methodology—matched control experiments and mechanism evidence tracking—has broad implications and could become a new paradigm in Agent evaluation. As personal AI Agents become more widespread, cross-session learning ability will become an important metric for assessing Agent intelligence, and PAST-Bench has taken a significant step in this direction. In the future, if PAST-Bench can expand to more task scenarios, reduce the barrier to entry, and integrate with other benchmarks to form an evaluation matrix, its influence will further grow.

7. Application Scenarios

• Academic Research: Provides a standardized, reproducible quantitative evaluation environment for Agent recursive self-improvement. Researchers can use PAST-Bench to compare the self-improvement effectiveness of different memory architectures (such as vector databases, structured memory, skill libraries), or to study the impact of base model scale on experience reuse capabilities. All experiments are conducted using a matched control design to ensure the reliability of conclusions, and Δ values and mechanism evidence scores can be cited as core metrics in papers.

• Framework Development: Accurately identifies shortcomings in Agent frameworks across dimensions such as memory, process reuse, information gathering, and state updates through four-dimensional capability diagnosis. For example, if an Agent has a low Δ value in the "Process Reuse" dimension, it indicates issues with its skill abstraction and reuse mechanisms. Developers can then optimize the workflow management module accordingly. The creation of the Hermes+ framework is a direct example of this scenario, as it was rebuilt based on runtime failures exposed by PAST-Bench.

• Model Selection: Under a fixed Agent framework, use PAST-Bench to compare the cross-session experience reuse performance of different base models (such as GPT-4, Claude, Llama, etc.). By comparing Δ values and mechanism evidence scores, one can identify the capability biases of each model in terms of memory capacity, retrieval accuracy, and skill transfer, providing data support for model selection. For instance, some models may excel in the "Information Gathering" task family but show deficiencies in "State Updates."

• Persistent Validation: Tests the practical effectiveness of different memory structures and retrieval strategies. For example, compares the performance of semantic similarity-based retrieval versus timestamp-based retrieval in the "Information Gathering" task family, verifying which strategy better supports experience reuse. The persistent artifact auditing feature of PAST-Bench can also help identify invalid persistence issues such as "stored but not found" or "stored but not used," directly guiding the optimization of memory system design.

• Product Iteration: Distinguishes whether score improvements in new Agent versions come from "cross-session experience accumulation" or "enhanced base model/Prompt optimization," ensuring that persistence features genuinely deliver value. Product teams can run PAST-Bench after each iteration to observe trends in Δ values and ensure the persistence improvement direction is correct. If Δ values do not increase, it indicates that the new version's persistence features have not led to substantive improvements, requiring a reevaluation of the design.

8. FAQ

Q: What does PAST-Bench primarily evaluate in an Agent?
A: PAST-Bench specifically evaluates the recursive self-improvement capability of personal AI Agents, i.e., whether an Agent can leverage experiences (memory, skills, task history) saved across sessions to enhance performance in subsequent tasks. It quantifies the net benefit of experience accumulation by comparing task scores under persistent and non-persistent conditions. Unlike single-session benchmarks, it focuses on whether the Agent becomes smarter with continued use.

Q: What hardware and software requirements are needed to run PAST-Bench?
A: Hardware-wise, you need a computer capable of running Docker (Linux is recommended, but macOS and Windows are also supported). The resource consumption of the evaluation mainly depends on the response speed of the model API being called. Software-wise, you need Python 3.8+, Docker 20.10+, Git, and access permissions to the relevant model APIs (e.g., OpenAI, Anthropic). It is recommended to have at least 8GB of memory and 20GB of disk space. If using a local model, you will need to configure a compatible API service separately.

Q: How should the Δ value and mechanism evidence score in the evaluation results be interpreted?
A: The Δ value (self-evolution difference) is calculated as "score with memory minus score without memory." A positive value indicates that experience accumulation has led to performance improvement, with larger values indicating more significant improvements. The Mechanism-Evidence Score measures whether the Agent truly follows the "store → retrieve → apply" path. A higher score means the improvement is supported by a real mechanism, rather than a coincidental correct answer. Together, these two metrics provide a comprehensive assessment of the Agent's self-improvement capability. For example, a high Δ value but low mechanism evidence score may suggest that the improvement comes from other factors, not genuine experience reuse.

Q: Which base models does PAST-Bench support?
A: PAST-Bench is not tied to any specific model. Any language model that supports tool calling and long context can be integrated by configuring the API Key via environment variables. Commonly supported models include the GPT-4 series, Claude series, and Llama series (via API services). It is important to note that the model's capabilities (e.g., reasoning, long context) will affect the evaluation results. Therefore, when comparing different Agent architectures, the same model should be used consistently. For local models, ensure that they expose an API compatible with the OpenAI format.

Q: What is the main difference between PAST-Bench and SWE-Bench?
A: The two benchmarks have different objectives: SWE-Bench evaluates an Agent's ability to fix code bugs in a single session, with each task being independent and without cross-session memory. PAST-Bench evaluates an Agent's ability to accumulate experience and improve subsequent behavior over continued use, with its core design being a persistence-based comparison across sessions. SWE-Bench is a result-level evaluation, while PAST-Bench is a mechanism-level attribution evaluation. Choose PAST-Bench if you are interested in long-term learning capabilities, and choose SWE-Bench if you are interested in single-time code fixing capabilities. The two can be used complementarily.

Q: How can a custom Agent be integrated with PAST-Bench?
A: Create an adapter class in the agents/ directory, inheriting from the BaseAgent interface, and implement required methods such as act(), persist(), and retrieve(). Additionally, the adapter must support the --compare-no-persistence switch, which disables all save and retrieval operations in non-persistent mode. After completing the adapter, run the evaluation using past-bench evolve --agent your_agent --task-family <family>. For detailed instructions, refer to the agents/README.md file in the GitHub repository. It is recommended to first read the example adapter (e.g., agents/openai_agent.py) to understand the interface conventions.

9. Project Links

• Project Homepage (GitHub Repository): https://github.com/Gen-Verse/PAST-Bench
• Technical Paper (arXiv): https://arxiv.org/pdf/2608.04003