MemHarness – A Memory Reconstruction Framework for LLM Agents Introduced by the Shanghai AI Lab and Others

1. What is MemHarness

MemHarness is a memory reconstruction framework for LLM Agents introduced jointly by the Shanghai Artificial Intelligence Lab and universities such as Zhejiang University, Fudan University, and Shanghai Jiao Tong University. Current memory-enhanced agents typically inject retrieved historical experiences directly into the context, which can lead to negative transfer when the old experiences are mismatched with the current state. Inspired by the human memory reconstruction mechanism, MemHarness inserts an explicit critical and reconstructive phase between retrieval and action generation. It retains, rewrites, or discards historical experiences based on the current context and achieves end-to-end training through the GRPO algorithm, without requiring additional human-labeled reconstruction data.

Technical Positioning and Domain: MemHarness belongs to the intersection of natural language processing and agent systems, focusing on how LLM agents can efficiently and robustly utilize historical experiences in long-term tasks. Unlike traditional memory-enhancement methods, which treat memory as static fragments to be replayed directly, MemHarness introduces a dynamic reconstruction mechanism, enabling memory usage to be context-aware. This framework is positioned as a general-purpose memory management middleware that can be embedded into various LLM-based decision systems, enhancing their performance in complex and non-deterministic environments.

Development Background: This framework was jointly developed by the Shanghai AI Lab and several top-tier universities. The research team has deep expertise in LLM agents, reinforcement learning, and memory systems. As LLM agents are increasingly applied in embodied intelligence and web navigation scenarios, avoiding negative transfer from experiences and improving out-of-distribution generalization have become key challenges. MemHarness was specifically designed to address this pain point, drawing inspiration from the reconstruction process of human memory in cognitive science—humans actively adjust memory content based on the current context during recall, rather than reproducing it verbatim.

Core Value: MemHarness solves the problem of performance degradation in existing memory-enhanced agents when outdated experiences are directly applied in mismatched states. Through an explicit critical and reconstructive phase, it transforms static memory fragments into context-sensitive state-aligned guidance, significantly improving task success rates. Experimental results show that a model with a 7B parameter scale outperforms Gemini-2.5-Pro by 23.1% and 39.7% on ALFWorld and WebShop, respectively, achieving an average success rate of 85.9% in out-of-distribution scenarios, demonstrating strong generalization capabilities.

Technical Features: The core technical features of MemHarness include a five-stage decision process (Observation – Retrieval – Critique – Reconstruction – Action), a unified policy model architecture (shared parameters for retrieval, reconstruction, and action generation), and the GRPO end-to-end training mechanism. It does not require separate labeling of data for the reconstruction phase, as reconstruction capabilities naturally emerge during training. Moreover, even when memory retrieval is turned off during testing, the reconstruction training objective still enhances the base policy's intrinsic reasoning ability, showcasing implicit reasoning augmentation.

2. Key Features

• Memory Retrieval: The Agent generates a query vector based on the current environmental observation and retrieves the top-k relevant historical experiences and their corresponding source states from the Milvus vector memory database. The retrieval process is based on semantic similarity, ensuring that the retrieved experiences have potential relevance to the current task, providing raw materials for subsequent critique and reconstruction.

• Memory Critique and Reconstruction: A unified policy model compares the source states of the retrieved memories with the current state to critique their applicability. For transferable knowledge, the model rewrites it into state-aligned guidance information; for mismatched content, the model performs adaptive adjustments; if completely irrelevant, the model outputs the <EMPTY> token to reject the memory and fall back to autonomous reasoning. This is the core component that differentiates MemHarness from traditional memory replay approaches.

• Action Generation: Based on the reconstructed guidance information or the results of autonomous reasoning, the policy model generates executable environmental actions. The action space is aligned with the specific task environment (e.g., text instructions in ALFWorld, click operations in WebShop), and the output format is learned during the cold-start SFT phase to ensure compatibility with the interaction environment.

• Experience Rewriting and Pruning: After each round of interaction, the system summarizes the current trajectory into structured experience and writes it back into the Milvus memory database. During the writing process, semantic deduplication is performed to avoid redundant storage; periodically, low-value memories are pruned based on experience utility scores to control the memory database size and maintain retrieval efficiency and quality.

• End-to-End GRPO Training: Since the reconstructed guidance information lacks ground-truth annotations, MemHarness employs GRPO (Group Relative Policy Optimization) to perform end-to-end optimization across the entire retrieval-reconstruction-action pipeline. The reward consists of sparse task outcomes (success/failure) and format rewards, using group-normalized advantages to assign trajectory-level credit to all tokens, while simultaneously training reasoning, reconstruction, and action generation capabilities.

• Cold-Start SFT (Optional): Before GRPO training, the framework provides a cold-start data construction script. By running scripts/build_**_coldstart_data.py, initial interaction data is generated, followed by executing scripts/cold_start_sft.sh to align the model with the interaction format and memory summary format. This step is not mandatory, but it can accelerate the convergence of subsequent GRPO training, especially for new task environments.

3. How to Use

1. Environment Requirements and Prerequisites: Use Python 3.12 to create a Conda environment and install vLLM, Flash Attention 2, and the MemHarness core. For hardware, it is recommended to have at least one NVIDIA A100 or equivalent GPU to support model inference and training; memory should be 64 GB or more to support the Milvus vector database. The operating system is recommended to be Linux (Ubuntu 20.04+).

2. Clone the Repository and Set Up the Environment: Execute git clone https://github.com/KnowledgeXLab/MemHarness.git to download the code, then use Conda to create a Python 3.12 environment and install vLLM, Flash Attention 2, and the project dependencies in sequence. Specific commands can be referenced in the installation instructions within the repository's README.

3. Deploy the Embedding Service: Start the BGE-M3 embedding model service using vllm serve BAAI/bge-m3 --port 8001, providing vector encoding capabilities for the Milvus memory database. This service must remain running before GRPO training begins to ensure that the Agent can convert text into vectors in real time for retrieval and storage.

4. Install the Target Environment and Cold Start (Optional): Install the ALFWorld or WebShop interactive environment based on the task requirements, and download the corresponding game files and pre-trained detectors. If you want to accelerate training, run scripts/build_**_coldstart_data.py to build cold start data, then execute scripts/cold_start_sft.sh to perform supervised fine-tuning on the model, aligning it with the interaction format.

5. GRPO End-to-End Training: Run run_scripts/train_alfworld.sh or run_scripts/train_webshop.sh, and the framework will automatically start the memory vector database, perform Agent retrieval, experience replay, and pruning to complete GRPO training. During training, monitor GPU memory usage and the scale of the memory database, adjusting the batch size or pruning threshold as necessary.

Notes: Ensure that the Milvus service and embedding service are running before training begins; for the first training session, it is recommended to use cold start SFT to reduce the difficulty of GRPO exploration; the action space varies significantly across different task environments, so the format reward portion in the reward function must be adjusted according to actual needs.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For Agent tasks requiring high robustness and strong out-of-distribution generalization (such as embodied intelligence and web navigation), MemHarness's reconstruction mechanism significantly outperforms direct replay or simple reflection methods. Its 7B model can achieve or even surpass the performance of large models, making it suitable for resource-constrained scenarios with high reliability requirements. If the team already has a mature LLM service and is not cost-sensitive, Reflexion's zero-training deployment is more advantageous, ideal for rapid prototyping and validation. For applications requiring ultra-long context memory (such as dialogue history management), MemGPT's virtual context approach is more straightforward, although it lacks explicit state alignment capabilities.

Scenario Matching: MemHarness is particularly suitable for dynamic environments where task states frequently change and historical experiences may partially become invalid. If the task state is relatively stable and experiences can be directly reused, EvolveR's evolutionary iteration mechanism may be simpler and more efficient. For applications requiring high interpretability (such as financial decision-making and medical diagnosis), MemHarness's transparent and controllable reconstruction process is superior to other black-box memory methods.

6. Editor's Summary

MemHarness introduces an innovative approach in the field of memory-enhanced agents by incorporating the cognitive mechanisms of human memory reconstruction into the decision-making process of LLM agents. Its core contribution lies in breaking the static paradigm of "memory as replay" through explicit critical and reconstructive stages, enabling historical experiences to dynamically adapt to the current context and effectively preventing negative transfer. This design not only improves task success rates but also demonstrates significant robustness in out-of-distribution scenarios. The result of the 7B model surpassing Gemini-2.5-Pro further proves that efficient memory utilization can compensate for limitations in model capacity.

In terms of practical value, MemHarness's end-to-end GRPO training framework allows reconstructive capabilities to emerge without the need for manual annotation, significantly reducing data costs. The unified policy model architecture integrates retrieval, reconstruction, and action generation within a single model, avoiding the engineering complexity of multi-module coordination. For researchers and engineers, this framework provides a reproducible and scalable baseline for memory management, particularly suitable for domains requiring long-term decision-making, such as embodied intelligence and web navigation.

In terms of target users, MemHarness is more suitable for researchers with a foundational understanding of reinforcement learning who wish to deeply explore memory mechanisms, as well as engineering teams requiring highly reliable agent implementations. Currently, it mainly supports two environments: ALFWorld and WebShop. For applications in other domains, adaptation is required, which imposes certain demands on the developers' environment integration capabilities.

Looking ahead, the reconstruction paradigm of MemHarness has the potential to be extended to a broader range of agent tasks, such as code generation, multi-turn dialogue, and robotic control. As more task environments are adapted and cold-start data accumulates, the framework's generalizability will further improve. Additionally, the automated tuning of memory pruning strategies and reward function design presents promising research directions. MemHarness offers a new path for memory management in LLM agents that balances performance with interpretability, and its influence is expected to continue growing in multi-agent systems and continuous learning scenarios.

7. Application Scenarios

• Embodied Intelligence for Household Tasks: In environments such as ALFWorld, the Agent can restructure past experiences related to object retrieval and cleaning to adapt to different room layouts and object placements, completing complex household tasks. For example, when the kitchen layout changes, the Agent does not mechanically apply old paths but instead reconstructs a strategy for finding kitchenware based on current observations.

• Autonomous Online Shopping: On e-commerce platforms like WebShop, the Agent restructures historical shopping experiences into current product search and filtering strategies, improving the success rate of goal-oriented shopping. When user needs or product inventory change, the restructuring mechanism ensures the Agent does not repeat outdated filtering conditions.

• Intelligent Customer Service Conversations: After retrieving historical similar tickets, the customer service Agent restructures solutions to avoid directly applying old responses that may be irrelevant. For example, for issues related to the same product across different versions, the Agent compares version differences and rewrites the solution to fit the current version.

• Code-Aided Development: The programming Agent restructures past bug-fixing experiences to adapt to the current code context, offering precise repair suggestions rather than copying old solutions. When the codebase structure changes, the restructuring mechanism ensures that the repair logic remains effective.

• Research Experiment Planning: The experiment Agent restructures historical experimental parameters and results into optimal configuration recommendations for current experimental conditions, reducing trial-and-error costs. For example, in material synthesis tasks, the Agent compares changes in reaction conditions and adjusts recommendations for parameters such as temperature and catalysts accordingly.

8. FAQ

Q1: What is the core difference between MemHarness and other memory-enhanced agents (such as Reflexion)?
A: Reflexion generates textual memories through post-hoc reflection but lacks an explicit comparison between the memory's source state and the current state, which may lead to directly applying mismatched experiences when the environment state changes. MemHarness, on the other hand, inserts a critical and reconstructive phase after retrieval, comparing state differences and retaining, rewriting, or discarding experiences accordingly, thus avoiding negative transfer and improving out-of-distribution generalization.

Q2: How much computational resources are required to train MemHarness?
A: The official recommendation is to use at least one NVIDIA A100 GPU (80 GB VRAM) and 64 GB or more of system memory. GRPO training for the 7B model on ALFWorld takes approximately 2-3 days (depending on group size and number of iterations). The cold-start SFT phase can be completed on a single V100 GPU, but GRPO training requires higher computational power.

Q3: Does it support custom task environments?
A: Currently, the official implementation only includes ALFWorld and WebShop environments. To support a new environment, you need to implement the environment interaction interface, define the action space and reward function, and adjust the cold-start data generation scripts. The framework provides modular interface design, and teams with some development experience can adapt it accordingly.

Q4: What vector database does the memory bank use?
A: MemHarness uses Milvus as the vector memory bank, converting text into vectors using the BGE-M3 embedding model for retrieval and storage. Milvus supports efficient similarity search and dynamic data management, making it suitable for large-scale memory scenarios. Users can also replace it with other vector databases based on their needs.

Q5: Does it support Chinese tasks?
A: MemHarness itself is not language-restricted, and both its base model (e.g., MiniCPM) and embedding model (BGE-M3) support Chinese. However, official evaluations are primarily based on English environments (ALFWorld, WebShop). For Chinese tasks, users need to build their own dataset and perform cold-start SFT and GRPO training.

Q6: How to deploy it in a production environment?
A: To deploy in production, export the trained policy model into a vLLM-compatible format and ensure that the Milvus memory bank and embedding service continue to run. The framework provides inference scripts that allow calling the Agent for decision-making via REST API. It is recommended to conduct thorough OOD testing before deployment to ensure the reconstruction mechanism is effective in new scenarios.

9. Project Links

• GitHub Repository: https://github.com/KnowledgeXLab/MemHarness
• HuggingFace Model Library: https://huggingface.co/KnowledgeXLab/MemHarness
• arXiv Technical Paper: https://arxiv.org/pdf/2607.28272