MemGUI-Agent – A Long-Horizon Mobile GUI Agent Jointly Developed by Zhejiang University and Kuaishou

1. What is MemGUI-Agent

MemGUI-Agent is a long-horizon mobile GUI agent jointly developed by Zhejiang University and Kuaishou, specifically designed for cross-app, multi-step, and long-chain mobile automation tasks. Traditional ReAct-style agents suffer from key information dilution due to linear expansion of historical records in long-horizon tasks. MemGUI-Agent addresses this issue through an innovative ConAct (Context-as-Action) protocol, allowing the model to simultaneously decide how to interact with the UI and how to organize memory at each step, thereby achieving compact and structured context management. Based on the Qwen3-VL-8B model with LoRA fine-tuning, MemGUI-8B-SFT demonstrates exceptional performance on long-horizon benchmarks, outperforming many 32B and even 235B baseline models with just 8B parameters, showcasing strong capabilities of a small model.

Image source: official article

Technical Positioning and Domain: MemGUI-Agent belongs to the intelligent agent system in the field of mobile GUI automation, focusing on solving complex mobile operation tasks that span multiple applications, long chains, and multiple steps. It differs from traditional UI automation testing tools or simple screen click simulators by integrating large language models (LLM) with vision-language models (VLM), granting the agent the ability for autonomous planning, memory management, and long-horizon reasoning. This tool innovatively elevates context management from an external rule module to a trainable strategy output within the domain of mobile GUI agents, filling a critical gap in solutions for information dilution in long-horizon tasks.

Development Background: This project was jointly developed by the School of Computer Science and Technology at Zhejiang University and Kuaishou Technology, leveraging the deep expertise of both parties in computer vision, natural language processing, and mobile AI. The motivation for development arose from the pain points in real-world mobile automation scenarios: existing agents often fail or hallucinate in long-horizon tasks with over 20 steps due to linear growth in historical records, which dilutes key UI facts (such as price, phone numbers, and specification parameters). The team found through analysis that traditional ReAct-style agents have passive and append-only context management, lacking the ability to actively compress and structure memory. Therefore, they proposed the ConAct protocol, treating memory operations as a first-class action on par with UI operations, and redesigned the agent's reasoning loop from the ground up.

Core Value: MemGUI-Agent resolves the core contradiction of context expansion and information loss in long-horizon mobile automation tasks. Through the ConAct protocol, the agent can actively decide which parts of the history to compress, which UI facts to retain, and how to describe the current step, ensuring that the context remains compact while preserving key information. In practical testing, ConAct saves approximately 1500 input tokens on average compared to ReAct in 150-step long-horizon tasks, while significantly improving the retention accuracy of key information. Additionally, the solution based on an 8B small model achieves efficient deployment, achieving a 17.9% success rate on the MobileWorld benchmark, an 8.5 percentage point improvement over the baseline, demonstrating excellent out-of-distribution generalization capabilities.

Technical Features: The core technical features of MemGUI-Agent include: the ConAct active context management protocol, three structured state fields (Folded Action History, Folded UI State, Recent Step Record), five structured output segments (thinking, folding, tool_call, ui_observation, action_intent), and explicit memory operations (memory_add/update/delete). These technologies collectively form a compact and trainable long-horizon reasoning framework, enabling an 8B model to perform tasks that previously required larger models.

2. Key Features

• Long-range Task Execution: Supports complex automation workflows across apps and pages, capable of handling over 30 consecutive steps on average. Through the ConAct protocol, the Agent actively manages context, maintaining stable task execution capabilities across trajectories of up to 150 steps. This significantly outperforms traditional ReAct-style Agents, which experience performance degradation after just 20 steps.

• Proactive Context Management: Using the ConAct protocol, the Agent autonomously decides when to compress history, when to write UI facts, and how to describe the current step. Unlike external rule modules or post-processing summarizers, this proactive management treats context updates as trainable policy outputs, allowing the model to dynamically adjust its memory structure based on task requirements.

• Structured Memory Operations: Natively supports memory actions such as memory_add, memory_update, and memory_delete, enabling the persistent storage of key UI facts like prices, phone numbers, and specifications. These memory operations are treated as first-class outputs, on par with UI clicks, swipes, and inputs, and are trained and inferred as part of the model's output, ensuring precise retrieval of critical information even after dozens of steps.

• End-to-end Model Training: Provides the MemGUI-8B-SFT LoRA fine-tuned model based on Qwen3-VL-8B, which is capable of handling long-range tasks with just 8B parameters. The training data includes the MemGUI-3K dataset, comprising 2,956 high-quality trajectories covering complex cross-app, multi-step scenarios. The model outperforms most 32B and even 235B baseline models on long-range benchmarks.

• Fully Open-source Pipeline: The code, MemGUI-3K dataset, training/evaluation pipeline, and model weights are all open-sourced. Researchers can fully reproduce the entire process from data construction to model training and evaluation, greatly reducing the research barrier for mobile GUI Agents.

• Out-of-Distribution Generalization: Achieves a 17.9% success rate on the MobileWorld benchmark (GUI-Only setting), an 8.5 percentage point improvement over the baseline. This demonstrates that the model not only performs well in training scenarios but also effectively generalizes to unseen apps and task types, showcasing strong cross-scenario adaptability.

3. How to Use

1. Environment Requirements and Prerequisites: A Linux or Windows machine equipped with a GPU is required, with an NVIDIA GPU (at least 8GB VRAM) recommended. It is advised to use Ubuntu 20.04 or higher as the operating system, along with Python 3.8+ and deep learning frameworks such as PyTorch installed. For Android environments, a physical smartphone with developer mode enabled and USB debugging activated is needed, or an Android emulator (recommended to use the emulator built into Android Studio or Genymotion).

2. Clone the Open-Source Code: Pull the complete code and training/evaluation pipeline from the GitHub repository github.com/kwai/MemGUI-Agent. Use the command git clone （link to be updated after official release） install -r requirements.txt.

3. Download the Dataset and Model Weights: Download the MemGUI-3K dataset (containing 2956 trajectories) and the MemGUI-8B-SFT model weights from the MemGUI-Agent collection page on HuggingFace. Load the dataset using the datasets library from HuggingFace, and load the model weights using the from_pretrained method from the transformers library.

4. Configure the Android Environment: Connect the Android device or start the emulator, ensuring that the adb devices command can recognize the device. Configure the ADB path and screen resolution parameters according to the project documentation. Some devices may require the installation of additional input methods or accessibility services to support UI operations.

5. Input Task Instructions and Execute: Describe long-term goals in natural language, such as “Open Taobao, search for 'wireless earphones,' filter products priced between 200-500 yuan, and record the names, prices, and ratings of the top three products in a memo.” The Agent will automatically plan and execute cross-app automation workflows, generating structured memory operations and UI action sequences during the process.

6. Notes and Best Practices: It is recommended to test for the first time on an emulator to avoid unintended operations on a physical device. For long-term tasks, regularly check the Agent's execution logs to confirm that memory operations are correct. If the task fails, adjust the granularity of the step-by-step task description or verify that the device's screen resolution matches the training data.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For scenarios requiring the processing of ultra-long (more than 30 steps) cross-App tasks, MemGUI-8B-SFT is the preferred choice. Its ConAct protocol demonstrates clear advantages in context management and information retention, and the full open-source nature of the entire pipeline facilitates customization and secondary development. If the tasks are primarily short-range (10–20 steps) and deployment convenience is a priority, OpenMobile-8B is also a solid option, as its ReAct-style implementation is simpler. However, be mindful of performance degradation caused by linear historical expansion. For commercial applications that do not prioritize open source and can accept API call costs, AppAgent offers strong visual understanding capabilities based on GPT-4V. However, it lacks explicit memory operations and may be less reliable in scenarios requiring persistent storage of UI facts compared to MemGUI-Agent.

6. Editor's Take

MemGUI-Agent has made significant technical innovations in the field of mobile GUI Agent research. Its core contribution lies in elevating context management from a passive post-processing step to an active, trainable strategy output—the ConAct protocol. This design fundamentally addresses the issue of critical information dilution caused by historical linear expansion in long-range tasks, enabling an 8B parameter small model to perform tasks that previously required 32B or even larger models. From a technical evaluation perspective, the experimental data showing a reduction of approximately 1500 input tokens in a 150-step long-range task is compelling. The 17.9% success rate on the MobileWorld benchmark, which represents an 8.5 percentage point improvement over the baseline, also confirms its out-of-distribution generalization capability.

In terms of practical value, MemGUI-Agent's fully open-source strategy significantly lowers the research barrier. From data construction (MemGUI-3K), model training (LoRA fine-tuning), to evaluation protocols, all components are reproducible, which is of great significance for promoting standardized research in the field of mobile GUI Agents. Additionally, the efficient deployment characteristics of the 8B small model make it a promising candidate for practical applications in resource-constrained scenarios.

The primary target users of this tool are: researchers in mobile automation, GUI Agent developers, RPA tool vendors, and advanced users with needs for cross-app automation. For academic researchers, the fully open-source nature provides an excellent research platform. For industrial developers, it can be used as a foundation to build customized mobile automation solutions.

Looking ahead, MemGUI-Agent has several promising directions for future development: first, expanding support to the iOS platform; second, further optimizing the triggering strategy for memory operations to reduce excessive compression; and third, combining it with more powerful foundational models (e.g., Qwen3-VL-72B) for knowledge distillation to further enhance performance.

7. Application Scenarios

• Cross-App Information Integration: Users can issue instructions such as "Search for 'wireless earbuds' on Taobao, view the top five prices and reviews, search for related reviews on Zhihu, and compile the integrated information into a memo." The Agent will automatically switch between e-commerce apps, social apps, and note-taking apps, collecting and persistently storing key facts, and ultimately generating a structured document.

• Long Form Filling: In complex registration or reimbursement processes, the Agent can extract verification codes from emails, retrieve contact information from the address book, and fetch dates from the calendar, automatically filling in multiple form pages. Through structured memory operations, the Agent maintains information consistency across multi-app workflows.

• Contact Management and Updates: Extract new phone numbers or email addresses from social media posts or chat records, automatically update contact information in the address book, and send SMS or email notifications. The Agent uses the memory_update operation to persistently store the history of contact changes.

• Product Price Comparison and Tracking: Browse multiple e-commerce platforms (Taobao, JD.com, Pinduoduo), record the price, specifications, and promotional information of specified products, generate a comparison document, and save it to a cloud note. The Agent uses the memory_add operation to incrementally accumulate data from each platform and finally outputs a consolidated summary.

• Automated Testing and RPA: Perform long-chain UI regression testing on mobile devices, such as simulating a user completing the full process from login, browsing, placing an order, to payment. The Agent can record the UI state and operation results at each step, facilitating test analysts in identifying failure causes. In enterprise-level RPA scenarios, cross-app business process automation solutions can be built based on MemGUI-Agent.

8. FAQ

Q: Does MemGUI-Agent support iOS devices?
A: Currently, MemGUI-Agent only supports the Android system, requiring connection to an Android device or launching an Android emulator. Support for iOS platforms has not yet been implemented, but it may be expanded in the future through Apple's accessibility APIs or the XCTest framework.

Q: How much computational resources are needed to train MemGUI-8B-SFT?
A: LoRA fine-tuning can be completed on a single GPU with 24GB VRAM (such as the NVIDIA RTX 4090 or A10G). Full fine-tuning requires a larger VRAM (recommended 80GB A100). For inference, 8GB VRAM is sufficient, but memory usage increases with the number of steps in long-range tasks.

Q: How was the MemGUI-3K dataset constructed?
A: The dataset contains 2,956 high-quality trajectories covering complex cross-App, multi-step scenarios. Each trajectory consists of a manually annotated sequence of UI operations and corresponding memory operations, and has undergone quality screening and consistency checks. The dataset is fully open-sourced on HuggingFace.

Q: What is the main difference between the ConAct protocol and ReAct?
A: ReAct linearly appends historical records to the context, leading to information bloat and dilution of key information. In contrast, ConAct treats context management as a component of the model's output, enabling compact and retrievable context representations through the collaborative work of three fields: folded history, structured memory, and recent records.

Q: Can MemGUI-Agent handle devices with different screen resolutions?
A: The model was trained using data from various resolutions, but optimal performance requires the device's screen resolution to match the training data. It is recommended to specify the target device's resolution in the configuration file, and the Agent will automatically adjust its UI element positioning strategy. If the resolution difference is too large, retraining or fine-tuning may be necessary.

Q: How is the task success rate of MemGUI-Agent evaluated?
A: The official evaluation of the success rate was conducted on the MobileWorld benchmark under the GUI-Only setting (17.9%), and detailed step-by-step success rate analysis is also provided. Researchers can use their own test sets and perform automatic evaluation using the evaluation pipeline in the project.

Q: Are there any restrictions on the commercial use of MemGUI-Agent?
A: The project uses the Apache 2.0 license, with the code and model weights fully open-sourced, supporting both commercial and academic use. However, note that the license for the base model Qwen3-VL-8B may differ. It is recommended to review its official license terms.

9. Project Links

• Project Website: https://memgui-agent.github.io/
• HuggingFace Model Library: https://huggingface.co/collections/lgy0404/memgui-agent
• arXiv Technical Paper: https://arxiv.org/pdf/2606.19926
• GitHub Code Repository: https://github.com/kwai/MemGUI-Agent