Qwen-UI-Agent – The GUI Agent Foundation Model Launched by Alibaba Tongyi

1. What is Qwen-UI-Agent

Qwen-UI-Agent is a GUI agent foundation model launched by the Alibaba Tongyi team, capable of operating interfaces on mobile phones, computers, and web browsers just like a real person. Trained on a real-device cluster comprising over 100 devices and more than 150 applications, this model supports the mixed execution of graphical interfaces and command-line interfaces, enabling the completion of complex, long-horizon tasks across multiple applications. It features a built-in hierarchical security mechanism that directly rejects dangerous requests and proactively prompts users for confirmation before performing sensitive operations such as payments or deletions. On multiple authoritative benchmarks such as MobileWorld and WebArena, its performance is either superior to or on par with flagship models like GPT-5.6 and Claude Opus, making it one of the leading GUI agents in terms of real-world usability today.

Image source: Official article
Image source: official article

Technical positioning and domain: Qwen-UI-Agent belongs to the intersection of multimodal agents and GUI automation, focusing on simulating human operational behavior on real digital devices to achieve complex task automation across platforms and applications. Unlike traditional RPA tools that rely on scripts and fixed rules, it leverages the reasoning capabilities of large models to enable dynamic decision-making and adaptive operations, covering three major terminals: mobile phones, computers, and browsers.

Development background: Developed by the Alibaba Tongyi team based on their deep expertise in the Tongyi Qianwen series of large models, Qwen-UI-Agent aims to bridge the performance gap between simulated environments and real devices, pushing GUI agents from the lab into practical use. The team previously accumulated experience in mobile GUI agents through projects such as MAI-UI, and Qwen-UI-Agent represents the culmination of their efforts in this direction.

Core value: It addresses the core pain points of existing GUI agents, including low usability in real environments, poor stability for long-horizon tasks, and weak security mechanisms, by providing a unified agent solution that can be directly deployed on mobile phones, computers, and browsers. The real-device training and mixed execution capabilities significantly enhance the success rate of tasks in real-world scenarios, while the hierarchical security mechanism ensures controllability of operations.

Technical features: It employs innovative technologies such as real-device training environments, hybrid action spaces (GUI + CLI), long-horizon online reinforcement learning, and hierarchical security alignment, achieving leading performance on multiple benchmarks. Notably, the Batched Actions mechanism allows for the parallel output of multiple actions in a single decision, and its long-horizon online RL with approximately 10,000 concurrent environments is a first in the industry, significantly improving task execution efficiency and long-horizon stability.

2. Key Features

• Real-device Multi-app Workflow: The model is trained on over 100 real smartphones and more than 150 real applications, enabling it to perform complex, long-chain tasks across apps—such as searching for restaurants on a map, comparing ratings on Dianping, and creating an event on DingTalk—all without human intervention. This training method based on real-device trajectories directly bridges the performance gap between simulated environments and real devices, resulting in significantly higher success rates in real-world scenarios compared to similar models trained solely on simulated data.

• Cross-device GUI Interaction: The model can automatically perform interface interactions such as clicking, typing, and swiping on Android phones, Windows/macOS computers, and mainstream browsers, unifying the interaction paradigms across different platforms. It adapts to various screen resolutions, layouts, and OS styles, achieving precise element localization and operation through multi-modal grounding technology, without requiring separate adaptation for each device type.

• GUI + CLI Hybrid Execution: The model is proficient in both GUI operations and command-line instructions, and introduces a Batched Actions mechanism that allows multiple actions to be output in a single decision. In computer-based tasks, CLI operations account for nearly half of the interactions, reducing the total number of execution steps by approximately 58% compared to pure GUI methods, significantly improving task completion efficiency. This hybrid execution approach enables the model to interact with traditional GUI applications while also directly executing scripts and system commands, expanding the boundaries of automation.

• Long-horizon Task Processing: Leveraging a long-horizon online reinforcement learning framework, the model can reliably execute complex tasks with over 100 steps, such as cross-website data collection and report generation. With approximately 10,000 concurrent environments performing parallel rollouts and adaptive curriculum learning, the model gradually increases task difficulty starting from simple tasks, continuously tackling high-complexity long-horizon tasks. On benchmarks like MobileWorld, its success rate for long-horizon tasks significantly outperforms other general-purpose models.

• Accurate Interface Localization: The model sets new SOTA records on grounding benchmarks such as ScreenSpot-Pro, accurately identifying the coordinates and types of screen elements. Even when faced with complex interface layouts (e.g., overlapping elements, dynamic content), it can precisely click on the target area. This capability is based on multi-modal feature alignment and fine-grained position encoding, forming the foundational prerequisite for reliable GUI operations.

• Security Boundary Control: A built-in layered security mechanism ensures safe operation: the first layer uses a pre-classifier to directly reject illegal or high-risk requests (e.g., violent or fraudulent content); the second layer handles sensitive operations such as payments, data deletion, and privacy authorization by triggering the ask_user function at key checkpoints, actively pausing execution and waiting for user confirmation. This design balances automation efficiency with user control.

3. How to Use

1. Read the Technical Report: Access the technical paper (PDF) provided on the project's official website to gain a detailed understanding of Qwen-UI-Agent's model architecture, training methods, and evaluation results on 8 benchmarks. It is recommended to focus on sections such as real-device training environment setup, hybrid action space design, long-horizon online reinforcement learning algorithms, and safety alignment strategies, as these are core content for understanding the model's design philosophy and performance boundaries.

2. Browse the Project Homepage: Visit the project's official website (tongyi-mai.github.io/Qwen-UI-Agent/), and watch the 5 real-world scenario demonstration videos, including examples such as cross-device invoice organization, proactive emergency handling for flight cancellations, and local life store exploration. These videos intuitively demonstrate the model's operational process and performance in real environments, helping to assess its applicability.

3. Track the Open-Source Repository: Follow the GitHub repository Tongyi-MAI/MAI-UI to obtain future model weights, inference code, and deployment documentation. The current repository already provides some benchmark evaluation code and Docker environment configuration, and will later be updated with model weights and the full inference pipeline. It is recommended to Star the repository to receive update notifications promptly.

4. Run Benchmark Evaluations: Use the Docker environment provided by the companion repository Tongyi-MAI/MobileWorld to reproduce the evaluation process of benchmarks such as MobileWorld on your local machine. You will need an NVIDIA GPU (with a recommended VRAM of 24GB or more) and a Docker environment. By following the steps in the repository's README and executing docker-compose up, you can launch the evaluation container and obtain reproducible evaluation results. This process can be used to verify the model's performance or compare the effects of different configurations.

5. Deploy Inference Service (After Model Release): Once the model weights are officially open-sourced, you can load the model using frameworks such as Hugging Face Transformers or vLLM, and provide GUI operation services via REST API. It is recommended to use a GPU with at least 80GB VRAM (such as the A100 80GB) to ensure inference speed; if using a consumer-grade GPU (such as the RTX 4090 24GB), you can optimize through quantization or batch processing, although throughput will decrease accordingly. When deploying, be sure to configure the safety filtering module and the ask_user callback interface.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

In scenarios requiring local deployment, customization, and high privacy standards, Qwen-UI-Agent is a more suitable choice. Its open-source nature and the robustness brought by real-device training give it a unique advantage in automating long-range cross-application tasks. It is particularly suitable for enterprise internal process automation, research institutions conducting secondary development, and teams requiring deep customization of GUI operation logic. Additionally, its hybrid execution capability provides significant efficiency advantages in desktop tasks, making it ideal for complex workflows that require simultaneous GUI and command-line operations.

If quick integration and a mature ecosystem are priorities, and the task complexity is low, closed-source API products like Claude Opus and GPT-5.6 are more convenient. They offer stable API services and rich documentation support, suitable for small and medium-sized teams to rapidly build GUI automation prototypes. However, it should be noted that these general-purpose models may not match the stability and real-world adaptability of Qwen-UI-Agent, which is specifically designed for such tasks. Additionally, there are concerns regarding data privacy and API costs. For scenarios primarily focused on mobile devices, the Android automation capabilities of Gemini 3.1 Pro are worth considering, although its cross-platform and CLI hybrid capabilities are relatively weak.

6. Editor's Summary

Qwen-UI-Agent demonstrates significant technological innovation in the field of GUI agents. Its real-device training environment directly captures operation trajectories from actual devices, fundamentally addressing the performance gap between simulated environments and the real world. This approach sets an important benchmark for future research. The hybrid action space design unifies GUI clicks with CLI commands and introduces the Batched Actions mechanism, reducing execution steps by nearly 60% in computer tasks, showcasing the depth of engineering optimization. The long-range online reinforcement learning framework supports end-to-end training with over 100 steps, combined with approximately 10,000 concurrent environments for parallel rollout and adaptive curriculum learning, enabling the model to tackle previously challenging complex long-range tasks.

In terms of practical value, Qwen-UI-Agent achieves leading performance on 8 core benchmarks, including MobileWorld, WebArena, and ScreenSpot-Pro. Its real-device demonstration cases highlight its broad applicability, ranging from local life exploration to enterprise office automation. For researchers and developers, this model provides a powerful foundational platform for exploring cutting-edge GUI automation technologies, such as unified cross-device operations, hybrid execution optimization, and safety alignment. For enterprise users, its ability to perform unified operations across devices and applications has the potential to significantly enhance office efficiency and process automation, especially in scenarios involving long-chain tasks across multiple applications. However, the model weights are not yet fully open-sourced, and documentation and community resources are still under development, which to some extent limits its adoption speed and depth of secondary development. In the future, as the open-source ecosystem matures, more application cases are accumulated, and inference efficiency is further optimized, Qwen-UI-Agent is expected to become a key infrastructure in the GUI agent domain, driving digital employees from concept to large-scale practical deployment.

7. Application Scenarios

• Local Life Exploration: After the user issues the instruction "Recommend a restaurant suitable for a date," the model automatically opens Gaode Map to search for nearby restaurants, switches to Dianping to check ratings and user reviews, then opens Xiaohongshu to read exploration notes. Finally, it synthesizes the information to generate a recommendation list and sends it to the user. The entire process does not require the user to manually switch between apps, achieving one-stop automation from information search to decision support.

• Travel and Meeting Scheduling Integration: When the user provides the meeting time and location, the model automatically queries 12306 to find available high-speed train departures, calculates the subway commute time from the train station to the meeting location, then creates a meeting schedule with reminders on DingTalk and pre-books a taxi. This realizes full-chain automation from travel planning to schedule management, significantly reducing manual operation steps.

• In-depth Research Reports: The model collects financial report data from multiple websites, runs a local Python script for data analysis, and automatically generates Excel spreadsheets, PowerPoint presentations, and Word documents with charts included. Finally, it packages the deliverables and sends them to the user. It supports stable execution across more than 100 steps, making it suitable for scenarios requiring multi-source data integration, such as research analysis and competitive benchmarking.

• Proactive Response to Flight Cancellations: When the model detects a flight cancellation via SMS or email, it proactively searches for remaining flights and high-speed train options for the same day. It then evaluates the best alternative based on time, cost, and travel connectivity, presents the recommendation to the user for confirmation, and automatically completes the rescheduling or refund process once confirmed. This demonstrates an upgrade from passive response to proactive service, significantly enhancing user experience.

8. FAQ

Q: Is Qwen-UI-Agent fully open source?
A: The project has currently open-sourced the technical report and part of the benchmark evaluation code, but the model weights and complete inference code have not been officially released yet. The GitHub repository Tongyi-MAI/MAI-UI will be updated in the future; please follow the official announcements. The model weights are expected to be released under an open-source license, with specific terms to be confirmed.

Q: What hardware configuration is required to run Qwen-UI-Agent?
A: The training phase requires a large number of real devices (more than 100 smartphones), which is not feasible for regular users to reproduce. For the inference phase, it is recommended to use an NVIDIA A100 (80GB) or a GPU with equivalent computing power. Consumer-grade GPUs with more than 24GB of VRAM (such as the RTX 4090) can also run the model, but performance will be significantly reduced. Specific dependencies will be clarified after the model is officially released; it is advised to refer to the official recommended configuration.

Q: Which operating systems and platforms does Qwen-UI-Agent support?
A: The model has been verified to work on Android phones, Windows/macOS computers, and mainstream browsers (Chrome, Edge), and is capable of interacting with GUI interfaces on these platforms. It may be expanded to more platforms such as iOS in the future, but currently focuses primarily on Android and desktop environments.

Q: How does Qwen-UI-Agent ensure the safety of operations?
A: It includes a layered security mechanism: the first layer uses a pre-classifier to directly reject illegal or high-risk requests (such as violent or fraudulent content). The second layer handles sensitive operations like payments, data deletion, and privacy authorization by triggering the ask_user function at critical points, pausing execution and waiting for user confirmation to ensure that sensitive operations are not executed without permission. Users can customize the list of sensitive operations.

Q: What is the difference between Qwen-UI-Agent and regular AI assistants (such as ChatGPT)?
A: Regular AI assistants mainly provide text-based conversations and knowledge-based Q&A, while Qwen-UI-Agent can directly interact with the graphical interfaces of phones, computers, and web pages, performing actions such as clicking, typing, and swiping, and completing complex cross-application tasks. It is closer to a "digital employee," capable of performing actual operational workflows on behalf of users rather than just providing information.

9. Project Links

• Product Homepage: https://tongyi-mai.github.io/Qwen-UI-Agent/
• GitHub Repository: https://github.com/Tongyi-MAI/MAI-UI
• Technical Paper: https://tongyi-mai.github.io/Qwen-UI-Agent/Qwen-UI-Agent-Technical-Report.pdf