Qwen-CUA – The Native Computer Use Agent Introduced by Alibaba Qwen and Others

1. What is Qwen-CUA

Qwen-CUA is a native Computer Use Agent introduced jointly by the Qwen team and XLang Lab. Based on a 397B-A17B Mixture-of-Experts (MoE) architecture, it perceives the interface state solely through screenshots, without relying on DOM trees or auxiliary function metadata, and directly outputs keyboard and mouse events to control browsers, desktop applications, and professional software. It achieved a score of 86.2 on the OSWorld-Verified benchmark, while the larger version, Qwen-CUA-Max, with over 10,000 billion parameters, reached 87.6, placing it at the forefront of open-source CUA technology.

Technical Positioning and Domain: Belongs to the field of GUI automation driven by multimodal interaction and computer vision, positioned as an end-to-end visual-action model. It directly overcomes the traditional perception barriers that rely on HTML structure or system APIs, enabling true cross-platform general control.

Development Background: Jointly launched by the Alibaba Qwen team and XLang Lab, leveraging the deep expertise of the Qwen series of large models in visual understanding and multimodal alignment. The goal is to address the over-reliance of existing Computer Use Agents on structured information or third-party APIs, and to advance open-source CUA technology toward practical application.

Core Value: By using pure visual perception, it breaks platform limitations and eliminates the need to write adaptation scripts for each application, significantly reducing the cost of automation development and maintenance. The end-to-end training paradigm enhances the stability and continuity of operation sequences, providing a high-cost-performance base model for scenarios such as RPA, accessibility assistance, and research automation.

Technical Features: Employs a 397B-A17B MoE architecture to balance expressiveness and inference costs; a hybrid attention mechanism reduces the computational complexity of long sequences from O(n²) to near-linear, increasing the 256K context throughput by 19 times; 512 fine-grained expert routes combined with shared experts enable flexible knowledge activation and information fusion.

2. Key Features

• Pure Visual Perception: Understands interface states solely through screenshots, without relying on HTML DOM trees, accessibility metadata, or operating system APIs. This design enables the model to control any software with a graphical interface, including browsers, desktop applications, and professional tools, offering significantly greater cross-platform universality than traditional solutions.

• Native Input Simulation: Directly outputs keyboard and mouse events (clicks, input, scrolling, dragging, etc.), without requiring intermediate API conversion or external automation frameworks. The model learns precise coordinate localization and action sequence generation during end-to-end training, resulting in low operational latency and high stability.

• MoE Architecture Foundation: Built on the Qwen3.5-397B-A17B Mixture-of-Experts model, with a total of 397B parameters. Only 17B parameters are activated during each forward inference. This design maintains strong expressive capabilities while controlling inference costs. The Max version exceeds trillions of parameters, further enhancing performance and adapting to different computational budgets.

• Hybrid Attention Mechanism: Combines GatedDeltaNet linear attention with Gated Full Attention in a 75:25 alternating configuration, reducing the computational complexity of long sequences from O(n²) to nearly linear. Under a 256K context size, throughput is 19 times higher than traditional architectures, supporting ultra-long operation history records.

• Fine-Grained Expert Routing: Each layer is equipped with 512 routing experts, activated through Top-10 selection and weight normalization. It is further complemented by one shared expert and a sigmoid gate for adaptive regulation of general knowledge contributions. This design prevents information loss and enhances the model's generalization capabilities across diverse interfaces.

• Visual Encoding and Localization: Inherits the ViT architecture from the Qwen-VL series, supporting dynamic resolution input. Combined with the MRoPE multimodal rotary position encoding, it simultaneously encodes three-dimensional information of time, height, and width, enabling precise localization of interface element coordinates and achieving millimeter-level click accuracy.

3. How to Use

1. Environment Requirements: It is recommended to use a server equipped with an NVIDIA A100 or higher-performance GPU, with a video memory capacity of no less than 80GB (for the 397B-A17B model). The operating system supports Linux (Ubuntu 20.04+), and Python 3.8+, PyTorch 2.0+, and CUDA 11.8+ must be installed. If using the Max version, higher computational power configurations are required.

2. Obtaining Code and Models: Access the GitHub repository xlang-ai/Qwen-CUA and use git clone to download the project code. Model weights can be downloaded via Hugging Face or official links, or automatically fetched using the scripts provided in the project. The technical report and demo code are both included in the repository.

3. Setting Up the Runtime Environment: Enter the project directory and run pip install -r requirements.txt to install dependencies. Follow the instructions in the README to configure the model path and inference parameters (such as context length and sampling temperature). It is recommended to first use the official configuration file template for adaptation.

4. Running the Reference Demo: Execute python run_demo.py to launch the example script. The demo will load the model and open a test interface, demonstrating the automated operation process through closed-loop input via screenshots and output via keyboard and mouse. This helps users verify whether the environment configuration is correct.

5. Integrating into Your Own Environment: Integrate the model into a local or cloud-based desktop or browser environment. Build a closed-loop task using screenshot input (supporting PNG/JPEG formats) and keyboard/mouse output (via PyAutoGUI or similar libraries). The official documentation provides API calling examples, facilitating integration into existing automation pipelines.

Notes: Pure visual perception may result in misidentification under low-resolution conditions or when interface elements are densely packed. It is recommended to maintain a screen resolution of 1920×1080 or higher. The first model load may take a long time, so it is advised to store weight files on an SSD. When deploying multiple instances, pay attention to the allocation of video memory.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For research teams or enterprises requiring high precision, cross-platform support, and sufficient budget, Qwen-CUA is the preferred choice due to its open-source architecture and leading benchmark scores, especially suitable for long-tail scenarios such as research automation and professional software control. If the team has limited computational resources and mainly focuses on browser automation, the closed-source API solution of OpenAI CUA/Operator can reduce deployment complexity, but it has higher long-term usage costs and limited flexibility. The multi-agent concatenation approach is suitable for rapid prototype verification, but its stability and end-to-end coherence are difficult to ensure, and it is not recommended for production-level tasks.

6. Editor's Summary

Qwen-CUA demonstrates a clear path of technological differentiation: by relying purely on visual perception, it completely eliminates dependence on DOM trees or system APIs, allowing the model to "see and operate on the screen" just like humans. This design philosophy inherently offers advantages in cross-platform universality. Its hybrid attention mechanism and fine-grained expert routing effectively address issues of computational efficiency for long sequences and knowledge activation at the engineering level. The data showing a 19-fold increase in context throughput with 256K context validates the advanced nature of its architectural design. In terms of practical value, the OSWorld benchmark score of 86.2 confirms the feasibility of the pure visual approach in complex GUI tasks. The open-source Apache 2.0 license reduces barriers for academic research and commercial integration. This tool is primarily aimed at research institutions, automation testing teams, and RPA developers with a certain level of computational resources, especially suitable for scenarios requiring control of professional software or legacy systems without API interfaces. In terms of future development potential, as community contributions grow and fine-tuning tools improve, Qwen-CUA has the potential to become a foundational model in the open-source CUA domain, driving more customized applications in vertical scenarios. However, the current documentation and ecosystem are still in an early stage, and hardware requirements limit its popularity among individual developers. Continued investment in model quantization, inference acceleration, and toolchain enhancement will be necessary in the future.

7. Application Scenarios

• Automated Software Testing: Identify UI elements through screenshots and perform actions such as clicks, input, and scrolling, enabling cross-platform UI regression testing without the need to write specific scripts. Particularly suitable for Web applications and desktop clients that undergo frequent iterations, significantly reducing the maintenance cost of test scripts.

• RPA Process Automation: Replace manual operations for repetitive desktop tasks such as data entry, report generation, and ERP operations, without the need to modify existing enterprise IT systems. Qwen-CUA can directly interact with any graphical interface, making it especially effective for legacy systems or software without API support.

• Accessibility Assistant Tools: Automatically recognize software interface elements for visually impaired users or the elderly and perform precise operations on their behalf, lowering the barrier to using professional software. The model understands the positions of components such as buttons and menus through screenshots, and with voice commands, it can achieve fully mouse-free operation.

• Scientific Research and Professional Software Control: Automatically control professional software such as MATLAB, SPSS, and Origin that lack API interfaces, enabling batch execution of long-term tasks like experimental data processing, chart generation, and statistical analysis. Researchers can define operation workflows and then delegate them to the model for automatic execution, improving work efficiency.

• Cloud-Based Agent Hosting Services: Deployed in a cloud desktop environment, it can operate 7×24 hours a day to perform background tasks such as website monitoring, data scraping, and scheduled report downloads. Qwen-CUA's purely visual characteristics make it unaffected by differences in cloud desktop environments, offering greater stability than solutions relying on DOM.

8. FAQ

Q: Which operating systems does Qwen-CUA support?
A: The model itself does not rely on a specific operating system and can run as long as the environment provides a screen capture interface and keyboard/mouse simulation capabilities. The official Demo has been validated on Linux, and Windows and macOS can be adapted using libraries such as PyAutoGUI.

Q: What hardware is required to deploy Qwen-CUA?
A: The 397B-A17B model is recommended to be deployed on an NVIDIA A100 80GB GPU or a GPU with equivalent computational power, with a VRAM recommendation of no less than 80GB. If using the quantized version or Max version, adjustments should be made based on actual conditions. Consumer-grade GPUs (such as the RTX 4090 24GB) may struggle to run the model directly; model distillation or sliced deployment can be attempted.

Q: What advantages does Qwen-CUA have compared to OpenAI Operator?
A: Qwen-CUA is an open-source model, with complete technical reports and Demo publicly available, supporting local deployment and secondary development. It scores significantly higher (86.2) on the OSWorld benchmark compared to Operator's historical performance (approximately 38.1%). Its pure visual perception capability enables it to control any graphical interface, while Operator is primarily optimized for browser-based tasks.

Q: Does the model support operations on Chinese interfaces?
A: Qwen-CUA's visual encoder has been trained on multiple languages and is capable of recognizing Chinese interface elements (such as button text and menu items). Actual performance depends on the clarity of the interface and font standardization, and targeted testing is recommended in Chinese scenarios.

Q: How can Qwen-CUA be integrated into an existing automation pipeline?
A: The project provides Python API examples, allowing the construction of a closed-loop system through screenshot input (image arrays or file paths) and action output (lists of keyboard and mouse events). It supports asynchronous inference and batch task scheduling, and can be encapsulated as a RESTful API or gRPC service.

9. Project Links

• GitHub Repository: https://github.com/xlang-ai/Qwen-CUA
• Technical Paper: https://github.com/xlang-ai/Qwen-CUA/blob/main/paper/Qwen-CUA.pdf
• Official Organization Homepage: https://github.com/xlang-ai (XLang Lab organization, follow for future updates)