PhoneBuddy – 4B Parameter Mobile Agent Model Open-Sourced by Tencent Hunyuan

1. What is PhoneBuddy

PhoneBuddy is a 4B parameter mobile Agent model open-sourced by the Tencent Hunyuan team, focusing on how to train a usable AI Agent in real mobile phone scenarios. The model adopts a "Real App + Mock App Hybrid Reinforcement Learning (RL)" training scheme. In 150 real-device evaluation tasks, its success rate for both single App and WeChat Mini Program tasks surpassed GPT-5.4, achieving 83.2% on the AndroidWorld benchmark. This proves that small parameter models, through optimized environment design, can possess capabilities competitive with closed-source large models. PhoneBuddy not only open-sources the model weights but also releases the training environment, execution framework, and safety evaluation tools, providing a reproducible technical baseline for directions like mobile automation, intelligent assistants, and accessibility support.

Image source: Official article

Technical Positioning and Domain: PhoneBuddy belongs to the mobile GUI Agent domain, focusing on executing operational tasks driven by natural language instructions in real Android phone environments. Its unique positioning lies in: using an open-source model with only 4B parameters, it surpasses closed-source models with parameter scales dozens of times larger (like GPT-5.4) on multiple benchmarks through an innovative real+simulated hybrid training strategy, challenging the conventional wisdom that "bigger models are necessarily stronger."

Development Background: Developed by the Tencent Hunyuan team, which has deep expertise in natural language processing and multimodal models. The research motivation stems from the core contradiction facing current mobile Agents: high training costs in real environments make scaling difficult, while models trained in purely simulated environments exhibit poor generalization in real scenarios. The hybrid training scheme proposed by PhoneBuddy aims to resolve this contradiction, pushing mobile Agents from the lab towards practical application.

Core Value: It solves the dilemma in mobile Agent training between being "real but hard to train" and "controllable but unrealistic." By constructing the PhoneWorld Mock environment, it provides resettable, verifiable, and scalable training signals while preserving the business logic and side effects of real Apps. This allows RL training to obtain stable reward signals, ultimately achieving high performance for a small model on real tasks. This methodology holds significant reference value for teams and scenarios with limited resources.

Technical Features: The two-stage training architecture (Shared SFT + RL) allows the model to learn both real and simulated operation trajectories simultaneously; the task-level verification mechanism directly checks the final state of a task (e.g., whether a message was sent, whether a document was saved), rather than just judging page similarity; the complete tech stack is open-sourced, from the environment and training to execution and safety evaluation, all publicly available, ensuring strong reproducibility.

2. Key Features

• Mobile GUI Understanding and Operation: The model can parse UI elements (buttons, input boxes, lists, etc.) in mobile screenshots and directly control the phone by predicting actions such as click coordinates, swipe directions, and text input. This capability is based on vision-language alignment technology, requires no reliance on native Accessibility APIs, and is applicable to any Android application.

• Single-App Task Execution: Completes operations such as search, settings, and content creation within a single native application. For example, enabling airplane mode in the Settings app, or searching for a specific keyword in a browser. PhoneBuddy achieves a 62.0% success rate on single-app tasks, surpassing GPT-5.4's 50.0%.

• Cross-App Information Flow: Capable of transferring information between multiple applications to complete complex tasks requiring collaboration. For example, extracting a meeting time from an email and adding it to a calendar, or selecting an image from the gallery and sending it to a chat application. The cross-app task success rate is 18.0%, making it the most challenging scenario currently, but initial capabilities have been demonstrated.

• WeChat Mini Program Operation: Supports completing tasks such as search, booking, and inquiry within the WeChat Mini Program ecosystem. The model can recognize the unique page structures and interaction logic of Mini Programs, achieving a 56.0% success rate on WeChat Mini Program tasks, significantly higher than GPT-5.4's 40.0%.

• Real-Environment Task Verification: A built-in task verifier directly checks whether the final task state meets the user's intent. For example, verifying whether a message was actually sent successfully (rather than just the page showing "sent"), or whether a document has been saved to the specified location. This "outcome-oriented" verification mechanism avoids false positives that rely solely on page similarity.

• Extensible Mock Environment Support: The PhoneWorld environment accompanying PhoneBuddy allows developers to reconstruct runnable Android Mock Apps from real GUI structures, preserving page layouts, navigation relationships, and interactive elements. This enables RL training to receive stable reward signals while supporting large-scale parallel training.

3. How to Use

1. Environment Setup: Download the PhoneBuddy-4B model weights (approximately 8GB) from HuggingFace and install the required dependencies (Python 3.9+, PyTorch 2.0+, transformers, etc.). Prepare a physical Android device (Android 10+) or an emulator (such as Android Studio Emulator) as the execution environment, ensuring Developer Options and USB Debugging are enabled.

2. Data Collection: To reproduce the training pipeline, you need to collect real App operation trajectories (screenshots + action sequences) for the Shared SFT phase. Additionally, you must build or connect to the PhoneWorld Mock App to obtain verifiable interaction data. The official resources provide PhoneWorld environment construction tools and sample Mock Apps.

3. Model Training: First, perform Shared SFT training to enable the model to master the unified operation format (e.g., [ACTION] tap (x,y)). Then, choose either the Real-only or Real+Mock environment for RL fine-tuning. The Real+Mock hybrid approach is recommended for optimal performance. Training hyperparameters can be referenced in the paper's appendix.

4. Task Deployment: Connect the trained model to the phone execution layer (e.g., the PhoneHarness framework) and issue task commands via a GUI or CLI. PhoneHarness is responsible for mapping the model's output actions to Android ADB commands and capturing screenshots to provide feedback to the model.

5. Result Verification: Use a task verifier to check the final execution state. The verifier requires specific judgment logic to be written for each task (e.g., checking if a file exists, or if a message appears in the chat history). The official resources provide a general verifier template that can be customized based on the task type.

Notes: The RL training phase requires significant computational resources (8×A100 80GB recommended), but the inference phase can run on a single consumer-grade GPU (such as an RTX 4090). Building Mock Apps requires familiarity with Android development; the official documentation provides detailed PhoneWorld guides and sample code.

4. Pros and Cons Analysis

5. Comparison with Similar Tools

Selection Advice: For academic research teams or developers seeking deep customization of phone Agents, PhoneBuddy is the top choice—it provides a complete open-source tech stack, and its 4B parameters allow inference on consumer-grade GPUs. For enterprises pursuing ultimate performance with sufficient budget, closed-source models like GPT-5.4 can be considered, but note their dependency on real environments and opaque training schemes. For rapid prototype validation, AppAgent's zero-shot approach is more convenient, but long-term performance is limited by the underlying model's capability and incurs higher costs.

6. Editor's Take

PhoneBuddy demonstrates significant innovation in its technical approach. Its core contribution lies in proposing a "Real + Simulated Hybrid RL" training paradigm, which cleverly balances the business complexity of real environments with the scalability of simulated ones. This concept is not only applicable to phone agents but also provides a reference for training in fields such as robot manipulation and web automation. From a practical value perspective, the results of a 4B parameter model surpassing GPT-5.4 on multiple benchmarks are encouraging. It shows that through carefully designed training environments, small models can compete with large models on specific tasks, lowering the barrier to entry for phone agent technology. However, the cross-app task success rate is only 18.0%, indicating that the model still has significant shortcomings in multi-step, multi-application collaboration scenarios, requiring more complex memory and planning mechanisms in the future. Regarding the target audience, this model is best suited for researchers and engineers with an Android development background, who can leverage the open-source code for secondary development; for purely business-oriented users, they may need to wait for official or community-packaged, more user-friendly products. The future development potential is considerable: with the expansion of the PhoneWorld environment library and community contributions, PhoneBuddy has the potential to become a benchmark platform in the phone agent field. Recommendation Index: ★★★★☆ (4.5/5). Points deducted are for cross-app capabilities and the high training barrier, but the overall technical value and open-source commitment are commendable.

7. Application Scenarios

• Smartphone Assistant: Users can instruct the Agent through natural language commands to complete daily operations such as checking deliveries, booking hotels, and sending messages. For example, by saying "Check where the package I bought yesterday is," PhoneBuddy will automatically open the logistics app and take a screenshot to identify the tracking information. Compared to traditional voice assistants, it can execute multi-step operations and is not limited to the API interfaces of specific apps.

• Automated Testing: Automatically execute UI testing processes in real apps to verify functional integrity and business logic. Test engineers can write test cases described in natural language (e.g., "Open settings, turn off Bluetooth, then verify the Bluetooth icon turns gray"), and PhoneBuddy will automatically execute and verify the results, significantly reducing the cost of writing test scripts.

• Accessibility Assistance: Help users with visual impairments or operational difficulties automatically complete complex multi-step operations on their phones. For example, if a user says, "Help me send yesterday's photo to Mom on WeChat," the Agent will automatically open WeChat, select the contact, choose the photo from the gallery, and send it. Since the model does not rely on the visual Accessibility API, it offers better compatibility.

• Cross-Application Workflows: Automatically extract information from emails to fill in forms, or transfer data between multiple office apps. For instance, extracting the date and time of a meeting invitation from a corporate email, automatically creating an event in the calendar app, and notifying attendees. Although PhoneBuddy's cross-app capabilities still have limitations, they are already usable in highly structured scenarios.

• Mini-Program Service Automation: Complete tasks such as inquiries, reservations, and ordering within WeChat mini-programs without installing native apps. For example, automatically placing an order in the "Meituan Waimai" mini-program or booking a department in the "Hospital Registration" mini-program. This is particularly valuable for lightweight service scenarios, as users do not need to download multiple apps.

8. FAQ

Q: Does PhoneBuddy support iOS?
A: Currently, it only supports Android. Model training and testing are based on Android real devices and emulators. Due to the closed nature of iOS, it is not supported at this time. Future expansion may be possible by adapting to iOS accessibility APIs.

Q: How much computational resource is needed to train PhoneBuddy?
A: For the Shared SFT phase, 4×A100 80GB GPUs are recommended; for the RL phase, 8×A100 80GB GPUs are recommended. During inference, it can run on a single RTX 4090 with approximately 12GB of VRAM usage, supporting real-time interaction (approximately 1-2 seconds per inference step).

Q: How can I build my own PhoneWorld Mock App?
A: We provide an official PhoneWorld building tool that can automatically generate the code framework for a Mock App from the GUI structure of a real app (obtained via the Accessibility Tree). Developers need to manually fill in the business logic and validators. For detailed steps, see the phoneworld/ directory in the GitHub repository.

Q: How does PhoneBuddy perform in Chinese-language scenarios?
A: The model supports Chinese instruction input, but the training data has a higher proportion of English-language apps. For native Chinese apps (e.g., WeChat, Alipay), the single-app task success rate is about 55%, slightly lower than in English scenarios. Due to sufficient training data for WeChat Mini Programs, the success rate for Chinese tasks approaches 56%. It is recommended to use Chinese UI screenshots when using Chinese instructions.

Q: What are PhoneBuddy's security and privacy mechanisms?
A: The model weights and training data do not contain user privacy information. During deployment, it is recommended to run in an isolated environment (e.g., a dedicated test phone) and limit the range of apps the Agent can operate. We provide a security evaluation tool that can detect if the model attempts to perform sensitive operations (e.g., sending SMS, accessing contacts).

Q: What are the advantages of PhoneBuddy compared to CogAgent?
A: CogAgent is a 9B-parameter open-source visual agent that also supports mobile phone operation. PhoneBuddy achieves 83.2% on AndroidWorld with 4B parameters, while CogAgent's public score is around 75%. PhoneBuddy's hybrid training method is also easier to scale. However, CogAgent supports more input modalities (e.g., processing text and images simultaneously) and may be stronger in complex visual understanding.

9. Project Links

• Official Website: https://phonebuddyai.github.io/
• GitHub Repository: https://github.com/PhoneBuddyAI/phonebuddy
• HuggingFace Model Hub: https://huggingface.co/PhoneBuddyAI/PhoneBuddy-4B
• Technical Paper: https://phonebuddyai.github.io/assets/paper.pdf