MMSkills – A Multimodal Skill Framework Jointly Launched by Shanghai Jiao Tong University and Xiaohongshu

1. What is MMSkills

MMSkills is a multimodal skill framework jointly developed by Shanghai Jiao Tong University and Xiaohongshu, specifically designed for general-purpose visual agents. This framework extends traditional text-only skills into multimodal procedural knowledge units that include text workflows, runtime state cards, and multi-view keyframes. Through a Branch Loading mechanism, it efficiently invokes visual evidence at runtime, avoiding decision pollution caused by injecting complete context all at once. On GUI and gaming tasks such as OSWorld, macOSWorld, and VAB-Minecraft, MMSkills delivers stable and significant performance improvements for multiple model families including Gemini, Qwen, Kimi, and GLM. Notably, it more than doubles the performance gains on smaller models, filling the gap in procedural prior knowledge for lightweight vision-language models.

Image source: Official article

Technical Positioning and Domain: MMSkills belongs to the field of multimodal AI agent frameworks, focusing on solving the problem of visual agents lacking reusable procedural knowledge in complex environments. Unlike traditional RPA or simple skill libraries, it represents skills as multimodal structures (text + state cards + keyframes), enabling agents to determine when to use a skill and how to verify execution progress based on real visual states, thereby transforming exploratory trial-and-error into state-aware, precise execution.

Development Background: Shanghai Jiao Tong University has deep expertise in vision-language model and agent research, while Xiaohongshu possesses extensive practical experience in multimodal content understanding and user interaction scenarios. The joint development of MMSkills aims to break through the technical bottleneck where current visual agents frequently fail in long-horizon tasks due to a lack of external procedural knowledge, driving the evolution of agents from "describing images" to "acting upon images."

Core Value: MMSkills addresses the common problems of "inexperience and disorientation" faced by visual agents in tasks such as GUI automation and game manipulation. Traditional agents rely on the model's internal priors, but lightweight models (e.g., 8B parameters) lack sufficient internal procedural knowledge; meanwhile, injecting complete demonstration trajectories leads to excessively long contexts where visual evidence interferes with primary decision-making. MMSkills, through multimodal skill packages and a branch loading mechanism, provides precise visual alignment guidance without polluting the main context, significantly improving task success rates and reducing ineffective actions.

Technical Features: Its core technology lies in the definition of multimodal skill packages (text workflow + state cards + multi-view keyframes) and the Branch Loading runtime mechanism. Skill packages are extracted from public trajectories through an automated five-stage process, rather than simply saving raw demonstrations. Branch Loading allows the main agent to temporarily open a branch at runtime, complete visual evidence alignment through View Selection and Branch Planning, and return compact, structured decision support (applicable, subgoal, plan, do_not_do, verify), achieving efficient context management and state-aware decision-making.

2. Key Features

• Multimodal Skill Package Construction: Expands traditional plain-text skills into complete units containing SKILL.md text procedures, State_cards.json runtime state cards, and Images multi-view keyframes. This structure enables skill packages to not only describe operational steps but also provide visual state judgment criteria, allowing the Agent to match current screenshots against state cards to determine skill applicability and execution progress correctness.

• Automated Skill Generation: Employs a five-stage process (Task Clustering, Intra-cluster Skill Planning, Skill Merging & Generalization, Text Draft Generation, Visual Grounding & Auditing) to automatically extract reusable diagnostic state knowledge from public non-test interaction trajectories. This process avoids the high cost of manual annotation, and the generated skill packages focus on critical state judgment rather than full demonstrations, offering greater generalization capability.

• Branch Loading: When the main Agent determines at runtime that a skill might be useful, it temporarily opens a skill branch, executes View Selection and Branch Planning within that branch, returns structured decision support after completing visual evidence alignment, and then closes the branch. This mechanism avoids injecting complete skill packages (especially large numbers of images) directly into the main context, preventing reference image contamination of the current decision-making process.

• View Selection: Within the Branch Loading branch, intelligently selects the most relevant perspective from the skill package's multi-view keyframes for alignment based on the current screenshot, historical actions, and state cards. This feature reduces the search space for visual matching, improves alignment efficiency, and minimizes interference from irrelevant visual information.

• Structured Decision Guidance: After the branch completes visual alignment, it returns a compact JSON structure to the main Agent, containing applicable (whether the skill is applicable), subgoal (current sub-goal), plan (next step plan), do_not_do (actions to avoid), and verify (verification steps). This structured output is directly embedded into the main Agent's decision loop without requiring additional parsing.

• Cross-Domain Task Coverage: Simultaneously supports GUI desktop automation (e.g., LibreOffice, file management) and visual gaming tasks (e.g., Minecraft, Super Mario Bros). Skill packages are transferable and reusable across different domains, for example, used for item acquisition and crafting planning in Minecraft, and for chart creation on macOS.

• Significant Gains for Small Models: On lightweight models like Qwen3-VL-8B, MMSkills delivers more than double the performance improvement. This is attributed to external multimodal procedural knowledge supplementing the insufficient internal process priors of small models, enabling them to maintain stable decision-making capabilities even in long-horizon tasks.

3. How to Use

1. Obtain the Skill Library: Visit the Hugging Face dataset page (zhangkangning/mmskills) to download pre-generated multimodal skill packages. Skill packages are organized in a standard format, containing SKILL.md, State_cards.json, and an Images directory, ready for direct Agent integration.

2. Integrate with the Main Agent: Connect MMSkills' branch loading mechanism into the inference pipeline of your existing visual Agent. A "Skill Matching & Branch Loading" module needs to be added to the Agent's decision loop. This module is responsible for determining at each step whether the current task might match a skill and triggering a Branch Loading request.

3. Trigger Skill Invocation: During runtime, based on the current screenshot and task description, the main Agent identifies potentially relevant skills through simple text matching or semantic retrieval. When the match score exceeds a threshold, the main Agent initiates a branch loading request, passing the current screenshot, historical action sequence, and candidate skill package to the branch processor.

4. Receive Structured Guidance: The branch processor executes View Selection and Branch Planning. After completing visual evidence alignment, it returns a JSON object containing applicable, subgoal, plan, do_not_do, and verify. The main Agent parses this structure and uses it as a reference for its next decision step.

5. Execute and Verify: The main Agent executes actions according to the returned plan and verifies task progress afterward using the state cards. If verification fails, it can re-trigger skill matching or fall back to the default exploration strategy.

6. Notes and Best Practices: It is recommended to validate the effectiveness of skill packages on a small-scale test set before integration. For different model families (e.g., Gemini, Qwen), the similarity threshold for View Selection may need adjustment. The overhead of Branch Loading is small, but frequent unnecessary branch triggers should be avoided; implementing a cooldown mechanism is recommended. Skill packages can be customized and extended; users can create new skills through an automated generation pipeline based on their own task trajectories.

4. Pros and Cons Analysis

5. Comparison of Similar Tools

Selection Advice: For visual Agent applications pursuing high task success rates and stability, MMSkills is the optimal choice, especially suitable for small model deployment scenarios (e.g., 8B models running on edge devices). If the task environment is highly fixed with minimal interface variation, Text-only Skills might be easier to integrate due to their simplicity and lightweight nature; however, if the interface changes dynamically (e.g., different resolutions, themes), MMSkills' state cards and keyframes provide stronger robustness. For scenarios requiring rapid prototype validation, Demo-based Agents might be more direct, but in the long run, their generalization ability and context management issues will limit scalability. Visual Agents without external skills, such as CogAgent, are suitable as baselines for comparison, but require large amounts of training data or model fine-tuning for complex, long-horizon tasks.

6. Editor's Take

MMSkills introduces a highly innovative paradigm for knowledge representation and invocation in the field of visual agents. Its core insight is that visual agent failures often stem not from insufficient model vision capabilities, but from a lack of reusable procedural knowledge to guide "when to do, what to do, and how to verify." Traditional methods either rely on purely textual skills (lacking visual grounding) or inject full demonstrations (polluting context and offering poor generalization). MMSkills cleverly strikes a balance between knowledge richness and context cleanliness through its multimodal skill packages and Branch Loading mechanism.

From a technical implementation perspective, the automated skill generation pipeline (five stages) ensures the scalability of the skill library, while View Selection and Branch Planning guarantee runtime efficiency. Experimental data shows that MMSkills brings an approximate 10 percentage point improvement to Gemini 3.1 Pro on OSWorld, and more than doubles the performance on Qwen3-VL-8B, fully validating its design philosophy of "supplementing internal priors with external knowledge." It is particularly noteworthy that the gains MMSkills provides for small models imply that lightweight visual agents can also handle long-horizon tasks previously requiring large models, which holds significant value for edge computing and privacy-sensitive scenarios.

Of course, MMSkills is not a panacea. The coverage of its skill library is currently limited to benchmark domains, and enterprise-level customization requires additional investment. Furthermore, the quality of visual alignment in View Selection is highly dependent on the capabilities of the underlying multimodal model; if the model itself has insufficient understanding of interface elements, the framework's effectiveness will be diminished. Future development directions should include community co-construction of the skill library, automated optimization for cross-domain transfer, and lighter-weight alignment algorithms.

Target Audience: Visual agent researchers, GUI automation developers, game AI engineers, RPA system architects. It is particularly suitable for teams looking to use small models (8B-14B) to accomplish complex desktop or gaming tasks.

7. Application Scenarios

• Desktop GUI Automation: Perform complex cross-application operations in Ubuntu/macOS environments, such as LibreOffice chart creation, file permission modification, and system settings adjustment. The Agent identifies the currently active window and button availability status through the MMSkills skill package, avoiding operations on incorrect interfaces, and can stably complete automation workflows that originally required manual demonstration.

• Game AI Agent: Execute long-range item acquisition and crafting planning in Minecraft survival mode, or decide jump timing based on visual state in Super Mario Bros. The skill package includes multi-view keyframes (e.g., trees and ores from different angles), allowing the Agent to accurately identify resource locations and plan paths, reducing ineffective exploration.

• Enterprise RPA Enhancement: Add visual state perception capabilities to existing Robotic Process Automation systems. Traditional RPA relies on coordinates or DOM element positioning, which easily fails when the interface changes; the state cards of the MMSkills skill package can detect whether interface elements exist and are clickable, enabling RPA to adapt to different versions of software interfaces.

• Visual Task Instruction: Distill manual operation experience into a reusable multimodal skill library for training new Agents or assisting other Agents in learning. For example, a senior designer can record Photoshop operation workflows as an MMSkills skill package, and a new Agent can quickly master complex editing techniques by loading the skill package.

8. FAQ

Q: Does MMSkills require a specific multimodal model to use?
A: The MMSkills framework itself is model-agnostic and supports various vision-language models such as Gemini, Qwen, Kimi, and GLM. However, the View Selection and visual alignment in Branch Loading depend on the model's visual understanding capabilities. It is recommended to use models with strong image-text matching abilities (e.g., Qwen2.5-VL, Gemini 2.0 Flash).

Q: How do I create a custom skill package?
A: Users can follow the documentation in the official GitHub repository to collect public interaction trajectories (screenshots + action sequences) and run the automated generation script (a five-stage process) to generate a skill package. Alternatively, you can manually write SKILL.md and State_cards.json, and prepare the corresponding keyframe images.

Q: How much inference latency does Branch Loading add?
A: In typical scenarios, the additional latency from Branch Loading is approximately 50-200ms, depending on the search space of View Selection and the computational cost of visual alignment. Compared to directly injecting a complete skill package (which might contain dozens of images), which leads to context length inflation and a sharp increase in inference time, the overall latency of Branch Loading is lower.

Q: How does MMSkills perform on macOS?
A: In the macOSWorld benchmark, MMSkills brought an improvement of about 8 percentage points for Gemini 3.1 Pro and about 10 percentage points for Qwen3-VL-235B. The skill packages already cover common tasks such as file management, system settings, and application operations.

Q: Can skill packages be used across languages?
A: The textual workflow (SKILL.md) of the skill package is currently primarily in English, but the state cards and keyframes are visual and language-independent. Users can translate the textual workflow into Chinese or other languages without affecting the visual alignment effect. Community members have already contributed Chinese versions of skill packages.

Q: What is the difference between MMSkills and RPA tools (like UiPath)?
A: Traditional RPA relies on fixed coordinates or DOM selectors, which easily fail when the interface changes. MMSkills is based on visual state perception and can adapt to different resolutions and themes. However, MMSkills is currently positioned as an Agent framework and does not offer a complete RPA orchestration interface, making it more suitable for developer integration.

9. Project Links

• Project Website: https://zkangning.github.io/MMSkills_for_Visual_Agents/
• GitHub Repository: https://github.com/zkangning/MMSkills_for_Visual_Agents
• HuggingFace Dataset: https://huggingface.co/datasets/zhangkangning/mmskills
• arXiv Technical Paper: https://arxiv.org/pdf/2605.13527