ACE-Ego – Open Embodied Manipulation VLA from ACERobotics and CUHK

1. What Is ACE-Ego

ACE-Ego is an open one-brain-multi-embodiment manipulation VLA model co-developed by ACERobotics (大晓机器人) and CUHK MMLab. Pretrained on 6.0K+ hours of egocentric human video, it uses camera-space actions and morphology-conditioned encoding so one model adapts across robot bodies. It sets SOTA on RoboCasa GR1 TableTop at 72.8%, reliably performing complex retail ops like bagging in plastic and packing shoes in boxes—cutting multi-embodiment deployment cost sharply.

Technical positioning and domain: Embodied VLA for robot manipulation. Core innovation: one-brain-multi-embodiment generalization—one pretrained model adapts to different hardware without retraining, addressing costly per-body training. Targets retail, logistics, home service with long-horizon contact-rich tasks; fills open-source gap on complex contact manipulation.

Research background: ACERobotics focuses on retail robotics; CUHK MMLab on vision and robot learning. Retail fulfillment needs many robot models in different stores—per-body training does not scale. Human-centric pretraining from daily egocentric video builds general manipulation representations with less robot demo data.

Core value: Cuts cost and barrier for multi-embodiment deployment. Human-centric pretrain + small robot finetune migrates skills efficiently. Camera-space actions and morphology conditioning let one weight set deploy across UR5, Franka, GR1, etc.—order-of-magnitude lower deploy cost. 72.8% on RoboCasa GR1 TableTop beats π0.5 (37.0%) and GR00T-N1.6 (47.6%) on contact-heavy tasks.

Technical characteristics: Human-centric pretrain on 6.0K+ hours egocentric video plus multi-embodiment robot data; camera-space actions unify prediction in camera frame; morphology conditioning encodes URDF, DOF, etc.; reliability-aware training filters low-confidence actions for safer long-horizon ops.

2. Key Features

• One-brain-multi-embodiment: One VLA adapts via morphology interface—register URDF, DOF, camera params, reuse weights without per-robot training. Ideal for retail chains with mixed robot fleets.

• Complex manipulation: Long-horizon contact-rich retail tasks—bagging, shoe boxing, shelf organizing—72.8% SOTA on RoboCasa GR1 TableTop with multi-step grasp, push, fold sequences.

• Bimanual coordination: 90.62% on RoboTwin 2.0 for dual-arm tasks like unboxing and assembly—camera-space actions unify both end-effector trajectories without extra coordination heuristics.

• Human-centric pretrain: 6.0K+ hours egocentric human video plus multi-robot and sim data—general ops representation from human video, few robot demos for transfer, far less robot teleop data needed.

• Unified camera-space actions: Predictions in camera frame remove body/view inconsistencies—same action space for UR5, Franka, GR1, simplifying cross-embodiment engineering.

• Reliability-aware validation: Confidence scoring in training; inference filters low-confidence actions for safer long retail fulfillment sequences.

• Open ecosystem: Weights, training code, inference scripts, paper under Apache 2.0—commercial use and community extension welcome.

3. How to Use

1. Environment and dependencies: Clone official GitHub repo (link to be updated upon official release). Python 3.10+, PyTorch 2.0+, CUDA 11.8+. A100 or RTX 4090 recommended, ≥24GB VRAM. pip install -r requirements.txt (torch, transformers, open3d, opencv-python, etc.).

2. Download and load model: Get ACE-Ego weights from GitHub Releases or Hugging Face. from ace_ego import ACE_Ego; model = ACE_Ego.from_pretrained("ace-ego-base") loads human-pretrained weights for direct inference.

3. Embodiment config and camera calibration: Register URDF, DOF (6/7-axis), end-effector type via morphology interface. Calibrate RGB-D intrinsics and extrinsics. Official toolkit supports RealSense, Kinect, etc.

4. Inference with reliability: Feed live RGB-D and language (e.g., “put shoes in the box”); model.infer(rgb_image, depth_image, instruction) returns camera-space end-effector actions (pose + gripper). model.set_reliability_threshold(0.8) filters low-confidence steps. IK converts to joint commands—validate in sim before real hardware.

4. Pros and Cons

5. Comparison with Similar Tools

Selection guidance: Retail/logistics with multi-body contact ops: ACE-Ego—SOTA, open, one-brain-multi-body. Budget-rich teams wanting closed π0.5 API: good in-distribution ops but weak cross-body. Humanoid research: GR00T-N1.6 for whole-body, weaker retail. RT-2: early VLA, weak on contact, closed—not for production manipulation.

6. Editor's Take

ACE-Ego breaks ground on embodied VLA. Human-centric pretrain + camera-space actions + morphology encoding solves multi-body generalization. vs π0.5’s massive robot data, 6.0K+ hours human video cuts data cost while beating performance (72.8% vs 37.0%)—strong evidence for human video pretrain. Reliability filtering fills a safety gap in open VLAs.

Fully open Apache 2.0 and one-brain-multi-body fit retail chains and warehouses—one weight set, URDF registration per robot. Stable on bagging and shoe boxing—production-ready retail fulfillment potential.

Clear audience: robot companies, embodied AI labs, retail automation integrators. Academics get full pretrain code and benchmarks; industry gets commercial license and workable inference stack.

Core framework (human video + morphology) extends to home service and assembly as community adds bodies and data. Dual-arm generality and hardware bar still need work.

Innovation 5/5, practical value 5/5, open ecosystem 5/5; −0.25 each for hardware requirements and doc gaps.

7. Use Cases

• Retail fulfillment: Supermarkets/convenience stores—shelving, bagging, shoe boxing from human-like ops learned from video.

• Logistics warehousing: Sorting, packing, palletizing with contact—90.62% bimanual on RoboTwin 2.0 for irregular items; camera-space actions across shelf heights.

• Home service: Tidying, organizing, light cleaning—one-brain-multi-body across wheeled and dual-arm home robots.

• Commercial service: Malls, hotels, restaurants—natural language (“move vase to windowsill”) for staff-free ops.

• Industrial manufacturing: Assembly, tool pick-place, material transfer—reliability module protects long assemblies; few industrial demos finetune per line.

8. FAQ

Q: What hardware is required?
A: A100 or RTX 4090, ≥24GB VRAM, CUDA 11.8+, 16+ CPU cores, 32GB RAM recommended. Real-time control needs <50ms latency—RTX 3060 often insufficient.

Q: Non-retail scenes—home or industrial?
A: Human video includes daily manipulation—reasonable home generalization. Industrial tasks need tens of demo finetunes; official scripts provided.

Q: Where to get human egocentric video?
A: Open release does not include raw video; pretrained weights already encode it. Reproduce pretrain per paper if needed—weights recommended for finetune.

Q: Adapt to my robot?
A: Morphology interface: URDF, DOF, gripper type, cameras. Examples for UR5, Franka Panda, GR1—write URDF and calibrate for new platforms.

Q: Inference speed for real-time control?
A: A100 ~30–50ms/frame; RTX 4090 ~60–80ms. TensorRT/ONNX can help.

Q: vs π0.5 main advantages?
A: One-brain-multi-body (π0.5 per-body finetune), fully open (π0.5 closed), higher retail performance (72.8% vs 37.0%). π0.5 smoother flow actions and larger pretrain for general closed-API use cases.

9. Project Links

• Project site: https://acerobotics-vla.github.io/ACE-Ego/
• GitHub: https://github.com/ACERobotics-VLA/ACE-Ego-0
• Paper: https://acerobotics-vla.github.io/ACE-Ego-0/paper.html