VitaBench 2.0 – Meituan LongCat’s Long-Horizon Dynamic Agent Benchmark

1. What Is VitaBench 2.0

VitaBench 2.0 from Meituan’s LongCat team is the first benchmark for long-horizon dynamic user modeling in real-life agent scenarios. It ships 56 statistically grounded synthetic users, 819 lifecycle-spanning tasks, 2,000+ evolving preferences, and 66 executable tools—average interaction span 1,580 days (max 2,974)—to measure personalization, proactive service, and adaptation over years—not single-shot accuracy.

Image source: Official article

Technical positioning and domain: AI evaluation for longitudinal user modeling. Unlike SWE-bench or AgentBench one-offs, time is a first-class axis: models must track drifting tastes across multi-year event streams.

Development background: LongCat’s lifestyle-AI experience exposed a gap—no standard way to score “ remembers you five years later.” VitaBench 2.0 fills that for assistants, concierge bots, and local services.

Core value: Quantifies long-term personalization—memory, preference inference, proactive clarification—pushing models from Q&A to companionship.

Technical characteristics: 3D decomposition (profile, prefs, history, tasks) on a strict timeline; memory arena pitting Agentic Memory vs RAG Memory on identical trajectories for fair architecture comparisons.

2. Key Features

• Realistic user trajectories: 56 users across delivery, in-store, travel, etc., with demographics mirroring real-world stats.

• Dynamic preference evolution: 2,000+ preference shifts (~48+ per user)—taste, spend, life-stage changes models must track.

• Complex task battery: 819 executable tasks tied to long-term prefs—decisions, retrieval, recommendations.

• Dual memory showdown: Agentic vs RAG vs full-context baselines on the same data/tasks with standardized hooks.

• Proactive tasks: Scenarios requiring clarifying questions when info is incomplete—mirrors real service dialogs.

• Noise injection: ~20% irrelevant/exploratory interactions—tests signal extraction under realistic clutter.

3. How to Use

1. Environment: Python 3.8+, ~5 GB dataset, 16 GB RAM recommended; Linux/macOS preferred.

2. Clone framework:

   git clone https://github.com/meituan-longcat/vitabench-2.0.git
   

3. Download data:

   from datasets import load_dataset
   dataset = load_dataset("meituan-longcat/VitaBench-2.0")
   

4. Install deps:

   cd vitabench-2.0
   pip install -r requirements.txt
   

5. Pick memory mode: Full context, Agentic Memory, or RAG Memory baseline.

6. Plug in your agent/model:

   python run_benchmark.py --model your_model_name --memory_mode agentic
   

7. Analyze reports: Review Avg@4, Pass@4, temporal decay curves for long-horizon weakness.

4. Pros and Cons

5. Comparison with Similar Tools

Selection guidance: Long-term personalization research → VitaBench 2.0. Code automation → SWE-bench. Broad agent scenarios → AgentBench. Tool-use depth → ToolBench.

6. Editor's Take

Introducing time and preference drift to agent eval is a milestone—SWE/AgentBench can’t tell if your assistant still knows you after a lifestyle change.

Directly actionable for personal AI, concierge, and recommendation teams; the memory arena is especially valuable for architecture choices.

Audience: Researchers, LLM vendors, agent engineers, lifestyle platform algo teams.

Future: More industries, multimodal traces, richer social graphs.

— Foundational bench despite hardware/learning curve costs.

7. Use Cases

• Personal assistant eval: Measure memory, habit tracking, proactive help.

• Customer support tuning: Test preference recall under evolving users.

• Recommendation QA: Realistic long-run behavior for food/travel/hotel stacks.

• Memory R&D: Compare Agentic vs RAG head-to-head.

• Long-context stress tests: Years of noisy events probe context limits.

8. FAQ

Q: vs VitaBench 1.0?
A: 2.0 scales users/tasks/time, adds dynamic prefs and noise separation vs smaller 1.0 scope.

Q: Integrate my model?
A: Implement the documented Agent API—ingest timed events, maintain memory, call tools, emit responses; samples in repo.

Q: Avg@4 and Pass@4?
A: Average success over 4 attempts vs at-least-once success in 4 tries—stability vs peak capability.

Q: Chinese models?
A: Dataset is Chinese lifestyle-centric; any model handling Chinese works; English-only models need adaptation.

Q: Runtime/hardware?
A: Hours in full-context mode; 16 GB+ RAM; GPU (A100/V100 class) recommended for local LLM inference.

Q: Privacy?
A: Fully synthetic users—no real PII; compliance-reviewed release.

Q: Can I run a subset of users or tasks for faster iteration?
A: Yes—the harness supports filtering by user ID, task category, or date range so you can smoke-test a new memory module on a handful of trajectories before launching the full 819-task sweep. Check run_benchmark.py flags and the README for --user_subset and --max_tasks options; partial runs still emit comparable Avg@4 metrics on the selected slice.

9. Project Links

• Site: https://vitabench2.github.io/
• GitHub: https://github.com/meituan-longcat/vitabench-2.0
• Hugging Face dataset: https://huggingface.co/datasets/meituan-longcat/VitaBench-2.0
• Paper: https://arxiv.org/pdf/2605.27141