PawBench – Tongyi Lab's General Agent Evaluation Benchmark

1. What Is PawBench

PawBench is a general agent evaluation benchmark from Tongyi Lab for personal assistant and agent scenarios, evaluating base models and runtime frameworks (Harness) together. PawBench v1.0 includes 150 real tasks and 4,050 test units across 9 models × 3 Harnesses, surfacing best model+Harness pairings, helping Harness developers pinpoint issues and validate fixes—a quantifiable, reproducible benchmark for co-evolution of agent systems.

Technical positioning and domain: AI evaluation that goes beyond model-only scores by including Harness in the same framework—first systematic quantification of whole agent system performance, filling the model–framework joint evaluation gap.

Development background: Tongyi Lab with deep NLP, LLM training, and agent building experience. Motivation: fast agent growth makes it hard to separate model vs framework failures—PawBench attributes failures and guides systemic evaluation.

Core value: PawBench resolves attribution when tasks fail—model reasoning vs framework misconfiguration. Cross-eval matrix and five-dimensional tagging quantify Harness impact (up to 6.4-point gaps), pinpointing Skill loading, path awareness, tool config defects, and optimization direction.

Technical characteristics: Docker sandbox isolation for reproducibility and traceability; hybrid scoring with deterministic assertions plus LLM-as-judge; five-dimensional tags for multi-axis drill-down (scenario, capability, complexity, modality, environment).

2. Key Features

• Joint evaluation: Models and Harness in one framework via 9×3 cross matrix—Harness gaps up to 6.4 points comparable to major model upgrades, enabling smaller models to win with better Harness.

• Real task set: 150 tasks, 4,050 units aggregated from six quality agent benchmarks (GAIA, WebArena, etc.) covering office collaboration, software engineering, automation, web search, Skill invocation—practical relevance.

• Five-dimensional tagging: Each item tagged by scenario, atomic capability, complexity (L1–L3), input modality, runtime environment—for sliced analysis of Harness defects per scenario.

• Hybrid scoring: Rule assertions plus LLM-as-judge semantic quality, score 0–1. Rules check files, fields, exit codes; LLM judges semantics—weighted final score balances objectivity and flexibility.

• Multi-slice leaderboards: Overall (150), Text (124), Multimodal (26)—focus text-only or multimodal teams separately.

• Full trajectory retention: Docker runs preserve trajectories, grader artifacts, environment snapshots—layered replay on failures for root cause.

• Deep diagnostic analysis: Slice by model size, modality, task type, skill domain to find Harness bugs in Skill loading, path sensing, default web tool config, etc.

3. How to Use

1. Prerequisites: Docker (Linux recommended; Docker Desktop on macOS/Windows). At least one model API key (Qwen, GPT-4, etc.) and Harness deps (Python 3.8+, Node.js, etc.).

2. Clone: git clone https://github.com/agentscope-ai/PawBench.git, then pip install -r requirements.txt.

3. Configure: Create .env with API keys (e.g., OPENAI_API_KEY=sk-xxx) and Harness env (endpoints, hyperparameters). Supports AgentScope, AutoGPT, etc.

4. Run slices: python run_benchmark.py --slicing text (text), --slicing multimodal (multimodal), --all (full). Docker preserves trajectories automatically.

5. Reports: Per-task rule score, LLM judge score, weighted total. python report.py --output report.html for HTML diagnostics with dimensional slices.

6. Submit results: Upload to https://agentscope-ai.github.io/PawBench/ per instructions to update Overall/Text/Multimodal leaderboards.

Notes: First run with default config (not max API keys) to mirror fresh clone experience. Keep model/Harness versions fixed across runs for comparability. ~500 MB disk per test unit during Docker runs.

4. Pros and Cons

5. Comparison with Similar Tools

Selection advice:

Harness builders and agent integrators should choose PawBench first—joint eval and diagnostics separate model vs framework failures unlike SWE-bench/GAIA. Multi-slice leaderboards help pick model+Harness by task type.

Code generation researchers: SWE-bench's 2000+ GitHub tasks richer for code but no Harness eval. GAIA: high-quality human tasks but small count and weak isolation. AgentBench: strong multi-env sim but simple scoring, no Harness focus. PawBench leads on Harness eval, failure diagnosis, and zero-config defaults.

6. Editor's Take

PawBench innovates by evaluating model+Harness jointly—first quantifying framework contribution to whole-system performance, addressing the key bottleneck of attributing failures. The 9×3 matrix isolates model vs framework via controlled comparison—methodologically novel in AI eval.

Diagnostic depth stands out: five-dimensional slicing quickly finds Skill loading, path, web tool defaults—"diagnose not just rank" makes it a Harness developer tool, not only a leaderboard. Docker isolation and full trajectories ensure reproducibility for research and industry.

Audience: Harness developers, model selection decision-makers, agent system researchers. Zero-config helps casual users; deep use needs technical background.

Future: could become agent eval standard as model–framework co-optimization matters; expanding vertical tasks and Harness support (LangChain, CrewAI) would widen impact.

Rationale: Excellent innovation and utility; − for 150 tasks and three Harnesses limiting vertical/professional coverage. Among the most innovative practical agent eval tools today.

7. Use Cases

• Harness optimization: Slice by atomic capability to fix Skill loading, path parsing, web tool defaults.

• Model selection: Pick best model+Harness for text, multimodal, Skill, or web search via Text/Multimodal boards.

• Regression testing: Re-run relevant slices after fixes to confirm improvement on targeted dimensions.

• Academic research: Reproducible cross-eval for model–framework co-evolution studies.

• Agent integration testing: Pre-deploy end-to-end tests on office, SWE, automation scenarios.

8. FAQ

Q: vs MMLU/HellaSwag?
A: Those test model reasoning only; PawBench evaluates model+Harness jointly and quantifies framework impact on success rate.

Q: Submit results?
A: Run locally, generate report, upload per https://agentscope-ai.github.io/PawBench/ instructions for leaderboard updates.

Q: Extend task set?
A: v1.0 fixed 150 tasks but framework supports custom tasks with five-dimensional tags locally; custom sets cannot join official leaderboard directly.

Q: Balance objectivity and flexibility in scoring?
A: Rules for hard checks; LLM-as-judge for semantics; weighted 0–1 score with tunable weights per task type.

Q: GPU needed?
A: No for the framework—inference via model APIs in cloud. Local needs Harness + Docker (~8 GB RAM, 50 GB disk).

Q: Supported Harnesses?
A: v1.0: AgentScope, AutoGPT, BabyAGI; LangChain Agent and CrewAI planned.

Q: What is "zero-config eval"?
A: Default post-clone setup—one API key, no max-key or heavy hyperparameter tuning required.

9. Project Links

• Official site: https://agentscope-ai.github.io/PawBench/
• GitHub: https://github.com/agentscope-ai/PawBench