PlanningBench – Open LLM Planning Evaluation Framework by Tencent Hunyuan and Partners

1. What Is PlanningBench

PlanningBench is an open framework from Tencent Hunyuan with Renmin University of China Gaoling School of Artificial Intelligence and partners, focused on evaluating and training large language model planning ability. Built from real scenarios, it provides a verifiable data generation system covering six categories and 30+ planning task types. Through constraint-driven closed-loop synthesis, model plans move from "looks reasonable" to "actually executable." Unlike static benchmarks or human scoring, PlanningBench uses structured verification checklists and dual metrics (Avg-pass and All-pass) to judge global executability and supply verifiable samples for reinforcement learning. It is not only an evaluation tool but a dynamic data generator that actively finds model capability boundaries—infrastructure for reliable complex constrained reasoning.

Technical positioning and domain: NLP and LLM evaluation focused specifically on planning—static planning problem solving with constraint reasoning, resource allocation, and state tracking—not multi-turn environment interaction like general agent benchmarks.

Development background: Led by Tencent Hunyuan with Renmin University Gaoling AI Institute. Motivation: benchmarks like AgentBench and GSM8K under-cover planning, lack verifiable difficulty-controlled trainable data, and weaken reliable signals for improving planning.

Core value: PlanningBench solves three issues: credible evaluation via automatic checklists vs subjective human scores; controllable difficulty via three constraint tiers (basic, medium, hard) affecting reasoning depth—not prompt length; effective training signals—~300 verifiable samples via GRPO improve external planning benchmarks and general instruction following, linking evaluation to training gain.

Technical characteristics: Generator-Responder-Critic closed-loop synthesis auto-generates and verifies planning problems and adjusts difficulty from model performance—a self-improving system finding capability boundaries. Dual metrics (Avg-pass, All-pass) catch "locally correct but globally inexecutable" outputs for finer diagnosis.

2. Key Features

• Real-scenario coverage: Six categories, 30+ tasks—scheduling, resource allocation, workforce rostering, routing, production ops, emergency services—abstracted from commuting, manufacturing, medical dispatch, etc., reducing overfitting to fixed item banks and improving real-world relevance.

• Three-tier constraint system: Basic (time windows, resource bounds), medium (multi-objective optimization, dependencies), hard (exception recovery, goal conflicts). Difficulty is controlled by constraint sampling distribution so complexity grows in planning depth, not text length or step count alone.

• Automatic verification: Each instance has a structured executable checklist with Avg-pass (partial constraint satisfaction) and All-pass (global executability). Dual metrics expose pseudo-success where local constraints pass but the full plan fails.

• Closed-loop data generation: Generator creates self-contained problems and checklists from task + constraint samples; Responder proposes plans; Critic validates against the checklist. Pass raises difficulty; fail retains hard samples—dynamic boundary seeking.

• Training signal output: Verifiable samples feed RL with stable planning optimization signals. ~300 samples via GRPO improve external planning and instruction tasks—seamless eval-to-train pipeline.

• Adaptive difficulty enhancement: When models pass easily, the loop increases constraint coupling and resource conflicts for harder samples, avoiding static dataset ceilings and pushing complex planning capability.

3. How to Use

1. Environment: Python 3.8+, 16 GB RAM recommended. CPU works; GPU (V100/A100) speeds model inference. Dependencies: transformers, torch, datasets, etc.

2. Get resources: Clone https://github.com/Tencent-Hunyuan/PlanningBench or download from HuggingFace https://huggingface.co/datasets/tencent/PlanningBench.

3. Run evaluation: Load the benchmark, generate plans with target model. Unified API computes Avg-pass and All-pass:

   python evaluate.py --model_name your_model --dataset_path ./data/planning_bench
   

   JSON output includes per task-family scores.

4. Generate custom data: Edit config.yaml with task_family (e.g., scheduling), difficulty_level (easy/medium/hard), num_samples, then:

   python generate_data.py --config config.yaml
   

   Samples include problem text, checklist, optional reference answers.

5. Train models: Use verifiable samples as RL rewards; see train_grpo.py. ~300 high-quality samples can show significant gains across planning tasks.

Notes: Test small batches with defaults before large generation. RL needs reward design—use built-in checklist validator as reward source.

4. Pros and Cons

5. Comparison with Similar Tools

Selection advice: For LLM planning (constraints, allocation, state tracking) with verifiable difficulty-controlled eval and train data, PlanningBench is best—closed-loop generation and dual metrics suit research diagnosis and iteration. For multi-turn real env interaction (OS, browser), AgentBench fits but with interaction randomness and no train data generation. GAIA suits multi-step reasoning and tools but relies more on human scoring and scales less.

6. Editor's Take

PlanningBench innovates by linking evaluation and training through verifiable checklists and closed-loop generation, fixing "test without train" in traditional benchmarks. Three-tier constraints quantify difficulty better than adding steps or length—which adds noise. Controlling coupling and conflict targets core planning reasoning—a model worth emulating. Avg-pass high with All-pass low signals local constraint skill without global coordination—actionable diagnosis.

Industry potential: scheduling, medical dispatch, etc., where planning quality drives decisions. Custom benchmark data helps enterprises assess models on real constraints and iterate with RL. ~300-sample effectiveness keeps cost low for adoption.

Audience: AI researchers (eval + RL), algorithm engineers improving planning, enterprise developers assessing scheduling reliability.

Future: standard benchmark if task families expand (multi-agent planning, dynamic replanning) and custom template barriers drop. Combining checklists with formal provers (Lean, Coq) could raise rigor.

Rationale: Strong innovation (closed-loop, dual metrics, train transfer) and high utility; − for static planning limits and custom extension cost.

7. Use Cases

• Capability diagnosis: Identify weak constraint reasoning, allocation, state tracking via Avg-pass vs All-pass gaps.

• Agent training: GRPO on verifiable samples for reliable scheduling, logistics, production planning.

• Academic research: Reproducible, extensible planning benchmarks for search-based or neuro-symbolic methods.

• Enterprise scheduling: Custom eval data for nurse rostering, production, logistics under real constraints with closed-loop iteration.

• Education grouping: Student grouping and resource distribution tasks to test fairness and optimality under hard/soft constraints.

8. FAQ

Q: PlanningBench vs AgentBench?
A: PlanningBench = static planning with checklist verification and train data generation. AgentBench = multi-turn simulated environments. PlanningBench supports verifiable dual metrics and native training data; AgentBench is eval-focused.

Q: Hardware requirements?
A: CPU OK; GPU recommended for inference speed. 7B+ models for generation: ~16 GB VRAM. Framework itself is lightweight; cost is mostly the target model.

Q: Ensure training data quality?
A: Generator-Responder-Critic keeps samples that pass Critic checklist validation. Manual spot checks or extra Critic rounds improve quality.

Q: Dynamic real-time scheduling?
A: Current version is static one-shot planning. Dynamic replanning from live feedback is out of scope; may come in future versions.

Q: Add new planning tasks?
A: Define task structure, constraint templates, and checklists; extend via interfaces in task_templates. Read "Custom tasks" in docs and engage community.

Q: GRPO training tips?
A: Use checklist validator as reward. Start ~100 samples to validate pipeline, then scale. Tune LR/batch for model size; defaults provided.

9. Project Links

• GitHub: https://github.com/Tencent-Hunyuan/PlanningBench
• HuggingFace dataset: https://huggingface.co/datasets/tencent/PlanningBench
• arXiv paper: https://arxiv.org/pdf/2605.20873