General365 – Meituan LongCat Team's Open General Reasoning Benchmark

1. What Is General365

General365 is an open general reasoning benchmark from Meituan's LongCat team, designed to evaluate large language models (LLMs) purely on logical reasoning in everyday scenarios. The benchmark includes 365 high-difficulty, human-authored seed questions and 1,095 extended variants, covering eight core reasoning challenge dimensions. Its key innovation is restricting required background knowledge to K–12 level, decoupling reasoning ability from domain expertise so models cannot mask weak reasoning with broad knowledge. In tests across 26 mainstream models, the best performer—Gemini 2.5 Pro—reached only 62.8% accuracy; most models failed to break the 60% bar, revealing significant gaps in general reasoning even among top systems.

Image source: Official article

Technical positioning and domain: General365 sits in NLP model evaluation and benchmarking. Unlike GSM8K or MMLU, which stress knowledge breadth or subject mastery, it measures generalized logical reasoning on complex, varied everyday problems—filling a gap in "general reasoning" evaluation.

Development background: Built by Meituan's LongCat team after observing strong benchmark scores but poor performance on unstructured logic in production—the classic "high score, low capability" problem. General365 was created as a stress test for true logical reasoning and to push the field from memorization toward reasoning.

Core value: Existing benchmarks like BBH and MMLU are saturating and struggle to separate top models on reasoning. General365 uses hard, diverse items to give developers a clear "health check" on reasoning, help enterprises choose models wisely, and offer a standardized framework for academic work on general reasoning.

Technical characteristics: t-SNE semantic distribution checks and reasoning-path similarity analysis keep items logically independent and reduce template memorization. A hybrid scoring system (rule-based + GPT-4.1 model scoring) with human verification reaches 99.6% scoring accuracy, balancing objectivity and flexibility on open-ended answers.

2. Key Features

• Eight-dimensional reasoning evaluation: General logic is split into complex constraints, branch enumeration, spatiotemporal reasoning, recursive backtracking, semantic distraction, implicit information, optimal strategy, and probabilistic uncertainty. Each item maps to at least one dimension; nearly 70% span two or more, enabling precise diagnosis of weak spots.

• Knowledge decoupling for pure reasoning: All items require only K–12 background knowledge, removing bias from deep domain corpora (medicine, law, etc.). Results reflect logical reasoning rather than retrieval or memorization.

• Hybrid scoring system: Numeric items with clear answers use math-verify for exact parsing; multiple-choice and open text use GPT-4.1 for semantic judgment. Human validation reports 99.6% accuracy for fair, reliable scores.

• Semi-public test set and contamination control: 180 seed questions and variants (720 items) are public for research; 185 seed sets and variants remain hidden to block training-set contamination and memorization, preserving long-term benchmark validity.

• Standardized multi-model comparison: Unified evaluation code supports reproducible head-to-head runs on OpenAI, Gemini, Claude, DeepSeek, Qwen, GLM, Kimi, LongCat, and 26+ models, with standardized reporting for cross-model comparison.

• Reasoning efficiency and token analysis: Tracks accuracy vs. output tokens to spot verbose, inefficient chains and optimize reasoning paths for better cost-performance tradeoffs.

3. How to Use

1. Environment and prerequisites:

   • Hardware: NVIDIA GPU (V100, A100, H100, etc.) with ≥16GB VRAM recommended. CPU mode works but is much slower.
   • Software: Python 3.8+, CUDA/cuDNN. Use conda for an isolated environment.
   • Models: Provide your own models. Commercial APIs need valid keys; open models need local weights.

2. Clone and install dependencies:

   git clone https://github.com/meituan-longcat/General365.git
   cd General365
   pip install -r requirements.txt
   

   Installs transformers, torch, openai, and other dependencies.

3. Configure evaluation:
   Create or edit config.yaml in the project root:

   • model_name: e.g. gpt-4o, gemini-2.5-pro
   • api_key: provider API key
   • model_type: api or local
   • data_path: dataset path (default: public 180 seeds)
   • output_dir: results directory

4. Run evaluation:

   python run_evaluation.py --config config.yaml
   

   Loads data, prompts the model per item, and collects answers with live progress in the console.

5. Review results:
   JSON/CSV outputs include per-item answers, reasoning traces, dimension scores, and totals. Analysis scripts can build radar charts to visualize strengths and weaknesses.

4. Pros and Cons

5. Comparison with Similar Tools

Selection guidance:

• Model R&D teams: Choose General365 to pinpoint logic gaps (spatiotemporal reasoning, semantic distraction, etc.) with finer granularity than BBH or MMLU-Pro. Use it as a key iteration benchmark.
• Enterprises selecting models: For contract analysis, routing, or planning-style logic, General365 beats knowledge-heavy benchmarks. High discrimination and contamination controls help pick models that are genuinely strong reasoners—not just good at trivia.

6. Editor's Take

General365 is an important correction to today's LLM evaluation stack. MMLU and BBH drove progress but also incentivized leaderboard chasing, widening the gap between benchmark gains and real-world capability. By decoupling knowledge and stressing hard general reasoning, it hits a real pain point.

Technically, the eight-dimension breakdown and hybrid scoring stand out: structured diagnosis plus flexible open-answer grading. Hidden holdouts are not new globally, but they matter especially for reasoning evals where contamination kills usefulness.

In practice, General365 is a high-quality mirror: even the best models fall short on everyday logic. That is both a warning and a roadmap—the next leap is not "remember more" but "think more flexibly."

Best for: AI researchers, LLM engineers, enterprise model buyers, and anyone tracking frontier capability honestly.

Future outlook: As models improve, difficulty may need refreshing; extending the framework to multimodal reasoning is a natural next step. Overall, a well-designed, sharply positioned benchmark with real industry value.

Rationale: Addresses a core eval gap with clear design, rigorous implementation, open tooling, and highly informative, discriminative scores. Essential for teams serious about real reasoning—not just benchmark scores.

7. Use Cases

• Model diagnosis and optimization: Run a "full checkup" across eight dimensions; low scores on recursive backtracking, for example, guide targeted data or chain-of-thought improvements for planning tasks.

• Enterprise model selection: Use General365 when picking models for support, analytics, or decision workflows—especially dimensions like optimal strategy and implicit information—to avoid being misled by trivia-heavy leaderboard wins.

• Academic research: Standardized platform to study architectures (MoE, Transformer variants) and training (RL, process supervision) on reasoning—not just knowledge.

• Efficiency and cost analysis: Compare tokens needed for equal accuracy across models or configs to balance API spend and latency.

8. FAQ

Q: How is General365 fundamentally different from MMLU, BBH, and similar benchmarks?

A: The focus differs. MMLU stresses knowledge breadth; BBH composite tasks; General365 caps knowledge at K–12 to measure everyday logical reasoning, decoupling "how smart" from "how much you know."

Q: A low General365 score—does that mean my model is "dumb"?

A: Not necessarily overall, but it exposes weak general reasoning. A model strong at code or creative writing may fail semantic distraction or spatiotemporal items—signaling limits on flexible logic in messy, unstructured daily problems.

Q: How is fairness preserved and "training on the test" prevented?

A: Only 180 seed sets (720 variants) are public; 185 seed sets and variants stay hidden permanently. Logical independence also blocks template memorization. Together, this reduces contamination-driven inflation.

Q: What does evaluation cost?

A: Depends on the model. Premium APIs on hard, long-chain items can consume millions of tokens. Local open models mainly cost compute. The public 720-item subset offers a faster, cheaper sanity check.

Q: How should I read the report?

A: Total score reflects overall general reasoning; dimension scores matter more for action. Low optimal-strategy scores imply weak planning under tradeoffs; low semantic-distraction scores imply susceptibility to irrelevant text—each points to concrete improvement targets.

9. Project Links

• Project site: https://general365.github.io/
• GitHub: https://github.com/meituan-longcat/General365
• HuggingFace dataset: https://huggingface.co/datasets/meituan-longcat/General365_Public
• arXiv paper: https://arxiv.org/pdf/2604.11778