CMMLU

The official website provides...

4.0| Editor Rating

Editor Review

CMMLU is a multi-task language understanding benchmark focused on the Chinese context, covering 67 topics including natural sciences, humanities, social sciences, and China-specific content. Its testing methods include five-shot and zero-shot evaluations, allowing for a comprehensive assessment of a model's reasoning and knowledge capabilities. However, the official website does not specify whether the complete dataset or usage documentation is provided, which may pose a barrier for non-technical users. Additionally, some model testing data is not publicly available, limiting its comprehensiveness. Overall, CMMLU is a practical Chinese evaluation tool suitable for research and comparing the performance of Chinese language models.

AI Tools Navigator Editorial TeamUpdated: 2026-08-19

What is CMMLU

CMMLU is a fully localized Chinese benchmark designed to comprehensively evaluate the knowledge and reasoning capabilities of language models in the Chinese context. It covers 67 topics, including natural sciences, humanities, social sciences, and China-specific subjects such as Chinese driving rules. Each task is designed to assess the model's performance across different domains, such as STEM requiring computational and logical reasoning, and humanities and social sciences requiring broad knowledge. Additionally, CMMLU focuses on content unique to China, which may not be common in other languages or regions. The benchmark uses five-shot and zero-shot testing methods to evaluate model performance under different conditions. The official website provides performance data for some models, but information on models not yet open for testing is not publicly available. The test data and results can be used for research and comparison of different language models.

More llm-benchmarks tools →

Basic Info

Category:

Best For

DevelopersResearchers

Difficulty: Intermediate

CMMLU Key Features

Covers 67 Topics
CMMLU includes 67 topics across various domains, ranging from basic subjects to advanced professional content, comprehensively evaluating the knowledge and reasoning capabilities of language models in the Chinese context. These topics cover natural sciences, humanities, social sciences, and some China-specific content requiring common sense, such as Chinese driving rules. This broad topic coverage allows CMMLU to effectively assess model performance across multiple domains.
China-Specific Content
CMMLU contains a large amount of content specific to China, which may not be applicable in other languages or regions. For example, Chinese driving rules, Chinese history, and Chinese law, among others, require models to demonstrate a higher level of understanding of the Chinese context.
Five-shot and Zero-shot Testing
CMMLU provides two testing methods: five-shot and zero-shot. Five-shot testing evaluates the model's performance with a small number of examples as guidance, while zero-shot testing assesses the model's zero-shot reasoning ability without any examples. These two methods allow for a more comprehensive evaluation of the model's performance under different conditions.
Partial Model Testing Data Available
The official website lists performance data for some models under five-shot and zero-shot testing, including scores in STEM, humanities, social sciences, other subjects, and China-specific topics. This data can be used to compare the performance of different models, but data for models not yet open for testing is not publicly available.

CMMLU Key Advantages

Covers 67 topics, providing a broad range of evaluation areas.
Includes a large amount of China-specific content, suitable for Chinese context assessments.
Offers diverse testing methods, including five-shot and zero-shot testing.

CMMLU Use Cases

Language Model Performance Evaluation
CMMLU can be used to evaluate the knowledge and reasoning capabilities of different language models in the Chinese context. By testing the model's performance across 67 topics, it is possible to determine its adaptability and accuracy in various domains.
Multi-Task Learning Research
CMMLU's multi-task structure makes it a useful tool for multi-task learning research. Researchers can use this benchmark to test the model's performance across multiple tasks, optimizing training strategies and task allocation.
Model Comparison in the Chinese Context
CMMLU focuses on the Chinese context, making it suitable for comparing the performance of different models on Chinese tasks. This comparison helps identify models that perform better in the Chinese environment.

Frequently Asked Questions

Does CMMLU support classical Chinese evaluation?▼

The official website does not specify whether CMMLU supports classical Chinese evaluation, but it mentions that if there is a need for classical Chinese evaluation, ACLUE can be used. Therefore, CMMLU primarily focuses on modern Chinese contexts, and classical Chinese evaluation may not be within its scope.

How can I obtain the CMMLU test data?▼

The test data and tasks for CMMLU can be obtained through its GitHub repository. Users need to visit the project page and download the relevant files, including the dataset, scripts, and test results. However, the official website does not specify whether the complete dataset is provided or if an application is required.

Is CMMLU applicable to all language models?▼

CMMLU primarily evaluates Chinese language models, but some models such as Llama-3.1- -Instruct and ChatGPT are also mentioned in the tests. However, the official website does not specify whether all language models are applicable to this benchmark, and it may require filtering based on the model's Chinese capabilities.