Chronicles-OCR – Cross-Temporal Visual Perception Benchmark for Chinese Script Evolution

1. What Is Chronicles-OCR

Chronicles-OCR is the first industry benchmark to cover the full evolutionary trajectory of Chinese script "seven-style transformation" (七体之变)—jointly released by Tencent Hunyuan, the Institute of Information Engineering (Chinese Academy of Sciences), Anyang Normal University, Nankai University, and the Palace Museum. The suite contains 2,800 high-quality images spanning oracle bone, bronze, seal, clerical, regular, running, and cursive scripts. Through stage-adaptive annotation and four evaluation tasks, it systematically measures how well vision–language models (VLLMs) perceive and generalize across millennia of character shape change.

Technical positioning and domain: Chronicles-OCR is a computer-vision × NLP evaluation benchmark focused on cross-temporal perception of VLLMs in historical document analysis. Unlike benchmarks limited to modern scans or a single ancient script, it extends evaluation across the entire Han character evolution timeline—filling a systematic gap in temporal robustness testing.

R&D background: Built by Tencent Hunyuan with CAS-IEE, Anyang Normal University's Oracle Bone Inscriptions Lab, Nankai, and the Palace Museum. Motivation: today's VLLMs often fail catastrophically on oracle and bronze scripts, yet no standardized cross-era benchmark existed to diagnose whether failures come from localization, decipherment, or layout understanding.

Core value: Chronicles-OCR offers a standardized, reproducible framework for temporal robustness of multimodal models. It exposes where frontier models break on fine-grained spatial grounding and semantic decoding of ancient glyphs, while stage-adaptive labels avoid forcing one annotation schema across incompatible historical stages.

Technical characteristics: Stage-adaptive labeling—character-level boxes plus modern-character mappings for early scripts; sequence-level layout and reading order for mature scripts. A visual referring mechanism (colored boxes highlighting targets) decouples decipherment from spatial search, pinpointing whether a model cannot see the glyph or cannot read it.

2. Key Features

• Seven-script full coverage: Oracle, bronze, seal, clerical, regular, running, and cursive—400 images each—from Shang dynasty materials through near-modern handwriting, ensuring every major morphological phase appears in evaluation—not just Song-regular prints.

• Stage-adaptive annotation: Early scripts get per-glyph bounding boxes with modern Han mappings; mature scripts get line/sequence layout with reading order. One-size-fits-all OCR boxes would erase stroke topology information critical for paleography metrics.

• Four core tasks: Cross-period character grounding, fine-grained ancient character recognition, ancient text parsing, and script classification—together forming a capability matrix instead of a single accuracy number that hides grounding failures.

• Visual referring mechanism: Color boxes mark the target glyph so the model only outputs the modern reading—localization is scored separately. This prevents models from gaming aggregate OCR scores via lucky full-page guesses.

• Expert multi-pass labeling: Paleography PhDs and graduate annotators cross-check script tags, boxes, and transcripts under academic review—reducing label noise that would otherwise dominate small-set variance.

• Authoritative provenance: Oracle data from Anyang's dedicated lab; bronze/seal from philology teams; regular/running/cursive from Palace Museum handwriting recognition test sets—each subset reflects real institutional collections, not synthetic font renders alone.

• Robustness diagnostics: Compare model curves across eras to see where modern-document pretraining fails to transfer—guiding data collection and architectural choices (e.g., higher-resolution crops for seal script).

3. How to Use

1. Environment requirements: GPU workstation recommended—NVIDIA A100 32 GB+ VRAM, Linux (Ubuntu 20.04+), Python 3.8+, PyTorch. CPU-only smoke tests are possible but impractical at VLLM scale.

2. Clone the repository: Visit https://github.com/VirtualLUOUCAS/Chronicles-OCR for data, paper, and eval scripts. Read README first for directory layout, license, and citation format.

3. Download the dataset: Fetch all 2,800 images (400 per script) plus JSON/XML annotations with script class, boxes, and modern mappings. Verify checksums if provided to avoid partial mirrors.

4. Understand label schemas: Early-stage samples = {bbox, modern_char} per glyph; mature-stage samples = sequence layout + reading order + transcript. Mixing schemas when writing dataloaders silently tanks metrics.

5. Pick evaluation tasks: Start with one task—e.g., visual referring recognition—before running the full quad. Each task defines prompt templates and metric scripts in the repo.

6. Prepare candidate VLLMs: Deploy models with image→text capability (GPT-4V class, Gemini Pro Vision, Qwen-VL, InternVL, etc.). Pin model weights and decoding params for reproducibility.

7. Run inference: Feed images per task spec; store raw outputs (boxes as normalized coords or JSON, transcripts as UTF-8 strings). Log failures/timeouts—ancient scripts can trigger long CoT on some LMs.

8. Execute official eval: Use bundled scripts to compute IoU for grounding, NED for transcription, accuracy for script ID, etc. Compare against published baselines in the paper appendix.

4. Pros and Cons

5. Comparison with Similar Tools

Selection guidance: Studying script evolution & VLLM temporal robustness → Chronicles-OCR. End-to-end ancient book digitization with QA/translation → AncientDoc's larger page corpus and task breadth. Structural/zero-shot character analysis → ACCID radical annotations.

6. Editor's Take

Chronicles-OCR is the benchmark paleography-aware multimodal evaluation needed but rarely built: it respects that oracle boxes ≠ Tang manuscript lines, and it refuses to let models hide weak grounding behind chain-of-thought bluffing. The Palace Museum + Anyang provenance gives institutional credibility rare in synthetic ancient OCR sets.

Practitioners should not expect SOTA ImageNet gains from fine-tuning on 400 oracle images—value is diagnostic. Teams shipping cultural-heritage OCR should treat sub-70 IoU on bronze as a product risk flag, not a leaderboard vanity metric.

Audience: VLLM researchers, digital humanities engineers, museum digitization vendors, and NLP groups probing long-tail visual generalization.

Outlook: Extensions to other extinct scripts and synthetic hard-negative augmentation could make this the "ImageNet of temporal OCR"—if the consortium keeps curating expert labels.

— Indispensable for Han evolution studies; size and task weight keep it from a perfect five for general OCR teams.

7. Application Scenarios

• Ancient character recognition R&D: Train/eval models assisting epigraphers on partially deciphered oracle/bronze forms—human-in-the-loop verification, not autonomous publication claims.

• Classics digitization: High-precision OCR on clerical/regular/running/cursive rubbings and manuscripts for searchable archives linked to catalog metadata.

• Script evolution research: Quantify how accuracy cliffs align with historical reforms (clerical regularization, running simplification)—supporting linguistics papers with reproducible VLLM curves.

• Model shopping for heritage vendors: Standardized PinchBench-style comparisons between Qwen-VL, GPT-4V, Gemini before procurement for museum SaaS.

• Artifact text extraction: Bronze inscriptions, bamboo slips, silk manuscripts—bootstrap transliteration pipelines for archaeology field reports.

• Public education: Interactive demos showing model failures on oracle vs. regular script—museum exhibits explaining AI limits in cultural AI literacy programs.

8. FAQ

Q: Can I fine-tune on Chronicles-OCR alone?

A: Possible for ablations, but 400 images/script is small—risk severe overfit. Use as held-out eval; combine with larger general ancient corpora (AncientDoc pages, etc.) for training if you need production-grade decoders.

Q: What do annotation files look like?

A: Early scripts: per-glyph pixel bboxes + Unicode modern mapping. Mature scripts: ordered polylines or line boxes + full transcript. Formats are JSON/XML as documented in the repo—do not assume COCO-only layouts.

Q: Which models should I benchmark first?

A: Any VLLM accepting image prompts: GPT-4V, Gemini Pro Vision, Qwen-VL-Max, InternVL2, LLaVA-NeXT variants. Report model version, prompt template, and temperature—ancient OCR is prompt-sensitive.

Q: How is this different from AncientDoc?

A: Chronicles-OCR = temporal morphology of Han characters across seven scripts. AncientDoc = document-level tasks (OCR + translation + QA) on printed/copied books. Complementary, not redundant.

Q: How to read a bad oracle score?

A: Low IoU with decent referring accuracy → localization/preprocessing issue. Low accuracy even with colored boxes → decipherment/lexicon gap. Compare both before blaming "the VLLM is bad at Chinese."

Q: Custom tasks allowed?

A: Yes—repo is open. Fork eval harness, add tasks (e.g., radical prediction), reuse annotation parsers, cite arXiv:2605.11960v1.

Q: Licensing and citation?

A: Follow LICENSE in GitHub; academic use requires citing arXiv:2605.11960v1. Commercial redistribution of Palace Museum subsets may carry additional institutional terms—read README carefully.

9. Project Links

• GitHub repository: https://github.com/VirtualLUOUCAS/Chronicles-OCR