CAPEval

A Decoupled Caption Evaluation across understanding and generation

Zhipeng Liu1* · Haochen Wang1,2*† · Zhaoxiang Zhang1,2†

1University of Chinese Academy of Sciences · 2Institute of Automation, Chinese Academy of Sciences

* Equal contribution  † Corresponding authors

Abstract Code Dataset Leaderboard Paper (soon)

Abstract

Captions serve as a primary supervision signal for both multimodal understanding and text-to-image generation. Previous evaluations treat caption quality as a single scalar, conflating two distinct properties: how much visual information a caption covers, and how reliably the image supports its stated claims. We introduce CAPEval (Coverage And Precision Evaluation), a decoupled caption evaluation benchmark with human-written ground-truth captions and human-verified atomic checklist items. CAPEval reports Coverage (C) — how thoroughly a caption covers ground-truth factual content — and Precision (P) — the factual correctness rate of claims expressed in the caption. Controlled downstream experiments with ten captioners show a consistent task-dependent dissociation: Coverage is the stronger correlate for understanding, while Precision dominates generation.

Overview

CAPEval builds dense atomic checklists from high-resolution images and long-form GT captions, scores each captioner with a checklist judge to obtain C and P, then runs controlled VLM / T2I training where the caption source is the only variable.

Figure 1: Overview of CAPEval benchmark, scoring, and downstream protocol
Figure 1. CAPEval constructs fine-grained checklists, scores captions into Coverage and Precision, and links caption quality to downstream understanding (U) and generation (G).

Metrics

Coverage C

How much of the checklist information is covered by the caption (yes or no vs. not_mentioned).

Precision P

Among checklist items the caption mentions, how often the claim is factually correct (yes vs. no).

Public scores are reported on a 0–100 percent scale. See the GitHub README for the exact formulas and evaluation code.

Key findings

  1. Caption quality profile matters more than model scale. Within InternVL3.5, smaller captioners can outperform larger ones when C / P better match the downstream objective.
  2. Understanding is dominated by Coverage. Broader semantic coverage is the consistent predictor of VLM understanding performance.
  3. Generation is driven by Precision. Factual precision is the significant predictor of T2I generation quality across captioners.
Figure 2: Coverage drives understanding; Precision drives generation
Figure 2. Task-dependent asymmetry: better Coverage → understanding; better Precision → generation.

Benchmark

Download the dataset on Hugging Face or clone the code repository. Public captioner C / P rankings: Leaderboard.

Citation

@misc{capeval2026,
  title        = {CAPEval: A Decoupled Caption Evaluation across Understanding and Generation},
  author       = {Zhipeng Liu and Haochen Wang and Zhaoxiang Zhang},
  year         = {2026},
  howpublished = {\url{https://github.com/liuzhipenggg/CAPEval}},
  note         = {Paper / arXiv TBD},
}