Clinical AI Benchmark Atlas (CABA)

CABA is a registry of the public benchmarks used to evaluate large language models in health care: the 58 benchmarks named by 13 evaluation suites and by frontier model reports, checked against five criteria on their downloaded items. Up to 200 items from each carry RCQ-Map labels, and every benchmark is compared with the task mix of real clinical queries that clinicians sent to an institutional AI assistant. Version 1.0 is the version described in the paper.

Benchmark items are not included. Each benchmark keeps its own license, and some ask that their items not be posted online. CABA releases item ids and labels; the scripts in the GitHub repository rebuild the items from each benchmark's own release.

Files

Config	Rows	Contents
registry	58	One row per benchmark: evaluation suites, frontier model reports, items sampled and labeled, split, version, license, source, rendering rule
per_benchmark	58	Task-category and kind-of-work shares per benchmark, and overlap with the real clinical task mix
labels	10,659	One row per benchmark item: item_id, benchmark and the 24 RCQ-Map fields. No item text

rcq_aggregate.json holds the aggregate shares of the 127,625 real clinical queries used for comparison.

from datasets import load_dataset
labels = load_dataset("NYU-OLAB/CABA", "labels", split="train")

How the labels were made

Up to 200 items per benchmark were sampled, rendered as a model receives them, and labeled with RCQ-Map by the paper's annotator (GPT-5.6 sol, reasoning effort none, the paper's guidelines, strict JSON schema). The 40 sampled Med-HALT items that failed the schema's consistency rules are listed in the GitHub repository. The protocol, the registry build stages and the comparison script are there as well, and scripts/compare.py reproduces the paper's medians from these labels.

License and citation

Registry, labels, summaries and reference shares: CC BY 4.0. The benchmarks themselves keep their own licenses, recorded in registry. If you use CABA, please cite the paper (Vishwanath et al., 2026; see the GitHub repository's CITATION.cff).